A framework for the emergence and analysis of language in social learning agents.
Journal
Nature communications
ISSN: 2041-1723
Titre abrégé: Nat Commun
Pays: England
ID NLM: 101528555
Informations de publication
Date de publication:
31 Aug 2024
31 Aug 2024
Historique:
received:
30
05
2023
accepted:
16
08
2024
medline:
1
9
2024
pubmed:
1
9
2024
entrez:
31
8
2024
Statut:
epublish
Résumé
Neural systems have evolved not only to solve environmental challenges through internal representations but also, under social constraints, to communicate these to conspecifics. In this work, we aim to understand the structure of these internal representations and how they may be optimized to transmit pertinent information from one individual to another. Thus, we build on previous teacher-student communication protocols to analyze the formation of individual and shared abstractions and their impact on task performance. We use reinforcement learning in grid-world mazes where a teacher network passes a message to a student to improve task performance. This framework allows us to relate environmental variables with individual and shared representations. We compress high-dimensional task information within a low-dimensional representational space to mimic natural language features. In coherence with previous results, we find that providing teacher information to the student leads to a higher task completion rate and an ability to generalize tasks it has not seen before. Further, optimizing message content to maximize student reward improves information encoding, suggesting that an accurate representation in the space of messages requires bi-directional input. These results highlight the role of language as a common representation among agents and its implications on generalization capabilities.
Identifiants
pubmed: 39217160
doi: 10.1038/s41467-024-51887-5
pii: 10.1038/s41467-024-51887-5
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
7590Subventions
Organisme : Deutsche Forschungsgemeinschaft (German Research Foundation)
ID : 227953431
Informations de copyright
© 2024. The Author(s).
Références
Bengio, Y., Courville, A. & Vincent, P. Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013).
doi: 10.1109/TPAMI.2013.50
pubmed: 23787338
Behrens, T. E. et al. What is a cognitive map? Organizing knowledge for flexible behavior. Neuron 100, 490–509 (2018).
doi: 10.1016/j.neuron.2018.10.002
pubmed: 30359611
Safaie, M. et al. Preserved neural dynamics across animals performing similar behaviour. Nature 623, 765–771 (2023).
doi: 10.1038/s41586-023-06714-0
pubmed: 37938772
pmcid: 10665198
Tomasello, M. The cultural origins of human cognition (Harvard University Press, 2009).
Dunbar, R. I. The social brain hypothesis. Evolut. Anthropol. 6, 178–190 (1998).
doi: 10.1002/(SICI)1520-6505(1998)6:5<178::AID-EVAN5>3.0.CO;2-8
Hasson, U., Ghazanfar, A. A., Galantucci, B., Garrod, S. & Keysers, C. Brain-to-brain coupling: a mechanism for creating and sharing a social world. Trends Cogn. Sci. 16, 114–121 (2012).
doi: 10.1016/j.tics.2011.12.007
pubmed: 22221820
pmcid: 3269540
Wilson, E. O. The Social Conquest of Earth (Liveright Publishing Corporation, 2012).
Ludwig, W., Anscombe, G. et al. Philosophical investigations. (Basic Blackwell,1953).
McKenzie, S. et al. Hippocampal representation of related and opposing memories develop within distinct, hierarchically organized neural schemas. Neuron 83, 202–215 (2014).
doi: 10.1016/j.neuron.2014.05.019
pubmed: 24910078
pmcid: 4082468
Kirby, S. & Hurford, J. Learning, culture and evolution in the origin of linguistic constraints. In Proc. Fourth European conference on artificial life 493-502 (MIT Press,1997).
Steels, L. The synthetic modeling of language origins. Evolut. Commun. 1, 1–34 (1997).
doi: 10.1075/eoc.1.1.02ste
Cangelosi, A. & Parisi, D. Computer simulation: A new scientific approach to the study of language evolution. Simulating the evolution of language (Springer, 2002).
Wagner, K., Reggia, J. A., Uriagereka, J. & Wilkinson, G. S. Progress in the simulation of emergent communication and language. Adapt. Behav. 11, 37–69 (2003).
doi: 10.1177/10597123030111003
Havrylov, S. & Titov, I. Emergence of language with multi-agent games: learning to communicate with sequences of symbols. In Proc. Advances in Neural Information Processing Systems. 30 (2017).
Kottur, S., Moura, J. M., Lee, S. & Batra, D. Natural language does not emerge 'naturally' in multi-agent dialog. arXiv preprint arXiv:1706.08502 (2017).
Jaques, N. et al. Social influence as intrinsic motivation for multi-agent deep reinforcement learning in International conference on machine learning. 3040–3049 (PMLR, 2019).
Lowe, R., Foerster, J., Boureau, Y.-L., Pineau, J. & Dauphin, Y. On the pitfalls of measuring emergent communication. arXiv preprint arXiv:1903.05168 (2019).
Kajić, I., Aygün, E. & Precup, D. Learning to cooperate: emergent communication in multi-agent navigation. arXiv preprint arXiv:2004.01097 (2020).
Yuan, L. et al. Emergence of pragmatics from referential game between theory of mind agents. arXiv preprint arXiv:2001.07752 (2020).
Andreas, J. Language models as agent models. In Findings of the Association for Computational Linguistics: EMNLP 2022. 5769–5779 (Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022).
Lazaridou, A., Peysakhovich, A. & Baroni, M. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182 (2016).
Lee, J., Cho, K., Weston, J. & Kiela, D. Emergent translation in multi-agent communication. arXiv preprint arXiv:1710.06922 (2017).
Sukhbaatar, S., Denton, E., Szlam, A. & Fergus, R. Learning goal embeddings via self-play for hierarchical reinforcement learning. arXiv preprint arXiv:1811.09083 (2018).
Fried, D., Tomlin, N., Hu, J., Patel, R. & Nematzadeh, A. Pragmatics in language grounding: phenomena, tasks, and modeling approaches. arXiv preprint arXiv:2211.08371 (2022).
Lazaridou, A. & Baroni, M. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.02419 (2020).
Oroojlooy, A. & Hajinezhad, D. A review of cooperative multi-agent deep reinforcement learning. Appl. Intell. 53, 1–46 (2022).
Rita, M., Chaabouni, R. & Dupoux, E. “LazImpa”: lazy and Impatient neural agents learn to communicate efficiently. arXiv preprint arXiv:2010.01878 (2020).
Tucker, M. et al. Emergent discrete communication in semantic spaces. Adv. Neural Inf. Process. Syst. 34, 10574–10586 (2021).
Sutton, R. S. & Barto, A. G. Reinforcement learning: an introduction (MIT Press, 2018).
Ndousse, K. K., Eck, D., Levine, S. & Jaques, N. Emergent social learning via multi-agent reinforcement learning. In Proc. International Conference on Machine Learning 7991–8004 (PMLR, 2021).
Hockett, C. F. & Hockett, C. D. The origin of speech. Sci. Am. 203, 88–97 (1960).
doi: 10.1038/scientificamerican0960-88
Galke, L., Ram, Y. & Raviv, L. Emergent communication for understanding human language evolution: What’s missing? arXiv preprint arXiv:2204.10590 (2022).
Haber, J. et al. The PhotoBook dataset: building common ground through visually-grounded dialogue. arXiv preprint arXiv:1906.01530 (2019).
Foerster, J., Assael, I. A., De Freitas, N. & Whiteson, S. Learning to communicate with deep multi-agent reinforcement learning. Adv. Neural Inf. Process. Syst. 29, (2016).
Lake, B. M. & Baroni, M. Human-like systematic generalization through a meta-learning neural network. Nature 623, 115–121 (2023).
doi: 10.1038/s41586-023-06668-3
pubmed: 37880371
pmcid: 10620072
Ku, A., Anderson, P., Patel, R., Ie, E. & Baldridge, J. Room-across-room: multilingual vision-and-language navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.07954 (2020).
Seyfarth, R. M. & Cheney, D. L. The evolution of language from social cognition. Curr. Opin. Neurobiol. 28, 5–9 (2014).
doi: 10.1016/j.conb.2014.04.003
pubmed: 24813180
Chaabouni, R., Kharitonov, E., Dupoux, E. & Baroni, M. Anti-efficient encoding in emergent communication. In Proc. Advances in Neural Information Processing Systems. Vol. 32, (2019).
Chaabouni, R., Kharitonov, E., Bouchacourt, D., Dupoux, E. & Baroni, M. Compositionality and generalization in emergent languages. arXiv preprint arXiv:2004.09124 (2020).
Bratman, J., Shvartsman, M., Lewis, R. L. & Singh, S. A new approach to exploring language emergence as boundedly optimal control in the face of environmental and cognitive constraints. In Proc. 10th International Conference on Cognitive Modeling 7–12 (Drexel University, 2010).
Mordatch, I. & Abbeel, P. Emergence of grounded compositional language in multi-agent populations. In Proc. AAAI conference on artificial intelligence, Vol. 32 (2018).
Chaabouni, R. et al. Emergent communication at scale. In Proc. International Conference on Learning Representations (2021).
Rita, M., Strub, F., Grill, J.-B., Pietquin, O. & Dupoux, E. On the role of population heterogeneity in emergent communication. arXiv preprint arXiv:2204.12982 (2022).
Rosenfeld, R. et al. A maximum entropy approach to adaptive statistical language modelling. Comput. Speech Lang. 10, 187 (1996).
doi: 10.1006/csla.1996.0011
Bengio, Y., Ducharme, R. & Vincent, P. A neural probabilistic language model. In Proc. Advances in Neural Information Processing Systems, Vol. 13 (2000).
Tinbergen, N. The evolution of signalling devices. Social behavior and organization among vertebrates, 206–230 (1964).
Dong, S., Lin, T., Nieh, J. C. & Tan, K. Social signal learning of the waggle dance in honey bees. Science 379, 1015–1018 (2023).
doi: 10.1126/science.ade1702
pubmed: 36893231
Tieleman, O., Lazaridou, A., Mourad, S., Blundell, C. & Precup, D. Shaping representations through communication: community size effect in artificial learning systems. arXiv preprint arXiv:1912.06208 (2019).
Silver, D. et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 1140–1144 (2018).
doi: 10.1126/science.aar6404
pubmed: 30523106
François-Lavet, V. et al. An introduction to deep reinforcement learning. Found. Trends Mach. Learn. 11, 219–354 (2018).
doi: 10.1561/2200000071
Lee, D., Seo, H. & Jung, M. W. Neural basis of reinforcement learning and decision making. Annu. Rev. Neurosci. 35, 287–308 (2012).
doi: 10.1146/annurev-neuro-062111-150512
pubmed: 22462543
pmcid: 3490621
Rocktäschel, T., Bošnjak, M., Singh, S. & Riedel, S. Low-Dimensional Embeddings of Logic. In Proc. Annual Meeting of the Association for Computational Linguistics (2014).
Antonello, R., Turek, J., Vo, V. A. & Huth, A. G. Low-dimensional structure in the space of language representations is reflected in brain responses. In Proc. Neural Information Processing Systems (2021).
Huth, A. G., De Heer, W. A., Griffiths, T. L., Theunissen, F. E. & Gallant, J. L. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532, 453–458 (2016).
doi: 10.1038/nature17637
pubmed: 27121839
pmcid: 4852309
Caucheteux, C. & King, J.-R. Brains and algorithms partially converge in natural language processing. Commun. Biol. 5, 134 (2022).
doi: 10.1038/s42003-022-03036-1
pubmed: 35173264
pmcid: 8850612
Robotka, H. et al. Sparse ensemble neural code for a complete vocal repertoire. Cell Rep. 42, 112034 (2023).
Ng, A. et al. Sparse autoencoder. CS294A Lect. Notes 72, 1–19 (2011).
Manning, C. & Schutze, H. Foundations of statistical natural language processing (MIT Press, 1999).
Olshausen, B. A. & Field, D. J. Sparse coding of sensory inputs. Curr. Opin. Neurobiol. 14, 481–487 (2004).
doi: 10.1016/j.conb.2004.07.007
pubmed: 15321069
Tampuu, A. et al. Multiagent cooperation and competition with deep reinforcement learning. PloS one 12, e0172395 (2017).
doi: 10.1371/journal.pone.0172395
pubmed: 28380078
pmcid: 5381785
Brighton, H. & Kirby, S. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artif. Life 12, 229–242 (2006).
doi: 10.1162/artl.2006.12.2.229
pubmed: 16539767
Kharitonov, E., Chaabouni, R., Bouchacourt, D. & Baroni, M. Entropy minimization in emergent languages in International Conference on Machine Learning, 5220–5230 (2020).
Mesoudi, A. & Whiten, A. The multiple roles of cultural transmission experiments in understanding human cultural evolution. Philos. Trans. R. Soc. B Biol. Sci. 363, 3489–3501 (2008).
doi: 10.1098/rstb.2008.0129
Flesch, T., Saxe, A. & Summerfield, C. Continual task learning in natural and artificial agents. Trends Neurosci. 46, P199-210 (2023).
Ten Cate, C. Assessing the uniqueness of language: animal grammatical abilities take center stage. Psychon. Bull. Rev. 24, 91–96 (2017).
doi: 10.3758/s13423-016-1091-9
pubmed: 27368632
Breithaupt, F., Li, B., Liddell, T. M., Schille-Hudson, E. B. & Whaley, S. Fact vs. affect in the telephone game: all levels of surprise are retold with high accuracy, even independently of facts. Front. Psychol. 9, 2210 (2018).
doi: 10.3389/fpsyg.2018.02210
pubmed: 30515116
pmcid: 6255933
McMahon, A. & McMahon, R. Evolutionary linguistics (Cambridge University Press, 2012).
Engesser, S. & Townsend, S. W. Combinatoriality in the vocal systems of nonhuman animals. Wiley Interdiscip. Rev. Cogn. Sci. 10, e1493 (2019).
doi: 10.1002/wcs.1493
pubmed: 30724476
Suzuki, T. N., Wheatcroft, D. & Griesser, M. Experimental evidence for compositional syntax in bird calls. Nat. Commun. 7, 10986 (2016).
doi: 10.1038/ncomms10986
pubmed: 26954097
pmcid: 4786783
Fitch, W. T. The evolution of language (Cambridge University Press, 2010).
Christiansen, M. H. & Chater, N. The now-or-never bottleneck: a fundamental constraint on language. Behav. Brain Sci. 39, e62 (2016).
doi: 10.1017/S0140525X1500031X
pubmed: 25869618
Mikolov, T., Yih, W.-t. & Zweig, G. Linguistic regularities in continuous space word representations. In Proc. 2013 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies, 746–751 (Association for Computational Linguistics, 2013).
Kharitonov, E. & Baroni, M. Emergent language generalization and acquisition speed are not tied to compositionality. arXiv preprint arXiv:2004.03420 (2020).
Nowak, M. A. & Krakauer, D. C. The evolution of language. Proc. Natl Acad. Sci. 96, 8028–8033 (1999).
doi: 10.1073/pnas.96.14.8028
pubmed: 10393942
pmcid: 22182
Spranger, M. The evolution of grounded spatial language. Computational Models of Language Evolution, Vol. 5 (Language Science Press, Berlin, 2016).
Janik, V. M. & Slater, P. J. Context-specific use suggests that bottlenose dolphin signature whistles are cohesion calls. Anim. Behav. 56, 829–838 (1998).
doi: 10.1006/anbe.1998.0881
pubmed: 9790693
Barker, A. J. et al. Cultural transmission of vocal dialect in the naked mole-rat. Science 371, 503–507 (2021).
doi: 10.1126/science.abc6588
pubmed: 33510025
Dupoux, E. Cognitive science in the era of artificial intelligence: a roadmap for reverse-engineering the infant language-learner. Cognition 173, 43–59 (2018).
doi: 10.1016/j.cognition.2017.11.008
pubmed: 29324240
Wieczorek, T. J., Tchumatchenko, T., Wert-Carvajal, C. & Eggl, M. F. A framework for the emergence and analysis of language in social learning agents version v5. Apr. https://doi.org/10.5281/zenodo.7885526 (2024).