How large language models can reshape collective intelligence.
Journal
Nature human behaviour
ISSN: 2397-3374
Titre abrégé: Nat Hum Behav
Pays: England
ID NLM: 101697750
Informations de publication
Date de publication:
20 Sep 2024
20 Sep 2024
Historique:
received:
06
11
2023
accepted:
17
07
2024
medline:
21
9
2024
pubmed:
21
9
2024
entrez:
20
9
2024
Statut:
aheadofprint
Résumé
Collective intelligence underpins the success of groups, organizations, markets and societies. Through distributed cognition and coordination, collectives can achieve outcomes that exceed the capabilities of individuals-even experts-resulting in improved accuracy and novel capabilities. Often, collective intelligence is supported by information technology, such as online prediction markets that elicit the 'wisdom of crowds', online forums that structure collective deliberation or digital platforms that crowdsource knowledge from the public. Large language models, however, are transforming how information is aggregated, accessed and transmitted online. Here we focus on the unique opportunities and challenges this transformation poses for collective intelligence. We bring together interdisciplinary perspectives from industry and academia to identify potential benefits, risks, policy-relevant considerations and open research questions, culminating in a call for a closer examination of how large language models affect humans' ability to collectively tackle complex problems.
Identifiants
pubmed: 39304760
doi: 10.1038/s41562-024-01959-9
pii: 10.1038/s41562-024-01959-9
doi:
Types de publication
Journal Article
Review
Langues
eng
Sous-ensembles de citation
IM
Subventions
Organisme : Deutsche Forschungsgemeinschaft (German Research Foundation)
ID : 458366841
Organisme : Deutsche Forschungsgemeinschaft (German Research Foundation)
ID : 458366841
Organisme : Deutsche Forschungsgemeinschaft (German Research Foundation)
ID : 458366841
Organisme : Deutsche Forschungsgemeinschaft (German Research Foundation)
ID : 390523135
Organisme : Irish Research Council (An Chomhairle um Thaighde in Éirinn)
ID : IRCLA/2022/3217
Informations de copyright
© 2024. Springer Nature Limited.
Références
Hu, K. ChatGPT sets record for fastest-growing user base—analyst note. Reuters (2 February 2023).
Mitchell, M. & Krakauer, D. C. The debate over understanding in AI's large language models. Proc. Natl Acad. Sci. USA 120, e2215907120 (2023).
pubmed: 36943882
pmcid: 10068812
doi: 10.1073/pnas.2215907120
Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. On the dangers of stochastic parrots: can language models be too big? In Proc. 2021 ACM Conference on Fairness, Accountability, and Transparency 610–623 (Association for Computing Machinery, 2021).
Weidinger, L. et al. Taxonomy of risks posed by language models. In Proc. 2022 ACM Conference on Fairness, Accountability, and Transparency 214–229 (Association for Computing Machinery, 2022).
Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with GPT-4. Preprint at arXiv https://doi.org/10.48550/arXiv.2303.12712 (2023).
Hilbert, M. What is the content of the world’s technologically mediated information and communication capacity: how much text, image, audio, and video? Inf. Soc. 30, 127–143 (2014).
doi: 10.1080/01972243.2013.873748
Malone, T. W. & Bernstein, M. S. Handbook of Collective Intelligence (MIT Press, 2015).
Flack, J., Ipeirotis, P., Malone, T. W., Mulgan, G. & Page, S. E. Editorial to the inaugural issue of Collective Intelligence. Collect. Intell. 1, 263391372211141 (2022).
doi: 10.1177/26339137221114179
Malone, T. W., Laubacher, R. & Dellarocas, C. The collective intelligence genome. IEEE Eng. Manage. Rev. 38, 38–52 (2010).
doi: 10.1109/EMR.2010.5559142
Suran, S., Pattanaik, V., Yahia, S. B. & Draheim, D. Exploratory analysis of collective intelligence projects developed within the EU-Horizon 2020 framework. In Computational Collective Intelligence: 11th International Conference, ICCCI 2019, Hendaye, France, September 4–6, 2019: Proceedings Pt 2 (eds Nguyen, N. T. et al.) 285–296 (Springer, 2019).
Suran, S., Pattanaik, V. & Draheim, D. Frameworks for collective intelligence: a systematic literature review. ACM Comput. Surv. 53, 1–36 (2021).
doi: 10.1145/3368986
Woolley, A. W. & Gupta, P. Understanding collective intelligence: investigating the role of collective memory, attention, and reasoning processes. Perspect. Psychol. Sci. 19, 344–354 (2024).
pubmed: 37642156
doi: 10.1177/17456916231191534
Salminen, J. Collective intelligence in humans: a literature review. Preprint at arXiv https://doi.org/10.48550/arXiv.1204.3401 (2012).
Smith, A. An Inquiry into the Nature and Causes of the Wealth of Nations (W. Strahan and T. Cadell, 1776).
Hayek, F. A. The use of knowledge in society. Am. Econ. Rev. 35, 519–530 (1945).
Hastie, R. & Kameda, T. The robust beauty of majority rules in group decisions. Psychol. Rev. 112, 494–508 (2005).
pubmed: 15783295
doi: 10.1037/0033-295X.112.2.494
Surowiecki, J. The Wisdom of Crowds (Anchor Books, 2005).
Dasgupta, P. & Maskin, E. On the robustness of majority rule. J. Eur. Econ. Assoc. 6, 949–973 (2008).
doi: 10.1162/JEEA.2008.6.5.949
Arrow, K. J. et al. The promise of prediction markets. Science 320, 877–878 (2008).
pubmed: 18487176
doi: 10.1126/science.1157679
Simon, H. A. Rational Decision-Making in Business Organizations (American Economic Association, 1978).
Vélez, N., Christian, B., Hardy, M., Thompson, B. D. & Griffiths, T. L. How do humans overcome individual computational limitations by working together? Cogn. Sci. 47, e13232 (2023).
pubmed: 36655981
pmcid: 10334258
doi: 10.1111/cogs.13232
Gupta, P., Nguyen, T. N., Gonzalez, C. & Woolley, A. W. Fostering collective intelligence in human–AI collaboration: laying the groundwork for COHUMAIN. Top. Cogn. Sci. https://doi.org/10.1111/tops.12679 (2023).
Riedl, C., Kim, Y. J., Gupta, P., Malone, T. W. & Woolley, A. W. Quantifying collective intelligence in human groups. Proc. Natl Acad. Sci. USA 118, e2005737118 (2021).
pubmed: 34001598
pmcid: 8166150
doi: 10.1073/pnas.2005737118
Hong, L. & Page, S. E. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proc. Natl Acad. Sci. USA 101, 16385–16389 (2004).
pubmed: 15534225
pmcid: 528939
doi: 10.1073/pnas.0403723101
Bang, D. & Frith, C. D. Making better decisions in groups. R. Soc. Open Sci. 4, 170193 (2017).
pubmed: 28878973
pmcid: 5579088
doi: 10.1098/rsos.170193
Harrison, D. A., Price, K. H., Gavin, J. H. & Florey, A. T. Time, teams, and task performance: changing effects of surface- and deep-level diversity on group functioning. Acad. Manage. J. 45, 1029–1045 (2002).
doi: 10.2307/3069328
Mohammed, S. & Angell, L. C. Surface‐ and deep‐level diversity in workgroups: examining the moderating effects of team orientation and team process on relationship conflict. J. Organ. Behav. 25, 1015–1039 (2004).
doi: 10.1002/job.293
Phillips, K. W. & Loyd, D. L. When surface and deep-level diversity collide: the effects on dissenting group members. Organ. Behav. Hum. Decis. Process. 99, 143–160 (2006).
doi: 10.1016/j.obhdp.2005.12.001
Condorcet, N. Essai sur l’Application de l’Analyse à la Probabilité des Décisions Rendues à la Pluralité des Voix (Chelsea, 1785).
Grofman, B., Owen, G. & Feld, S. L. Thirteen theorems in search of the truth. Theory Decis. 15, 261–278 (1983).
doi: 10.1007/BF00125672
Page, S. The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies New Edn (Princeton Univ. Press, 2008).
Hong, L. & Page, S. E. in Collective Wisdom (eds Landemore, H. & Elster, J.) 56–71 (Cambridge Univ. Press, 2012).
Ladha, K. K. The Condorcet jury theorem, free speech, and correlated votes. Am. J. Polit. Sci. 36, 617–634 (1992).
doi: 10.2307/2111584
Kameda, T., Toyokawa, W. & Tindale, R. S. Information aggregation and collective intelligence beyond the wisdom of crowds. Nat. Rev. Psychol. 1, 345–357 (2022).
doi: 10.1038/s44159-022-00054-y
Laan, A., Madirolas, G. & De Polavieja, G. G. Rescuing collective wisdom when the average group opinion is wrong. Front. Robot. AI 4, 56 (2017).
doi: 10.3389/frobt.2017.00056
Lyon, A. & Pacuit, E. in Handbook of Human Computation (ed. Michelucci, P.) 599–614 (Springer New York, 2013).
Landemore, H. & Page, S. E. Deliberation and disagreement: problem solving, prediction, and positive dissensus. Polit. Phil. Econ. 14, 229–254 (2015).
doi: 10.1177/1470594X14544284
List, C. The theory of judgment aggregation: an introductory review. Synthese 187, 179–207 (2012).
doi: 10.1007/s11229-011-0025-3
Centola, D. The network science of collective intelligence. Trends Cogn. Sci. 26, 923–941 (2022).
pubmed: 36180361
doi: 10.1016/j.tics.2022.08.009
Baron, R. S. So right it’s wrong: groupthink and the ubiquitous nature of polarized group decision making. Adv. Exp. Soc. Psychol. 37, 219–253 (2005).
doi: 10.1016/S0065-2601(05)37004-3
Hahn, U., Von Sydow, M. & Merdes, C. How communication can make voters choose less well. Top. Cogn. Sci. 11, 194–206 (2019).
pubmed: 30585433
doi: 10.1111/tops.12401
Hahn, U., Hansen, J. U. & Olsson, E. J. Truth tracking performance of social networks: how connectivity and clustering can make groups less competent. Synthese 197, 1511–1541 (2020).
doi: 10.1007/s11229-018-01936-6
Jönsson, M. L., Hahn, U. & Olsson, E. J. The kind of group you want to belong to: effects of group structure on group accuracy. Cognition 142, 191–204 (2015).
pubmed: 26046424
doi: 10.1016/j.cognition.2015.04.013
Becker, J., Brackbill, D. & Centola, D. Network dynamics of social influence in the wisdom of crowds. Proc. Natl Acad. Sci. USA 114, E5070–E5076 (2017).
pubmed: 28607070
pmcid: 5495222
doi: 10.1073/pnas.1615978114
Zollman, K. J. S. The communication structure of epistemic communities. Phil. Sci. 74, 574–587 (2007).
doi: 10.1086/525605
Zollman, K. J. S. The epistemic benefit of transient diversity. Erkenntnis 72, 17–35 (2010).
doi: 10.1007/s10670-009-9194-6
Zollman, K. J. S. Network epistemology: communication in epistemic communities. Phil. Compass 8, 15–27 (2013).
doi: 10.1111/j.1747-9991.2012.00534.x
Smaldino, P. E., Moser, C., Pérez Velilla, A. & Werling, M. Maintaining transient diversity is a general principle for improving collective problem solving. Perspect. Psychol. Sci. 19, 454–464 (2023).
pubmed: 37369100
pmcid: 10913329
doi: 10.1177/17456916231180100
Bak-Coleman, J. B. et al. Stewardship of global collective behavior. Proc. Natl Acad. Sci. USA 118, e2025764118 (2021).
pubmed: 34155097
pmcid: 8271675
doi: 10.1073/pnas.2025764118
Suran, S. et al. Building global societies on collective intelligence: challenges and opportunities. Digit. Gov. Res. Pract. 3, 1–6 (2022).
doi: 10.1145/3568169
Tsvetkova, M., Yasseri, T., Pescetelli, N. & Werner, T. Human-machine social systems. Nat. Hum. Behav. https://doi.org/10.48550/arXiv.2402.14410 (in press).
Cui, H. & Yasseri, T. AI-enhanced collective intelligence. Patterns https://doi.org/10.48550/arXiv.2403.10433 (in press).
Ovadya, A. ‘Generative CI’ through collective response systems. Preprint at arXiv https://doi.org/10.48550/arXiv.2302.00672 (2023).
Zaremba, W. et al. Democratic inputs to AI. OpenAI https://openai.com/blog/democratic-inputs-to-ai (2023).
Lorenz-Spreen, P., Oswald, L., Lewandowsky, S. & Hertwig, R. A systematic review of worldwide causal and correlational evidence on digital media and democracy. Nat. Hum. Behav. 7, 74–101 (2022).
pubmed: 36344657
pmcid: 9883171
doi: 10.1038/s41562-022-01460-1
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B. & Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. Preprint at arXiv https://doi.org/10.48550/arXiv.2305.14325 (2023).
Wu, Q. et al. AutoGen: enabling next-gen LLM applications via multi-agent conversation. Preprint at arXiv https://doi.org/10.48550/arXiv.2308.08155 (2023).
Yoffe, L., Amayuelas, A. & Wang, W. Y. DebUnc: mitigating hallucinations in large language model agent communication with uncertainty estimations. Preprint at arXiv https://doi.org/10.48550/arXiv.2407.06426 (2024).
Bommasani, R. et al. On the opportunities and risks of foundation models. Preprint at arXiv http://arxiv.org/abs/2108.07258 (2022).
Törnberg, P. ChatGPT-4 outperforms experts and crowd workers in annotating political Twitter messages with zero-shot learning. Preprint at arXiv http://arxiv.org/abs/2304.06588 (2023).
Rathje, S. et al. GPT is an effective tool for multilingual psychological text analysis. Proc. Natl Acad. Sci. USA 131, e2308950121 (2024).
doi: 10.1073/pnas.2308950121
Goldstein, D. G., McAfee, R. P. & Suri, S. The wisdom of smaller, smarter crowds. In Proc. 15th ACM Conference on Economics and Computation 471–488 (Association for Computing Machinery, 2014).
Mannes, A. E., Soll, J. B. & Larrick, R. P. The wisdom of select crowds. J. Pers. Soc. Psychol. 107, 276–299 (2014).
pubmed: 25090129
doi: 10.1037/a0036677
Wu, L., Wang, D. & Evans, J. A. Large teams develop and small teams disrupt science and technology. Nature 566, 378–382 (2019).
pubmed: 30760923
doi: 10.1038/s41586-019-0941-9
Hahn, U. Collectives and epistemic rationality. Top. Cogn. Sci. 14, 602–620 (2022).
pubmed: 35285151
doi: 10.1111/tops.12610
Lafont, C. Deliberation, participation, and democratic legitimacy: should deliberative mini-publics shape public policy? J. Polit. Phil. 23, 40–63 (2015).
doi: 10.1111/jopp.12031
Landemore, H. Can AI Bring Deliberation To The Masses? (Stanford Institute for Human-Centered Artificial Intelligence, 2022).
Cohen, R. L. Procedural justice and participation. Hum. Relat. 38, 643–663 (1985).
doi: 10.1177/001872678503800703
Greenberg, J. & Folger, R. in Basic Group Processes (ed. Paulus, P. B.) 235–256 (Springer New York, 1983).
El Zein, M., Bahrami, B. & Hertwig, R. Shared responsibility in collective decisions. Nat. Hum. Behav. 3, 554–559 (2019).
pubmed: 31011162
doi: 10.1038/s41562-019-0596-4
Fishkin, J. S. When the People Speak: Deliberative Democracy and Public Consultation (Oxford Univ. Press, 2009).
Steiner, I. D. Models for inferring relationships between group size and potential group productivity. Behav. Sci. 11, 273–283 (1966).
pubmed: 5945374
doi: 10.1002/bs.3830110404
Steiner, I. D. Group Process and Productivity (Academic Press, 1972).
Hill, G. W. Group versus individual performance: are N + 1 heads better than one? Psychol. Bull. 91, 517–539 (1982).
doi: 10.1037/0033-2909.91.3.517
Almaatouq, A., Alsobay, M., Yin, M. & Watts, D. J. Task complexity moderates group synergy. Proc. Natl Acad. Sci. USA 118, e2101062118 (2021).
pubmed: 34479999
pmcid: 8433503
doi: 10.1073/pnas.2101062118
Straub, V. J., Tsvetkova, M. & Yasseri, T. The cost of coordination can exceed the benefit of collaboration in performing complex tasks. Collect. Intell. 2, 263391372311569 (2023).
doi: 10.1177/26339137231156912
Zhu, W. et al. Multilingual machine translation with large language models: empirical results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2024 (eds. Duh, K., Gomez, H. & Bethard, S.) 2765–2781 (Association for Computational Linguistics, 2024).
Bawden, R. & Yvon, F. Investigating the translation performance of a large multilingual language model: the case of BLOOM. In Proc. 24th Annual Conference of the European Association for Machine Translation (eds. Nurminen, M. et al.) 157–170 (European Association for Machine Translation, 2023).
Berdejo-Espinola, V. & Amano, T. AI tools can improve equity in science. Science 379, 991 (2023).
pubmed: 36893248
doi: 10.1126/science.adg9714
Katsnelson, A. Poor English skills? New AIs help researchers to write better. Nature 609, 208–209 (2022).
pubmed: 36038730
doi: 10.1038/d41586-022-02767-9
Romero-Olivares, A. L. Reviewers, don’t be rude to nonnative English speakers. Science https://doi.org/10.1126/science.caredit.aaz7179 (2019).
Ramírez-Castañeda, V. Disadvantages in preparing and publishing scientific papers caused by the dominance of the English language in science: the case of Colombian researchers in biological sciences. PLoS ONE 15, e0238372 (2020).
pubmed: 32936821
pmcid: 7494110
doi: 10.1371/journal.pone.0238372
Brabham, D. C. Crowdsourcing as a model for problem solving: an introduction and cases. Converg. Int. J. Res. N. Media Technol. 14, 75–90 (2008).
von Hippel, E. in Handbook of the Economics of Innovation (eds Hall, B. H. & Rosenberg, N.) Vol. 1, 411–427 (Elsevier, 2010).
Boussioux, L., Lane, J. N., Zhang, M., Jacimovic, V. & Lakhani, K. R. The crowdless future? Generative AI and creative problem solving. Organ. Sci. 0, 1–19 (2024).
Girotra, K., Meincke, L., Terwiesch, C. & Ulrich, K. T. Ideas are dimes a dozen: large language models for idea generation in innovation. SSRN Electron. J. https://doi.org/10.2139/ssrn.4526071 (2023).
Argyle, L. P. et al. Out of one, many: using language models to simulate human samples. Polit. Anal. 31, 337–351 (2023).
doi: 10.1017/pan.2023.2
Jiang, H., Zhang, X., Cao, X., Breazeal, C., Roy, D. & Kabbara, J. PersonaLLM: investigating the ability of large language models to express personality traits. In Findings of the Association for Computational Linguistics: NAACL 2024 (eds. Duh, K. et al.) 3605–3627 (Association for Computational Linguistics, 2024).
Salewski, L., Alaniz, S., Rio-Torto, I., Schulz, E. & Akata, Z. In-context impersonation reveals large language models’ strengths and biases. In Adv. Neur. Inf. Process. Syst. 36 (NeurIPS 2023) (eds Oh, A. et al.) 72044–720579 (2023).
Ashkinaze, J., Mendelsohn, J., Qiwei, L., Budak, C. & Gilbert, E. How AI ideas affect the creativity, diversity, and evolution of human ideas: evidence from a large, dynamic experiment. Preprint at arXiv https://doi.org/10.48550/arXiv.2401.13481 (2024).
Fink, A. et al. Stimulating creativity via the exposure to other people’s ideas. Hum. Brain Mapp. 33, 2603–2610 (2012).
pubmed: 23074077
doi: 10.1002/hbm.21387
Doshi, A. R. & Hauser, O. Generative artificial intelligence enhances creativity. SSRN Electron. J. https://doi.org/10.2139/ssrn.4535536 (2023).
Herzog, S. M. & Hertwig, R. The wisdom of many in one mind: improving individual judgments with dialectical bootstrapping. Psychol. Sci. 20, 231–237 (2009).
pubmed: 19170937
doi: 10.1111/j.1467-9280.2009.02271.x
Herzog, S. M. & Hertwig, R. Harnessing the wisdom of the inner crowd. Trends Cogn. Sci. 18, 504–506 (2014).
pubmed: 25261350
doi: 10.1016/j.tics.2014.06.009
Schilling, M. A. & Green, E. Recombinant search and breakthrough idea generation: an analysis of high impact papers in the social sciences. Res. Policy 40, 1321–1331 (2011).
doi: 10.1016/j.respol.2011.06.009
Porciello, J., Ivanina, M., Islam, M., Einarson, S. & Hirsh, H. Accelerating evidence-informed decision-making for the Sustainable Development Goals using machine learning. Nat. Mach. Intell. 2, 559–565 (2020).
doi: 10.1038/s42256-020-00235-5
Weitzman, M. L. Recombinant growth. Q. J. Econ. 113, 331–360 (1998).
doi: 10.1162/003355398555595
Hibbing, J. R. & Theiss-Morse, E. Stealth Democracy: Americans’ Beliefs about How Government Should Work (Cambridge Univ. Press, 2002).
Rosenberg, S. W. in Deliberative Democracy (eds Elstub, S. & McLaverty, P.) 98–117 (Edinburgh Univ. Press, 2014).
Achen, C. H. & Bartels, L. M. Democracy for Realists: Why Elections Do Not Produce Responsive Government (Princeton Univ. Press, 2017).
Sunstein, C. R. On a danger of deliberative democracy. Daedalus 131, 120–124 (2002).
Neblo, M. A., Esterling, K. M., Kennedy, R. P., Lazer, D. M. J. & Sokhey, A. E. Who wants to deliberate—and why? Am. Polit. Sci. Rev. 104, 566–583 (2010).
doi: 10.1017/S0003055410000298
Maćkowiak, B., Matějka, F. & Wiederholt, M. Rational inattention: a review. J. Econ. Lit. 61, 226–273 (2023).
doi: 10.1257/jel.20211524
Shim, J. P. et al. Past, present, and future of decision support technology. Decis. Support Syst. 33, 111–126 (2002).
doi: 10.1016/S0167-9236(01)00139-7
Donohoe, H., Stellefson, M. & Tennant, B. Advantages and limitations of the e-Delphi technique. Am. J. Health Educ. 43, 38–46 (2012).
doi: 10.1080/19325037.2012.10599216
Dalkey, N. & Helmer, O. An experimental application of the Delphi method to the use of experts. Manage. Sci. 9, 458–467 (1963).
doi: 10.1287/mnsc.9.3.458
Tetlock, P. E., Mellers, B. A., Rohrbaugh, N. & Chen, E. Forecasting tournaments: tools for increasing transparency and improving the quality of debate. Curr. Dir. Psychol. Sci. 23, 290–295 (2014).
doi: 10.1177/0963721414534257
McAndrew, T. et al. Early human judgment forecasts of human monkeypox, May 2022. Lancet Digit. Health 4, e569–e571 (2022).
pubmed: 35811294
pmcid: 9534486
doi: 10.1016/S2589-7500(22)00127-3
Lin, X. Exploring the role of ChatGPT as a facilitator for motivating self-directed learning among adult learners. Adult Learn. 35, 56–166 (2023).
Argyle, L. P. et al. AI chat assistants can improve conversations about divisive topics. Preprint at arXiv https://doi.org/10.48550/arXiv.2302.07268 (2023).
Hadfi, R. et al. Conversational agents enhance women’s contribution in online debates. Sci. Rep. 13, 14534 (2023).
pubmed: 37666917
pmcid: 10477209
doi: 10.1038/s41598-023-41703-3
Small, C. T. et al. Opportunities and risks of LLMs for scalable deliberation with polis. Preprint at arXiv https://doi.org/10.48550/arXiv.2306.11932 (2023).
Fishkin, J. et al. Deliberative democracy with the online deliberation platform. In 7th AAAI Conference on Human Computation and Crowdsourcing, https://www.humancomputation.com/2019/assets/papers/144.pdf (Association for the Advancement of Artificial Intelligence, 2019).
Miller, K. A moderator ChatBot for civic discourse. Stanford HAI https://hai.stanford.edu/news/moderator-chatbot-civic-discourse (2020).
Jackson, M. O. & Xing, Y. Culture-dependent strategies in coordination games. Proc. Natl Acad. Sci. USA 111, 10889–10896 (2014).
pubmed: 25024196
pmcid: 4113921
doi: 10.1073/pnas.1400826111
Coavoux, M., Elsahar, H. & Gallé, M. Unsupervised aspect-based multi-document abstractive summarization. In Proc. 2nd Workshop on New Frontiers in Summarization (eds Wang, L. et al.) 42–47 (Association for Computational Linguistics, 2019).
Angelidis, S., Amplayo, R. K., Suhara, Y., Wang, X. & Lapata, M. Extractive opinion summarization in quantized transformer spaces. Trans. Assoc. Comput. Linguist. 9, 277–293 (2021).
doi: 10.1162/tacl_a_00366
Suhara, Y., Wang, X., Angelidis, S. & Tan, W.-C. OpinionDigest: a simple framework for opinion summarization. In Proc. 58th Annual Meeting of the Association for Computational Linguistics (eds Jurafsky, D. et al.) 5789–5798 (Association for Computational Linguistics, 2020).
Bakker, M. et al. Fine-tuning language models to find agreement among humans with diverse preferences. In Adv. Neur. Inf. Process. Syst. 35 (NeurIPS 2022) (eds Koyejo, S. et al.) 38176–38189 (2022).
Huang, S. & Siddarth, D. Generative AI and the digital commons. Preprint at arXiv https://doi.org/10.48550/arXiv.2303.11074 (2023).
Veselovsky, V., Ribeiro, M. H. & West, R. Artificial artificial artificial intelligence: crowd workers widely use large language models for text production tasks. Preprint at arXiv https://doi.org/10.48550/arXiv.2306.07899 (2023).
del Rio-Chanona, M., Laurentsyeva, N. & Wachs, J. Are large language models a threat to digital public goods? Evidence from activity on Stack Overflow. Preprint at arXiv https://doi.org/10.48550/arXiv.2307.07367 (2023).
Farič, N. & Potts, H. W. Motivations for contributing to health-related articles on Wikipedia: an interview study. J. Med. Internet Res. 16, e260 (2014).
pubmed: 25498308
pmcid: 4275502
doi: 10.2196/jmir.3569
Javanmardi, S., Ganjisaffar, Y., Lopes, C. & Baldi, P. User contribution and trust in Wikipedia. In Proc. 5th International ICST Conference on Collaborative Computing: Networking, Applications, Worksharing (eds Joshi, J. & Zhang, T.) https://doi.org/10.4108/ICST.COLLABORATECOM2009.8376 (Institute of Electrical and Electronics Engineers, 2009).
Adaji, I. & Vassileva, J. in Social Informatics (eds Spiro, E. & Ahn, Y.-Y.) 3–13 (Springer International, 2016).
Blincoe, K., Sheoran, J., Goggins, S., Petakovic, E. & Damian, D. Understanding the popular users: following, affiliation influence and leadership on GitHub. Inf. Softw. Technol. 70, 30–39 (2016).
doi: 10.1016/j.infsof.2015.10.002
Franceschelli, G. & Musolesi, M. Copyright in generative deep learning. Data Policy 4, e17 (2022).
doi: 10.1017/dap.2022.10
Desai, S. C., Xie, B. & Hayes, B. K. Getting to the source of the illusion of consensus. Cognition 223, 105023 (2022).
doi: 10.1016/j.cognition.2022.105023
Noelle-Neumann, E. The spiral of silence: a theory of public opinion. J. Commun. 24, 43–51 (1974).
doi: 10.1111/j.1460-2466.1974.tb00367.x
Wikipedia: five pillars. Wikipedia https://en.wikipedia.org/w/index.php?title=Wikipedia:Five_pillars (2023).
Yasseri, T. & Kertész, J. Value production in a collaborative environment: sociophysical studies of Wikipedia. J. Stat. Phys. 151, 414–439 (2013).
doi: 10.1007/s10955-013-0728-6
Hecht, B. & Gergle, D. The tower of Babel meets Web 2.0: user-generated content and its applications in a multilingual context. In Proc. SIGCHI Conference on Human Factors in Computing Systems 291–300 (Association for Computing Machinery, 2010).
Graham, M. The problem with Wikidata. Atlantic (6 April 2012).
Clemen, R. T. & Winkler, R. L. Limits for the precision and value of information from dependent sources. Oper. Res. 33, 427–442 (1985).
doi: 10.1287/opre.33.2.427
Davis-Stober, C. P., Budescu, D. V., Dana, J. & Broomell, S. B. When is a crowd wise?. Decision 1, 79–101 (2014).
doi: 10.1037/dec0000004
Herzog, S. M., Litvinova, A., Yahosseini, K. S., Tump, A. N. & Kurvers, R. H. J. M. in Taming Uncertainty (eds Hertwig, R. et al.) 245–262 (MIT Press, 2019).
Kurvers, R. H. J. M. et al. How to detect high-performing individuals and groups: decision similarity predicts accuracy. Sci. Adv. 5, eaaw9011 (2019).
pubmed: 31976366
pmcid: 6957221
doi: 10.1126/sciadv.aaw9011
Palley, A. B. & Soll, J. B. Extracting the wisdom of crowds when information is shared. Manage. Sci. 65, 2291–2309 (2019).
Walzner, D. D., Fuegener, A. & Gupta, A. Managing AI advice in crowd decision-making. In ICIS 2022 Proceedings https://aisel.aisnet.org/icis2022/hci_robot/hci_robot/7 (Association for Information Systems, 2022).
Padmakumar, V. & He, H. Does writing with language models reduce content diversity? Preprint at arXiv https://doi.org/10.48550/arXiv.2309.05196 (2023).
Kleinberg, J. & Raghavan, M. Algorithmic monoculture and social welfare. Proc. Natl Acad. Sci. USA 118, e2018340118 (2021).
pubmed: 34035166
pmcid: 8179131
doi: 10.1073/pnas.2018340118
Campbell, C. M., Izquierdo, E. J. & Goldstone, R. L. Partial copying and the role of diversity in social learning performance. Collect. Intell. 1, 263391372210818 (2022).
doi: 10.1177/26339137221081849
Toyokawa, W., Whalen, A. & Laland, K. N. Social learning strategies regulate the wisdom and madness of interactive crowds. Nat. Hum. Behav. 3, 183–193 (2019).
pubmed: 30944445
doi: 10.1038/s41562-018-0518-x
Almaatouq, A., Rahimian, M. A., Burton, J. W. & Alhajri, A. The distribution of initial estimates moderates the effect of social influence on the wisdom of the crowd. Sci. Rep. 12, 16546 (2022).
pubmed: 36192623
pmcid: 9530231
doi: 10.1038/s41598-022-20551-7
Brinkmann, L. et al. Machine culture. Nat. Hum. Behav. 7, 1855–1868 (2023).
pubmed: 37985914
doi: 10.1038/s41562-023-01742-2
OpenAI. GPT-4 technical report. Preprint at arXiv https://doi.org/10.48550/arXiv.2303.08774 (2023).
Semnani, S., Yao, V., Zhang, H. & Lam, M. WikiChat: stopping the hallucination of large language model chatbots by few-shot grounding on wikipedia. In Findings of the Association for Computational Linguistics: EMNLP 2023 (eds. Bouamor, H., Pino, J. & Bali, K.) 2387–2413 (Association for Computational Linguistics, 2023).
Goldstein, J. A. et al. Generative language models and automated influence operations: emerging threats and potential mitigations. Preprint at arXiv https://doi.org/10.48550/arXiv.2301.04246 (2023).
Simon, F. M., Altay, S. & Mercier, H. Misinformation reloaded? Fears about the impact of generative AI on misinformation are overblown. Harv. Kennedy Sch. Misinform. Rev. https://doi.org/10.37016/mr-2020-127 (2023).
Musser, M. A cost analysis of generative language models and influence operations. Preprint at arXiv https://doi.org/10.48550/arXiv.2308.03740 (2023).
Kapoor, S. & Narayanan, A. How to Prepare for the Deluge of Generative AI on Social Media (Knight First Amendment Institute, 2023).
Solaiman, I. The gradient of generative AI release: methods and considerations. In 2023 ACM Conference on Fairness, Accountability, and Transparency 111–122 (Association for Computing Machinery, 2023).
Warso, Z. & Keller, P. Open source AI and the paradox of open. Open Future https://openfuture.eu/blog/open-source-ai-and-the-paradox-of-open (2023).
Kapoor, S. et al. On the societal impact of open foundation models. Preprint at arXiv https://doi.org/10.48550/arXiv.2403.07918 (2024).
Widder, D. G., West, S. & Whittaker, M. Open (for business): big tech, concentrated power, and the political economy of open AI. SSRN Electron. J. https://doi.org/10.2139/ssrn.4543807 (2023).
National Artificial Intelligence Initiative https://www.ai.gov/nairrtf/ (National Artificial Intelligence Research Resource Task Force, 2024).
Artificial Intelligence Act (European Parliament, 2023).
Kapoor, S. & Narayanan, A. Generative AI Companies Must Publish Transparency Reports (Knight First Amendment Institute, 2023).
Mökander, J., Schuett, J., Kirk, H. R. & Floridi, L. Auditing large language models: a three-layered approach. AI Ethics https://doi.org/10.1007/s43681-023-00289-2 (2023).
Chang, Y. et al. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol. 15, 39 (2024).
doi: 10.1145/3641289
Knott, A. et al. Generative AI models should include detection mechanisms as a condition for public release. Ethics Inf. Technol. 25, 55 (2023).
doi: 10.1007/s10676-023-09728-4
Berditchevskaia, A., Malliaraki, E. & Peach, K. Participatory AI for Humanitarian Innovation: A Briefing Paper (Nesta, 2021).
Meta’s Third-Party Fact-Checking Program https://www.facebook.com/formedia/mjp/programs/third-party-fact-checking (Meta Journalism Project, accessed 29 March 2024).
Porter, E. & Wood, T. J. The global effectiveness of fact-checking: evidence from simultaneous experiments in Argentina, Nigeria, South Africa, and the United Kingdom. Proc. Natl Acad. Sci. USA 118, e2104235118 (2021).
pubmed: 34507996
pmcid: 8449384
doi: 10.1073/pnas.2104235118
Walter, N., Cohen, J., Holbert, R. L. & Morag, Y. Fact-checking: a meta-analysis of what works and for whom. Polit. Commun. 37, 350–375 (2020).
doi: 10.1080/10584609.2019.1668894
Carnahan, D. & Bergan, D. E. Correcting the misinformed: the effectiveness of fact-checking messages in changing false beliefs. Polit. Commun. 39, 166–183 (2022).
doi: 10.1080/10584609.2021.1963358
Ecker, U. K. H. et al. The psychological drivers of misinformation belief and its resistance to correction. Nat. Rev. Psychol. 1, 13–29 (2022).
doi: 10.1038/s44159-021-00006-y
Cai, A. et al. DesignAID: using generative AI and semantic diversity for design inspiration. In Proc. ACM Collective Intelligence Conference (eds Bernstein, M. et al.) 1–11 (Association for Computing Machinery, 2023).
Griebel, M., Flath, C. & Friesike, S. Augmented creativity: leveraging artificial intelligence for idea generation in the creative sphere. ECIS 2020 Research-in-Progress Papers https://aisel.aisnet.org/ecis2020_rip/77 (Association for Information Systems, 2020).
Wittenberg, C., Tappin, B. M., Berinsky, A. J. & Rand, D. G. The (minimal) persuasive advantage of political video over text. Proc. Natl Acad. Sci. USA 118, e2114388118 (2021).
pubmed: 34782473
pmcid: 8617416
doi: 10.1073/pnas.2114388118
Radford, A. et al. Language models are unsupervised multitask learners. OpenAI Blog (2019).
Hoffmann, J. et al. Training compute-optimal large language models. In 36th Conf Neural Information Processing, https://proceedings.neurips.cc/paper_files/paper/2022/file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf (NeurIPS, 2022).
Ouyang, L. et al. Training language models to follow instructions with human feedback. In Adv. Neur. Inf. Process. Syst. 35 (NeurIPS 2022) (eds Koyejo, S. et al.) 27730–27744 (2022).
Lee, A., Miranda, B. & Koyejo, S. Beyond scale: the diversity coefficient as a data quality metric demonstrates LLMs are pre-trained on formally diverse data. Preprint at arXiv http://arxiv.org/abs/2306.13840 (2023).
Atari, M., Xue, M. J., Park, P. S., Blasi, D. E. & Henrich, J. Which humans? Preprint at PsyArXiv https://doi.org/10.31234/osf.io/5b26t (2023).
Cao, Y. et al. Assessing cross-cultural alignment between ChatGPT and human societies: an empirical study. In Proc. First Workshop on Cross-Cultural Considerations in NLP (C3NLP) 53–67 (Association for Computational Linguistics, 2023).
Haller, P., Aynetdinov, A. & Akbik, A. OpinionGPT: modelling explicit biases in instruction-tuned LLMs. Preprint at arXiv https://doi.org/10.48550/arXiv.2309.03876 (2023).
Levy, S. et al. Comparing biases and the impact of multilingual training across multiple languages. In Proc. 2023 Conference on Empirical Methods in Natural Language Processing (eds Bouamor, H. et al.) 10260–10280 (Association for Computational Linguistics, 2023).
Arora, A., Kaffee, L.-A. & Augenstein, I. Probing pre-trained language models for cross-cultural differences in values. Preprint at arXiv https://doi.org/10.48550/arXiv.2203.13722 (2023).
Dietterich, T. G. in Multiple Classifier Systems. MCS 2000. Lecture Notes in Computer Science, vol 1857, https://doi.org/10.1007/3-540-45014-9_1 (Springer Berlin Heidelberg, 2000).
Grossmann, I. et al. AI and the transformation of social science research. Science 380, 1108–1109 (2023).
pubmed: 37319216
doi: 10.1126/science.adi1778
Bail, C. A. Can generative AI improve social science? Proc. Natl Acad. Sci. USA 121, e2314021121 (2024).
pubmed: 38722813
pmcid: 11127003
doi: 10.1073/pnas.2314021121
Aher, G. V., Arriaga, R. I. & Kalai, A. T. Using large language models to simulate multiple humans and replicate human subject studies. In Proc. 40th International Conference on Machine Learning 337–371 (Proceedings of Machine Learning Research, 2023).
Dillion, D., Tandon, N., Gu, Y. & Gray, K. Can AI language models replace human participants? Trends Cogn. Sci. 27, 597–600 (2023).
pubmed: 37173156
doi: 10.1016/j.tics.2023.04.008
Epstein, J. M. & Axtell, R. Growing Artificial Societies: Social Science from the Bottom Up (Brookings Institution Press, 1996).
Bonabeau, E. Agent-based modeling: methods and techniques for simulating human systems. Proc. Natl Acad. Sci. USA 99, 7280–7287 (2002).
pubmed: 12011407
pmcid: 128598
doi: 10.1073/pnas.082080899
Park, J. S. et al. Social simulacra: creating populated prototypes for social computing systems. In Proc. 35th Annual ACM Symposium on User Interface Software and Technology (eds Agrawala, M. et al.) 1–18 (Association for Computing Machinery, 2022).
Gao, C. et al. S
Horton, J. J. Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? Working Paper 31122 (National Bureau of Economic Research, 2023).
Chen, L., Zaharia, M. & Zou, J. How is ChatGPT’s behavior changing over time? Harvard Data Sci. Rev. 6, https://doi.org/10.1162/99608f92.5317da47 (2024).
Burton, J. W., Stein, M. & Jensen, T. B. A systematic review of algorithm aversion in augmented decision making. J. Behav. Decis. Mak. 33, 220–239 (2020).
doi: 10.1002/bdm.2155
Glikson, E. & Woolley, A. W. Human trust in artificial intelligence: review of empirical research. Acad. Manage. Ann. 14, 627–660 (2020).
doi: 10.5465/annals.2018.0057