Testing the capacity of Bard and ChatGPT for writing essays on ethical dilemmas: A cross-sectional study.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
30 10 2024
30 10 2024
Historique:
received:
18
01
2024
accepted:
23
10
2024
medline:
30
10
2024
pubmed:
30
10
2024
entrez:
30
10
2024
Statut:
epublish
Résumé
Recent advancements in artificial intelligence (AI) have led to concerns about its potential misuse in education. As large language models (LLMs) such as ChatGPT and Bard can generate human-like text, researchers and educators noted the potential redundancy of tasking students with writing academic essays. We aimed to explore if the two LLMs could generate unstructured essays on medical students' personal experiences of challenges and ethical dilemmas that are indistinguishable from human-written texts. We collected 47 original student-written essays from which we extracted keywords to develop prompts for the LLMs. We then used these prompts to generate an equivalent number of essays using ChatGPT and Bard. We analysed the essays using the Language Inquiry and Word Count (LIWC) 22 software, extracting the main LIWC summary measures and variables related to social and psychological processes. We also conducted sub-analyses for sixteen student essays that were presumably written entirely or in part by AI, according to two AI detectors. We found that AI-written essays used more language related to affect, authenticity, and analytical thinking compared to student-written essays after we removed AI-co-written student essays from the analysis. We observed that, despite the differences in language characteristics compared to student-written essays, both LLMs are highly effective in generating essays on students' personal experiences and opinions regarding real-life ethical dilemmas.
Identifiants
pubmed: 39472544
doi: 10.1038/s41598-024-77576-3
pii: 10.1038/s41598-024-77576-3
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
26046Informations de copyright
© 2024. The Author(s).
Références
Guleria, A., Krishan, K., Sharma, V. & Kanchan, T. ChatGPT: ethical concerns and challenges in academics and research. J Infect Dev Ctries. 17, 1292–1299. https://doi.org/10.3855/jidc.18738 (2023).
doi: 10.3855/jidc.18738
pubmed: 37824352
Watters, C. & Lemanski, M. K. Universal skepticism of ChatGPT: a review of early literature on chat generative pre-trained transformer. Front. Big Data 6, 1224976. https://doi.org/10.3389/fdata.2023.1224976 (2023).
doi: 10.3389/fdata.2023.1224976
pubmed: 37680954
pmcid: 10482048
European University Association. Artificial intelligence tools and their responsible use in higher education learning and teaching. https://eua.eu/resources/publications/1059:artificial-intelligence-tools-and-their-responsible-use-in-higher-education-learning-and-teaching.html (2023).
Open Universities Australia. How you should—and shouldn’t—use ChatGPT as a student, https://www.open.edu.au/advice/insights/ethical-way-to-use-chatgpt-as-a-student (2023).
University of Toronto. ChatGPT and Generative AI in the Classroom, https://www.viceprovostundergrad.utoronto.ca/strategic-priorities/digital-learning/special-initiative-artificial-intelligence (2023).
Caulfield, J. University Policies on AI Writing Tools | Overview & List, https://www.scribbr.com/ai-tools/chatgpt-university-policies/ (2023).
United Nations Children’s Fund. UNESCO survey: Less than 10% of schools and universities have formal guidance on AI, https://www.unesco.org/en/articles/unesco-survey-less-10-schools-and-universities-have-formal-guidance-ai (2023).
Brin, D. et al. Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci Rep. 13, 16492. https://doi.org/10.1038/s41598-023-43436-9 (2023).
doi: 10.1038/s41598-023-43436-9
pubmed: 37779171
pmcid: 10543445
Bommarito, M. J. & Katz, D. M. GPT Takes the Bar Exam. SSRN. https://doi.org/10.2139/ssrn.4314839 (2022).
doi: 10.2139/ssrn.4314839
Stokel-Walker, C. AI bot ChatGPT writes smart essays - should professors worry?. Nature https://doi.org/10.1038/d41586-022-04397-7 (2022).
doi: 10.1038/d41586-022-04397-7
pubmed: 36494443
Graham, F. Daily briefing: Will ChatGPT kill the essay assignment?. Nature https://doi.org/10.1038/d41586-022-04437-2 (2022).
doi: 10.1038/d41586-022-04437-2
pubmed: 36600000
pmcid: 9684080
von Garrel, J. & Mayer, J. Artificial Intelligence in studies—use of ChatGPT and AI-based tools among students in Germany. Humanit. Soc. Sci. Commun. 10, 799. https://doi.org/10.1057/s41599-023-02304-7 (2023).
doi: 10.1057/s41599-023-02304-7
Malik, A. R. et al. Exploring Artificial Intelligence in Academic Essay: Higher Education Student’s Perspective. Int. J. Educ. Res. Open 5, 100296. https://doi.org/10.1016/j.ijedro.2023.100296 (2023).
doi: 10.1016/j.ijedro.2023.100296
Bašić, Ž, Banovac, A., Kružić, I. & Jerković, I. ChatGPT-3.5 as writing assistance in students’ essays. Humanit. Soc. Sci. Commun. 10, 750. https://doi.org/10.1057/s41599-023-02269-7 (2023).
doi: 10.1057/s41599-023-02269-7
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z. & Trautsch, A. A large-scale comparison of human-written versus ChatGPT-generated essays. Sci. Rep. 13, 18617. https://doi.org/10.1038/s41598-023-45644-9 (2023).
doi: 10.1038/s41598-023-45644-9
pubmed: 37903836
pmcid: 10616290
Meyer, J. G. et al. ChatGPT and large language models in academia: opportunities and challenges. BioData Min. 16(1), 20. https://doi.org/10.1186/s13040-023-00339-9 (2023).
doi: 10.1186/s13040-023-00339-9
pubmed: 37443040
pmcid: 10339472
Hwang, S. I. et al. Is ChatGPT a “fire of prometheus” for non-native English-speaking researchers in academic writing?. Korean. J. Radiol. 24(10), 952. https://doi.org/10.3348/kjr.2023.0773 (2023).
doi: 10.3348/kjr.2023.0773
pubmed: 37793668
pmcid: 10550740
Huang, J. & Tan, M. The role of ChatGPT in scientific communication: writing better scientific, review articles. Am. J. Cancer. Res. 13(4), 1148 (2023).
pubmed: 37168339
pmcid: 10164801
Ng, A. L. & Ong, J. ChatGPT vs Bard: Which is a Better Writer? 2024 IEEE Conference on Artericial Intelligence. https://doi.org/10.1109/CAI59869.2024.00066 (2024).
Ahmet, I. et al. ChatGPT versus Bard: A comparative study. Engineering Reports. https://doi.org/10.1002/eng2.12890 (2024).
doi: 10.1002/eng2.12890
Li, Y. et al. Can large language models write reflectively. Comput. Educ. Artif. Intell. 4, 100140. https://doi.org/10.1016/j.caeai.2023.100140 (2023).
doi: 10.1016/j.caeai.2023.100140
Jiang, H. et al. PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits. Preprint at arXiv.2305.02547 (2024).
Sandler, M., Choung, H., Ross, A. & David, P. A Linguistic Comparison between Human and ChatGPT-Generated Conversations. Preprint at arXiv:2401.16587 (2024).
Perry, A. AI will never convey the essence of human empathy. Nat. Hum. Behav. 7, 1808–1809. https://doi.org/10.1038/s41562-023-01675-w (2023).
doi: 10.1038/s41562-023-01675-w
pubmed: 37474839
da Silva, G.S. & Ulbricht, V.R. ChatGPT and Bard in Education: A Comparative Review. 20th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA, 2023).
Motlagh, N.Y., Khajavi, M., Sharifi, A. & Ahmadi, M. The impact of artificial intelligence on the evolution of digital education: A comparative study of openAI text generation tools including ChatGPT, Bing Chat, Bard, and Ernie. Preprint at arXiv:2309.02029 (2023).
Patil, N. S., Huang, R. S., van der Pol, C. B. & Larocque, N. Comparative performance of ChatGPT and bard in a text-based radiology knowledge assessment. Can. Assoc. Radiol. J. 75(2), 344–350. https://doi.org/10.1177/08465371231193716 (2024).
doi: 10.1177/08465371231193716
pubmed: 37578849
Singh, S.K., Kumar. S. & Mehra, P.S. Chat GPT & Google Bard AI: A Review. International Conference on IoT, Communication and Automation Technology (ICICAT) 23 (pp. 1–6). IEEE https://doi.org/10.1109/ICICAT57735.2023.10263706 (2023).
Borji, A. & Mohammadian, M. Battle of the Wordsmiths: Comparing ChatGPT, GPT-4, Claude, and Bard. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4476855 (2023).
doi: 10.2139/ssrn.4476855
Australian Government - Tertiary Education Quality and Standards Agency. Assessment reform for the age of artificial intelligence. (TEQSA, 2023).
United Nations Children’s Fund. Guidance for generative AI in education and research. (UNESCO, 2023).
Hayawi, K., Shahriar, S. & Mathew, S. S. The imitation game: detecting human and AI-generated texts in the era of ChatGPT and BARD. J. Inf. Sci. https://doi.org/10.48550/arXiv.2307.12166 (2024).
doi: 10.48550/arXiv.2307.12166
Kabakus, A. T. & Dogru, I. The battle of Chatbot Giants: an experimental comparison of ChatGPT and Bard. IJERAD 16, 679–691. https://doi.org/10.29137/umagd.1390083 (2024).
doi: 10.29137/umagd.1390083
Boyd, R. L., Ashokkumar, A., Seraj, S. & Pennebaker, J. W. The Development and Psychometric Properties of LIWC-22. (University of Texas at Austin, 2022).