Speech recognition technology for assessing team debriefing communication and interaction patterns: An algorithmic toolkit for healthcare simulation educators.
Affective computing
Debriefing
Education
Healthcare
Human-computer interaction
Simulation
Sociograms
Speaker diarization
Journal
Advances in simulation (London, England)
ISSN: 2059-0628
Titre abrégé: Adv Simul (Lond)
Pays: England
ID NLM: 101700425
Informations de publication
Date de publication:
09 Oct 2024
09 Oct 2024
Historique:
received:
08
04
2024
accepted:
20
09
2024
medline:
10
10
2024
pubmed:
10
10
2024
entrez:
10
10
2024
Statut:
epublish
Résumé
Debriefings are central to effective learning in simulation-based medical education. However, educators often face challenges when conducting debriefings, which are further compounded by the lack of empirically derived knowledge on optimal debriefing processes. The goal of this study was to explore the technical feasibility of audio-based speaker diarization for automatically, objectively, and reliably measuring debriefing interaction patterns among debriefers and participants. Additionally, it aimed to investigate the ability to automatically create statistical analyses and visualizations, such as sociograms, solely from the audio recordings of debriefings among debriefers and participants. We used a microphone to record the audio of debriefings conducted during simulation-based team training with third-year medical students. The debriefings were led by two healthcare simulation instructors. We processed the recorded audio file using speaker diarization machine learning algorithms and validated the results manually to showcase its accuracy. We selected two debriefings to compare the speaker diarization results between different sessions, aiming to demonstrate similarities and differences in interaction patterns. Ten debriefings were analyzed, each lasting about 30 min. After data processing, the recorded data enabled speaker diarization, which in turn facilitated the automatic creation of visualized interaction patterns, such as sociograms. The findings and data visualizations demonstrated the technical feasibility of implementing audio-based visualizations of interaction patterns, with an average accuracy of 97.78%.We further analyzed two different debriefing cases to uncover similarities and differences between the sessions. By quantifying the response rate from participants, we were able to determine and quantify the level of interaction patterns triggered by instructors in each debriefing session. In one session, the debriefers triggered 28% of the feedback from students, while in the other session, this percentage increased to 36%. Our results indicate that speaker diarization technology can be applied accurately and automatically to provide visualizations of debriefing interactions. This application can be beneficial for the development of simulation educator faculty. These visualizations can support instructors in facilitating and assessing debriefing sessions, ultimately enhancing learning outcomes in simulation-based healthcare education.
Sections du résumé
BACKGROUND
BACKGROUND
Debriefings are central to effective learning in simulation-based medical education. However, educators often face challenges when conducting debriefings, which are further compounded by the lack of empirically derived knowledge on optimal debriefing processes. The goal of this study was to explore the technical feasibility of audio-based speaker diarization for automatically, objectively, and reliably measuring debriefing interaction patterns among debriefers and participants. Additionally, it aimed to investigate the ability to automatically create statistical analyses and visualizations, such as sociograms, solely from the audio recordings of debriefings among debriefers and participants.
METHODS
METHODS
We used a microphone to record the audio of debriefings conducted during simulation-based team training with third-year medical students. The debriefings were led by two healthcare simulation instructors. We processed the recorded audio file using speaker diarization machine learning algorithms and validated the results manually to showcase its accuracy. We selected two debriefings to compare the speaker diarization results between different sessions, aiming to demonstrate similarities and differences in interaction patterns.
RESULTS
RESULTS
Ten debriefings were analyzed, each lasting about 30 min. After data processing, the recorded data enabled speaker diarization, which in turn facilitated the automatic creation of visualized interaction patterns, such as sociograms. The findings and data visualizations demonstrated the technical feasibility of implementing audio-based visualizations of interaction patterns, with an average accuracy of 97.78%.We further analyzed two different debriefing cases to uncover similarities and differences between the sessions. By quantifying the response rate from participants, we were able to determine and quantify the level of interaction patterns triggered by instructors in each debriefing session. In one session, the debriefers triggered 28% of the feedback from students, while in the other session, this percentage increased to 36%.
CONCLUSION
CONCLUSIONS
Our results indicate that speaker diarization technology can be applied accurately and automatically to provide visualizations of debriefing interactions. This application can be beneficial for the development of simulation educator faculty. These visualizations can support instructors in facilitating and assessing debriefing sessions, ultimately enhancing learning outcomes in simulation-based healthcare education.
Identifiants
pubmed: 39385298
doi: 10.1186/s41077-024-00315-1
pii: 10.1186/s41077-024-00315-1
doi:
Types de publication
Journal Article
Langues
eng
Pagination
42Subventions
Organisme : Eidgenössische Technische Hochschule Zürich
ID : Grant number
Informations de copyright
© 2024. The Author(s).
Références
Abegglen S, Greif R, Balmer Y, Znoj HJ, Nabecker S. Debriefing interaction patterns and learning outcomes in simulation: an observational mixed-methods network study. Adv Simul. 2022;7(1):1–10.
doi: 10.1186/s41077-022-00222-3
Ali AA, Musallam E. Debriefing quality evaluation in nursing simulation-based education: an integrative review. Clin Simul Nurs. 2018;16:15–24.
doi: 10.1016/j.ecns.2017.09.009
Allen JA, Lehmann-Willenbrock N. The science of workplace meetings: Integrating findings, building new theoretical angles, and embracing cross-disciplinary research. Organ Psychol Rev. 2023;13(4):351–4. https://doi.org/10.1177/20413866221122896 .
Arora S, Ahmed M, Paige J, Nestel D, Runnacles J, Hull L, et al. Objective structured assessment of debriefing: bringing science to the art of debriefing in surgery. Ann Surg. 2012;256(6):982–8.
doi: 10.1097/SLA.0b013e3182610c91
pubmed: 22895396
Hervé Bredin. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. 24th INTERSPEECH Conference (INTERSPEECH 2023). Dublin, Ireland. 2023;1983–7. https://doi.org/10.21437/Interspeech.2023-105 .
Brett-Fleegler M, Rudolph J, Eppich W, Monuteaux M, Fleegler E, Cheng A, et al. Debriefing assessment for simulation in healthcare: development and psychometric properties. Simul Healthc. 2012;7(5):288–94.
doi: 10.1097/SIH.0b013e3182620228
pubmed: 22902606
Cheng A, Morse KJ, Rudolph J, Arab AA, Runnacles J, Eppich W. Learner-centered debriefing for health care simulation education: lessons for faculty development. Simul Healthc. 2016;11(1):32–40.
doi: 10.1097/SIH.0000000000000136
pubmed: 26836466
Coggins A, Hong SS, Baliga K, Halamek LP. Immediate faculty feedback using debriefing timing data and conversational diagrams. Adv Simul. 2022;7(1):1–10.
doi: 10.1186/s41077-022-00203-6
Dieckmann P, Molin Friis S, Lippert A, Østergaard D. The art and science of debriefing in simulation: ideal and practice. Med Teach. 2009;31(7):e287–94.
doi: 10.1080/01421590902866218
pubmed: 19811136
Doas M, et al. Are we losing the art of actively listening to our patients? Connecting the art of active listening with emotionally competent behaviors. Open J Nurs. 2015;5(06):566.
doi: 10.4236/ojn.2015.56060
Dubey H, Gopal V, Cutler R, Aazami A, Matusevych S, Braun S, et al. ICASSP 2022 deep noise suppression challenge. In: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022. pp. 9271–9275. https://doi.org/10.1109/ICASSP43922.2022.9747230 .
Duff JP, Morse KJ, Seelandt J, Gross IT, Lydston M, Sargeant J, et al. Debriefing methods for simulation in healthcare: a systematic review. Simul Healthc. 2024;19(1S):S112–21.
doi: 10.1097/SIH.0000000000000765
pubmed: 38240623
Duvivier V, Carosin E, Derobertmasure A, Demeuse M. Simulation-Oriented Training: Analysis and Modeling of Trainer Activity During Post-simulation Debriefing (D-STAM) ISSN: 2758-0962 The Paris Conference on Education 2023: Official Conference Proceedings. 2023;147–67. https://doi.org/10.22492/issn.2758-0962.2023.15 .
Eppich W, Cheng A. Promoting Excellence and Reflective Learning in Simulation (PEARLS): development and rationale for a blended approach to health care simulation debriefing. Simul Healthc. 2015;10(2):106–15.
doi: 10.1097/SIH.0000000000000072
pubmed: 25710312
Fanning RM, Gaba DM. The role of debriefing in simulation-based learning. Simul Healthc. 2007;2(2):115–25.
doi: 10.1097/SIH.0b013e3180315539
pubmed: 19088616
Havyer RD, Wingo MT, Comfere NI, Nelson DR, Halvorsen AJ, McDonald FS, et al. Teamwork assessment in internal medicine: a systematic review of validity evidence and outcomes. J Gen Intern Med. 2014;29:894–910.
doi: 10.1007/s11606-013-2686-8
pubmed: 24327309
Jaye P, Thomas L, Reedy G. ‘The Diamond’: a structure for simulation debrief. Clin Teach. 2015;12(3):171–5.
doi: 10.1111/tct.12300
pubmed: 26009951
pmcid: 4497353
Kainth, Ranjev MBBS, MRCPCH, MA; Reedy, Gabriel PhD, MEd. Transforming Professional Identity in Simulation Debriefing: A Systematic Metaethnographic Synthesis of the Simulation Literature. Simulation in Healthcare: J Soc Simul Healthcare. 2024;19(2):90–104. https://doi.org/10.1097/SIH.0000000000000734 .
Kim YJ, Yoo JH. The utilization of debriefing for simulation in healthcare: a literature review. Nurse Educ Pract. 2020;43:102698.
doi: 10.1016/j.nepr.2020.102698
pubmed: 32004851
Kolbe M, Weiss M, Grote G, Knauth A, Dambach M, Spahn DR, et al. TeamGAINS: a tool for structured debriefings for simulation-based team trainings. BMJ Qual Saf. 2013;22(7):541–53.
doi: 10.1136/bmjqs-2012-000917
pubmed: 23525093
Kolbe M, Grande B, Spahn DR. Briefing and debriefing during simulation-based training and beyond: content, structure, attitude and setting. Best Pract Res Clin Anaesthesiol. 2015;29(1):87–96. https://doi.org/10.1016/j.bpa.2015.01.002 .
Kolbe M, Marty A, Seelandt J, Grande B. How to debrief teamwork interactions: using circular questions to explore and change team interaction patterns. Adv Simul. 2016;1:1–8.
doi: 10.1186/s41077-016-0029-7
Kolbe M, Eppich W, Rudolph J, Meguerdichian M, Catena H, Cripps A, et al. Managing psychological safety in debriefings: a dynamic balancing act. BMJ Simul Technol Enhanc Learn. 2020;6(3):164.
doi: 10.1136/bmjstel-2019-000470
pubmed: 35518370
pmcid: 8936758
Kolbe M, Grande B, Lehmann-Willenbrock N, Seelandt JC. Helping healthcare teams to debrief effectively: associations of debriefers’ actions and participants’ reflections during team debriefings. BMJ Qual Saf. 2023;32(3):160–72.
doi: 10.1136/bmjqs-2021-014393
pubmed: 35902231
Kolbe M, Goldhahn J, Useini M, Grande B. “Asking for help is a strength”—how to promote undergraduate medical students’ teamwork through simulation training and interprofessional faculty. Front Psychol. 2023;14.
Luctkar-Flude M, Tyerman J, Verkuyl M, Goldsworthy S, Harder N, Wilson-Keates B, et al. Effectiveness of debriefing methods for virtual simulation: a systematic review. Clin Simul Nurs. 2021;57:18–30.
doi: 10.1016/j.ecns.2021.04.009
Lyons R, Lazzara EH, Benishek LE, Zajac S, Gregory M, Sonesh SC, et al. Enhancing the effectiveness of team debriefings in medical simulation: more best practices. Joint Comm J Qual Patient Saf. 2015;41(3):115–25.
Okuda Y, Bryson EO, DeMaria S Jr, Jacobson L, Quinones J, Shen B, et al. The utility of simulation in medical education: what is the evidence? Mt Sinai J Med J Transl Personalized Med. 2009;76(4):330–43.
doi: 10.1002/msj.20127
Plaquet A, Bredin H. Powerset multi-class cross entropy loss for neural speaker diarization. 24th Interspeech Conference (INTERSPEECH 2023), ISCA: International Speech Communication As- sociation. Dublin; 2023. pp. 3222–6. https://doi.org/10.21437/Interspeech.2023-205 .
Reed SJ. Measuring learning and engagement during debriefing: a new instrument. Clin Simul Nurs. 2020;46:15–21.
doi: 10.1016/j.ecns.2020.03.002
Rochester SR. The significance of pauses in spontaneous speech. J Psycholinguist Res. 1973;2:51–81.
doi: 10.1007/BF01067111
pubmed: 24197795
Rosser AA, Qadadha YM, Thompson RJ, Jung HS, Jung S. Measuring the impact of simulation debriefing on the practices of interprofessional trauma teams using natural language processing. Am J Surg. 2023;225(2):394–9.
doi: 10.1016/j.amjsurg.2022.09.018
pubmed: 36207174
Rudolph JW, Simon R, Rivard P, Dufresne RL, Raemer DB. Debriefing with good judgment: combining rigorous feedback with genuine inquiry. Anesthesiol Clin. 2007;25(2):361–76.
doi: 10.1016/j.anclin.2007.03.007
pubmed: 17574196
Rudolph JW, Simon R, Raemer DB, Eppich WJ. Debriefing as formative assessment: closing performance gaps in medical education. Acad Emerg Med. 2008;15(11):1010–6.
doi: 10.1111/j.1553-2712.2008.00248.x
pubmed: 18945231
Salas E, Rosen MA, Weaver SJ, Held JD, Weissmuller JJ. Guidelines for performance measurement in simulation-based training. Ergon Des. 2009;17(4):12Y18.
Salik I, Paige JT. Debriefing the Interprofessional Team in Medical Simulation. In: StatPearls. Treasure Island (FL): StatPearls Publishing; 2023.
Sawyer T, Eppich W, Brett-Fleegler M, Grant V, Cheng A. More than one way to debrief: a critical review of healthcare simulation debriefing methods. Simul Healthc. 2016;11(3):209–17.
doi: 10.1097/SIH.0000000000000148
pubmed: 27254527
Seelandt JC, Grande B, Kriech S, Kolbe M. DE-CODE: a coding scheme for assessing debriefing interactions. BMJ Simul Technol Enhanc Learn. 2018;4(2):51.
doi: 10.1136/bmjstel-2017-000233
pubmed: 35515884
pmcid: 8990183
Van Schaik S, Plant J, Bridget O, et al. Challenges of interprofessional team training: a qualitative analysis of residents’ perceptions. Educ Health. 2015;28(1):52–7.
doi: 10.4103/1357-6283.161883
Weiss KE, Kolbe M, Lohmeyer Q, Meboldt M. Measuring teamwork for training in healthcare using eye tracking and pose estimation. Front Psychol. 2023;14:1169940.
doi: 10.3389/fpsyg.2023.1169940
pubmed: 37325757
pmcid: 10264622
Woolley AW, Chabris CF, Pentland A, Hashmi N, Malone TW. Evidence for a collective intelligence factor in the performance of human groups. Science. 2010;330(6004):686–8.
doi: 10.1126/science.1193147
pubmed: 20929725
Zhao S, Ma B, Watcharasupat KN, Gan WS. FRCRN: boosting feature representation using frequency recurrence for monaural speech enhancement. In: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022. pp. 9281–9285. https://doi.org/10.1109/ICASSP43922.2022.9747578 .