The application of objective clinical human reliability analysis (OCHRA) in the assessment of basic robotic surgical skills.
Assessment tools
Error analysis
OCHRA
Robotic surgery
Surgical training
Journal
Surgical endoscopy
ISSN: 1432-2218
Titre abrégé: Surg Endosc
Pays: Germany
ID NLM: 8806653
Informations de publication
Date de publication:
06 Nov 2023
06 Nov 2023
Historique:
received:
30
06
2023
accepted:
01
10
2023
medline:
7
11
2023
pubmed:
7
11
2023
entrez:
6
11
2023
Statut:
aheadofprint
Résumé
Using a validated, objective, and standardised assessment tool to assess progression and competency is essential for basic robotic surgical training programmes. Objective clinical human reliability analysis (OCHRA) is an error-based assessment tool that provides in-depth analysis of individual technical errors. We conducted a feasibility study to assess the concurrent validity and reliability of OCHRA when applied to basic, generic robotic technical skills assessment. Selected basic robotic surgical skill tasks, in virtual reality (VR) and dry lab equivalent, were performed by novice robotic surgeons during an intensive 5-day robotic surgical skills course on da Vinci® X and Xi surgical systems. For each task, we described a hierarchical task analysis. Our developed robotic surgical-specific OCHRA methodology was applied to error events in recorded videos with a standardised definition. Statistical analysis to assess concurrent validity with existing tools and inter-rater reliability were performed. OCHRA methodology was applied to 272 basic robotic surgical skills tasks performed by 20 novice robotic surgeons. Performance scores improved from the start of the course to the end using all three assessment tools; Global Evaluative Assessment of Robotic Skills (GEARS) [VR: t(19) = - 9.33, p < 0.001] [dry lab: t(19) = - 10.17, p < 0.001], OCHRA [VR: t(19) = 6.33, p < 0.001] [dry lab: t(19) = 10.69, p < 0.001] and automated VR [VR: t(19) = - 8.26, p < 0.001]. Correlation analysis, for OCHRA compared to GEARS and automated VR scores, shows a significant and strong inverse correlation in every VR and dry lab task; OCHRA vs GEARS [VR: mean r = - 0.78, p < 0.001] [dry lab: mean r = - 0.82, p < 0.001] and OCHRA vs automated VR [VR: mean r = - 0.77, p < 0.001]. There is very strong and significant inter-rater reliability between two independent reviewers (r = 0.926, p < 0.001). OCHRA methodology provides a detailed error analysis tool in basic robotic surgical skills with high reliability and concurrent validity with existing tools. OCHRA requires further evaluation in more advanced robotic surgical procedures.
Sections du résumé
BACKGROUND
BACKGROUND
Using a validated, objective, and standardised assessment tool to assess progression and competency is essential for basic robotic surgical training programmes. Objective clinical human reliability analysis (OCHRA) is an error-based assessment tool that provides in-depth analysis of individual technical errors. We conducted a feasibility study to assess the concurrent validity and reliability of OCHRA when applied to basic, generic robotic technical skills assessment.
METHODS
METHODS
Selected basic robotic surgical skill tasks, in virtual reality (VR) and dry lab equivalent, were performed by novice robotic surgeons during an intensive 5-day robotic surgical skills course on da Vinci® X and Xi surgical systems. For each task, we described a hierarchical task analysis. Our developed robotic surgical-specific OCHRA methodology was applied to error events in recorded videos with a standardised definition. Statistical analysis to assess concurrent validity with existing tools and inter-rater reliability were performed.
RESULTS
RESULTS
OCHRA methodology was applied to 272 basic robotic surgical skills tasks performed by 20 novice robotic surgeons. Performance scores improved from the start of the course to the end using all three assessment tools; Global Evaluative Assessment of Robotic Skills (GEARS) [VR: t(19) = - 9.33, p < 0.001] [dry lab: t(19) = - 10.17, p < 0.001], OCHRA [VR: t(19) = 6.33, p < 0.001] [dry lab: t(19) = 10.69, p < 0.001] and automated VR [VR: t(19) = - 8.26, p < 0.001]. Correlation analysis, for OCHRA compared to GEARS and automated VR scores, shows a significant and strong inverse correlation in every VR and dry lab task; OCHRA vs GEARS [VR: mean r = - 0.78, p < 0.001] [dry lab: mean r = - 0.82, p < 0.001] and OCHRA vs automated VR [VR: mean r = - 0.77, p < 0.001]. There is very strong and significant inter-rater reliability between two independent reviewers (r = 0.926, p < 0.001).
CONCLUSION
CONCLUSIONS
OCHRA methodology provides a detailed error analysis tool in basic robotic surgical skills with high reliability and concurrent validity with existing tools. OCHRA requires further evaluation in more advanced robotic surgical procedures.
Identifiants
pubmed: 37932602
doi: 10.1007/s00464-023-10510-2
pii: 10.1007/s00464-023-10510-2
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2023. Crown.
Références
Intuitive Surgical Inc (2021) Intuitive reaches 10 million procedures performed using da Vinci Surgical Systems. https://isrg.intuitive.com/news-releases/news-release-details/intuitive-reaches-10-million-procedures-performed-using-da-vinci
Dixon F, Keeler BD (2020) Robotic surgery: training, competence assessment and credentialing. Bull R Coll Surg Engl 102:302–306. https://doi.org/10.1308/rcsbull.2020.185
doi: 10.1308/rcsbull.2020.185
Chen R, Rodrigues Armijo P, Krause C, Siu KC, Oleynikov D (2020) A comprehensive review of robotic surgery curriculum and training for residents, fellows, and postgraduate surgical education. Surg Endosc 34:361–367. https://doi.org/10.1007/s00464-019-06775-1
doi: 10.1007/s00464-019-06775-1
pubmed: 30953199
Alemzadeh H, Raman J, Leveson N, Kalbarczyk Z, Iyer RK (2016) Adverse events in robotic surgery: a retrospective study of 14 years of FDA data. PLoS ONE 11:e0151470. https://doi.org/10.1371/journal.pone.0151470
doi: 10.1371/journal.pone.0151470
pubmed: 27097160
pmcid: 4838256
Vanlander AE, Mazzone E, Collins JW, Mottrie AM, Rogiers XM, van der Poel HG, Van Herzeele I, Satava RM, Gallagher AG (2020) Orsi Consensus Meeting on European Robotic Training (OCERT): results from the first multispecialty consensus meeting on training in robot-assisted surgery. Eur Urol 78:713–716
doi: 10.1016/j.eururo.2020.02.003
pubmed: 32089358
Challacombe B, Ahmed K, Soomro N, Dasgupta P, Shamim Khan M, Cross W, Weston R, Sanger V, Joyce A, O’Flynn K, Speakman M (2015) British Association of Urological Surgeons (BAUS) Robotic surgery curriculum—guidelines for training. https://www.baus.org.uk/professionals/baus_business/publications/83/robotic_surgery_curriculum/
Puliatti S, Mazzone E, Dell’Oglio P (2020) Training in robot-assisted surgery. Curr Opin Urol 30:65–72. https://doi.org/10.1097/MOU.0000000000000687
doi: 10.1097/MOU.0000000000000687
pubmed: 31725006
Foster JD, Miskovic D, Allison AS, Conti JA, Ockrim J, Cooper EJ, Hanna GB, Francis NK (2016) Application of objective clinical human reliability analysis (OCHRA) in assessment of technical performance in laparoscopic rectal cancer surgery. Tech Coloproctol 20:361–367. https://doi.org/10.1007/s10151-016-1444-4
doi: 10.1007/s10151-016-1444-4
pubmed: 27154295
Tang B, Hanna GB, Cuschieri A (2005) Analysis of errors enacted by surgical trainees during skills training courses. Surgery 138:14–20. https://doi.org/10.1016/J.SURG.2005.02.014
doi: 10.1016/J.SURG.2005.02.014
pubmed: 16003310
Joice P, Hanna GB, Cuschieri A (1998) Errors enacted during endoscopic surgery—a human reliability analysis. Appl Ergon 29:409–414. https://doi.org/10.1016/S0003-6870(98)00016-7
doi: 10.1016/S0003-6870(98)00016-7
pubmed: 9796785
Gauba V, Tsangaris P, Tossounis C, Mitra A, McLean C, Saleh GM (2008) Human reliability analysis of cataract surgery. Arch Ophthalmol 126:173–177. https://doi.org/10.1001/ARCHOPHTHALMOL.2007.47
doi: 10.1001/ARCHOPHTHALMOL.2007.47
pubmed: 18268206
Curtis NJ, Dennison G, Brown CSB, Hewett PJ, Hanna GB, Stevenson ARL, Francis NK (2021) Clinical evaluation of intraoperative near misses in laparoscopic rectal cancer surgery. Ann Surg 273:778–784. https://doi.org/10.1097/SLA.0000000000003452
doi: 10.1097/SLA.0000000000003452
pubmed: 31274657
Miskovic D, Ni M, Wyles SM, Parvaiz A, Hanna GB (2012) Observational clinical human reliability analysis (OCHRA) for competency assessment in laparoscopic colorectal surgery at the specialist level. Surg Endosc 26:796–803. https://doi.org/10.1007/S00464-011-1955-Z/TABLES/4
doi: 10.1007/S00464-011-1955-Z/TABLES/4
pubmed: 22042584
Lyman WB, Passeri MJ, Murphy K, Siddiqui IA, Khan AS, Iannitti DA, Martinie JB, Baker EH, Vrochides D (2021) An objective approach to evaluate novice robotic surgeons using a combination of kinematics and stepwise cumulative sum (CUSUM) analyses. Surg Endosc 35:2765–2772. https://doi.org/10.1007/s00464-020-07708-z
doi: 10.1007/s00464-020-07708-z
pubmed: 32556751
Hung AJ, Chen J, Gill IS (2018) Automated performance metrics and machine learning algorithms tomeasure surgeon performance and anticipate clinical outcomes in robotic surgery. JAMA Surg 153:770–771
doi: 10.1001/jamasurg.2018.1512
pubmed: 29926095
pmcid: 9084629
Hung AJ, Liu Y, Anandkumar A (2021) Deep learning to automate technical skills assessment in robotic surgery. JAMA Surg 156:1059–1060
doi: 10.1001/jamasurg.2021.3651
pubmed: 34524401
Hung AJ, Ma R, Cen S, Nguyen JH, Lei X, Wagner C (2021) Surgeon automated performance metrics as predictors of early urinary continence recovery after robotic radical prostatectomy—a prospective bi-institutional study. Eur Urol Open Sci 27:65–72. https://doi.org/10.1016/j.euros.2021.03.005
doi: 10.1016/j.euros.2021.03.005
pubmed: 33959725
pmcid: 8095672
Chen J, Cheng N, Cacciamani G, Oh P, Lin-Brande M, Remulla D, Gill IS, Hung AJ (2019) Objective assessment of robotic surgical technical skill: a systematic review. J Urol 201:461–469
doi: 10.1016/j.juro.2018.06.078
pubmed: 30053510
Hertz AM, George EI, Vaccaro CM, Brand TC (2018) Head-to-head comparison of three virtual-reality robotic surgery simulators. J Soc Laparosc Robot Surg 22:e2017.00081. https://doi.org/10.4293/JSLS.2017.00081
doi: 10.4293/JSLS.2017.00081
Abboudi H, Khan MS, Aboumarzouk O, Guru KA, Challacombe B, Dasgupta P, Ahmed K (2013) Current status of validation for robotic surgery simulators—a systematic review. BJU Int 111:194–205. https://doi.org/10.1111/j.1464-410X.2012.11270.x
doi: 10.1111/j.1464-410X.2012.11270.x
pubmed: 22672340
Schmidt MW, Köppinger KF, Fan C, Kowalewski K-F, Schmidt LP, Vey J, Proctor T, Probst P, Bintintan VV, Müller-Stich B-P, Nickel F (2021) Virtual reality simulation in robot-assisted surgery: meta-analysis of skill transfer and predictability of skill. BJS Open. https://doi.org/10.1093/bjsopen/zraa066
doi: 10.1093/bjsopen/zraa066
pubmed: 33864069
pmcid: 8052560
Guerin S, Huaulmé A, Lavoue V, Jannin P, Timoh KN (2022) Review of automated performance metrics to assess surgical technical skills in robot-assisted laparoscopy. Surg Endosc 36:853–870. https://doi.org/10.1007/s00464-021-08792-5
doi: 10.1007/s00464-021-08792-5
pubmed: 34750700
Alshuaibi M, Perrenot C, Hubert J, Perez M (2020) Concurrent, face, content, and construct validity of the RobotiX Mentor simulator for robotic basic skills. Int J Med Robot Comput Assist Surg. https://doi.org/10.1002/rcs.2100
doi: 10.1002/rcs.2100
Sánchez R, Rodríguez O, Rosciano J, Vegas L, Bond V, Rojas A, Sanchez-Ismayel A (2016) Robotic surgery training: construct validity of Global Evaluative Assessment of Robotic Skills (GEARS). J Robot Surg 10:227–231. https://doi.org/10.1007/s11701-016-0572-1
doi: 10.1007/s11701-016-0572-1
pubmed: 27039189
Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187:247–252. https://doi.org/10.1016/j.juro.2011.09.032
doi: 10.1016/j.juro.2011.09.032
pubmed: 22099993
Tang B, Hanna GB, Joice P, Cuschieri A (2004) Identification and categorization of technical errors by observational clinical human reliability assessment (OCHRA) during laparoscopic cholecystectomy. Arch Surg 139:1215. https://doi.org/10.1001/archsurg.139.11.1215
doi: 10.1001/archsurg.139.11.1215
pubmed: 15545569
Francis NK, Curtis NJ, Conti JA, Foster JD, Bonjer HJ, Hanna GB, Abu-Hilal M, Agresta F, Antoniu SA, Arezzo A, Balagúe C, Boni L, Bouvy N, Carus T, Edwin B, Diana M, Faria G, Ignjatovic D, de Manzini N, Margallo FM, Martinek L, Matveev N, Mintz Y, Nakajima K, Popa DE, Schijven PJ, Sedman P, Yiannakopoulou E (2018) EAES classification of intraoperative adverse events in laparoscopic surgery. Surg Endosc 32:3822–3829. https://doi.org/10.1007/s00464-018-6108-1
doi: 10.1007/s00464-018-6108-1
pubmed: 29435754
Salmon P, Stanton NA, Walker G (2003) Human factors design methods review. Human Factors Integration Defence Technology Centre. Ref HFIDTC/WPI.3./1. https://eprints.soton.ac.uk/368316/1/__soton.ac.uk_ude_personalfiles_users_jr1d11_mydesktop_ePrints_hf-design-methods-review.pdf
Guni A, Raison N, Challacombe B, Khan S, Dasgupta P, Ahmed K (2018) Development of a technical checklist for the assessment of suturing in robotic surgery. Surg Endosc 32:4402–4407. https://doi.org/10.1007/S00464-018-6407-6/TABLES/3
doi: 10.1007/S00464-018-6407-6/TABLES/3
pubmed: 30194643
Sridhar AN, Briggs TP, Kelly JD, Nathan S (2017) Training in robotic surgery—an overview. Curr Urol Rep 18:58. https://doi.org/10.1007/s11934-017-0710-y
doi: 10.1007/s11934-017-0710-y
pubmed: 28647793
pmcid: 5486586
Husslein H, Shirreff L, Shore EM, Lefebvre GG, Grantcharov TP (2015) The Generic Error Rating Tool: a novel approach to assessment of performance and surgical education in gynecologic laparoscopy. J Surg Educ 72:1259–1265. https://doi.org/10.1016/J.JSURG.2015.04.029
doi: 10.1016/J.JSURG.2015.04.029
pubmed: 26111823
Curry M, Malpani A, Li R, Tantillo T, Jog A, Blanco R, Ha PK, Califano J, Kumar R, Richmon J (2012) Objective assessment in residency-based training for transoral robotic surgery. Laryngoscope 122:2184–2192. https://doi.org/10.1002/lary.23369
doi: 10.1002/lary.23369
pubmed: 22915265
pmcid: 4572744
Gomez ED, Aggarwal R, McMahan W, Bark K, Kuchenbecker KJ (2016) Objective assessment of robotic surgical skill using instrument contact vibrations. Surg Endosc 30:1419–1431. https://doi.org/10.1007/s00464-015-4346-z
doi: 10.1007/s00464-015-4346-z
pubmed: 26201410
Liu M, Purohit S, Mazanetz J, Allen W, Kreaden US, Curet M (2018) Assessment of Robotic Console Skills (ARCS): construct validity of a novel global rating scale for technical skills in robotically assisted surgery. Surg Endosc 32:526–535. https://doi.org/10.1007/s00464-017-5694-7
doi: 10.1007/s00464-017-5694-7
pubmed: 28667546
Lovegrove C, Novara G, Mottrie A, Guru KA, Brown M, Challacombe B, Popert R, Raza J, Van der Poel H, Peabody J, Dasgupta P, Ahmed K (2016) Structured and modular training pathway for robot-assisted radical prostatectomy (RARP): validation of the RARP assessment score and learning curve assessment. Eur Urol 69:526–535. https://doi.org/10.1016/j.eururo.2015.10.048
doi: 10.1016/j.eururo.2015.10.048
pubmed: 26585582
Hussein AA, Hinata N, Dibaj S, May PR, Kozlowski JD, Abol-Enein H, Abaza R, Eun D, Khan MS, Mohler JL, Agarwal P, Pohar K, Sarle R, Boris R, Mane SS, Hutson A, Guru KA (2017) Development, validation and clinical application of Pelvic Lymphadenectomy Assessment and Completion Evaluation: intraoperative assessment of lymph node dissection after robot-assisted radical cystectomy for bladder cancer. BJU Int 119:879–884. https://doi.org/10.1111/bju.13748
doi: 10.1111/bju.13748
pubmed: 27987527
pmcid: 8177005
Iqbal U, Jing Z, Ahmed Y, Elsayed AS, Rogers C, Boris R, Porter J, Allaf M, Badani K, Stifelman M, Kaouk J, Terakawa T, Hinata N, Aboumohamed AA, Kauffman E, Li Q, Abaza R, Guru KA, Hussein AA, Eun D (2022) Development and validation of an objective scoring tool for robot-assisted partial nephrectomy: scoring for partial nephrectomy. J Endourol 36:647–653. https://doi.org/10.1089/end.2021.0706
doi: 10.1089/end.2021.0706
pubmed: 34809491
Frederick PJ, Szender JB, Hussein AA, Kesterson JP, Shelton JA, Anderson TL, Barnabei VM, Guru K (2017) Surgical competency for robot-assisted hysterectomy: development and validation of a robotic hysterectomy assessment score (RHAS). J Minim Invasive Gynecol 24:55–61. https://doi.org/10.1016/j.jmig.2016.10.004
doi: 10.1016/j.jmig.2016.10.004
pubmed: 27780777
Goldenberg MG, Lee JY, Kwong JCC, Grantcharov TP, Costello A (2018) Implementing assessments of robot-assisted technical skill in urological education: a systematic review and synthesis of the validity evidence. BJU Int 122:501–519. https://doi.org/10.1111/bju.14219
doi: 10.1111/bju.14219
pubmed: 29603869
Bonrath EM, Zevin B, Dedy NJ, Grantcharov TP (2013) Error rating tool to identify and analyse technical errors and events in laparoscopic surgery. Br J Surg 100:1080–1088. https://doi.org/10.1002/bjs.9168
doi: 10.1002/bjs.9168
pubmed: 23754649
Peters JH, Fried GM, Swanstrom LL, Soper NJ, Sillin LF, Schirmer B, Hoffman K, the SAGES FLS Committee (2004) Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery 135:21–27
doi: 10.1016/S0039-6060(03)00156-9
pubmed: 14694297
Bann S, Datta V, Khan M, Darzi A (2003) The surgical error examination is a novel method for objective technical knowledge assessment. Am J Surg 185:507–511. https://doi.org/10.1016/S0002-9610(03)00081-3
doi: 10.1016/S0002-9610(03)00081-3
pubmed: 12781875
Bonrath EM, Dedy NJ, Zevin B, Grantcharov TP (2013) Defining technical errors in laparoscopic surgery: a systematic review. Surg Endosc 27:2678–2691. https://doi.org/10.1007/S00464-013-2827-5/FIGURES/2
doi: 10.1007/S00464-013-2827-5/FIGURES/2
pubmed: 23436086
Curtis NJ, Foster JD, Miskovic D, Brown CSB, Hewett PJ, Abbott S, Hanna GB, Stevenson ARL, Francis NK (2020) Association of surgical skill assessment with clinical outcomes in cancer surgery. JAMA Surg 155:590–598. https://doi.org/10.1001/jamasurg.2020.1004
doi: 10.1001/jamasurg.2020.1004
pubmed: 32374371
Tang B (2020) Observational clinical human reliability analysis (OCHRA) for assessing and improving quality of surgical performance: the current status and future. J Surg Simul. https://doi.org/10.1102/2051-7726.2020.0009
doi: 10.1102/2051-7726.2020.0009
Coleman M (2009) LAPCO: national training programme for laparoscopic colorectal surgery. Bull R Coll Surg Engl 91:274–275. https://doi.org/10.1308/147363509X467302
doi: 10.1308/147363509X467302
Hanna GB, Mackenzie H, Miskovic D, Ni M, Wyles S, Aylin P, Parvaiz A, Cecil T, Gudgeon A, Griffith J, Robinson JM, Selvasekar C, Rockall T, Acheson A, Maxwell-Armstrong C, Jenkins JT, Horgan A, Cunningham C, Lindsey I, Arulampalam T, Motson RW, Francis NK, Kennedy RH, Coleman MG (2022) Laparoscopic colorectal surgery outcomes improved after national training program (LAPCO) for specialists in England. Ann Surg 275:1149–1155. https://doi.org/10.1097/SLA.0000000000004584
doi: 10.1097/SLA.0000000000004584
pubmed: 33086313