Technical skill assessment in minimally invasive surgery using artificial intelligence: a systematic review.
Artificial intelligence
Minimally invasive surgery
Surgical data science
Surgical skill assessment
Technical skill assessment
Journal
Surgical endoscopy
ISSN: 1432-2218
Titre abrégé: Surg Endosc
Pays: Germany
ID NLM: 8806653
Informations de publication
Date de publication:
10 2023
10 2023
Historique:
received:
15
05
2023
accepted:
20
07
2023
medline:
4
10
2023
pubmed:
16
8
2023
entrez:
16
8
2023
Statut:
ppublish
Résumé
Technical skill assessment in surgery relies on expert opinion. Therefore, it is time-consuming, costly, and often lacks objectivity. Analysis of intraoperative data by artificial intelligence (AI) has the potential for automated technical skill assessment. The aim of this systematic review was to analyze the performance, external validity, and generalizability of AI models for technical skill assessment in minimally invasive surgery. A systematic search of Medline, Embase, Web of Science, and IEEE Xplore was performed to identify original articles reporting the use of AI in the assessment of technical skill in minimally invasive surgery. Risk of bias (RoB) and quality of the included studies were analyzed according to Quality Assessment of Diagnostic Accuracy Studies criteria and the modified Joanna Briggs Institute checklists, respectively. Findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. In total, 1958 articles were identified, 50 articles met eligibility criteria and were analyzed. Motion data extracted from surgical videos (n = 25) or kinematic data from robotic systems or sensors (n = 22) were the most frequent input data for AI. Most studies used deep learning (n = 34) and predicted technical skills using an ordinal assessment scale (n = 36) with good accuracies in simulated settings. However, all proposed models were in development stage, only 4 studies were externally validated and 8 showed a low RoB. AI showed good performance in technical skill assessment in minimally invasive surgery. However, models often lacked external validity and generalizability. Therefore, models should be benchmarked using predefined performance metrics and tested in clinical implementation studies.
Sections du résumé
BACKGROUND
Technical skill assessment in surgery relies on expert opinion. Therefore, it is time-consuming, costly, and often lacks objectivity. Analysis of intraoperative data by artificial intelligence (AI) has the potential for automated technical skill assessment. The aim of this systematic review was to analyze the performance, external validity, and generalizability of AI models for technical skill assessment in minimally invasive surgery.
METHODS
A systematic search of Medline, Embase, Web of Science, and IEEE Xplore was performed to identify original articles reporting the use of AI in the assessment of technical skill in minimally invasive surgery. Risk of bias (RoB) and quality of the included studies were analyzed according to Quality Assessment of Diagnostic Accuracy Studies criteria and the modified Joanna Briggs Institute checklists, respectively. Findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.
RESULTS
In total, 1958 articles were identified, 50 articles met eligibility criteria and were analyzed. Motion data extracted from surgical videos (n = 25) or kinematic data from robotic systems or sensors (n = 22) were the most frequent input data for AI. Most studies used deep learning (n = 34) and predicted technical skills using an ordinal assessment scale (n = 36) with good accuracies in simulated settings. However, all proposed models were in development stage, only 4 studies were externally validated and 8 showed a low RoB.
CONCLUSION
AI showed good performance in technical skill assessment in minimally invasive surgery. However, models often lacked external validity and generalizability. Therefore, models should be benchmarked using predefined performance metrics and tested in clinical implementation studies.
Identifiants
pubmed: 37584774
doi: 10.1007/s00464-023-10335-z
pii: 10.1007/s00464-023-10335-z
pmc: PMC10520175
doi:
Types de publication
Systematic Review
Journal Article
Review
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
7412-7424Informations de copyright
© 2023. The Author(s).
Références
Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJO (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369:1434–1442. https://doi.org/10.1056/NEJMsa1300625
doi: 10.1056/NEJMsa1300625
pubmed: 24106936
Fecso AB, Bhatti JA, Stotland PK, Quereshy FA, Grantcharov TP (2019) Technical performance as a predictor of clinical outcomes in laparoscopic gastric cancer surgery. Ann Surg 270:115–120. https://doi.org/10.1097/SLA.0000000000002741
doi: 10.1097/SLA.0000000000002741
pubmed: 29578907
Stulberg JJ, Huang R, Kreutzer L, Ban K, Champagne BJ, Steele SR, Johnson JK, Holl JL, Greenberg CC, Bilimoria KY (2020) Association between surgeon technical skills and patient outcomes. JAMA Surg 155:960–968. https://doi.org/10.1001/jamasurg.2020.3007
doi: 10.1001/jamasurg.2020.3007
pubmed: 32838425
pmcid: 7439214
Curtis NJ, Foster JD, Miskovic D, Brown CSB, Hewett PJ, Abbott S, Hanna GB, Stevenson ARL, Francis NK (2020) Association of surgical skill assessment with clinical outcomes in cancer surgery. JAMA Surg 155:590–598. https://doi.org/10.1001/jamasurg.2020.1004
doi: 10.1001/jamasurg.2020.1004
pubmed: 32374371
Lendvay TS, White L, Kowalewski T (2015) Crowdsourcing to assess surgical skill. JAMA Surg 150:1086–1087. https://doi.org/10.1001/jamasurg.2015.2405
doi: 10.1001/jamasurg.2015.2405
pubmed: 26421369
Aggarwal R, Grantcharov T, Moorthy K, Milland T, Darzi A (2008) Toward feasible, valid, and reliable video-based assessments of technical surgical skills in the operating room. Ann Surg 247:372–379. https://doi.org/10.1097/SLA.0b013e318160b371
doi: 10.1097/SLA.0b013e318160b371
pubmed: 18216547
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. https://doi.org/10.1136/bmj.n71
doi: 10.1136/bmj.n71
pubmed: 33782057
pmcid: 8005924
Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A (2016) Rayyan—a web and mobile app for systematic reviews. Syst Rev 5:210. https://doi.org/10.1186/s13643-016-0384-4
doi: 10.1186/s13643-016-0384-4
pubmed: 27919275
pmcid: 5139140
Whiting PF (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155:529. https://doi.org/10.7326/0003-4819-155-8-201110180-00009
doi: 10.7326/0003-4819-155-8-201110180-00009
pubmed: 22007046
Anteby R, Horesh N, Soffer S, Zager Y, Barash Y, Amiel I, Rosin D, Gutman M, Klang E (2021) Deep learning visual analysis in laparoscopic surgery: a systematic review and diagnostic test accuracy meta-analysis. Surg Endosc 35:1521–1533. https://doi.org/10.1007/s00464-020-08168-1
doi: 10.1007/s00464-020-08168-1
pubmed: 33398560
Kwong MT, Colopy GW, Weber AM, Ercole A, Bergmann JHM (2019) The efficacy and effectiveness of machine learning for weaning in mechanically ventilated patients at the intensive care unit: a systematic review. Bio-Des Manuf 2:31–40. https://doi.org/10.1007/s42242-018-0030-1
doi: 10.1007/s42242-018-0030-1
Alonso-Silverio GA, Perez-Escamirosa F, Bruno-Sanchez R, Ortiz-Simon JL, Munoz-Guerrero R, Minor-Martinez A, Alarcon-Paredes A (2018) Development of a laparoscopic box trainer based on open source hardware and artificial intelligence for objective assessment of surgical psychomotor skills. Surg Innov 25:380–388. https://doi.org/10.1177/1553350618777045
doi: 10.1177/1553350618777045
pubmed: 29809097
Anastasiou D, Jin Y, Stoyanov D, Mazomenos E (2023) Keep your eye on the best: contrastive regression transformer for skill assessment in robotic surgery. IEEE Robot Autom Lett 8:1755–1762. https://doi.org/10.1109/LRA.2023.3242466
doi: 10.1109/LRA.2023.3242466
Anh NX, Chauhan S, Nataraja RM (2020) Towards near real-time assessment of surgical skills: A comparison of feature extraction techniques. Comput Methods Programs Biomed 187:105234. https://doi.org/10.1016/j.cmpb.2019.105234
doi: 10.1016/j.cmpb.2019.105234
pubmed: 31794913
Baghdadi A, Hussein AA, Ahmed Y, Guru KA, Cavuoto LA (2019) A computer vision technique for automated assessment of surgical performance using surgeons’ console-feed videos. Int J Comput Assist Radiol Surg 14:697–707. https://doi.org/10.1007/s11548-018-1881-9
doi: 10.1007/s11548-018-1881-9
pubmed: 30460490
Benmansour M, Handouzi W, Malti A (2018) A neural network architecture for automatic and objective surgical skill assessment. In: Proceedings of 2018 3rd international conference on electrical sciences and technologies in Maghreb (CISTEM), pp 1–5. https://doi.org/10.1109/CISTEM.2018.8613550
Benmansour M, Malti A, Jannin P (2023) Deep neural network architecture for automated soft surgical skills evaluation using objective structured assessment of technical skills criteria. Int J Comput Assist Radiol Surg 18:929–937. https://doi.org/10.1007/s11548-022-02827-5
doi: 10.1007/s11548-022-02827-5
pubmed: 36694051
Brown JD, Brien CEO, Leung SC, Dumon KR, Lee DI, Kuchenbecker KJ (2017) Using contact forces and robot arm accelerations to automatically rate surgeon skill at peg transfer. IEEE Trans Biomed Eng 64:2263–2275. https://doi.org/10.1109/TBME.2016.2634861
doi: 10.1109/TBME.2016.2634861
pubmed: 28113295
Castro D, Pereira D, Zanchettin C, Macêdo D, Bezerra BLD (2019) Towards optimizing convolutional neural networks for robotic surgery skill evaluation. In: Proceedings of 2019 international joint conference on neural networks (IJCNN), pp 1–8. https://doi.org/10.1109/IJCNN.2019.8852341
Fard MJ, Ameri S, Darin Ellis R, Chinnam RB, Pandya AK, Klein MD (2018) Automated robot-assisted surgical skill evaluation: predictive analytics approach. Int J Med Robot 14:e1850. https://doi.org/10.1002/rcs.1850
doi: 10.1002/rcs.1850
Fathabadi FR, Grantner JL, Shebrain SA, Abdel-Qader I (2021) Surgical skill assessment system using fuzzy logic in a multi-class detection of laparoscopic box-trainer instruments. In: Proceedings of 2021 IEEE international conference on systems, man, and cybernetics (SMC), pp 1248–1253. https://doi.org/10.1109/SMC52423.2021.9658766
Fawaz HI, Forestier G, Weber J, Idoumghar L, Muller PA (2019) Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks. Int J Comput Assist Radiol Surg 14:1611–1617. https://doi.org/10.1007/s11548-019-02039-4
doi: 10.1007/s11548-019-02039-4
Forestier G, Petitjean F, Senin P, Despinoy F, Huaulme A, Fawaz HI, Weber J, Idoumghar L, Muller PA, Jannin P (2018) Surgical motion analysis using discriminative interpretable patterns. Artif Intell Med 91:3–11. https://doi.org/10.1016/j.artmed.2018.08.002
doi: 10.1016/j.artmed.2018.08.002
pubmed: 30172445
French A, Kowalewski TM, Lendvay TS, Sweet RM (2017) Predicting surgical skill from the first N seconds of a task: value over task time using the isogony principle. Int J Comput Assist Radiol Surg 12:1161–1170. https://doi.org/10.1007/s11548-017-1606-5
doi: 10.1007/s11548-017-1606-5
pubmed: 28516300
Funke I, Speidel S, Mees ST, Weitz J (2019) Video-based surgical skill assessment using 3D convolutional neural networks. Int J Comput Assist Radiol Surg 14:1217–1225. https://doi.org/10.1007/s11548-019-01995-1
doi: 10.1007/s11548-019-01995-1
pubmed: 31104257
Gao Y, Yan P, Kruger U, Cavuoto L, Schwaitzberg S, De S, Intes X (2021) Functional brain imaging reliably predicts bimanual motor skill performance in a standardized surgical task. IEEE Trans Biomed Eng 68:2058–2066. https://doi.org/10.1109/TBME.2020.3014299
doi: 10.1109/TBME.2020.3014299
pubmed: 32755850
pmcid: 8265734
Islam G, Kahol K, Li BX, Smith M, Patel VL (2016) Affordable, web-based surgical skill training and evaluation tool. J Biomed Inform 59:102–114. https://doi.org/10.1016/j.jbi.2015.11.002
doi: 10.1016/j.jbi.2015.11.002
pubmed: 26556643
Jin A, Yeung S, Jopling J, Krause J, Azagury D, Milstein A, Fei-Fei L (2018) Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. In: Proceedings of 2018 IEEE winter conference on applications of computer vision (WACV), pp 691–699. https://doi.org/10.1109/WACV.2018.00081
Juarez-Villalobos L, Hevia-Montiel N, Perez-Gonzalez J (2021) Machine learning based classification of local robotic surgical skills in a training tasks set. In: Annual international conference of the IEEE Engineering in Medicine & Biology Society 2021, pp 4596–4599. https://doi.org/10.1109/EMBC46164.2021.9629579
Keles HO, Cengiz C, Demiral I, Ozmen MM, Omurtag A (2021) High density optical neuroimaging predicts surgeons’s subjective experience and skill levels. PLoS ONE 16:e0247117. https://doi.org/10.1371/journal.pone.0247117
doi: 10.1371/journal.pone.0247117
pubmed: 33600502
pmcid: 7891714
Kelly JD, Kowalewski TM, Petersen A, Lendvay TS (2020) Bidirectional long short-term memory for surgical skill classification of temporally segmented tasks. Int J Comput Assist Radiol Surg 15:2079–2088. https://doi.org/10.1007/s11548-020-02269-x
doi: 10.1007/s11548-020-02269-x
pubmed: 33000365
pmcid: 7677176
Khalid S, Goldenberg M, Grantcharov T, Taati B, Rudzicz F (2020) Evaluation of deep learning models for identifying surgical actions and measuring performance. JAMA Netw OPEN. https://doi.org/10.1001/jamanetworkopen.2020.1664
doi: 10.1001/jamanetworkopen.2020.1664
pubmed: 33337494
pmcid: 7749442
Kitaguchi D, Takeshita N, Matsuzaki H, Igaki T, Hasegawa H, Ito M (2021) Development and validation of a 3-dimensional convolutional neural network for automatic surgical skill assessment based on spatiotemporal video analysis. JAMA Netw Open 4:e2120786. https://doi.org/10.1001/jamanetworkopen.2021.20786
doi: 10.1001/jamanetworkopen.2021.20786
pubmed: 34387676
pmcid: 8363914
Kiyasseh D, Ma R, Haque TF, Miles BJ, Wagner C, Donoho DA, Anandkumar A, Hung AJ (2023) A vision transformer for decoding surgeon activity from surgical videos. Nat Biomed Eng. https://doi.org/10.1038/s41551-023-01010-8
doi: 10.1038/s41551-023-01010-8
pubmed: 36997732
pmcid: 10307635
Kowalewski KF, Garrow CR, Schmidt MW, Benner L, Muller-Stich BP, Nickel F (2019) Sensor-based machine learning for workflow detection and as key to detect expert level in laparoscopic suturing and knot-tying. Surg Endosc Interv Tech 33:3732–3740. https://doi.org/10.1007/s00464-019-06667-4
doi: 10.1007/s00464-019-06667-4
Kuo RJ, Chen H-J, Kuo Y-H (2022) The development of an eye movement-based deep learning system for laparoscopic surgical skills assessment. Sci Rep 12:11036. https://doi.org/10.1038/s41598-022-15053-5
doi: 10.1038/s41598-022-15053-5
pubmed: 35970911
pmcid: 9378740
Lajko G, NagyneElek R, Haidegger T (2021) Endoscopic image-based skill assessment in robot-assisted minimally invasive surgery. Sensors. https://doi.org/10.3390/s21165412
doi: 10.3390/s21165412
pubmed: 34450854
pmcid: 8398563
Lam K, Lo FP-W, An Y, Darzi A, Kinross JM, Purkayastha S, Lo B (2022) Deep learning for instrument detection and assessment of operative skill in surgical videos. IEEE Trans Med Robot Bionics 4:1068–1071. https://doi.org/10.1109/TMRB.2022.3214377
doi: 10.1109/TMRB.2022.3214377
Lavanchy JL, Zindel J, Kirtac K, Twick I, Hosgor E, Candinas D, Beldi G (2021) Automation of surgical skill assessment using a three-stage machine learning algorithm. Sci Rep 11:5197. https://doi.org/10.1038/s41598-021-84295-6
doi: 10.1038/s41598-021-84295-6
pubmed: 33664317
pmcid: 7933408
Laverde R, Rueda C, Amado L, Rojas D, Altuve M (2018) Artificial neural network for laparoscopic skills classification using motion signals from apple watch. In: Proceedings of 2018 40th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 5434–5437. https://doi.org/10.1109/EMBC.2018.8513561
Law H, Zhang Y, Kim T-K, Miller D, Montie J, Deng J, Ghani K (2018) Surgeon technical skill assessment using computer vision-based analysis. J Urol 199:e1138
Lazar A, Sroka G, Laufer S (2023) Automatic assessment of performance in the FLS trainer using computer vision. Surg Endosc. https://doi.org/10.1007/s00464-023-10132-8
doi: 10.1007/s00464-023-10132-8
pubmed: 37253868
Lee D, Yu HW, Kwon H, Kong H-J, Lee KE, Kim HC (2020) Evaluation of surgical skills during robotic surgery by deep learning-based multiple surgical instrument tracking in training and actual operations. J Clin Med 9:1–15. https://doi.org/10.3390/jcm9061964
doi: 10.3390/jcm9061964
Liu D, Jiang T, Wang Y, Miao R, Shan F, Li Z (2020) Clearness of operating field: a surrogate for surgical skills on in vivo clinical data. Int J Comput Assist Radiol Surg 15:1817–1824. https://doi.org/10.1007/s11548-020-02267-z
doi: 10.1007/s11548-020-02267-z
pubmed: 33044734
Liu D, Li Q, Jiang T, Wang Y, Miao R, Shan F, Li Z (2021) Towards unified surgical skill assessment. In: Proceedings of 2021 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 9517–9526. https://doi.org/10.1109/CVPR46437.2021.00940
Lyman WB, Passeri MJ, Murphy K, Iannitti DA, Martinie JB, Baker EH, Vrochides D, Siddiqui IA, Khan AS (2021) An objective approach to evaluate novice robotic surgeons using a combination of kinematics and stepwise cumulative sum (CUSUM) analyses. Surg Endosc 35:2765–2772. https://doi.org/10.1007/s00464-020-07708-z
doi: 10.1007/s00464-020-07708-z
pubmed: 32556751
Nguyen XA, Ljuhar D, Pacilli M, Nataraja RM, Chauhan S (2019) Surgical skill levels: classification and analysis using deep neural network model and motion signals. Comput Methods Programs Biomed 177:1–8. https://doi.org/10.1016/j.cmpb.2019.05.008
doi: 10.1016/j.cmpb.2019.05.008
pubmed: 31319938
Oğul BB, Gilgien M, Özdemir S (2022) Ranking surgical skills using an attention-enhanced Siamese network with piecewise aggregated kinematic data. Int J Comput Assist Radiol Surg 17:1039–1048. https://doi.org/10.1007/s11548-022-02581-8
doi: 10.1007/s11548-022-02581-8
pubmed: 35286585
Oquendo YA, Riddle EW, Hiller D, Blinman TA, Kuchenbecker KJ (2018) Automatically rating trainee skill at a pediatric laparoscopic suturing task. Surg Endosc Interv Tech 32:1840–1857. https://doi.org/10.1007/s00464-017-5873-6
doi: 10.1007/s00464-017-5873-6
Pan M, Wang S, Li J, Li J, Yang X, Liang K (2023) An automated skill assessment framework based on visual motion signals and a deep neural network in robot-assisted minimally invasive surgery. Sensors 23:4496. https://doi.org/10.3390/s23094496
doi: 10.3390/s23094496
pubmed: 37177699
pmcid: 10181496
Perez-Escamirosa F, Alarcon-Paredes A, Alonso-Silverio GA, Oropesa I, Camacho-Nieto O, Lorias-Espinoza D, Minor-Martinez A (2020) Objective classification of psychomotor laparoscopic skills of surgeons based on three different approaches. Int J Comput Assist Radiol Surg 15:27–40. https://doi.org/10.1007/s11548-019-02073-2
doi: 10.1007/s11548-019-02073-2
pubmed: 31605351
Sasaki S, Kitaguchi D, Takenaka S, Nakajima K, Sasaki K, Ogane T, Takeshita N, Gotohda N, Ito M (2022) Machine learning-based automatic evaluation of tissue handling skills in laparoscopic colorectal surgery: a retrospective experimental study. Ann Surg. https://doi.org/10.1097/SLA.0000000000005731
doi: 10.1097/SLA.0000000000005731
pubmed: 36453261
Shafiei SB, Shadpour S, Mohler JL, Attwood K, Liu Q, Gutierrez C, Toussi MS (2023) Developing surgical skill level classification model using visual metrics and a gradient boosting algorithm. Ann Surg Open 4:e292. https://doi.org/10.1097/AS9.0000000000000292
doi: 10.1097/AS9.0000000000000292
pubmed: 37305561
pmcid: 10249659
Soangra R, Sivakumar R, Anirudh ER, Sai Viswanth Reddy Y, John EB (2022) Evaluation of surgical skill using machine learning with optimal wearable sensor locations. PLOS ONE 17:e0267936. https://doi.org/10.1371/journal.pone.0267936
doi: 10.1371/journal.pone.0267936
pubmed: 35657912
pmcid: 9165861
Soleymani A, Asl AAS, Yeganejou M, Dick S, Tavakoli M, Li X (2021) Surgical skill evaluation from robot-assisted surgery recordings. In: Proceedings of 2021 international symposium on medical robotics (ISMR), pp 1–6. https://doi.org/10.1109/ISMR48346.2021.9661527
Soleymani A, Li X, Tavakoli M (2022) A domain-adapted machine learning approach for visual evaluation and interpretation of robot-assisted surgery skills. IEEE Robot Autom Lett 7:8202–8208. https://doi.org/10.1109/LRA.2022.3186769
doi: 10.1109/LRA.2022.3186769
Uemura M, Tomikawa M, Akahoshi T, Lefor AK, Hashizume M, Miao T, Souzaki R, Ieiri S (2018) Feasibility of an AI-based measure of the hand motions of expert and novice surgeons. Comput Math Methods Med 2018:9873273. https://doi.org/10.1155/2018/9873273
doi: 10.1155/2018/9873273
pubmed: 29686724
pmcid: 5857335
Wang Y, Dai J, Morgan TN, Elsaied M, Garbens A, Qu X, Steinberg R, Gahan J, Larson EC (2021) Evaluating robotic-assisted surgery training videos with multi-task convolutional neural networks. J Robot Surg. https://doi.org/10.1007/s11701-021-01316-2
doi: 10.1007/s11701-021-01316-2
pubmed: 34709538
Wang ZH, Fey AM (2018) Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery. Int J Comput Assist Radiol Surg 13:1959–1970. https://doi.org/10.1007/s11548-018-1860-1
doi: 10.1007/s11548-018-1860-1
pubmed: 30255463
Wang Z, Fey AM (2018) SATR-DL: improving surgical skill assessment and task recognition in robot-assisted surgery with deep neural networks. In: Annual international conference of the IEEE Engineering in Medicine and Biology Society 2018, pp 1793–1796. https://doi.org/10.1109/EMBC.2018.8512575
Zia A, Essa I (2018) Automated surgical skill assessment in RMIS training. Int J Comput Assist Radiol Surg 13:731–739. https://doi.org/10.1007/s11548-018-1735-5
doi: 10.1007/s11548-018-1735-5
pubmed: 29549553
Zendejas B, Ruparel RK, Cook DA (2016) Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool: a systematic review. Surg Endosc 30:512–520. https://doi.org/10.1007/s00464-015-4233-7
doi: 10.1007/s00464-015-4233-7
pubmed: 26091982
Fried GM, Feldman LS, Vassiliou MC, Fraser SA, Stanbridge D, Ghitulescu G, Andrew CG (2004) Proving the value of simulation in laparoscopic surgery. Ann Surg 240:518–528. https://doi.org/10.1097/01.sla.0000136941.46529.56
doi: 10.1097/01.sla.0000136941.46529.56
pubmed: 15319723
pmcid: 1356442
Gao Y, Vedula SS, Reiley CE, Ahmidi N, Varadarajan B, Lin HC, Tao L, Zappella L, Béjar B, Yuh DD, Chen CCG, Vidal R, Khudanpur S, Hager GD (2014) JHU-ISI gesture and skill assessment working set (JIGSAWS): a surgical activity dataset for human motion modeling. In: Modeling and monitoring of computer assisted interventions (M2CAI)—MICCAI Workshop
Rivas-Blanco I, Pérez-del-Pulgar C, Mariani A, Quaglia C, Tortora G, Reina AJ V (2021) A surgical dataset from the da Vinci Research Kit for task automation and recognition. arXiv preprint. https://doi.org/10.48550/arXiv.2102.03643
Twinanda AP, Shehata S, Mutter D, Marescaux J, Marescaux J, de Mathelin M, Padoy N (2017) EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36:86–97. https://doi.org/10.1109/tmi.2016.2593957
doi: 10.1109/tmi.2016.2593957
pubmed: 27455522
Hussein AA, Ghani KR, Peabody J, Sarle R, Abaza R, Eun D, Hu J, Fumo M, Lane B, Montgomery JS, Hinata N, Rooney D, Comstock B, Chan HK, Mane SS, Mohler JL, Wilding G, Miller D, Guru KA, Michigan Urological Surgery Improvement Collaborative and Applied Technology Laboratory for Advanced Surgery Program (2017) Development and validation of an objective scoring tool for robot-assisted radical prostatectomy: prostatectomy assessment and competency evaluation. J Urol 197:1237–1244. https://doi.org/10.1016/j.juro.2016.11.100
doi: 10.1016/j.juro.2016.11.100
pubmed: 27913152
Peters JH, Fried GM, Swanstrom LL, Soper NJ, Sillin LF, Schirmer B, Hoffman K, the SAGES FLS Committee (2004) Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery 135:21–27. https://doi.org/10.1016/S0039-6060(03)00156-9
doi: 10.1016/S0039-6060(03)00156-9
pubmed: 14694297
Mori T, Kimura T, Kitajima M (2010) Skill accreditation system for laparoscopic gastroenterologic surgeons in Japan. Minim Invasive Ther Allied Technol 19:18–23. https://doi.org/10.3109/13645700903492969
doi: 10.3109/13645700903492969
pubmed: 20095893
Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents: objective structured assessment of technical skill. Br J Surg 84:273–278. https://doi.org/10.1046/j.1365-2168.1997.02502.x
doi: 10.1046/j.1365-2168.1997.02502.x
pubmed: 9052454
Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187:247–252. https://doi.org/10.1016/j.juro.2011.09.032
doi: 10.1016/j.juro.2011.09.032
pubmed: 22099993
Winkler-Schwartz A, Mirchi N, Ponnudurai N, Yilmaz R, Ledwos N, Karlik B, Del Maestro RF, Bissonnette V, Siyar S, Azarnoush H (2019) Artificial intelligence in medical education: best practices using machine learning to assess surgical expertise in virtual reality simulation. J Surg Educ 76:1681–1690. https://doi.org/10.1016/j.jsurg.2019.05.015
doi: 10.1016/j.jsurg.2019.05.015
pubmed: 31202633
Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, Denniston AK, Faes L, Geerts B, Ibrahim M, Liu X, Mateen BA, Mathur P, McCradden MD, Morgan L, Ordish J, Rogers C, Saria S, Ting DSW, Watkinson P, Weber W, Wheatstone P, McCulloch P (2022) Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ 377:e070904. https://doi.org/10.1136/bmj-2022-070904
doi: 10.1136/bmj-2022-070904
pubmed: 35584845
pmcid: 9116198
Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, SPIRIT-AI and CONSORT-AI Working Group (2020) Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med 26:1364–1374. https://doi.org/10.1038/s41591-020-1034-x
doi: 10.1038/s41591-020-1034-x
pubmed: 32908283
pmcid: 7598943
Maier-Hein L, Reinke A, Godau P, Tizabi MD, Büttner F, Christodoulou E, Glocker B, Isensee F, Kleesiek J, Kozubek M, Reyes M, Riegler MA, Wiesenfarth M, Kavur AE, Sudre CH, Baumgartner M, Eisenmann M, Heckmann-Nötzel D, Rädsch AT, Acion L, Antonelli M, Arbel T, Bakas S, Benis A, Blaschko M, Cardoso MJ, Cheplygina V, Cimini BA, Collins GS, Farahani K, Ferrer L, Galdran A, van Ginneken B, Haase R, Hashimoto DA, Hoffman MM, Huisman M, Jannin P, Kahn CE, Kainmueller D, Kainz B, Karargyris A, Karthikesalingam A, Kenngott H, Kofler F, Kopp-Schneider A, Kreshuk A, Kurc T, Landman BA, Litjens G, Madani A, Maier-Hein K, Martel AL, Mattson P, Meijering E, Menze B, Moons KGM, Müller H, Nichyporuk B, Nickel F, Petersen J, Rajpoot N, Rieke N, Saez-Rodriguez J, Sánchez CI, Shetty S, van Smeden M, Summers RM, Taha AA, Tiulpin A, Tsaftaris SA, Van Calster B, Varoquaux G, Jäger PF (2022) Metrics reloaded: pitfalls and recommendations for image analysis validation. arXiv preprint. https://doi.org/10.48550/arXiv.2206.01653
Kitaguchi D, Watanabe Y, Madani A, Hashimoto DA, Meireles OR, Takeshita N, Mori K, Ito M, on behalf of the Computer Vision in Surgery International Collaborative (2022) Artificial intelligence for computer vision in surgery: a call for developing reporting guidelines. Ann Surg 275:e609–e611. https://doi.org/10.1097/SLA.0000000000005319
doi: 10.1097/SLA.0000000000005319
pubmed: 35129482