Surgical optomics: hyperspectral imaging and deep learning towards precision intraoperative automatic tissue recognition-results from the EX-MACHYNA trial.
Convolutional neural networks
Deep learning
Hyperspectral imaging
Image-guided surgery
Semantic scene segmentation
Surgical data science
Journal
Surgical endoscopy
ISSN: 1432-2218
Titre abrégé: Surg Endosc
Pays: Germany
ID NLM: 8806653
Informations de publication
Date de publication:
24 May 2024
24 May 2024
Historique:
received:
03
02
2024
accepted:
23
04
2024
medline:
25
5
2024
pubmed:
25
5
2024
entrez:
24
5
2024
Statut:
aheadofprint
Résumé
Hyperspectral imaging (HSI), combined with machine learning, can help to identify characteristic tissue signatures enabling automatic tissue recognition during surgery. This study aims to develop the first HSI-based automatic abdominal tissue recognition with human data in a prospective bi-center setting. Data were collected from patients undergoing elective open abdominal surgery at two international tertiary referral hospitals from September 2020 to June 2021. HS images were captured at various time points throughout the surgical procedure. Resulting RGB images were annotated with 13 distinct organ labels. Convolutional Neural Networks (CNNs) were employed for the analysis, with both external and internal validation settings utilized. A total of 169 patients were included, 73 (43.2%) from Strasbourg and 96 (56.8%) from Verona. The internal validation within centers combined patients from both centers into a single cohort, randomly allocated to the training (127 patients, 75.1%, 585 images) and test sets (42 patients, 24.9%, 181 images). This validation setting showed the best performance. The highest true positive rate was achieved for the skin (100%) and the liver (97%). Misclassifications included tissues with a similar embryological origin (omentum and mesentery: 32%) or with overlaying boundaries (liver and hepatic ligament: 22%). The median DICE score for ten tissue classes exceeded 80%. To improve automatic surgical scene segmentation and to drive clinical translation, multicenter accurate HSI datasets are essential, but further work is needed to quantify the clinical value of HSI. HSI might be included in a new omics science, namely surgical optomics, which uses light to extract quantifiable tissue features during surgery.
Sections du résumé
BACKGROUND
BACKGROUND
Hyperspectral imaging (HSI), combined with machine learning, can help to identify characteristic tissue signatures enabling automatic tissue recognition during surgery. This study aims to develop the first HSI-based automatic abdominal tissue recognition with human data in a prospective bi-center setting.
METHODS
METHODS
Data were collected from patients undergoing elective open abdominal surgery at two international tertiary referral hospitals from September 2020 to June 2021. HS images were captured at various time points throughout the surgical procedure. Resulting RGB images were annotated with 13 distinct organ labels. Convolutional Neural Networks (CNNs) were employed for the analysis, with both external and internal validation settings utilized.
RESULTS
RESULTS
A total of 169 patients were included, 73 (43.2%) from Strasbourg and 96 (56.8%) from Verona. The internal validation within centers combined patients from both centers into a single cohort, randomly allocated to the training (127 patients, 75.1%, 585 images) and test sets (42 patients, 24.9%, 181 images). This validation setting showed the best performance. The highest true positive rate was achieved for the skin (100%) and the liver (97%). Misclassifications included tissues with a similar embryological origin (omentum and mesentery: 32%) or with overlaying boundaries (liver and hepatic ligament: 22%). The median DICE score for ten tissue classes exceeded 80%.
CONCLUSION
CONCLUSIONS
To improve automatic surgical scene segmentation and to drive clinical translation, multicenter accurate HSI datasets are essential, but further work is needed to quantify the clinical value of HSI. HSI might be included in a new omics science, namely surgical optomics, which uses light to extract quantifiable tissue features during surgery.
Identifiants
pubmed: 38789623
doi: 10.1007/s00464-024-10880-1
pii: 10.1007/s00464-024-10880-1
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2024. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.
Références
Tang B, Hanna GB, Joice P et al (2004) Identification and categorization of technical errors by Observational Clinical Human Reliability Assessment (OCHRA) during laparoscopic cholecystectomy. Arch Surg 139:1215–1220
pubmed: 15545569
doi: 10.1001/archsurg.139.11.1215
Strasberg SM (2008) Error traps and vasculo-biliary injury in laparoscopic and open cholecystectomy. J Hepatobiliary Pancreat Surg 15:284–292
pubmed: 18535766
doi: 10.1007/s00534-007-1267-9
Francis NK, Curtis NJ, Conti JA et al (2018) EAES classification of intraoperative adverse events in laparoscopic surgery. Surg Endosc 32:3822–3829
pubmed: 29435754
doi: 10.1007/s00464-018-6108-1
Mascagni P, Longo F, Barberio M et al (2018) New intraoperative imaging technologies: innovating the surgeon’s eye toward surgical precision. J Surg Oncol 118:265–282
pubmed: 30076724
doi: 10.1002/jso.25148
Barberio M, Benedicenti S, Pizzicannella M et al (2021) Intraoperative guidance using hyperspectral imaging: a review for surgeons. Diagnostics 11:2066
pubmed: 34829413
pmcid: 8624094
doi: 10.3390/diagnostics11112066
Clancy NT, Jones G, Maier-Hein L et al (2020) Surgical spectral imaging. Med Image Anal 63:101699
pubmed: 32375102
pmcid: 7903143
doi: 10.1016/j.media.2020.101699
Shapey J, Xie Y, Nabavi E et al (2019) Intraoperative multispectral and hyperspectral label-free imaging: a systematic review of in vivo clinical studies. J Biophotonics 12:e201800455
pubmed: 30859757
pmcid: 6736677
doi: 10.1002/jbio.201800455
Barberio M, Felli E, Pizzicannella M et al (2021) Quantitative serosal and mucosal optical imaging perfusion assessment in gastric conduits for esophageal surgery: an experimental study in enhanced reality. Surg Endosc 35:5827–5835
pubmed: 33026514
doi: 10.1007/s00464-020-08077-3
Wakabayashi T, Barberio M, Urade T et al (2021) Intraoperative perfusion assessment in enhanced reality using quantitative optical imaging: an experimental study in a pancreatic partial ischemia model. Diagnostics 11:93
pubmed: 33430038
pmcid: 7826658
doi: 10.3390/diagnostics11010093
Jansen-Winkeln B, Barberio M, Chalopin C et al (2021) Feedforward artificial neural network-based colorectal cancer detection using hyperspectral imaging: a step towards automatic optical biopsy. Cancers 13:967
pubmed: 33669082
pmcid: 7956537
doi: 10.3390/cancers13050967
Barberio M, Collins T, Bencteux V et al (2021) Deep learning analysis of in vivo hyperspectral images for automated intraoperative nerve detection. Diagnostics 11:1508
pubmed: 34441442
pmcid: 8391550
doi: 10.3390/diagnostics11081508
Felli E, Al-Taher M, Collins T et al (2021) Automatic liver viability scoring with deep learning and hyperspectral imaging. Diagnostics 11:1527
pubmed: 34573869
pmcid: 8472457
doi: 10.3390/diagnostics11091527
Seidlitz S, Sellner J, Odenthal J et al (2022) Robust deep learning-based semantic organ segmentation in hyperspectral images. Med Image Anal 80:102488
pubmed: 35667327
doi: 10.1016/j.media.2022.102488
Studier-Fischer A, Seidlitz S, Sellner J et al (2021) Spectral organ fingerprints for intraoperative tissue classification with hyperspectral imaging. bioRxiv 469943
Okamoto N, Rodríguez-Luna MR, Bencteux V et al (2022) Computer-assisted differentiation between Colon–Mesocolon and retroperitoneum using Hyperspectral Imaging (HSI) technology. Diagnostics (Basel) 12:2225
pubmed: 36140626
doi: 10.3390/diagnostics12092225
Köhler H, Jansen-Winkeln B, Maktabi M et al (2019) Evaluation of hyperspectral imaging (HSI) for the measurement of ischemic conditioning effects of the gastric conduit during esophagectomy. Surg Endosc 33:3775–3782
pubmed: 30675658
doi: 10.1007/s00464-019-06675-4
Barberio M, Longo F, Fiorillo C et al (2020) HYPerspectral Enhanced Reality (HYPER): a physiology-based surgical guidance tool. Surg Endosc 34:1736–1744
pubmed: 31309313
doi: 10.1007/s00464-019-06959-9
Jansen-Winkeln B, Dvorak M, Köhler H et al (2022) Border line definition using hyperspectral imaging in colorectal resections. Cancers 14:1188
pubmed: 35267496
pmcid: 8909141
doi: 10.3390/cancers14051188
Urade T, Felli E, Barberio M et al (2021) HYPerspectral Enhanced Reality (HYPER) for anatomical liver resection. Surg Endosc 35:1844–1850
pubmed: 32342212
doi: 10.1007/s00464-020-07586-5
Barberio M, Lapergola A, Benedicenti S et al (2022) Intraoperative bowel perfusion quantification with hyperspectral imaging: a guidance tool for precision colorectal surgery. Surg Endosc 36:8520–8532
pubmed: 35836033
doi: 10.1007/s00464-022-09407-3
Aboughaleb IH, Aref MH, El-Sharkawy YH (2020) Hyperspectral imaging for diagnosis and detection of ex-vivo breast cancer. Photodiagn Photodyn Ther 31:101922
doi: 10.1016/j.pdpdt.2020.101922
Akbari H, Halig LV, Schuster DM et al (2012) Hyperspectral imaging and quantitative analysis for prostate cancer detection. J Biomed Opt 17:076005
pubmed: 22894488
pmcid: 3608529
doi: 10.1117/1.JBO.17.7.076005
Collins T, Bencteux V, Benedicenti S et al (2022) Automatic optical biopsy for colorectal cancer using hyperspectral imaging and artificial neural networks. Surg Endosc. https://doi.org/10.1007/s00464-022-09524-z
doi: 10.1007/s00464-022-09524-z
pubmed: 36266482
pmcid: 9510278
Johansen TH, Møllersen K, Ortega S et al (2020) Recent advances in hyperspectral imaging for melanoma detection. WIREs Comput Stat 12:e1465
doi: 10.1002/wics.1465
Liu Z, Wang H, Li Q (2012) Tongue tumor detection in medical hyperspectral images. Sensors 12:162–174
pubmed: 22368462
doi: 10.3390/s120100162
De Landro M, Felli E, Collins T et al (2021) Prediction of in vivo laser-induced thermal damage with hyperspectral imaging using deep learning. Sensors 21:6934
pubmed: 34696147
pmcid: 8539534
doi: 10.3390/s21206934
Ramspek CL, Jager KJ, Dekker FW et al (2021) External validation of prognostic models: what, why, how, when and where? Clin Kidney J 14:49–58
pubmed: 33564405
doi: 10.1093/ckj/sfaa188
Steyerberg EW, Harrell FE (2016) Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol 69:245–247
pubmed: 25981519
doi: 10.1016/j.jclinepi.2015.04.005
von Elm E, Altman DG, Egger M et al (2007) The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med 147:573–577
doi: 10.7326/0003-4819-147-8-200710160-00010
Felli E, Al-Taher M, Collins T et al (2020) Hyperspectral evaluation of hepatic oxygenation in a model of total vs arterial liver ischaemia. Sci Rep. 10:15441
pubmed: 32963333
pmcid: 7509803
doi: 10.1038/s41598-020-72915-6
Karimi D, Dou H, Warfield SK et al (2020) Deep learning with noisy labels: exploring techniques and remedies in medical image analysis. Med Image Anal 65:101759
pubmed: 32623277
pmcid: 7484266
doi: 10.1016/j.media.2020.101759
Minaee S, Boykov Y, Porikli F et al (2022) Image segmentation using deep learning: a survey. IEEE Trans Pattern Anal Mach Intell 44:3523–3542
pubmed: 33596172
Shen D, Wu G, Suk H-I (2017) Deep learning in medical image analysis. Annu Rev Biomed Eng 19:221–248
pubmed: 28301734
pmcid: 5479722
doi: 10.1146/annurev-bioeng-071516-044442
Wang R, Lei T, Cui R et al (2022) Medical image segmentation using deep learning: a survey. IET Image Proc 16:1243–1267
doi: 10.1049/ipr2.12419
Goodfellow I, Bengio Y, Courville A (2016) Deep learning, Illustrated. The MIT Press, Cambridge
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444
pubmed: 26017442
doi: 10.1038/nature14539
Siontis GCM, Tzoulaki I, Castaldi PJ et al (2015) External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 68:25–34
pubmed: 25441703
doi: 10.1016/j.jclinepi.2014.09.007
He K, Zhang X, Ren S et al (2015) Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. arXiv:1502.01852 [cs]. http://arxiv.org/abs/1502.01852 . Accessed 9 May 2022
Eigen D, Fergus R (2015) Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2650–2658
Lafferty JD, McCallum A, Pereira FCN (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. Proceedings of the eighteenth international conference on machine learning. Morgan Kaufmann Publishers Inc, San Francisco, pp 282–289
Alam FI, Zhou J, Liew AW-C et al (2019) Conditional random field and deep feature learning for hyperspectral image segmentation. IEEE Trans Geosci Remote Sens. 57:1612
doi: 10.1109/TGRS.2018.2867679
Signoroni A, Savardi M, Baronio A et al (2019) Deep learning meets hyperspectral image analysis: a multidisciplinary review. J Imaging 5:52
pubmed: 34460490
pmcid: 8320953
doi: 10.3390/jimaging5050052
Fu Y, Lei Y, Wang T et al (2021) A review of deep learning based methods for medical image multi-organ segmentation. Physica Med 85:107–122
doi: 10.1016/j.ejmp.2021.05.003
Thompson ML, Zucchini W (1989) On the statistical analysis of ROC curves. Stat Med 8:1277–1290
pubmed: 2814075
doi: 10.1002/sim.4780081011
Scheikl PM, Laschewski S, Kisilenko A et al (2020) Deep learning for semantic segmentation of organs and tissues in laparoscopic surgery. Curr Directions Biomed Eng. https://doi.org/10.1515/cdbme-2020-0016
doi: 10.1515/cdbme-2020-0016
Felli E, Cinelli L, Bannone E et al (2022) Hyperspectral imaging in major hepatectomies: preliminary results from the ex-machyna trial. Cancers (Basel) 14:5591
pubmed: 36428685
doi: 10.3390/cancers14225591
Yoon J, Joseph J, Waterhouse DJ et al (2019) A clinically translatable hyperspectral endoscopy (HySE) system for imaging the gastrointestinal tract. Nat Commun 10:1902
pubmed: 31015458
pmcid: 6478902
doi: 10.1038/s41467-019-09484-4
Maier-Hein L, Eisenmann M, Sarikaya D et al (2022) Surgical data science—from concepts toward clinical translation. Med Image Anal 76:102306
pubmed: 34879287
doi: 10.1016/j.media.2021.102306
Bar O, Neimark D, Zohar M et al (2020) Impact of data on generalization of AI for surgical intelligence applications. Sci Rep 10:22208
pubmed: 33335191
pmcid: 7747564
doi: 10.1038/s41598-020-79173-6
Taylor AM, Bordoni B (2022) Histology, blood vascular system. In: StatPearls. Treasure Island: StatPearls Publishing. http://www.ncbi.nlm.nih.gov/books/NBK553217/ . Accessed 30 Jan 2023
Mahour GH, Wakim KG, Soule EH et al (1967) Structure of the common bile duct in man: presence or absence of smooth muscle. Ann Surg 166:91–94
pubmed: 4165872
pmcid: 1477349
doi: 10.1097/00000658-196707000-00011
Nema S, Vachhani L (2022) Surgical instrument detection and tracking technologies: automating dataset labeling for surgical skill assessment. Front Robot AI 9:1030846
pubmed: 36405072
pmcid: 9671944
doi: 10.3389/frobt.2022.1030846
Rodrigues M, Mayo M, Patros P (2022) Surgical tool datasets for machine learning research: a survey. Int J Comput Vis 130:2222–2248
doi: 10.1007/s11263-022-01640-6
Collins T, Pizarro D, Gasparini S et al (2021) Augmented reality guided laparoscopic surgery of the uterus. IEEE Trans Med Imaging 40:371–380
pubmed: 32986548
doi: 10.1109/TMI.2020.3027442
Modrzejewski R, Collins T, Seeliger B et al (2019) An in vivo porcine dataset and evaluation methodology to measure soft-body laparoscopic liver registration accuracy with an extended algorithm that handles collisions. Int J CARS 14:1237–1245
doi: 10.1007/s11548-019-02001-4
Quero G, Lapergola A, Soler L et al (2019) Virtual and augmented reality in oncologic liver surgery. Surg Oncol Clin 28:31–44
doi: 10.1016/j.soc.2018.08.002
Gumbs AA, Alexander F, Karcz K et al (2022) White paper: definitions of artificial intelligence and autonomous actions in clinical surgery. Artificial Intelligence Surgery 2:93–100
doi: 10.20517/ais.2022.10
Mo Y, Wu Y, Yang X et al (2022) Review the state-of-the-art technologies of semantic segmentation based on deep learning. Neurocomputing 493:626–646
doi: 10.1016/j.neucom.2022.01.005
Sun C, Shrivastava A, Singh S et al (2017) Revisiting unreasonable effectiveness of data in deep learning era. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 843–852.
Castiglioni I, Rundo L, Codari M et al (2021) AI applications to medical images: From machine learning to deep learning. Phys Med 83:9–24
pubmed: 33662856
doi: 10.1016/j.ejmp.2021.02.006
Pfahl A, Köhler H, Thomaßen MT et al (2022) Video: clinical evaluation of a laparoscopic hyperspectral imaging system. Surg Endosc 36:7794–7799
pubmed: 35546207
pmcid: 9485189
doi: 10.1007/s00464-022-09282-y