Anatomic Depth Estimation and 3-Dimensional Reconstruction of Microsurgical Anatomy Using Monoscopic High-Definition Photogrammetry and Machine Learning.
Journal
Operative neurosurgery (Hagerstown, Md.)
ISSN: 2332-4260
Titre abrégé: Oper Neurosurg (Hagerstown)
Pays: United States
ID NLM: 101635417
Informations de publication
Date de publication:
01 04 2023
01 04 2023
Historique:
received:
17
06
2022
accepted:
17
09
2022
pubmed:
27
1
2023
medline:
21
3
2023
entrez:
26
1
2023
Statut:
ppublish
Résumé
Immersive anatomic environments offer an alternative when anatomic laboratory access is limited, but current three-dimensional (3D) renderings are not able to simulate the anatomic detail and surgical perspectives needed for microsurgical education. To perform a proof-of-concept study of a novel photogrammetry 3D reconstruction technique, converting high-definition (monoscopic) microsurgical images into a navigable, interactive, immersive anatomy simulation. Images were acquired from cadaveric dissections and from an open-access comprehensive online microsurgical anatomic image database. A pretrained neural network capable of depth estimation from a single image was used to create depth maps (pixelated images containing distance information that could be used for spatial reprojection and 3D rendering). Virtual reality (VR) experience was assessed using a VR headset, and augmented reality was assessed using a quick response code-based application and a tablet camera. Significant correlation was found between processed image depth estimations and neuronavigation-defined coordinates at different levels of magnification. Immersive anatomic models were created from dissection images captured in the authors' laboratory and from images retrieved from the Rhoton Collection. Interactive visualization and magnification allowed multiple perspectives for an enhanced experience in VR. The quick response code offered a convenient method for importing anatomic models into the real world for rehearsal and for comparing other anatomic preparations side by side. This proof-of-concept study validated the use of machine learning to render 3D reconstructions from 2-dimensional microsurgical images through depth estimation. This spatial information can be used to develop convenient, realistic, and immersive anatomy image models.
Sections du résumé
BACKGROUND
Immersive anatomic environments offer an alternative when anatomic laboratory access is limited, but current three-dimensional (3D) renderings are not able to simulate the anatomic detail and surgical perspectives needed for microsurgical education.
OBJECTIVE
To perform a proof-of-concept study of a novel photogrammetry 3D reconstruction technique, converting high-definition (monoscopic) microsurgical images into a navigable, interactive, immersive anatomy simulation.
METHODS
Images were acquired from cadaveric dissections and from an open-access comprehensive online microsurgical anatomic image database. A pretrained neural network capable of depth estimation from a single image was used to create depth maps (pixelated images containing distance information that could be used for spatial reprojection and 3D rendering). Virtual reality (VR) experience was assessed using a VR headset, and augmented reality was assessed using a quick response code-based application and a tablet camera.
RESULTS
Significant correlation was found between processed image depth estimations and neuronavigation-defined coordinates at different levels of magnification. Immersive anatomic models were created from dissection images captured in the authors' laboratory and from images retrieved from the Rhoton Collection. Interactive visualization and magnification allowed multiple perspectives for an enhanced experience in VR. The quick response code offered a convenient method for importing anatomic models into the real world for rehearsal and for comparing other anatomic preparations side by side.
CONCLUSION
This proof-of-concept study validated the use of machine learning to render 3D reconstructions from 2-dimensional microsurgical images through depth estimation. This spatial information can be used to develop convenient, realistic, and immersive anatomy image models.
Identifiants
pubmed: 36701667
doi: 10.1227/ons.0000000000000544
pii: 01787389-202304000-00012
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
432-444Informations de copyright
Copyright © Congress of Neurological Surgeons 2022. All rights reserved.
Références
Khan F, Salahuddin S, Javidnia H. Deep learning-based monocular depth estimation methods-A state-of-the-art review. Sensors (Basel). 2020;20(8):2272.
Tomlinson SB, Hendricks BK, Cohen-Gadol AA. Editorial. Innovations in neurosurgical education during the COVID-19 pandemic: is it time to reexamine our neurosurgical training models? J Neurosurg. 2020;133(1):14-15.
Sorenson J, Khan N, Couldwell W, Robertson J. The Rhoton Collection. World Neurosurg. 2016;92:649-652.
Hanalioglu S, Romo NG, Mignucci-Jimenez G, et al. Development and validation of a novel methodological pipeline to integrate neuroimaging and photogrammetry for immersive 3D cadaveric neurosurgical simulation. Front Surg. 2022;9:878378.
Bernardo A. Virtual reality and simulation in neurosurgical training. World Neurosurg. 2017;106:1015-1029.
Tomlinson SB, Hendricks BK, Cohen-Gadol A. Immersive three-dimensional modeling and virtual reality for enhanced visualization of operative neurosurgical anatomy. World Neurosurg. 2019;131:313-320.
Gurses ME, Gungor A, Hanalioglu S, et al. Qlone®: a simple method to create 360-degree photogrammetry-based 3-dimensional model of cadaveric specimens. Oper Neurosurg. 2021;21(6):E488-E493.
Ranftl R, Lasinger K, Hafner D, Schindler K, Koltun V. Towards robust monocular depth estimation: mixing datasets for zero-shot cross-dataset transfer. IEEE Trans Pattern Anal Mach Intell. 2022;44(3):1623-1637.
Ranftl R, Bochkovskiy A, Koltun V. Vision Transformers For Dense Prediction. ArXiv. 2021; abs/2103.13413.
Zhou Q-Y, Park J, Koltun V. Open3D: A Modern Library For 3D Data Processing. ArXiv. 2018; abs/1801.09847.
Chen D, Zhang Q, Deng J, et al. A shortage of cadavers: the predicament of regional anatomy education in mainland China. Anatomical Sci Educ. 2018;11(4):397-402.
Tatar I, Huri E, Selcuk I, Moon YL, Paoluzzi A, Skolarikos A. Review of the effect of 3D medical printing and virtual reality on urology training with ‘MedTRain3DModsim’ Erasmus + European Union Project. Turk J Med Sci. 2019;49(5):1257-1270.
Brassett C, Cosker T, Davies DC, et al. COVID-19 and anatomy: stimulus and initial response. J Anat. 2020;237(3):393-403.
Jones DG. Anatomy in a post-Covid-19 world: tracing a new trajectory. Anat Sci Educ. 2021;14(2):148-153.
Patra A, Asghar A, Chaudhary P, Ravi KS. Integration of innovative educational technologies in anatomy teaching: new normal in anatomy education. Surg Radiol Anat. 2022;44(1):25-32.
Sahin B, Hanalioglu S. The continuing impact of coronavirus disease 2019 on neurosurgical training at the 1-year mark: results of a nationwide survey of neurosurgery residents in Turkey. World Neurosurg. 2021;151:e857-e870.
Alaraj A, Luciano CJ, Bailey DP, et al. Virtual reality cerebral aneurysm clipping simulation with real-time haptic feedback. Oper Neurosurg. 2015;11(1):52-58.
de Faria JWV, Teixeira MJ, de Moura Sousa Junior L, Otoch JP, Figueiredo EG. Virtual and stereoscopic anatomy: when virtual reality meets medical education. J Neurosurg. 2016;125(5):1105-1111.
Gmeiner M, Dirnberger J, Fenz W, et al. Virtual cerebral aneurysm clipping with real-time haptic force feedback in neurosurgical education. World Neurosurg. 2018;112:e313-e323.
Teodoro-Vite S, Perez-Lomeli JS, Dominguez-Velasco CF, Hernandez-Valencia AF, Capurso-Garcia MA, Padilla-Castaneda MA. A high-fidelity hybrid virtual reality simulator of aneurysm clipping repair with brain sylvian fissure exploration for vascular neurosurgery training. Simul Healthc. 2021;16(4):285-294.
Henn JS, Lemole GM Jr, Ferreira MAT, et al. Interactive stereoscopic virtual reality: a new tool for neurosurgical education. Technical note. J Neurosurg. 2002;96(1):144-149.
Balogh AA, Preul MC, Laszlo K, et al. Multilayer image grid reconstruction technology: four-dimensional interactive image reconstruction of microsurgical neuroanatomic dissections. Oper Neurosurg. 2006;58(1 suppl):ONS157-ONS165.
Balogh A, Preul MC, Schornak M, Hickman M, Spetzler RF. Intraoperative stereoscopic QuickTime virtual reality. J Neurosurg. 2004;100(4):591-596.
Rodriguez Rubio R, Xie W, Vigo V, et al. Immersive surgical anatomy of the retrosigmoid approach. Cureus. 2021;13(6):e16068.
Rodriguez Rubio R, Chae R, Kournoutas I, Abla A, McDermott M. Immersive surgical anatomy of the frontotemporal-orbitozygomatic approach. Cureus. 2019;11(11):e6053.
Rodriguez Rubio R, Chae R, Vigo V, Abla AA, McDermott M. Immersive surgical anatomy of the pterional approach. Cureus. 2019;11(7):e5216.
Kournoutas I, Vigo V, Chae R, et al. Acquisition of volumetric models of skull base anatomy using endoscopic endonasal approaches: 3D scanning of deep corridors via photogrammetry. World Neurosurg. 2019;129:372-377.
Roh TH, Oh JW, Jang CK, et al. Virtual dissection of the real brain: integration of photographic 3D models into virtual reality and its effect on neurosurgical resident education. Neurosurg Focus. 2021;51(2):E16.
Rhoton AL Jr. Rhoton's Cranial Anatomy and Surgical Approaches. Oxford University Press; 2019.
Farhadi DS, Jubran JH, Zhao X, et al. The neuroanatomic studies of Albert L. Rhoton Jr. in historical context: an analysis of origin, evolution, and application. World Neurosurg. 2021;151:258-276.
Shimizu S, Tanaka R, Rhoton AL Jr, et al. Anatomic dissection and classic three-dimensional documentation: a unit of education for neurosurgical anatomy revisited. Neurosurgery. 2006;58(5):E1000; discussion E1000.
Alkadri S, Ledwos N, Mirchi N, et al. Utilizing a multilayer perceptron artificial neural network to assess a virtual reality surgical procedure. Comput Biol Med. 2021;136:104770.
Winkler-Schwartz A, Yilmaz R, Mirchi N, et al. Machine learning identification of surgical and operative factors associated with surgical expertise in virtual reality simulation. JAMA Netw Open. 2019;2(8):e198363.