Deep-learning-based instrument detection for intra-operative robotic assistance.
Data augmentation
Dataset
Mask R-CNN
Mask-based object insertion
Robot-assisted surgery
Robotic scrub nurse
Journal
International journal of computer assisted radiology and surgery
ISSN: 1861-6429
Titre abrégé: Int J Comput Assist Radiol Surg
Pays: Germany
ID NLM: 101499225
Informations de publication
Date de publication:
Sep 2022
Sep 2022
Historique:
received:
14
01
2022
accepted:
30
06
2022
pubmed:
28
7
2022
medline:
14
9
2022
entrez:
27
7
2022
Statut:
ppublish
Résumé
Robotic scrub nurses have the potential to become an attractive solution for the operating room. Surgical instrument detection is a fundamental task for these systems, which is the focus of this work. We address the detection of the complete surgery set for wisdom teeth extraction, and propose a data augmentation technique tailored for this task. Using a robotic scrub nurse system, we create a dataset of 369 unique multi-instrument images with manual annotations. We then propose the Mask-Based Object Insertion method, capable of automatically generating a large amount of synthetic images. By using both real and artificial data, different Mask R-CNN models are trained and evaluated. Our experiments reveal that models trained on the synthetic data created with our method achieve comparable performance to that of models trained on real images. Moreover, we demonstrate that the combination of real and our artificial data can lead to a superior level of generalization. The proposed data augmentation technique is capable of dramatically reducing the labelling work required for training a deep-learning-based detection algorithm. A dataset for the complete instrument set for wisdom teeth extraction is made available for the scientific community, as well as the raw information required for the generation of the synthetic data ( https://github.com/Jorebs/Deep-learning-based-instrument-detection-for-intra operative-robotic-assistance ).
Identifiants
pubmed: 35896914
doi: 10.1007/s11548-022-02715-y
pii: 10.1007/s11548-022-02715-y
pmc: PMC9463311
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
1685-1695Informations de copyright
© 2022. The Author(s).
Références
Aini FAN, Purwalaksana AZ, Manalu IP (2019) Object detection of surgical instruments for assistant robot surgeon using knn. In: 2019 international conference on advanced mechatronics, intelligent manufacture and industrial automation (ICAMIMIA), pp 37–40. IEEE
Carpintero E, Perez C, Morales R, Garcia N, Candela A, Azorin J (2010) Development of a robotic scrub nurse for the operating theatre. In: 2010 3rd IEEE RAS and EMBS international conference on biomedical robotics and biomechatronics, pp 504–509
Dalal R, Moh T (2018) Fine-grained object detection using transfer learning and data augmentation. In: IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM), pp 893–896
Dutta A, Gupta A, Zissermann A (2016) VGG image annotator (VIA). http://www.robots.ox.ac.uk/vgg/software/via/
Dwibedi D, Misra I, Hebert M (2017) Cut, paste and learn: Surprisingly easy synthesis for instance detection. In: Proceedings of the IEEE international conference on computer vision, pp 1301–1310
He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969
Jacob MG, Li YT, Wachs JP (2013) Surgical instrument handling and retrieval in the operating room with a multimodal robotic assistant. In: IEEE international conference on robotics and automation, pp 2140–2145. IEEE
Jacob MG, Li YT, Wachs JP (2012) Gestonurse: a multimodal robotic scrub nurse. In: ACM/IEEE international conference on human-robot interaction, pp 153–154
Jo H, Na YH, Song JB (2017) Data augmentation using synthesized images for object detection. In: International conference on control, automation and systems, pp 1035–1038
Kan N, Kondo N, Chinsatit W, Saitoh T (2018) Effectiveness of data augmentation for cnn-based pupil center point detection. In: 57th annual conference of the society of instrument and control engineers of Japan (SICE), pp 41–46
Li H, Rao J, Zhou L, Zhang J (2019) Valid data augmentation by patch alpha matting. In: 2019 IEEE 4th international conference on signal and image processing (ICSIP), pp 361–366
Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: common objects in context. In: European conference on computer vision, pp 740–755. Springer
Matrox Electronic Systems: Matrox imaging library, https://integrys.com/product/matrox-imaging-library-mil/
Miyawaki F, Masamune K, Suzuki S, Yoshimitsu K, Vain J (2005) Scrub nurse robot system-intraoperative motion analysis of a scrub nurse and timed-automata-based model for surgery. IEEE Trans Industr Electron 52(5):1227–1235
doi: 10.1109/TIE.2005.855692
Murillo PCU, Moreno RJ, Arenas JOP (2017) Comparison between cnn and haar classifiers for surgical instrumentation classification. Contemp Eng Sci 10(28):1351–1363
doi: 10.12988/ces.2017.711157
Nakano A, Nagamune K (2022) A development of robotic scrub nurse system-detection for surgical instruments using faster region-based convolutional neural network-. J Adv Comput Intell Intell Inf 26(1):74–82
doi: 10.20965/jaciii.2022.p0074
Shin H, Lee K, Lee C (2020) Data augmentation method of object detection for deep learning in maritime image. In: IEEE international conference on big data and smart computing (BigComp), pp 463–466
Talukdar J, Biswas A, Gupta S (2018) Data augmentation on synthetic images for transfer learning using deep CNNs. In: 5th international conference on signal processing and integrated networks (SPIN), pp 215–219
Uysal MC, Karapinar T, Benligiray B, Topal C (2018) Dataset augmentation for accurate object detection. In: 26th signal processing and communications applications conference (SIU), pp 1–4
Wang Y, Zhou Y, Luo C (2019) Deep learning for topmost roller chain detection using data augmentation. In: 4th international conference on mechanical, control and computer engineering (ICMCCE), pp 443–4433
Xu Y, Mao Y, Tong X, Tan H, Griffin WB, Kannan B, DeRose LA (2015) Robotic handling of surgical instruments in a cluttered tray. IEEE Trans Autom Sci Eng 12(2):775–780
Zemmar A, Lozano AM, Nelson BJ (2020) The rise of robots in surgical environments during covid-19. Nat Mach Intell 2(10):566–572
doi: 10.1038/s42256-020-00238-2
Zhou T, Wachs JP (2017) Finding a needle in a haystack: recognizing surgical instruments through vision and manipulation. Electron Imaging 2017(9):37–45
doi: 10.2352/ISSN.2470-1173.2017.9.IRIACV-264
Zhou T, Wachs JP (2017) Needle in a haystack: interactive surgical instrument recognition through perception and manipulation. Robot Auton Syst 97:182–192
doi: 10.1016/j.robot.2017.08.013