Incremental retraining, clinical implementation, and acceptance rate of deep learning auto-segmentation for male pelvis in a multiuser environment.
deep learning auto-segmentation
inter-observer contour variation
prostate radiotherapy
Journal
Medical physics
ISSN: 2473-4209
Titre abrégé: Med Phys
Pays: United States
ID NLM: 0425746
Informations de publication
Date de publication:
Jul 2023
Jul 2023
Historique:
revised:
02
05
2023
received:
07
03
2023
accepted:
23
05
2023
medline:
11
7
2023
pubmed:
8
6
2023
entrez:
8
6
2023
Statut:
ppublish
Résumé
Deep learning auto-segmentation (DLAS) models have been adopted in the clinic; however, they suffer from performance deterioration owing to the clinical practice variability. Some commercial DLAS software provide an incremental retraining function that enables users to train a custom model using their institutional data to account for clinical practice variability. This study was performed to evaluate and implement the commercial DLAS software with the incremental retraining function for definitive treatment of patients with prostate cancer in a multi-user environment. CT-based target organs and organs-at-risk (OAR) delineation of 215 prostate cancer patients were utilized. The performance of three commercial DLAS software built-in models was validated with 20 patients. A retrained custom model was developed using 100 patients and evaluated on the remaining data (n = 115). Dice similarity coefficient (DSC), Hausdorff distance (HD), mean surface distance (MSD), and surface DSC (SDSC) were utilized for quantitative evaluation. A multi-rater qualitative evaluation was blindly performed with a five-level scale. Visual inspection was performed in consensus and non-consensus unacceptable cases to identify the failure modes. Three commercial DLAS vendor built-in models achieved sub-optimal performance in 20 patients. The retrained custom model had a mean DSC of 0.82 for prostate, 0.48 for seminal vesicles (SV), and 0.92 for rectum, respectively. This represents a significant improvement over the built-in model with DSC of 0.73, 0.37, and 0.81 for the corresponding structures. Compared to the acceptance rate of 96.5% and consensus unacceptable rate (i.e., both reviewers rated as unacceptable) of 3.5% achieved by manual contours, the custom model achieved a 91.3% acceptance rate and 8.7% consensus unacceptable rate. The failure modes of retrained custom model were attributed to the following: cystogram (n = 2), hip prosthesis (n = 2), low dose rate brachytherapy seeds (n = 2), air in endorectal balloon(n = 1), non-iodinated spacer (n = 2), and giant bladder(n = 1). The commercial DLAS software with the incremental retraining function was validated and clinically adopted for prostate patients in a multi-user environment. AI-based auto-delineation of the prostate and OARs is shown to achieve improved physician acceptance, overall clinical utility, and accuracy.
Sections du résumé
BACKGROUND
BACKGROUND
Deep learning auto-segmentation (DLAS) models have been adopted in the clinic; however, they suffer from performance deterioration owing to the clinical practice variability. Some commercial DLAS software provide an incremental retraining function that enables users to train a custom model using their institutional data to account for clinical practice variability.
PURPOSE
OBJECTIVE
This study was performed to evaluate and implement the commercial DLAS software with the incremental retraining function for definitive treatment of patients with prostate cancer in a multi-user environment.
METHODS
METHODS
CT-based target organs and organs-at-risk (OAR) delineation of 215 prostate cancer patients were utilized. The performance of three commercial DLAS software built-in models was validated with 20 patients. A retrained custom model was developed using 100 patients and evaluated on the remaining data (n = 115). Dice similarity coefficient (DSC), Hausdorff distance (HD), mean surface distance (MSD), and surface DSC (SDSC) were utilized for quantitative evaluation. A multi-rater qualitative evaluation was blindly performed with a five-level scale. Visual inspection was performed in consensus and non-consensus unacceptable cases to identify the failure modes.
RESULTS
RESULTS
Three commercial DLAS vendor built-in models achieved sub-optimal performance in 20 patients. The retrained custom model had a mean DSC of 0.82 for prostate, 0.48 for seminal vesicles (SV), and 0.92 for rectum, respectively. This represents a significant improvement over the built-in model with DSC of 0.73, 0.37, and 0.81 for the corresponding structures. Compared to the acceptance rate of 96.5% and consensus unacceptable rate (i.e., both reviewers rated as unacceptable) of 3.5% achieved by manual contours, the custom model achieved a 91.3% acceptance rate and 8.7% consensus unacceptable rate. The failure modes of retrained custom model were attributed to the following: cystogram (n = 2), hip prosthesis (n = 2), low dose rate brachytherapy seeds (n = 2), air in endorectal balloon(n = 1), non-iodinated spacer (n = 2), and giant bladder(n = 1).
CONCLUSION
CONCLUSIONS
The commercial DLAS software with the incremental retraining function was validated and clinically adopted for prostate patients in a multi-user environment. AI-based auto-delineation of the prostate and OARs is shown to achieve improved physician acceptance, overall clinical utility, and accuracy.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
4079-4091Subventions
Organisme : NCI NIH HHS
ID : 75N91020C00048
Pays : United States
Organisme : NCI NIH HHS
ID : 75N91020C00048
Pays : United States
Informations de copyright
© 2023 American Association of Physicists in Medicine.
Références
Feng X, Qing K, Tustison NJ, Meyer CH, Chen Q. Deep convolutional neural network for segmentation of thoracic organs-at-risk using cropped 3D images [published online ahead of print 2019/03/05]. Med Phys. 2019;46(5):2169-2180.
Kaderka R, Gillespie EF, Mundt RC, et al. Geometric and dosimetric evaluation of atlas based auto-segmentation of cardiac structures in breast cancer patients [published online ahead of print 2018/08/16]. Radiother Oncol. 2019;131:215-220.
Raudaschl PF, Zaffino P, Sharp GC, et al. Evaluation of segmentation methods on head and neck CT: auto-segmentation challenge 2015 [published online ahead of print 2017/03/09]. Med Phys. 2017;44(5):2020-2036.
Nikolov S, Blackwell S, Zverovitch A, et al. Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy. arXiv preprint arXiv:180904430. 2018.
Yang J, Veeraraghavan H. Autosegmentation for thoracic radiation treatment planning: a grand challenge at AAPM 2017 [published online ahead of print 2018/08/26]. Med Phys. 2018;45(10):4568-4581.
Isambert A, Dhermain F, Bidault F, et al. Evaluation of an atlas-based automatic segmentation software for the delineation of brain organs at risk in a radiation therapy clinical context. Radiother Oncol. 2008;87(1):93-99.
Lustberg T, van Soest J, Gooding M, et al. Clinical evaluation of atlas and deep learning based automatic contouring for lung cancer [published online ahead of print 2017/12/07]. Radiother Oncol. 2018;126(2):312-317.
Kosmin M, Ledsam J, Romera-Paredes B, et al. Rapid advances in auto-segmentation of organs at risk and target volumes in head and neck cancer. Radiother Oncol. 2019;135:130-140.
Cardenas CE, Yang J, Anderson BM, Court LE, Brock KB. Advances in auto-segmentation [published online ahead of print 2019/04/28]. Semin Radiat Oncol. 2019;29(3):185-197.
Wong J, Fong A, McVicar N, et al. Comparing deep learning-based auto-segmentation of organs at risk and clinical target volumes to expert inter-observer variability in radiotherapy planning [published online ahead of print 2019/12/10]. Radiother Oncol. 2020;144:152-158.
Tao CJ, Yi JL, Chen NY, et al. Multi-subject atlas-based auto-segmentation reduces interobserver variation and improves dosimetric parameter consistency for organs at risk in nasopharyngeal carcinoma: a multi-institution clinical study [published online ahead of print 2015/05/31]. Radiother Oncol. 2015;115(3):407-411.
Thor M, Apte A, Haq R, Iyer A, LoCastro E, Deasy JO. Using auto-segmentation to reduce contouring and dose inconsistency in clinical trials: the simulated impact on RTOG 0617 [published online ahead of print 2020/11/17]. Int J Radiat Oncol Biol Phys. 2021;109(5):1619-1626.
Chen X, Sun S, Bai N, et al. A deep learning-based auto-segmentation system for organs-at-risk on whole-body computed tomography images for radiation therapy [published online ahead of print 2021/05/08]. Radiother Oncol. 2021;160:175-184.
Brouwer CL, Dinkla AM, Vandewinckele L, et al. Machine learning applications in radiation oncology: current use and needs to support clinical implementation [published online ahead of print 2021/01/19]. Phys Imaging Radiat Oncol. 2020;16:144-148.
Liang X, Nguyen D, Jiang SB. Generalizability issues with deep learning models in medicine and their potential solutions: illustrated with cone-beam computed tomography (CBCT) to computed tomography (CT) image conversion. Mach Learn Sci Technol. 2020;2(1):015007.
Chen C, Bai W, Davies RH, et al. Improving the generalizability of convolutional neural network-based segmentation on CMR images. Front Cardiovasc Med. 2020;7:105.
Wang X, Liang G, Zhang Y, Blanton H, Bessinger Z, Jacobs N. Inconsistent performance of deep learning models on mammogram classification. J Am Coll Radiol. 2020;17(6):796-803.
Wang B, Dohopolski M, Bai T, et al. Performance deterioration of deep learning models after clinical deployment: a case study with auto-segmentation for definitive prostate cancer radiotherapy. arXiv preprint arXiv:221005673. 2022.
Eche T, Schwartz LH, Mokrane F-Z, Dercle L. Toward generalizability in the deployment of artificial intelligence in radiology: role of computation stress testing to overcome underspecification. Radiol Artif Intell. 2021;3(6):e210097.
Azulay A, Weiss Y, Why do deep convolutional networks generalize so poorly to small image transformations?. arXiv preprint arXiv:180512177. 2018.
Feng X, Bernard ME, Hunter T, Chen Q. Improving accuracy and robustness of deep convolutional neural network based thoracic OAR segmentation [published online ahead of print 2020/02/23]. Phys Med Biol. 2020;65(7):07NT01.
Finlayson SG, Subbaswamy A, Singh K, et al. The clinician and dataset shift in artificial intelligence. N Engl J Med. 2021;385(3):283.
Pan I, Agarwal S, Merck D. Generalizable inter-institutional classification of abnormal chest radiographs using efficient convolutional neural networks [published online ahead of print 2019/03/07]. J Digit Imaging. 2019;32(5):888-896.
Brouwer CL, Steenbakkers RJHM, Gort E, et al. Differences in delineation guidelines for head and neck cancer result in inconsistent reported dose and corresponding NTCP. Radiother Oncol. 2014;111(1):148-152.
Roach D, Holloway LC, Jameson MG, et al. Multi-observer contouring of male pelvic anatomy: highly variable agreement across conventional and emerging structures of interest [published online ahead of print 2019/01/05]. J Med Imaging Radiat Oncol. 2019;63(2):264-271.
Patrick H, Souhami L, Kildea J. Reduction of inter-observer contouring variability in daily clinical practice through a retrospective, evidence-based intervention. Acta Oncol. 2021;60(2):229-236.
Savenije MHF, Maspero M, Sikkes GG, et al. Clinical implementation of MRI-based organs-at-risk auto-segmentation with convolutional networks for prostate radiotherapy [published online ahead of print 2020/05/13]. Radiat Oncol. 2020;15(1):104.
Balagopal A, Kazemifar S, Nguyen D, et al. Fully automated organ segmentation in male pelvic CT images. Phys Med Biol. 2018;63(24):245015.
Balagopal A, Morgan H, Dohopolski M, et al. PSA-Net: deep learning-based physician style-aware segmentation network for postoperative prostate cancer clinical target volumes. Artif Intell Med. 2021;121:102195.
Balagopal A, Nguyen D, Mashayekhi M, et al. Dosimetric impact of physician style variations in contouring CTV for post-operative prostate cancer: a deep learning-based simulation study. arXiv preprint arXiv:210201006. 2021.
Duan J, Bernard M, Downes L, et al. Evaluating the clinical acceptability of deep learning contours of prostate and organs-at-risk in an automated prostate treatment planning process. Med Phys. 2022;49(4):2570-2581.
Liu C, Gardner SJ, Wen N, et al. Automatic segmentation of the prostate on CT images using deep neural networks (DNN) [published online ahead of print 2019/03/21]. Int J Radiat Oncol Biol Phys. 2019;104(4):924-932.
Almeida G, Tavares JMR. Deep learning in radiation oncology treatment planning for prostate cancer: a systematic review. J Med Syst. 2020;44(10):1-15.
Kiljunen T, Akram S, Niemelä J, et al. A deep learning-based automated CT segmentation of prostate cancer anatomy for radiation therapy planning-a retrospective multicenter study. Diagnostics. 2020;10(11):959.
Kalantar R, Lin G, Winfield JM, et al. Automatic segmentation of pelvic cancers using deep learning: state-of-the-art approaches and challenges. Diagnostics. 2021;11(11):1964.
Kawula M, Hadi I, Nierer L, et al. Patient-specific transfer learning for auto-segmentation in adaptive 0.35 T MRgRT of prostate cancer: a bi-centric evaluation. Med Phys. 2023;50(3):1573-1585.
Avanzo M, Trianni A, Botta F, Talamonti C, Stasi M, Iori M. Artificial intelligence and the medical physicist: welcome to the machine. Appl Sci. 2021;11(4):1691.
Nakatsugawa M, Cheng Z, Kiess A, et al. The needs and benefits of continuous model updates on the accuracy of RT-induced toxicity prediction models within a learning health system. Int J Radiat Oncol Biol Phys. 2019;103(2):460-467.
Sherer MV, Lin D, Elguindi S, et al. Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: a critical review [published online ahead of print 2021/05/14]. Radiother Oncol. 2021;160:185-191.
Vaassen F, Hazelaar C, Vaniqui A, et al. Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy. Phys Imaging Radiat Oncol. 2020;13:1-6.
Cha E, Elguindi S, Onochie I, et al. Clinical implementation of deep learning contour autosegmentation for prostate radiotherapy [published online ahead of print 2021/03/06]. Radiother Oncol. 2021;159:1-7.
Jarrett D, Stride E, Vallis K, Gooding MJ. Applications and limitations of machine learning in radiation oncology. Br J Radiol. 2019;92(1100):20190001.
Sartor H, Minarik D, Enqvist O, et al. Auto-segmentations by convolutional neural network in cervical and anorectal cancer with clinical structure sets as the ground truth. Clin Transl Radiat Oncol. 2020;25:37-45.
Bayman NA, Wylie JP. When should the seminal vesicles be included in the target volume in prostate radiotherapy? Clin Oncol. 2007;19(5):302-307.
Kestin LL, Goldstein NS, Vicini FA, Yan D, Korman HJ, Martinez AA. Treatment of prostate cancer with radiotherapy: should the entire seminal vesicles be included in the clinical target volume? Int J Radiat Oncol Biol Phys. 2002;54(3):686-697.
Rhee DJ, Cardenas CE, Elhalawani H, et al. Automatic detection of contouring errors using convolutional neural networks. Med Phys. 2019;46(11):5086-5097.
McCarroll RE, Beadle BM, Balter PA, et al. Retrospective validation and clinical implementation of automated contouring of organs at risk in the head and neck: a step toward automated radiation treatment planning for low-and middle-income countries. J Glob Oncol. 2018;4:1-11.
Teguh DN, Levendag PC, Voet PW, et al. Clinical validation of atlas-based auto-segmentation of multiple target volumes and normal tissue (swallowing/mastication) structures in the head and neck [published online ahead of print 2010/10/12]. Int J Radiat Oncol Biol Phys. 2011;81(4):950-957.
Huyskens DP, Maingon P, Vanuytsel L, et al. A qualitative and a quantitative analysis of an auto-segmentation module for prostate cancer. Radiother Oncol. 2009;90(3):337-345.