End-to-end reproducible AI pipelines in radiology using the cloud.
Journal
Nature communications
ISSN: 2041-1723
Titre abrégé: Nat Commun
Pays: England
ID NLM: 101528555
Informations de publication
Date de publication:
13 Aug 2024
13 Aug 2024
Historique:
received:
05
07
2023
accepted:
30
07
2024
medline:
14
8
2024
pubmed:
14
8
2024
entrez:
13
8
2024
Statut:
epublish
Résumé
Artificial intelligence (AI) algorithms hold the potential to revolutionize radiology. However, a significant portion of the published literature lacks transparency and reproducibility, which hampers sustained progress toward clinical translation. Although several reporting guidelines have been proposed, identifying practical means to address these issues remains challenging. Here, we show the potential of cloud-based infrastructure for implementing and sharing transparent and reproducible AI-based radiology pipelines. We demonstrate end-to-end reproducibility from retrieving cloud-hosted data, through data pre-processing, deep learning inference, and post-processing, to the analysis and reporting of the final results. We successfully implement two distinct use cases, starting from recent literature on AI-based biomarkers for cancer imaging. Using cloud-hosted data and computing, we confirm the findings of these studies and extend the validation to previously unseen data for one of the use cases. Furthermore, we provide the community with transparent and easy-to-extend examples of pipelines impactful for the broader oncology field. Our approach demonstrates the potential of cloud resources for implementing, sharing, and using reproducible and transparent AI pipelines, which can accelerate the translation into clinical solutions.
Identifiants
pubmed: 39138215
doi: 10.1038/s41467-024-51202-2
pii: 10.1038/s41467-024-51202-2
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
6931Subventions
Organisme : EC | EU Framework Programme for Research and Innovation H2020 | H2020 Priority Excellent Science | H2020 European Research Council (H2020 Excellent Science - European Research Council)
ID : 866504
Organisme : Foundation for the National Institutes of Health (Foundation for the National Institutes of Health, Inc.)
ID : HHSN261201500003l
Informations de copyright
© 2024. The Author(s).
Références
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H. & Aerts, H. J. W. L. Artificial intelligence in radiology. Nat. Rev. Cancer 18, 500–510 (2018).
doi: 10.1038/s41568-018-0016-5
pubmed: 29777175
pmcid: 6268174
Kann, B. H., Hosny, A. & Aerts, H. J. W. L. Artificial intelligence for clinical oncology. Cancer Cell 39, 916–927 (2021).
doi: 10.1016/j.ccell.2021.04.002
pubmed: 33930310
pmcid: 8282694
Huynh, E. et al. Artificial intelligence in radiation oncology. Nat. Rev. Clin. Oncol. 17, 771–781 (2020).
doi: 10.1038/s41571-020-0417-8
pubmed: 32843739
Niazi, M. K. K., Parwani, A. V. & Gurcan, M. N. Digital pathology and artificial intelligence. Lancet Oncol. 20, e253–e261 (2019).
doi: 10.1016/S1470-2045(19)30154-8
pubmed: 31044723
pmcid: 8711251
Haibe-Kains, B. et al. Transparency and reproducibility in artificial intelligence. Nature 586, E14–E16 (2020).
doi: 10.1038/s41586-020-2766-y
pubmed: 33057217
pmcid: 8144864
Benaich, N. & Hogarth, I. State of AI report 2021. https://www.stateof.ai/ (2021).
Simko, A., Garpebring, A., Jonsson, J., Nyholm, T. & Löfstedt, T. Reproducibility of the methods in medical imaging with deep learning. Medical Imaging with Deep Learning 227, 95–106 (2024).
Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454 (2016).
doi: 10.1038/533452a
pubmed: 27225100
Yu, K.-H., Beam, A. L. & Kohane, I. S. Artificial intelligence in healthcare. Nat. Biomed. Eng. 2, 719–731 (2018).
doi: 10.1038/s41551-018-0305-z
pubmed: 31015651
Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447–453 (2019).
doi: 10.1126/science.aax2342
pubmed: 31649194
Adamson, A. S. & Smith, A. Machine learning and health care disparities in dermatology. JAMA Dermatol. 154, 1247–1248 (2018).
doi: 10.1001/jamadermatol.2018.2348
pubmed: 30073260
Roberts, M. et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat. Mach. Intell. 3, 199–217 (2021).
doi: 10.1038/s42256-021-00307-0
Liu, X. et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health 1, e271–e297 (2019).
doi: 10.1016/S2589-7500(19)30123-2
pubmed: 33323251
Wynants, L. et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 369, m1328 (2020).
doi: 10.1136/bmj.m1328
pubmed: 32265220
pmcid: 7222643
Yu, A. C., Mohajer, B. & Eng, J. External validation of deep learning algorithms for radiologic diagnosis: a systematic review. Radio. Artif. Intell. 4, e210064 (2022).
doi: 10.1148/ryai.210064
Wong, A. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern. Med. 181, 1065–1070 (2021).
doi: 10.1001/jamainternmed.2021.2626
pubmed: 34152373
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G. & King, D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 17, 195 (2019).
doi: 10.1186/s12916-019-1426-2
pubmed: 31665002
pmcid: 6821018
Walsh, I. et al. DOME: recommendations for supervised machine learning validation in biology. Nat. Methods 18, 1122–1127 (2021).
doi: 10.1038/s41592-021-01205-4
pubmed: 34316068
Gebru, T. et al. Datasheets for datasets. Commun. ACM 64, 86–92 (2021).
Mitchell, M. et al. Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, 220–229 (2019).
Laine, R. F., Arganda-Carreras, I., Henriques, R. & Jacquemet, G. Avoiding a replication crisis in deep-learning-based bioimage analysis. Nat. Methods 18, 1136–1144 (2021).
doi: 10.1038/s41592-021-01284-3
pubmed: 34608322
pmcid: 7611896
Varoquaux, G. & Cheplygina, V. Machine learning for medical imaging: methodological failures and recommendations for the future. NPJ Digit. Med. 5, 48 (2022).
doi: 10.1038/s41746-022-00592-y
pubmed: 35413988
pmcid: 9005663
Gottesman, O. et al. Guidelines for reinforcement learning in healthcare. Nat. Med. 25, 16–18 (2019).
doi: 10.1038/s41591-018-0310-5
pubmed: 30617332
Mongan, J., Moy, L. & Kahn, C. E. Jr Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radio. Artif. Intell. 2, e200029 (2020).
doi: 10.1148/ryai.2020200029
Matschinske, J. et al. The AIMe registry for artificial intelligence in biomedical research. Nat. Methods 18, 1128–1131 (2021).
doi: 10.1038/s41592-021-01241-0
pubmed: 34433960
Mateus, P. et al. Image based prognosis in head and neck cancer using convolutional neural networks: a case study in reproducibility and optimization. Sci. Rep. 13, 18176 (2023).
doi: 10.1038/s41598-023-45486-5
pubmed: 37875663
pmcid: 10598263
Botvinik-Nezer, R. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020).
doi: 10.1038/s41586-020-2314-9
pubmed: 32483374
pmcid: 7771346
Moradmand, H., Aghamiri, S. M. R. & Ghaderi, R. Impact of image preprocessing methods on reproducibility of radiomic features in multimodal magnetic resonance imaging in glioblastoma. J. Appl. Clin. Med. Phys. 21, 179–190 (2020).
doi: 10.1002/acm2.12795
pubmed: 31880401
Poirot, M. G. et al. Robustness of radiomics to variations in segmentation methods in multimodal brain MRI. Sci. Rep. 12, 16712 (2022).
doi: 10.1038/s41598-022-20703-9
pubmed: 36202934
pmcid: 9537186
Brazma, A. et al. Minimum information about a microarray experiment (MIAME)-toward standards for microarray data. Nat. Genet. 29, 365–371 (2001).
doi: 10.1038/ng1201-365
pubmed: 11726920
Grossman, R. L. et al. Toward a shared vision for cancer genomic data. New Engl. J. Med. 375, 1109–1112 (2016).
doi: 10.1056/NEJMp1607591
pubmed: 27653561
Zhang, Z. et al. Uniform genomic data analysis in the NCI Genomic Data Commons. Nat. Commun. 12, 1226 (2021).
doi: 10.1038/s41467-021-21254-9
pubmed: 33619257
pmcid: 7900240
McDermott, M. B. A. et al. Reproducibility in machine learning for health research: still a ways to go. Sci. Transl. Med. 13, eabb1655 (2021).
doi: 10.1126/scitranslmed.abb1655
pubmed: 33762434
Schacherer, D. P. et al. The NCI Imaging Data Commons as a platform for reproducible research in computational pathology. Comput. Methods Prog. Biomed. 242, 107839 (2023).
doi: 10.1016/j.cmpb.2023.107839
Fedorov, A. et al. National Cancer Institute Imaging Data Commons: toward transparency, reproducibility, and scalability in imaging artificial intelligence. Radiographics 43, e230180 (2023).
doi: 10.1148/rg.230180
pubmed: 37999984
Hosny, A. et al. Deep learning for lung cancer prognostication: a retrospective multi-cohort radiomics study. PLoS Med. 15, e1002711 (2018).
doi: 10.1371/journal.pmed.1002711
pubmed: 30500819
pmcid: 6269088
Pai, S. et al. Foundation models for quantitative biomarker discovery in cancer imaging. Nat. Mach. Intell. 6, 354–367 (2024).
Aerts, H. J. W. L. et al. Data from NSCLC-Radiomics. https://doi.org/10.7937/K9/TCIA.2015.PF0M9REI (The Cancer Imaging Archive, 2019).
Amazon Web Services. SageMaker studio lab. https://studiolab.sagemaker.aws/ (2024).
Radboud University Medical Center. Grand challenge https://grand-challenge.org/ (2024).
Wolff, T. The future of community coalition building. Am. J. Community Psychol. 29, 263–268 (2001).
doi: 10.1023/A:1010330730421
pubmed: 11446280
The Binder Team. The binder project. https://mybinder.org/ (2024).
Sinha, V. The collaborative data notebook for EDA & ETL. Noteable https://noteable.io (2023).
The Imaging Data Common Team. IDC Porta. https://portal.imaging.datacommons.cancer.gov/ (2024).
Aerts, H. J. W. L. et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat. Commun. 5, 4006 (2014).
doi: 10.1038/ncomms5006
pubmed: 24892406
Bakr, S. et al. Data for NSCLC Radiogenomics. https://doi.org/10.7937/K9/TCIA.2017.7hs46erv (The Cancer Imaging Archive, 2017).
Bakr, S. et al. A radiogenomic dataset of non-small cell lung cancer. Sci. Data 5, 180202 (2018).
doi: 10.1038/sdata.2018.202
pubmed: 30325352
pmcid: 6190740
Bisong, E. Google colaboratory. In Building Machine Learning and Deep Learning Models on Google Cloud Platform: A Comprehensive Guide for Beginners (ed. Bisong, E.) 59–64 (Apress, Berkeley, 2019).
Shackleford, J., Kandasamy, N. & Sharp, G. Chapter 6—Plastimatch—an open-source software for radiotherapy imaging. In High Performance Deformable Image Registration Algorithms for Manycore Processors (eds Shackleford, J. et al.) 107–114 (Morgan Kaufmann, Boston, 2013).
Harris, C. R. et al. Array programming with NumPy. Nature 585, 357–362 (2020).
Lowekamp, B. C., Chen, D. T., Ibáñez, L. & Blezek, D. The design of SimpleITK. Front. Neuroinform. 7, 45 (2013).
doi: 10.3389/fninf.2013.00045
pubmed: 24416015
pmcid: 3874546
McCormick, M., Liu, X., Jomier, J., Marion, C. & Ibanez, L. ITK: enabling reproducible research and open science. Front. Neuroinform. 8, 13 (2014).
doi: 10.3389/fninf.2014.00013
pubmed: 24600387
pmcid: 3929840
Insight Software Consortium. Itkwidgets: An Elegant Python Interface for Visualization on the Web Platform to Interactively Generate Insights into Multidimensional Images, Point Sets, and Geometry. https://github.com/InsightSoftwareConsortium/itkwidgets (2024).
Ziegler, E. et al. Open health imaging foundation viewer: an extensible open-source framework for building web-based imaging applications to support cancer research. JCO Clin. Cancer Inf. 4, 336–345 (2020).
doi: 10.1200/CCI.19.00131
The Peak team. s5cmd: Parallel S3 and Local Filesystem Execution Tool. https://github.com/peak/s5cmd (2024).
Pieper, S. Dicomsort: A Project to Provide Custom Sorting and Renaming of Dicom Files. https://github.com/pieper/dicomsort (2024).
Mason, D. L. et al. Pydicom: An open source DICOM library. https://github.com/pydicom/pydicom (2024).
Phil, T., Albrecht, T., Gay, S. & Rasmussen, M. E. Sikerdebaard/dcmrtstruct2nii: dcmrtstruct2nii. https://doi.org/10.5281/zenodo.4037864 (2023).
Virtanen, P. et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods 17, 261–272 (2020).
Pedregosa, F. et al. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12, 2825–2830 (2011).
Davidson-Pilon, C. Lifelines: Survival Analysis in Python. https://github.com/CamDavidsonPilon/lifelines (2024).
The Onnx Team. Onnx: Open Standard for Machine Learning Interoperability. https://github.com/onnx/onnx (2024).
Bontempi, D. ImagingDataCommons/idc-Radiomics-Reproducibility. https://doi.org/10.5281/zenodo.10123555 (2024).
Clark, K. et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26, 1045–1057 (2013).
doi: 10.1007/s10278-013-9622-7
pubmed: 23884657
pmcid: 3824915