METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII.
Artificial intelligence
Deep learning
Guideline
Machine learning
Radiomics
Journal
Insights into imaging
ISSN: 1869-4101
Titre abrégé: Insights Imaging
Pays: Germany
ID NLM: 101532453
Informations de publication
Date de publication:
17 Jan 2024
17 Jan 2024
Historique:
received:
01
08
2023
accepted:
20
11
2023
medline:
17
1
2024
pubmed:
17
1
2024
entrez:
16
1
2024
Statut:
epublish
Résumé
To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies. We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated. In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community. In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers. A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning. • A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score ( https://metricsscore.github.io/metrics/METRICS.html ) and a repository created to collect feedback from the radiomics community ( https://github.com/metricsscore/metrics ).
Identifiants
pubmed: 38228979
doi: 10.1186/s13244-023-01572-w
pii: 10.1186/s13244-023-01572-w
doi:
Types de publication
Journal Article
Langues
eng
Pagination
8Informations de copyright
© 2024. The Author(s).
Références
Gillies RJ, Kinahan PE, Hricak H (2016) Radiomics: images are more than pictures, they are data. Radiology 278:563–577. https://doi.org/10.1148/radiol.2015151169
doi: 10.1148/radiol.2015151169
pubmed: 26579733
Kocak B, Baessler B, Cuocolo R et al (2023) Trends and statistics of artificial intelligence and radiomics research in Radiology, Nuclear Medicine, and Medical Imaging: bibliometric analysis. Eur Radiol. https://doi.org/10.1007/s00330-023-09772-0
doi: 10.1007/s00330-023-09772-0
pubmed: 37740080
Kocak B, Bulut E, Bayrak ON et al (2023) NEgatiVE results in Radiomics research (NEVER): a meta-research study of publication bias in leading radiology journals. Eur J Radiol 163:110830. https://doi.org/10.1016/j.ejrad.2023.110830
doi: 10.1016/j.ejrad.2023.110830
pubmed: 37119709
Pinto Dos Santos D, Dietzel M, Baessler B (2021) A decade of radiomics research: are images really data or just patterns in the noise? Eur Radiol 31:1–4. https://doi.org/10.1007/s00330-020-07108-w
doi: 10.1007/s00330-020-07108-w
pubmed: 32767103
Papanikolaou N, Matos C, Koh DM (2020) How to develop a meaningful radiomic signature for clinical use in oncologic patients. Cancer Imaging 20:33. https://doi.org/10.1186/s40644-020-00311-4
doi: 10.1186/s40644-020-00311-4
pubmed: 32357923
pmcid: 7195800
Buvat I, Orlhac F (2019) The dark side of radiomics: on the paramount importance of publishing negative results. J Nucl Med 60:1543–1544. https://doi.org/10.2967/jnumed.119.235325
doi: 10.2967/jnumed.119.235325
pubmed: 31541033
Vallières M, Zwanenburg A, Badic B et al (2018) Responsible radiomics research for faster clinical translation. J Nucl Med 59:189–193. https://doi.org/10.2967/jnumed.117.200501
doi: 10.2967/jnumed.117.200501
pubmed: 29175982
pmcid: 5807530
Kocak B, Yardimci AH, Yuzkan S et al (2022) Transparency in artificial intelligence research: a systematic review of availability items related to open science in radiology and nuclear medicine. Acad Radiol S1076–6332(22):00635–3. https://doi.org/10.1016/j.acra.2022.11.030
doi: 10.1016/j.acra.2022.11.030
deSouza NM, van der Lugt A, Deroose CM et al (2022) Standardised lesion segmentation for imaging biomarker quantitation: a consensus recommendation from ESR and EORTC. Insights Imaging 13:159. https://doi.org/10.1186/s13244-022-01287-4
doi: 10.1186/s13244-022-01287-4
pubmed: 36194301
pmcid: 9532485
Lambin P, Leijenaar RTH, Deist TM et al (2017) Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol 14:749–762. https://doi.org/10.1038/nrclinonc.2017.141
doi: 10.1038/nrclinonc.2017.141
pubmed: 28975929
Spadarella G, Stanzione A, Akinci D’Antonoli T et al (2023) Systematic review of the radiomics quality score applications: an EuSoMII Radiomics Auditing Group Initiative. Eur Radiol 33:1884–1894. https://doi.org/10.1007/s00330-022-09187-3
doi: 10.1007/s00330-022-09187-3
pubmed: 36282312
Sanduleanu S, Woodruff HC, de Jong EEC et al (2018) Tracking tumor biology with radiomics: a systematic review utilizing a radiomics quality score. Radiother Oncol 127:349–360. https://doi.org/10.1016/j.radonc.2018.03.033
doi: 10.1016/j.radonc.2018.03.033
pubmed: 29779918
Akinci D’Antonoli T, Cavallo AU, Vernuccio F et al (2023) Reproducibility of radiomics quality score: an intra- and inter-rater reliability study. Eur Radiol. https://doi.org/10.1007/s00330-023-10217-x
doi: 10.1007/s00330-023-10217-x
pubmed: 37740080
Welch ML, McIntosh C, Haibe-Kains B et al (2019) Vulnerabilities of radiomic signature development: the need for safeguards. Radiother Oncol 130:2–9. https://doi.org/10.1016/j.radonc.2018.10.027
doi: 10.1016/j.radonc.2018.10.027
pubmed: 30416044
Kocak B, Baessler B, Bakas S et al (2023) CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII. Insights Imaging 14:75. https://doi.org/10.1186/s13244-023-01415-8
doi: 10.1186/s13244-023-01415-8
pubmed: 37142815
pmcid: 10160267
Caulley L, Catalá-López F, Whelan J et al (2020) Reporting guidelines of health research studies are frequently used inappropriately. J Clin Epidemiol 122:87–94. https://doi.org/10.1016/j.jclinepi.2020.03.006
doi: 10.1016/j.jclinepi.2020.03.006
pubmed: 32184126
Logullo P, MacCarthy A, Kirtley S, Collins GS (2020) Reporting guideline checklists are not quality evaluation forms: they are guidance for writing. Health Sci Rep 3:e165. https://doi.org/10.1002/hsr2.165
doi: 10.1002/hsr2.165
pubmed: 32373717
pmcid: 7196677
Moher D, Schulz KF, Simera I, Altman DG (2010) Guidance for developers of health research reporting guidelines. PLoS Med 7:e1000217. https://doi.org/10.1371/journal.pmed.1000217
doi: 10.1371/journal.pmed.1000217
pubmed: 20169112
pmcid: 2821895
Diamond IR, Grant RC, Feldman BM et al (2014) Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 67:401–409. https://doi.org/10.1016/j.jclinepi.2013.12.002
doi: 10.1016/j.jclinepi.2013.12.002
pubmed: 24581294
Roszkowska E (2013) Rank Ordering Criteria Weighting Methods – a Comparative Overview. Optim Stud Ekon 14–33
Stillwell WG, Seaver DA, Edwards W (1981) A comparison of weight approximation techniques in multiattribute utility decision making. Organ Behav Hum Perform 28:62–77. https://doi.org/10.1016/0030-5073(81)90015-5
doi: 10.1016/0030-5073(81)90015-5
Whiting PF, Rutjes AWS, Westwood ME et al (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155:529–536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009
doi: 10.7326/0003-4819-155-8-201110180-00009
pubmed: 22007046
Bossuyt PM, Reitsma JB, Bruns DE et al (2015) STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ 351:h5527. https://doi.org/10.1136/bmj.h5527
doi: 10.1136/bmj.h5527
pubmed: 26511519
pmcid: 4623764
Collins GS, Reitsma JB, Altman DG, Moons KG (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med 13:1. https://doi.org/10.1186/s12916-014-0241-z
doi: 10.1186/s12916-014-0241-z
pubmed: 25563062
pmcid: 4284921
Luo W, Phung D, Tran T et al (2016) Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res 18:e323. https://doi.org/10.2196/jmir.5870
doi: 10.2196/jmir.5870
pubmed: 27986644
pmcid: 5238707
Martin J (2017) © Joanna Briggs Institute 2017 Critical Appraisal Checklist for Analytical Cross Sectional Studies
Mongan J, Moy L, Kahn CE (2020) Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell 2:e200029. https://doi.org/10.1148/ryai.2020200029
doi: 10.1148/ryai.2020200029
pubmed: 33937821
pmcid: 8017414
Zwanenburg A, Vallières M, Abdalah MA et al (2020) The image biomarker standardization initiative: standardized quantitative radiomics for high-throughput image-based phenotyping. Radiology 295:328–338. https://doi.org/10.1148/radiol.2020191145
doi: 10.1148/radiol.2020191145
pubmed: 32154773
Orlhac F, Nioche C, Klyuzhin I et al (2021) Radiomics in PET imaging: a practical guide for newcomers. PET Clin 16:597–612. https://doi.org/10.1016/j.cpet.2021.06.007
doi: 10.1016/j.cpet.2021.06.007
pubmed: 34537132
Pfaehler E, Zhovannik I, Wei L et al (2021) A systematic review and quality of reporting checklist for repeatability and reproducibility of radiomic features. Phys Imaging Radiat Oncol 20:69–75. https://doi.org/10.1016/j.phro.2021.10.007
doi: 10.1016/j.phro.2021.10.007
pubmed: 34816024
pmcid: 8591412
Shur JD, Doran SJ, Kumar S et al (2021) Radiomics in Oncology: a practical guide. Radiographics 41:1717–1732. https://doi.org/10.1148/rg.2021210037
doi: 10.1148/rg.2021210037
pubmed: 34597235
Sollini M, Cozzi L, Ninatti G et al (2021) PET/CT radiomics in breast cancer: mind the step. Methods 188:122–132. https://doi.org/10.1016/j.ymeth.2020.01.007
doi: 10.1016/j.ymeth.2020.01.007
pubmed: 31978538
Volpe S, Pepa M, Zaffaroni M et al (2021) Machine learning for head and neck cancer: a safe bet?-a clinically oriented systematic review for the radiation oncologist. Front Oncol 11:772663. https://doi.org/10.3389/fonc.2021.772663
doi: 10.3389/fonc.2021.772663
pubmed: 34869010
pmcid: 8637856
Jha AK, Bradshaw TJ, Buvat I et al (2022) Nuclear medicine and artificial intelligence: best practices for evaluation (the RELAINCE Guidelines). J Nucl Med 63:1288–1299. https://doi.org/10.2967/jnumed.121.263239
doi: 10.2967/jnumed.121.263239
pubmed: 35618476
pmcid: 9454473
Hatt M, Krizsan AK, Rahmim A et al (2023) Joint EANM/SNMMI guideline on radiomics in nuclear medicine: jointly supported by the EANM Physics Committee and the SNMMI Physics, Instrumentation and Data Sciences Council. Eur J Nucl Med Mol Imag 50:352–375. https://doi.org/10.1007/s00259-022-06001-6
doi: 10.1007/s00259-022-06001-6
Cerdá-Alberich L, Solana J, Mallol P et al (2023) MAIC–10 brief quality checklist for publications using artificial intelligence and medical images. Insights Imaging 14:11. https://doi.org/10.1186/s13244-022-01355-9
doi: 10.1186/s13244-022-01355-9
pubmed: 36645542
pmcid: 9842808
Heus P, Damen JAAG, Pajouheshnia R et al (2019) Uniformity in measuring adherence to reporting guidelines: the example of TRIPOD for assessing completeness of reporting of prediction model studies. BMJ Open 9:e025611. https://doi.org/10.1136/bmjopen-2018-025611
doi: 10.1136/bmjopen-2018-025611
pubmed: 31023756
pmcid: 6501951
Tejani AS, Klontzas ME, Gatti AA et al (2023) Updating the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) for reporting AI research. Nat Mach Intell 5:950–951. https://doi.org/10.1038/s42256-023-00717-2
doi: 10.1038/s42256-023-00717-2
Klontzas ME, Gatti AA, Tejani AS, Kahn CE (2023) AI Reporting Guidelines: how to select the best one for your research. Radiol Artif Intell 5:e230055. https://doi.org/10.1148/ryai.230055
doi: 10.1148/ryai.230055
pubmed: 37293341
pmcid: 10245184
Gidwani M, Chang K, Patel JB et al (2023) Inconsistent partitioning and unproductive feature associations yield idealized radiomic models. Radiology 307:e220715. https://doi.org/10.1148/radiol.220715
doi: 10.1148/radiol.220715
pubmed: 36537895
Zwanenburg A (2019) Radiomics in nuclear medicine: robustness, reproducibility, standardization, and how to avoid data analysis traps and replication crisis. Eur J Nucl Med Mol Imag 46:2638–2655. https://doi.org/10.1007/s00259-019-04391-8
doi: 10.1007/s00259-019-04391-8
Park JE, Park SY, Kim HJ, Kim HS (2019) Reproducibility and generalizability in radiomics modeling: possible strategies in radiologic and statistical perspectives. Korean J Radiol 20:1124–1137. https://doi.org/10.3348/kjr.2018.0070
doi: 10.3348/kjr.2018.0070
pubmed: 31270976
pmcid: 6609433
National Academies of Sciences Engineering, Medicine (2019) Reproducibility and Replicability in Science. The National Academies Press, Washington, DC
Kocak B, Keles A, Akinci D’Antonoli T (2023) Self-reporting with checklists in artificial intelligence research on medical imaging: a systematic review based on citations of CLAIM. Eur Radiol. https://doi.org/10.1007/s00330-023-10243-9
doi: 10.1007/s00330-023-10243-9
pubmed: 37740080
Akinci D’Antonoli T, Mercaldo ND (2023) Obsolescence of nomograms in radiomics research. Eur Radiol. https://doi.org/10.1007/s00330-023-09728-4
doi: 10.1007/s00330-023-09728-4
pubmed: 37740080