Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls.

To investigate the impact of the following three methodological pitfalls on model generalizability: ( The authors used retrospective CT, histopathologic analysis, and radiography datasets to develop machine learning models with and without the three methodological pitfalls to quantitatively illustrate their effect on model performance and generalizability. F1 score was used to measure performance, and differences in performance between models developed with and without errors were assessed using the Wilcoxon rank sum test when applicable. Violation of the independence assumption by applying oversampling, feature selection, and data augmentation before splitting data into training, validation, and test sets seemingly improved model F1 scores by 71.2% for predicting local recurrence and 5.0% for predicting 3-year overall survival in head and neck cancer and by 46.0% for distinguishing histopathologic patterns in lung cancer. Randomly distributing data points for a patient across datasets superficially improved the F1 score by 21.8%. High model performance metrics did not indicate high-quality lung segmentation. In the presence of a batch effect, a model built for pneumonia detection had an F1 score of 98.7% but correctly classified only 3.86% of samples from a new dataset of healthy patients. Machine learning models developed with these methodological pitfalls, which are undetectable during internal evaluation, produce inaccurate predictions; thus, understanding and avoiding these pitfalls is necessary for developing generalizable models.

Informations de copyright

© 2022 by the Radiological Society of North America, Inc.

Déclaration de conflit d'intérêts

Disclosures of conflicts of interest: F.M. No relevant relationships. K.O. No relevant relationships. R.G. No relevant relationships. C.R. Grants or contracts from Imagia. A.S. No relevant relationships. R.F. Research grant from McGill University Health Centre Foundation, TD Bank, GE Healthcare, and Intel for Artificial Intelligence for Urgent Care: Providing Better Care to Remote Communities, payments made to institution (Research Institute of the McGill University Health Centre); Canadian Cancer Society/Canadian Institutes of Health Research/Brain Canada Foundation Spark Grant: Novel Technology Applications in Cancer Prevention and Early Detection (SPARK-21), research grant, payments made to institution (McGill); Fonds de recherche en santé du Québec and Fondation de l’Association des radiologistes du Québec grant providing research salary support and time (payments to author) and general research operating grant (payments to institution), not directly funding this work.