A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study.
Random Forest
cross-validation
hyperparameter tuning
penalized regression
prediction models
Journal
Statistics in medicine
ISSN: 1097-0258
Titre abrégé: Stat Med
Pays: England
ID NLM: 8215016
Informations de publication
Date de publication:
08 Jan 2024
08 Jan 2024
Historique:
revised:
10
09
2023
received:
11
11
2022
accepted:
21
09
2023
medline:
8
1
2024
pubmed:
8
1
2024
entrez:
8
1
2024
Statut:
aheadofprint
Résumé
Tuning hyperparameters, such as the regularization parameter in Ridge or Lasso regression, is often aimed at improving the predictive performance of risk prediction models. In this study, various hyperparameter tuning procedures for clinical prediction models were systematically compared and evaluated in low-dimensional data. The focus was on out-of-sample predictive performance (discrimination, calibration, and overall prediction error) of risk prediction models developed using Ridge, Lasso, Elastic Net, or Random Forest. The influence of sample size, number of predictors and events fraction on performance of the hyperparameter tuning procedures was studied using extensive simulations. The results indicate important differences between tuning procedures in calibration performance, while generally showing similar discriminative performance. The one-standard-error rule for tuning applied to cross-validation (1SE CV) often resulted in severe miscalibration. Standard non-repeated and repeated cross-validation (both 5-fold and 10-fold) performed similarly well and outperformed the other tuning procedures. Bootstrap showed a slight tendency to more severe miscalibration than standard cross-validation-based tuning procedures. Differences between tuning procedures were larger for smaller sample sizes, lower events fractions and fewer predictors. These results imply that the choice of tuning procedure can have a profound influence on the predictive performance of prediction models. The results support the application of standard 5-fold or 10-fold cross-validation that minimizes out-of-sample prediction error. Despite an increased computational burden, we found no clear benefit of repeated over non-repeated cross-validation for hyperparameter tuning. We warn against the potentially detrimental effects on model calibration of the popular 1SE CV rule for tuning prediction models in low-dimensional settings.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2024 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.
Références
Van Smeden M, Reitsma JB, Riley RD, Collins GS, Moons KGM. Clinical prediction models: diagnosis versus prognosis. J Clin Epidemiol. 2021;132:142-145.
Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019;110:12-22.
Ellenbach N, Boulesteix AL, Bischl B, Unger K, Hornung R. Improved outcome prediction across data sources through robust parameter tuning. J Classif. 2020;38:212-231.
Probst P, Boulesteix AL, Bischl B. Tunability: importance of hyperparameters of machine learning algorithms. Journal of Machine Learning Research. 2019;20:1-32.
Witten IH, Frank E, Hall MA. Data Mining: Practical Machine Learning Tools and Techniques. Massachusetts, USA: Morgan Kaufmann; 2011.
Breiman L, Friedman J, Stone CJ, Olshen RA. Classification and Regression Trees. Boca Raton: CRC press; 1984.
Hastie T, Tibshirani R, Friedman J. In the Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer; 2009.
Léger C, Romano JP. Bootstrap choice of tuning parameters. Ann Instit Stat Math. 1990;42:709-735.
Fan Y, Tang CY. Tuning parameter selection in high-dimensional penalized likelihood. J R Stat Soc Ser B. 2013;75:531-552.
Cessie SL, Houwelingen JC. Ridge estimators in logistic regression. Appl Stat. 1992;41:191-201.
Tibshirani R. Regression shrinkage and selection via the Lasso. J R Stat Soc B. 1996;58:267-288.
Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc B. 2005;67:301-320.
Breiman L. Random forests. Mach Learn. 2001;45:5-32.
Agresti A. Categorical Data Analysis. 2. New Jersey: John Wiley & Sons, Inc.; 2002.
Hoerl AE, Kennard RW. Ridge regression: biased estimation for nonorthogonal problems. Technometrics. 2000;42:80-86.
Probst P, Wright MN, Boulesteix AL. Hyperparameters and tuning strategies for random forest. WIREs Data Min Knowl Discov. 2019;9(e1301):1-15.
Geurts P, Ernst D, Wehenkel L. Extremely randomized trees. Mach Learn. 2006;63:3-42.
Probst P, Boulesteix AL. To tune or not to tune the number of trees in random forest. J Mach Learn Res. 2018;18:1-18.
Anguita D, Boni A, Ridella S, Rivieccio F, Sterpi D. Theoretical and Practical Model Selection Methods for Support Vector Classifiers. Berlin: Springer; 2005:177.
Kuhn M. Building predictive models in R using the caret package. J Stat Softw. 2008;28:1-26.
Riley RD, Ensor J, Snell KIE, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020;368:1-12.
Harrell FE Jr. Regression Modeling Strategies. New York: Springer; 2001.
Van Calster B, McLernon DJ, Van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(230):1-7.
Van Calster B, Van Smeden M, De Cock B, Steyerberg EW. Regression shrinkage methods for clinical prediction models do not guarantee improved performance: simulation study. Stat Methods Med Res. 2020;29:1-13.
Team RC. A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2020.
Friedman J, Hastie T, Tibshirani R. Regularization paths for generalized linear models via coordinate descent. J Stat Softw. 2010;33:1-22.
Wright MN, Wager S, Probst P. A fast implementation of random forests. 2019: 123-136.
Kaijser J, Bourne T, Valentin L, et al. Improving strategies for diagnosing ovarian cancer: a summary of the International Ovarian Tumor Analysis (IOTA) studies. Ultrasound Obstet Gynecol. 2013;41:9-20.
Van Calster B, Van Hoorde K, Valentin L, et al. Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study. BMJ. 2014;349:g5920.
Duarte E, Wainer J. Empirical comparison of cross-validation and internal metrics for tuning SVM hyperparameters. Pattern Recognit Lett. 2017;88:6-11.
Steyerberg EW, Harrell FE Jr, Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54:774-781.
Austin PC, Steyerberg EW. Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Stat Methods Med Res. 2017;26:796-808.
Riley RD, Snell KIE, Martin GP, et al. Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small. J Clin Epidemiol. 2021;132:88-96.
Šinkovec H, Heinze G, Blagus R, Geroldinger A. To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets. BMC Med Res Methodol. 2021;21:199.
Van Smeden M, Moons KGM, Groot dJAH, et al. Sample size for binary logistic prediction models: Beyond events per variable criteria. Stat Methods Med Res. 2019;28:2455-2474.
Akaike H. Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory; 1973; Budapest.