Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model.
Journal
Applied clinical informatics
ISSN: 1869-0327
Titre abrégé: Appl Clin Inform
Pays: Germany
ID NLM: 101537732
Informations de publication
Date de publication:
03 2022
03 2022
Historique:
entrez:
4
5
2022
pubmed:
5
5
2022
medline:
7
5
2022
Statut:
ppublish
Résumé
The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital. The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes. The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data ( This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.
Identifiants
pubmed: 35508197
doi: 10.1055/s-0042-1746168
pmc: PMC9068274
doi:
Types de publication
Evaluation Study
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
431-438Informations de copyright
Thieme. All rights reserved.
Déclaration de conflit d'intérêts
None declared.
Références
N Engl J Med. 2019 Apr 4;380(14):1347-1358
pubmed: 30943338
BMJ. 1995 Jan 21;310(6973):170
pubmed: 7833759
Blood Purif. 2016;41(1-3):211-7
pubmed: 26766175
BMJ Qual Saf. 2019 Mar;28(3):231-237
pubmed: 30636200
Clin J Am Soc Nephrol. 2013 Mar;8(3):476-83
pubmed: 23471414
Heart. 2012 May;98(9):691-8
pubmed: 22397946
Nephron Clin Pract. 2012;120(4):c179-84
pubmed: 22890468
N Engl J Med. 2020 Apr 23;382(17):1583-1586
pubmed: 32320568
Kidney Int. 2012 Mar;81(5):442-8
pubmed: 22113526
Stat Med. 2000 Feb 29;19(4):453-73
pubmed: 10694730
J Clin Epidemiol. 2015 Mar;68(3):279-89
pubmed: 25179855
Multivariate Behav Res. 2011 May;46(3):399-424
pubmed: 21818162
JAMA Intern Med. 2021 Aug 1;181(8):1065-1070
pubmed: 34152373
Kidney Res Clin Pract. 2021 Mar;40(1):40-51
pubmed: 33663033
Pediatr Rev. 2014 Jan;35(1):16-29
pubmed: 24385562
Ann Intern Med. 1999 Mar 16;130(6):515-24
pubmed: 10075620
Pediatr Res. 2017 Sep;82(3):465-473
pubmed: 28486440
N Engl J Med. 2017 Jan 5;376(1):11-20
pubmed: 27959707
Stat Med. 2008 Jan 30;27(2):157-72; discussion 207-12
pubmed: 17569110
Biostatistics. 2020 Apr 1;21(2):345-352
pubmed: 31742354
Pediatrics. 2003 Jun;111(6 Pt 1):1416-21
pubmed: 12777562
Appl Clin Inform. 2017 Aug 09;8(3):826-831
pubmed: 28837212
Ann Intern Med. 2020 Jun 2;172(11 Suppl):S137-S144
pubmed: 32479180
BMC Nephrol. 2016 Jul 15;17(1):83
pubmed: 27422016
J Am Med Inform Assoc. 2017 Nov 01;24(6):1052-1061
pubmed: 28379439
JAMIA Open. 2020 Apr 11;3(2):252-260
pubmed: 32734166
Pediatrics. 1976 Aug;58(2):259-63
pubmed: 951142
NPJ Digit Med. 2021 Jun 14;4(1):98
pubmed: 34127786
Pediatrics. 2013 Sep;132(3):e756-67
pubmed: 23940245
Kidney Int. 2016 Jul;90(1):212-21
pubmed: 27217196
Can J Kidney Health Dis. 2015 Oct 06;2:36
pubmed: 26445676
P T. 2008 Dec;33(12):700-11
pubmed: 19750051
N Engl J Med. 2021 Jul 15;385(3):283-286
pubmed: 34260843
Kidney Int. 2020 Mar;97(3):580-588
pubmed: 31980139