Estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theory.
Antibody test
Cut point
Extreme value theory
Serology
Threshold
Journal
BMC medical research methodology
ISSN: 1471-2288
Titre abrégé: BMC Med Res Methodol
Pays: England
ID NLM: 100968545
Informations de publication
Date de publication:
08 Feb 2024
08 Feb 2024
Historique:
received:
11
09
2023
accepted:
28
12
2023
medline:
9
2
2024
pubmed:
9
2
2024
entrez:
8
2
2024
Statut:
epublish
Résumé
Rapidly developing tests for emerging diseases is critical for early disease monitoring. In the early stages of an epidemic, when low prevalences are expected, high specificity tests are desired to avoid numerous false positives. Selecting a cutoff to classify positive and negative test results that has the desired operating characteristics, such as specificity, is challenging for new tests because of limited validation data with known disease status. While there is ample statistical literature on estimating quantiles of a distribution, there is limited evidence on estimating extreme quantiles from limited validation data and the resulting test characteristics in the disease testing context. We propose using extreme value theory to select a cutoff with predetermined specificity by fitting a Pareto distribution to the upper tail of the negative controls. We compared this method to five previously proposed cutoff selection methods in a data analysis and simulation study. We analyzed COVID-19 enzyme linked immunosorbent assay antibody test results from long-term care facilities and skilled nursing staff in Colorado between May and December of 2020. We found the extreme value approach had minimal bias when targeting a specificity of 0.995. Using the empirical quantile of the negative controls performed well when targeting a specificity of 0.95. The higher target specificity is preferred for overall test accuracy when prevalence is low, whereas the lower target specificity is preferred when prevalence is higher and resulted in less variable prevalence estimation. While commonly used, the normal based methods showed considerable bias compared to the empirical and extreme value theory-based methods. When determining disease testing cutoffs from small training data samples, we recommend using the extreme value based-methods when targeting a high specificity and the empirical quantile when targeting a lower specificity.
Sections du résumé
BACKGROUND
BACKGROUND
Rapidly developing tests for emerging diseases is critical for early disease monitoring. In the early stages of an epidemic, when low prevalences are expected, high specificity tests are desired to avoid numerous false positives. Selecting a cutoff to classify positive and negative test results that has the desired operating characteristics, such as specificity, is challenging for new tests because of limited validation data with known disease status. While there is ample statistical literature on estimating quantiles of a distribution, there is limited evidence on estimating extreme quantiles from limited validation data and the resulting test characteristics in the disease testing context.
METHODS
METHODS
We propose using extreme value theory to select a cutoff with predetermined specificity by fitting a Pareto distribution to the upper tail of the negative controls. We compared this method to five previously proposed cutoff selection methods in a data analysis and simulation study. We analyzed COVID-19 enzyme linked immunosorbent assay antibody test results from long-term care facilities and skilled nursing staff in Colorado between May and December of 2020.
RESULTS
RESULTS
We found the extreme value approach had minimal bias when targeting a specificity of 0.995. Using the empirical quantile of the negative controls performed well when targeting a specificity of 0.95. The higher target specificity is preferred for overall test accuracy when prevalence is low, whereas the lower target specificity is preferred when prevalence is higher and resulted in less variable prevalence estimation.
DISCUSSION
CONCLUSIONS
While commonly used, the normal based methods showed considerable bias compared to the empirical and extreme value theory-based methods.
CONCLUSIONS
CONCLUSIONS
When determining disease testing cutoffs from small training data samples, we recommend using the extreme value based-methods when targeting a high specificity and the empirical quantile when targeting a lower specificity.
Identifiants
pubmed: 38331732
doi: 10.1186/s12874-023-02139-5
pii: 10.1186/s12874-023-02139-5
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
30Informations de copyright
© 2024. The Author(s).
Références
Levin AT, Owusu-Boaitey N, Pugh S, Fosdick BK, Zwi AB, Malani A, et al. Assessing the burden of COVID-19 in developing countries: Systematic review, meta-analysis and public policy implications. BMJ Glob Health. 2022;7(5):e008477.
doi: 10.1136/bmjgh-2022-008477
pubmed: 35618305
Takahashi S, Greenhouse B, Rodríguez-Barraquer I. Are seroprevalence estimates for severe acute respiratory syndrome coronavirus 2 biased? J Infect Dis. 2020;222(11):1772–5.
doi: 10.1093/infdis/jiaa523
pubmed: 32856712
Klumpp-Thomas C, Kalish H, Drew M, Hunsberger S, Snead K, Fay MP, et al. Standardization of ELISA protocols for serosurveys of the SARS-CoV-2 pandemic using clinical and at-home blood sampling. Nat Commun. 2021;12(1):113.
doi: 10.1038/s41467-020-20383-x
pubmed: 33397956
pmcid: 7782755
Centers for Disease Control and Prevention. Interim Guidelines for COVID-19 Antibody Testing. Published May 23, 2020. Updated August 1, 2020. https://www.cdc.gov/coronavirus/2019-ncov/lab/resources/antibody-tests-guidelines.html . Accessed 7 Jan 2021.
Devanarayan V, Smith WC, Brunelle RL, Seger ME, Krug K, Bowsher RR. Recommendations for systematic statistical computation of immunogenicity cut points. AAPS J. 2017;19(5):1487–98.
doi: 10.1208/s12248-017-0107-3
pubmed: 28733862
Hoffman D, Berger M. Statistical considerations for calculation of immunogenicity screening assay cut points. J Immunol Methods. 2011;373(1–2):200–8.
doi: 10.1016/j.jim.2011.08.019
pubmed: 21906599
Zhang L, Zhang JJ, Kubiak RJ, Yang H. Statistical methods and tool for cut point analysis in immunogenicity assays. J Immunol Methods. 2013;389(1–2):79–87.
doi: 10.1016/j.jim.2012.12.008
pubmed: 23305975
Pickands III J. Statistical inference using extreme order statistics. Ann Stat. 1975;3(1):119–31.
Cooley D, Nychka D, Naveau P. Bayesian spatial modeling of extreme precipitation return levels. J Am Stat Assoc. 2007;102(479):824–40.
doi: 10.1198/016214506000000780
Martín J, Parra MI, Pizarro MM, Sanjuán EL. Baseline methods for the parameter estimation of the generalized Pareto distribution. Entropy. 2022;24(2):178.
doi: 10.3390/e24020178
pubmed: 35205473
pmcid: 8870988
Kiriliouk A, Rootzén H, Segers J, Wadsworth JL. Peaks over thresholds modeling with multivariate generalized Pareto distributions. Technometrics. 2019;61(1):123–35.
doi: 10.1080/00401706.2018.1462738
Bewley KR, Coombes NS, Gagnon L, McInroy L, Baker N, Shaik I, et al. Quantification of SARS-CoV-2 neutralizing antibody by wild-type plaque reduction neutralization, microneutralization and pseudotyped virus neutralization assays. Nat Protoc. 2021;16(6):3114–40.
doi: 10.1038/s41596-021-00536-y
pubmed: 33893470
Cohen B, Doblas D, Andrews N. Comparison of plaque reduction neutralisation test (PRNT) and measles virus-specific IgG ELISA for assessing immunogenicity of measles vaccination. Vaccine. 2008;26(50):6392–7.
doi: 10.1016/j.vaccine.2008.08.074
pubmed: 18834911
Eyal O, Olshevsky U, Lustig S, Paran N, Halevy M, Schneider P, et al. Development of a tissue-culture-based enzyme-immunoassay method for the quantitation of anti-vaccinia-neutralizing antibodies in human sera. J Virol Methods. 2005;130(1–2):15–21.
doi: 10.1016/j.jviromet.2005.05.027
pubmed: 16024096
Gallichotte EN, Nehring M, Young MC, Pugh S, Sexton NR, Fitzmeyer E, et al. Durable antibody responses in staff at two long-term care facilities, during and post SARS-CoV-2 outbreaks. Microbiol Spectr. 2021;9(1):e00224-21.
doi: 10.1128/Spectrum.00224-21
Nehring M, Pugh S, Dihle T, Gallichotte E, Nett T, Weber E, et al. Laboratory-based SARS-CoV-2 receptor-binding domain serologic assays perform with equivalent sensitivity and specificity to commercial FDA-EUA approved tests. Viruses. 2023;15(1):106.
doi: 10.3390/v15010106
Jordan G, Staack RF. An alternative data transformation approach for ADA cut point determination: Why not use a Weibull transformation? AAPS J. 2021;23(5):97.
doi: 10.1208/s12248-021-00625-6
pubmed: 34389881
Mire-Sluis AR, Barrett YC, Devanarayan V, Koren E, Liu H, Maia M, et al. Recommendations for the design and optimization of immunoassays used in the detection of host antibodies against biotechnology products. J Immunol Methods. 2004;289(1–2):1–16.
doi: 10.1016/j.jim.2004.06.002
pubmed: 15251407
Balkema AA, De Haan L. Residual life time at great age. Ann Probab. 1974;2(5):792–804.
doi: 10.1214/aop/1176996548
Rosbjerg D, Madsen H, Rasmussen PF. Prediction in partial duration series with generalized Pareto-distributed exceedances. Water Resour Res. 1992;28(11):3001–10.
doi: 10.1029/92WR01750
DuMouchel WH. Estimating the stable index [Formula: see text] in order to measure tail thickness: A critique. Ann Stat. 1983;11(4):1019–31.
Durán-Rosal AM, Carbonero M, Gutiérrez PA, Hervás-Martínez C. A mixed distribution to fix the threshold for Peak-Over-Threshold wave height estimation. Sci Rep. 2022;12(1):17327.
doi: 10.1038/s41598-022-22243-8
pubmed: 36243880
pmcid: 9569348
Rogan WJ, Gladen B. Estimating prevalence from the results of a screening test. Am J Epidemiol. 1978;107(1):71–6.
doi: 10.1093/oxfordjournals.aje.a112510
pubmed: 623091
Blostein M, Miljkovic T. On modeling left-truncated loss data using mixtures of distributions. Insur Math Econ. 2019;85:35–46.
doi: 10.1016/j.insmatheco.2018.12.001
Felder S, Mayrhofer T. The Optimal Cutoff of a Diagnostic Test. In: Medical Decision Making. Springer; Berlin, Heidelberg: 2022. https://doi.org/10.1007/978-3-662-64654-0_8
Greiner M, Pfeiffer D, Smith RD. Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Prev Vet Med. 2000;45(1–2):23–41.
doi: 10.1016/S0167-5877(00)00115-X
pubmed: 10802332
Hajian-Tilaki K. The choice of methods in determining the optimal cut-off value for quantitative diagnostic test evaluation. Stat Methods Med Res. 2018;27(8):2374–83.
doi: 10.1177/0962280216680383
pubmed: 28673124
Linnet K, Brandt E. Assessing diagnostic tests once an optimal cutoff point has been selected. Clin Chem. 1986;32(7):1341–6.
doi: 10.1093/clinchem/32.7.1341
pubmed: 3719943
Cheng X, Liu Y, Wang J, Chen Y, Robertson AG, Zhang X, et al. cSurvival: A web resource for biomarker interactions in cancer outcomes and in cell lines. Brief Bioinforma. 2022;23(3):bbac090.
Lan L, Cheng X, Xing L, Zhang X. BOSS–Biomarker Optimal Segmentation System. 2023. arXiv preprint arXiv:230509090.
Lausen B, Schumacher M. Maximally selected rank statistics. Biometrics. 1992;48(1):73–85.
Bottomley C, Otiende M, Uyoga S, Gallagher K, Kagucia E, Etyang A, et al. Quantifying previous SARS-CoV-2 infection through mixture modelling of antibody levels. Nat Commun. 2021;12(1):6196.
doi: 10.1038/s41467-021-26452-z
pubmed: 34702829
pmcid: 8548402
Bouman JA, Riou J, Bonhoeffer S, Regoes RR. Estimating the cumulative incidence of SARS-CoV-2 with imperfect serological tests: Exploiting cutoff-free approaches. PLoS Comput Biol. 2021;17(2):e1008728.
doi: 10.1371/journal.pcbi.1008728
pubmed: 33635863
pmcid: 7946301
Hitchings MDT, Patel EU, Khan R, Srikrishnan AK, Anderson M, Kumar KS, et al. A mixture model to estimate SARS-CoV-2 seroprevalence in Chennai. India Am J Epidemiol. 2023;192(9):1552–61.
doi: 10.1093/aje/kwad103
pubmed: 37084085
Schaarschmidt F, Hofmann M, Jaki T, Grün B, Hothorn LA. Statistical approaches for the determination of cut points in anti-drug antibody bioassays. J Immunol Methods. 2015;418:84–100.
doi: 10.1016/j.jim.2015.02.004
pubmed: 25733352
Vink MA, van de Kassteele J, Wallinga J, Teunis PF, Bogaards JA. Estimating seroprevalence of human papillomavirus type 16 using a mixture model with smoothed age-dependent mixing proportions. Epidemiology. 2015;26(1):8–16.
doi: 10.1097/EDE.0000000000000196
pubmed: 25380503
Kostoulas P, Eusebi P, Hartnack S. Diagnostic accuracy estimates for COVID-19 real-time polymerase chain reaction and lateral flow immunoassay tests with Bayesian latent-class models. Am J Epidemiol. 2021;190(8):1689–95.
doi: 10.1093/aje/kwab093
pubmed: 33823529
pmcid: 8083455
Laurin E, Morrison D, Gardner IA, Siah A, Powell JF, Kamaitis M. Bayesian latent class analysis of ELISA and RT-rPCR diagnostic accuracy for subclinical Renibacterium salmoninarum infection in Atlantic salmon (Salmo salar) broodstock. J Fish Dis. 2019;42(2):303–13.
doi: 10.1111/jfd.12933
pubmed: 30549278
Symons R, Beath K, Dangis A, Lefever S, Smismans A, De Bruecker Y, et al. A statistical framework to estimate diagnostic test performance for COVID-19. Clin Radiol. 2021;76(1):75-e1.
doi: 10.1016/j.crad.2020.10.004
Jain R, Chlamtac I. The P2 algorithm for dynamic calculation of quantiles and histograms without storing observations. Commun ACM. 1985;28(10):1076–85.
doi: 10.1145/4372.4378
Karnin Z, Lang K, Liberty E, Optimal quantile approximation in streams. In: 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS). IEEE; 2016. p. 71–8.