Understanding the heterogeneous performance of variant effect predictors across human protein-coding genes.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
30 10 2024
30 10 2024
Historique:
received:
12
06
2024
accepted:
11
10
2024
medline:
31
10
2024
pubmed:
31
10
2024
entrez:
31
10
2024
Statut:
epublish
Résumé
Variant effect predictors (VEPs) are computational tools developed to assess the impacts of genetic mutations, often in terms of likely pathogenicity, employing diverse algorithms and training data. Here, we investigate the performance of 35 VEPs in the discrimination between pathogenic and putatively benign missense variants across 963 human protein-coding genes. We observe considerable gene-level heterogeneity as measured by the widely used area under the receiver operating characteristic curve (AUROC) metric. To investigate the origins of this heterogeneity and the extent to which gene-level VEP performance is predictable, for each VEP, we train random forest models to predict the gene-level AUROC. We find that performance as measured by AUROC is related to factors such as gene function, protein structure, and evolutionary conservation. Notably, intrinsic disorder in proteins emerged as a significant factor influencing apparent VEP performance, often leading to inflated AUROC values due to their enrichment in weakly conserved putatively benign variants. Our results suggest that gene-level features may be useful for identifying genes where VEP predictions are likely to be more or less reliable. However, our work also shows that AUROC, despite being independent of class balance, still has crucial limitations when used for comparing VEP performance across different genes.
Identifiants
pubmed: 39478110
doi: 10.1038/s41598-024-76202-6
pii: 10.1038/s41598-024-76202-6
doi:
Substances chimiques
Proteins
0
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
26114Subventions
Organisme : HORIZON EUROPE European Research Council
ID : 101001169
Organisme : Medical Research Council
ID : MC_UU_00035/9
Pays : United Kingdom
Informations de copyright
© 2024. The Author(s).
Références
Mardis, E. R. Next-generation sequencing platforms. Annu. Rev. Anal. Chem.6 https://doi.org/10.1146/annurev-anchem-062012-092628 (2013).
Shendure, J. & Ji, H. Next-generation DNA sequencing. Nat. Biotechnol.26, 1135–1145. https://doi.org/10.1038/nbt1486 (2008).
doi: 10.1038/nbt1486
pubmed: 18846087
Slatko, B. E., Gardner, A. F. & Ausubel, F. M. Overview of next-generation sequencing technologies. Curr. Protoc. Mol. Biol.122 https://doi.org/10.1002/cpmb.59 (2018).
Eilbeck, K., Quinlan, A. & Yandell, M. Settling the score: Variant prioritization and mendelian disease. Nat. Rev. Genet.18, 599–612. https://doi.org/10.1038/nrg.2017.52 (2017).
doi: 10.1038/nrg.2017.52
pubmed: 28804138
Niroula, A. & Vihinen, M. Variation interpretation predictors: Principles, types, performance, and choice. Hum. Mutat.37, 579–597. https://doi.org/10.1002/humu.22987 (2016).
doi: 10.1002/humu.22987
pubmed: 26987456
Katsonis, P., Wilhelm, K., Williams, A. & Lichtarge, O. Genome interpretation using in silico predictors of variant impact. Hum. Genet.141, 1549–1577. https://doi.org/10.1007/s00439-022-02457-6 (2022).
doi: 10.1007/s00439-022-02457-6
pubmed: 35488922
Livesey, B. J. & Marsh, J. A. Interpreting protein variant effects with computational predictors and deep mutational scanning. Dis. Model. Mech.15, dmm049510. https://doi.org/10.1242/dmm.049510 (2022).
doi: 10.1242/dmm.049510
pubmed: 35736673
Liu, Y., Yeung, W. S. B., Chiu, P. C. N. & Cao, D. Computational approaches for predicting variant impact: An overview from resources, principles to applications. Front. Genet.13 https://doi.org/10.3389/fgene.2022.981005 (2022).
Livesey, B. J. et al. Guidelines for Releasing a Variant Effect Predictor (2024). arXiv: https://doi.org/10.48550/arXiv.2404.10807
Ng, P. C. & Henikoff, S. SIFT: Predicting amino acid changes that affect protein function. Nucl. Acids Res.31 https://doi.org/10.1093/nar/gkg509 (2003).
Riesselman, A. J., Ingraham, J. B. & Marks, D. S. Deep generative models of genetic variation capture the effects of mutations. Nat. Methods. 15 https://doi.org/10.1038/s41592-018-0138-4 (2018).
Meier, J. et al. Language models enable zero-shot prediction of the effects of mutations on protein function. Adv. Neural Inf. Process. Syst.34, 29287–29303 (2021).
Livesey, B. J. & Marsh, J. A. Advancing variant effect prediction using protein language models. Nat. Genet.55, 1426–1427. https://doi.org/10.1038/s41588-023-01470-3 (2023).
doi: 10.1038/s41588-023-01470-3
pubmed: 37563330
Grimm, D. G. et al. The evaluation of tools used to predict the impact of missense variants is hindered by two types of circularity. Hum. Mutat.36 https://doi.org/10.1002/humu.22768 (2015).
Pathak, A. K. et al. Pervasive ancestry bias in variant effect predictors. (2024). bioRxiv: 2024.05.20.594987. https://doi.org/10.1101/2024.05.20.594987 .
Livesey, M. Updated benchmarking of variant effect predictors using deep mutational scanning. Mol. Syst. Biol.19, e11474. https://doi.org/10.15252/msb.202211474 (2023).
doi: 10.15252/msb.202211474
pubmed: 37310135
Cheng, J. et al. Accurate proteome-wide missense variant effect prediction with AlphaMissense. Science. 381, eadg7492. https://doi.org/10.1126/science.adg7492 (2023).
doi: 10.1126/science.adg7492
pubmed: 37733863
Marsh, J. A. & Teichmann, S. A. Predicting pathogenic protein variants. Science. 381, 1284–1285. https://doi.org/10.1126/science.adj8672 (2023).
doi: 10.1126/science.adj8672
pubmed: 37725046
Richards, S. et al. Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet. Med.17 https://doi.org/10.1038/gim.2015.30 (2015).
Gunning, A. C. et al. Assessing performance of pathogenicity predictors using clinically relevant variant datasets. J. Med. Genet.58 https://doi.org/10.1136/jmedgenet-2020-107003 (2021).
Pejaver, V. et al. Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations for PP3/BP4 criteria. Am. J. Hum. Genet.109, 2163–2177. https://doi.org/10.1016/j.ajhg.2022.10.013 (2022).
doi: 10.1016/j.ajhg.2022.10.013
pubmed: 36413997
Miosge, L. A. et al. Comparison of predicted and actual consequences of missense mutations. Proc. Natl. Acad. Sci. U S A. 112 https://doi.org/10.1073/pnas.1511585112 (2015).
Katsanis, N. The continuum of causality in human genetic disorders. Genome Biol.17 https://doi.org/10.1186/s13059-016-1107-9 (2016).
Wright, C. F. et al. Assessing the pathogenicity, penetrance, and expressivity of putative disease-causing variants in a population setting. Am. J. Hum. Genet.104 https://doi.org/10.1016/j.ajhg.2018.12.015 (2019).
Livesey, B. J. & Marsh, J. A. Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations. Mol. Syst. Biol.16 https://doi.org/10.15252/msb.20199380 (2020).
Notin, P. et al. ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design. In Advances in Neural Information Processing Systems. 36. Available: (2023). https://proceedings.neurips.cc/paper_files/paper/2023/hash/cac723e5ff29f65e3fcbb0739ae91bee-Abstract-Datasets_and_Benchmarks.html
Livesey, B. J. & Marsh, J. A. Variant effect predictor correlation with functional assays is reflective of clinical classification performance. (2024). bioRxiv: https://doi.org/10.1101/2024.05.12.593741
Gerasimavicius, L., Livesey, B. J. & Marsh, J. A. Loss-of-function, gain-of-function and dominant-negative mutations have profoundly different effects on protein structure. Nat. Commun.13 https://doi.org/10.1038/s41467-022-31686-6 (2022).
Mahmood, K. et al. Variant effect prediction tools assessed using independent, functional assay-based datasets: implications for discovery and diagnostics. Hum. Genomics. 11 https://doi.org/10.1186/s40246-017-0104-8 (2017).
Miller, M., Wang, Y. & Bromberg, Y. What went wrong with variant effect predictor performance for the PCM1 challenge. Hum. Mutat.40, 1486–1494. https://doi.org/10.1002/humu.23832 (2019).
doi: 10.1002/humu.23832
pubmed: 31268618
Castellana, S. & Mazza, T. Congruency in the prediction of pathogenic missense mutations: State-of-the-art web-based tools. Brief. Bioinform. 14 https://doi.org/10.1093/bib/bbt013 (2013).
Qorri, E. et al. A comprehensive evaluation of the performance of Prediction algorithms on clinically relevant missense variants. Int. J. Mol. Sci.23 https://doi.org/10.3390/ijms23147946 (2022).
Landrum, M. J. et al. ClinVar: public archive of relationships among sequence variation and human phenotype. Nucl. Acids Res.42 https://doi.org/10.1093/nar/gkt1113 (2014).
Karczewski, K. J. et al. The mutational constraint spectrum quantified from variation in 141,456 humans. Nature. 581 https://doi.org/10.1038/s41586-020-2308-7 (2020).
Attard, T. J., Welburn, J. P. I. & Marsh, J. A. Understanding molecular mechanisms and predicting phenotypic effects of pathogenic tubulin mutations. PLoS Comput. Biol.18 https://doi.org/10.1371/journal.pcbi.1010611 (2022).
Ioannidis, N. M. et al. REVEL: An ensemble method for predicting the pathogenicity of rare missense variants. Am. J. Hum. Genet.99, 877–885. https://doi.org/10.1016/j.ajhg.2016.08.016 (2016).
doi: 10.1016/j.ajhg.2016.08.016
pubmed: 27666373
Wu, Y., Li, R., Sun, S., Weile, J. & Roth, F. P. Improved pathogenicity prediction for rare human missense variants. Am. J. Hum. Genet.108 https://doi.org/10.1016/j.ajhg.2021.08.012 (2021).
Liu, X., Li, C., Mou, C., Dong, Y. & Tu, Y. dbNSFP v4: A comprehensive database of transcript-specific functional predictions and annotations for human nonsynonymous and splice-site SNVs. Genome Med.12, 103. https://doi.org/10.1186/s13073-020-00803-9 (2020).
doi: 10.1186/s13073-020-00803-9
pubmed: 33261662
Brandes, N., Goldman, G., Wang, C. H., Ye, C. J. & Ntranos, V. Genome-wide prediction of disease variant effects with a deep protein language model. Nat. Genet.55, 1512–1522. https://doi.org/10.1038/s41588-023-01465-0 (2023).
doi: 10.1038/s41588-023-01465-0
pubmed: 37563329
Greiner, M., Pfeiffer, D. & Smith, R. D. Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Prev. Vet. Med.45, 23–41. https://doi.org/10.1016/S0167-5877(00)00115-X (2000).
doi: 10.1016/S0167-5877(00)00115-X
pubmed: 10802332
Richardson, E. et al. The receiver operating characteristic curve accurately assesses imbalanced datasets. Patterns. 5, 100994. https://doi.org/10.1016/j.patter.2024.100994 (2024).
doi: 10.1016/j.patter.2024.100994
pubmed: 39005487
Breiman, L. Random forests. Mach. Learn.45 https://doi.org/10.1023/A:1010933404324 (2001).
Lundberg, S. & Lee, S-I. A Unified Approach to Interpreting Model Predictions. Available: (2017). arXiv: http://arxiv.org/abs/1705.07874
Lundberg, S. M. et al. From local explanations to global understanding with explainable AI for trees. Nat. Mach. Intell.2, 56–67. https://doi.org/10.1038/s42256-019-0138-9 (2020).
doi: 10.1038/s42256-019-0138-9
pubmed: 32607472
pmcid: 7326367
Janzing, D., Minorics, L. & Blöbaum, P. Feature relevance quantification in explainable AI: A causal problem. Available: (2019). arXiv: http://arxiv.org/abs/1910.13413
Frazer, J. et al. Disease variant prediction with deep generative models of evolutionary data. Nature. 599, 91–95. https://doi.org/10.1038/s41586-021-04043-8 (2021).
doi: 10.1038/s41586-021-04043-8
pubmed: 34707284
Schymkowitz, J. et al. The FoldX web server: an online force field. Nucl. Acids Res.33 https://doi.org/10.1093/nar/gki387 (2005).
Badonyi, M. & Marsh, J. A. Proteome-scale prediction of molecular mechanisms underlying dominant genetic diseases. PLOS ONE 19, e0307312. https://doi.org/10.1371/journal.pone.0307312 (2024).
Zeng, T., Spence, J. P., Mostafavi, H. & Pritchard, J. K. Bayesian estimation of gene constraint from an evolutionary model with gene features. Nat. Genet. 56, 163201643 https://doi.org/10.1038/s41588-024-01820-9 (2024).
Jumper, J. et al. Highly accurate protein structure prediction with AlphaFold. Nature. 596 https://doi.org/10.1038/s41586-021-03819-2 (2021).
Mariani, V., Biasini, M., Barbato, A. & Schwede, T. lDDT: A local superposition-free score for comparing protein structures and models using distance difference tests. Bioinformatics. 29, 2722–2728. https://doi.org/10.1093/bioinformatics/btt473 (2013).
doi: 10.1093/bioinformatics/btt473
pubmed: 23986568
Guo, H-B. et al. AlphaFold2 models indicate that protein sequence determines both structure and dynamics. Sci. Rep.12, 10696. https://doi.org/10.1038/s41598-022-14382-9 (2022).
doi: 10.1038/s41598-022-14382-9
pubmed: 35739160
Alderson, T. R., Pritišanac, I., Kolarić, Đ., Moses, A. M. & Forman-Kay, J. D. Systematic identification of conditionally folded intrinsically disordered regions by AlphaFold2. Proc. Natl. Acad. Sci.120, e2304302120. https://doi.org/10.1073/pnas.2304302120 (2023).
doi: 10.1073/pnas.2304302120
pubmed: 37878721
Van Der Lee, R. et al. Classification of intrinsically disordered regions and proteins. Chem. Rev.114, 6589–6631. https://doi.org/10.1021/cr400525m (2014).
doi: 10.1021/cr400525m
pubmed: 24773235
Williamson, K. A. et al. Recurrent heterozygous PAX6 missense variants cause severe bilateral microphthalmia via predictable effects on DNA–protein interaction. Genet. Med.22, 598–609. https://doi.org/10.1038/s41436-019-0685-9 (2020).
doi: 10.1038/s41436-019-0685-9
pubmed: 31700164
Breuza, L. et al. The UniProtKB guide to the human proteome. Database. (2016). https://doi.org/10.1093/database/bav120
Varadi, M. et al. AlphaFold protein structure database: Massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucl. Acids Res.50, D439–D444. https://doi.org/10.1093/nar/gkab1061 (2022).
doi: 10.1093/nar/gkab1061
pubmed: 34791371
Mi, H., Muruganujan, A. & Thomas, P. D. PANTHER in 2013: Modeling the evolution of gene function, and other gene attributes, in the context of phylogenetic trees. Nucl. Acids Res.41, D377–D386. https://doi.org/10.1093/nar/gks1118 (2012).
doi: 10.1093/nar/gks1118
pubmed: 23193289
Thomas, P. D. et al. PANTHER: making genome-scale phylogenetics accessible to all. Protein Sci.31, 8–22. https://doi.org/10.1002/pro.4218 (2022).
doi: 10.1002/pro.4218
pubmed: 34717010