Transfer learning with graph neural networks for improved molecular property prediction in the multi-fidelity setting.
Journal
Nature communications
ISSN: 2041-1723
Titre abrégé: Nat Commun
Pays: England
ID NLM: 101528555
Informations de publication
Date de publication:
26 Feb 2024
26 Feb 2024
Historique:
received:
28
06
2023
accepted:
25
01
2024
medline:
27
2
2024
pubmed:
27
2
2024
entrez:
26
2
2024
Statut:
epublish
Résumé
We investigate the potential of graph neural networks for transfer learning and improving molecular property prediction on sparse and expensive to acquire high-fidelity data by leveraging low-fidelity measurements as an inexpensive proxy for a targeted property of interest. This problem arises in discovery processes that rely on screening funnels for trading off the overall costs against throughput and accuracy. Typically, individual stages in these processes are loosely connected and each one generates data at different scale and fidelity. We consider this setup holistically and demonstrate empirically that existing transfer learning techniques for graph neural networks are generally unable to harness the information from multi-fidelity cascades. Here, we propose several effective transfer learning strategies and study them in transductive and inductive settings. Our analysis involves a collection of more than 28 million unique experimental protein-ligand interactions across 37 targets from drug discovery by high-throughput screening and 12 quantum properties from the dataset QMugs. The results indicate that transfer learning can improve the performance on sparse tasks by up to eight times while using an order of magnitude less high-fidelity training data. Moreover, the proposed methods consistently outperform existing transfer learning strategies for graph-structured data on drug discovery and quantum mechanics datasets.
Identifiants
pubmed: 38409255
doi: 10.1038/s41467-024-45566-8
pii: 10.1038/s41467-024-45566-8
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
1517Informations de copyright
© 2024. The Author(s).
Références
Yang, K. et al. Analyzing learned molecular representations for property prediction. J. Chem. Inf. Model. 59, 3370–3388 (2019). PMID: 31361484.
pubmed: 31361484
pmcid: 6727618
doi: 10.1021/acs.jcim.9b00237
Stokes, J. M. et al. A deep learning approach to antibiotic discovery. Cell 180, 688–702.e13 (2020).
pubmed: 32084340
pmcid: 8349178
doi: 10.1016/j.cell.2020.01.021
Heid, E. et al. Chemprop: A machine learning package for chemical property prediction. Journal of Chemical Information and Modeling 64, 9–17 (2023).
Wong, F., Omori, S., Donghia, N. M., Zheng, E. J. & Collins, J. J. Discovering small-molecule senolytics with deep neural networks. Nat. Aging 3, 734–750 (2023).
pubmed: 37142829
doi: 10.1038/s43587-023-00415-z
Merchant, A. et al. Scaling deep learning for materials discovery. Nature 624, 80–85 (2023).
pubmed: 38030720
pmcid: 10700131
doi: 10.1038/s41586-023-06735-9
Buterez, D., Bica, I., Tariq, I., Andrés-Terré, H. & Lió, P. CellVGAE: an unsupervised scRNA-seq analysis workflow with graph attention networks. Bioinformatics 38, 1277–1286 (2021).
pmcid: 8825872
doi: 10.1093/bioinformatics/btab804
Chen, C., Zuo, Y., Ye, W., Li, X. & Ong, S. P. Learning properties of ordered and disordered materials from multi-fidelity data. Nat. Comput. Sci. 1, 46–53 (2021).
pubmed: 38217148
doi: 10.1038/s43588-020-00002-x
Buterez, D., Janet, J. P., Kiddle, S. J., Oglic, D. & Liò, P. Graph neural networks with adaptive readouts. In Advances in Neural Information Processing Systems, vol. 35 (eds Koyejo, S. et al.) 19746–19758 (Curran Associates, Inc., 2022).
Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30 (eds Guyon, I. et al.) (Curran Associates, Inc., 2017).
Avsec, Ž. et al. Effective gene expression prediction from sequence by integrating long-range interactions. Nat. Methods 18, 1196–1203 (2021).
pubmed: 34608324
pmcid: 8490152
doi: 10.1038/s41592-021-01252-x
Vig, J. et al. BERTology meets biology: Interpreting attention in protein language models. In International Conference on Learning Representations (2021).
Schwaller, P. et al. Molecular transformer: a model for uncertainty-calibrated chemical reaction prediction. ACS Cent. Sci. 5, 1572–1583 (2019). PMID: 31572784.
pubmed: 31572784
pmcid: 6764164
doi: 10.1021/acscentsci.9b00576
Buterez, D. Scaling up DNA digital data storage by efficiently predicting DNA hybridisation using deep learning. Sci. Rep. 11, 20517 (2021).
pubmed: 34654863
pmcid: 8519920
doi: 10.1038/s41598-021-97238-y
Smith, J. S. et al. Approaching coupled cluster accuracy with a general-purpose neural network potential through transfer learning. Nat. Commun. 10, 2903 (2019).
pubmed: 31263102
pmcid: 6602931
doi: 10.1038/s41467-019-10827-4
Schütt, K. T. et al. Schnet: a continuous-filter convolutional neural network for modeling quantum interactions. In Proc. 31st International Conference on Neural Information Processing Systems, NIPS’17 992–1002 (Curran Associates Inc., Red Hook, NY, USA, 2017).
Schütt, K. T., Sauceda, H. E., Kindermans, P.-J., Tkatchenko, A. & Müller, K.-R. Schnet – a deep learning architecture for molecules and materials. J. Chem. Phys. 148, 241722 (2018).
pubmed: 29960322
doi: 10.1063/1.5019779
Perola, E. An analysis of the binding efficiencies of drugs and their leads in successful drug discovery programs. J. Med. Chem. 53, 2986–2997 (2010).
pubmed: 20235539
doi: 10.1021/jm100118x
Macarron, R. et al. Impact of high-throughput screening in biomedical research. Nat. Rev. Drug Discov. 10, 188–195 (2011).
pubmed: 21358738
doi: 10.1038/nrd3368
Brown, D. G. & Boström, J. Where do recent small molecule clinical development candidates come from? J. Med. Chem. 61, 9442–9468 (2018). PMID: 29920198.
pubmed: 29920198
doi: 10.1021/acs.jmedchem.8b00675
Wexler, P. Omics and related recent technologies. In Encyclopedia of Toxicology (Academic, 2014).
Hansel, C. S., Plant, D. L., Holdgate, G. A., Collier, M. J. & Plant, H. Advancing automation in high-throughput screening: modular unguarded systems enable adaptable drug discovery. Drug Discov. Today 27, 2051–2056 (2022).
pubmed: 35304338
doi: 10.1016/j.drudis.2022.03.010
Ramakrishnan, R., Dral, P. O., Rupp, M. & von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data 1, 1–7 (2014).
doi: 10.1038/sdata.2014.22
Isert, C., Atz, K., Jiménez-Luna, J. & Schneider, G. Qmugs, quantum mechanical properties of drug-like molecules. Sci. Data 9, 273 (2022).
pubmed: 35672335
pmcid: 9174255
doi: 10.1038/s41597-022-01390-7
Khrabrov, K. et al. nabladft: large-scale conformational energy and hamiltonian prediction benchmark and dataset. Phys. Chem. Chem. Phys. 24, 25853–25863 (2022).
pubmed: 36279016
doi: 10.1039/D2CP03966D
Smith, J. S. et al. The ani-1ccx and ani-1x data sets, coupled-cluster and density functional theory properties for molecules. Sci. Data 7, 134 (2020).
pubmed: 32358545
pmcid: 7195467
doi: 10.1038/s41597-020-0473-z
Chmiela, S., Sauceda, H. E., Müller, K.-R. & Tkatchenko, A. Towards exact molecular dynamics simulations with machine-learned force fields. Nat. Commun. 9, 3887 (2018).
pubmed: 30250077
pmcid: 6155327
doi: 10.1038/s41467-018-06169-2
Qiao, Z., Welborn, M., Anandkumar, A., Manby, F. R. & Miller, T. F. Orbnet: deep learning for quantum chemistry using symmetry-adapted atomic-orbital features. J. Chem. Phys. 153, 124111 (2020).
pubmed: 33003742
doi: 10.1063/5.0021955
Ramakrishnan, R., Dral, P. O., Rupp, M. & von Lilienfeld, O. A. Big data meets quantum chemistry approximations: the δ-machine learning approach. J. Chem. Theory Comput. 11, 2087–2096 (2015).
pubmed: 26574412
doi: 10.1021/acs.jctc.5b00099
Buterez, D., Janet, J. P., Kiddle, S. J. & Lió, P. Mf-pcba: Multifidelity high-throughput screening benchmarks for drug discovery and machine learning. J. Chem. Inf. Model. 63, 2667–2678 (2023). PMID: 37058588.
pubmed: 37058588
pmcid: 10170507
doi: 10.1021/acs.jcim.2c01569
Chen, G. et al. Alchemy: a quantum chemistry dataset for benchmarking AI models. Preprint at CoRRabs/1906.09427. http://arxiv.org/abs/1906.09427 (2019).
Hoja, J. et al. QM7-x, a comprehensive dataset of quantum-mechanical properties spanning the chemical space of small organic molecules. Sci. Data 8, 43 (2021).
pubmed: 33531509
pmcid: 7854709
doi: 10.1038/s41597-021-00812-2
Ramsundar, B. et al. Massively multitask networks for drug discovery. https://arxiv.org/abs/1502.02072 (2015).
Tran-Nguyen, V.-K., Jacquemard, C. & Rognan, D. Lit-pcba: an unbiased data set for machine learning and virtual screening. J. Chem. Inf. Model. 60, 4263–4273 (2020). PMID: 32282202.
pubmed: 32282202
doi: 10.1021/acs.jcim.0c00155
Petrone, P. M. et al. Rethinking molecular similarity: comparing compounds on the basis of biological activity. ACS Chem. Biol. 7, 1399–1409 (2012). PMID: 22594495.
pubmed: 22594495
doi: 10.1021/cb3001028
Helal, K. Y., Maciejewski, M., Gregori-Puigjané, E., Glick, M. & Wassermann, A. M. Public domain hts fingerprints: design and evaluation of compound bioactivity profiles from pubchem’s bioassay repository. J. Chem. Inf. Model. 56, 390–398 (2016). PMID: 26898267.
pubmed: 26898267
doi: 10.1021/acs.jcim.5b00498
Laufkötter, O., Sturm, N., Bajorath, J., Chen, H. & Engkvist, O. Combining structural and bioactivity-based fingerprints improves prediction performance and scaffold hopping capability. J. Cheminf. 11, 54 (2019).
doi: 10.1186/s13321-019-0376-1
Sturm, N. et al. Application of bioactivity profile-based fingerprints for building machine learning models. J.Chem. Inf. Model. 59, 962–972 (2019).
pubmed: 30408959
doi: 10.1021/acs.jcim.8b00550
Yang, C.-H. et al. Multi-fidelity machine learning models for structure–property mapping of organic electronics. Computat. Mater. Sci. 213, 111599 (2022).
doi: 10.1016/j.commatsci.2022.111599
Gómez-Bombarelli, R. et al. Automatic chemical design using a data-driven continuous representation of molecules. ACS Cent. Sci. 4, 268–276 (2018).
pubmed: 29532027
pmcid: 5833007
doi: 10.1021/acscentsci.7b00572
Meng, X. & Karniadakis, G. E. A composite neural network that learns from multi-fidelity data: application to function approximation and inverse pde problems. J. Comput. Phys. 401, 109020 (2020).
doi: 10.1016/j.jcp.2019.109020
Fare, C., Fenner, P., Benatan, M., Varsi, A. & Pyzer-Knapp, E. O. A multi-fidelity machine learning approach to high throughput materials screening. npj Comput. Mater. 8, 257 (2022).
doi: 10.1038/s41524-022-00947-9
Patra, A. et al. A multi-fidelity information-fusion approach to machine learn and predict polymer bandgap. Comput. Mater. Sci. 172, 109286 (2020).
doi: 10.1016/j.commatsci.2019.109286
Li, S., Xing, W., Kirby, R. & Zhe, S. Multi-fidelity Bayesian optimization via deep neural networks. In Advances in Neural Information Processing Systems, Vol. 33, 8521–8531 (Curran Associates, Inc., 2020).
Chen, K., Kunkel, C., Cheng, B., Reuter, K. & Margraf, J. T. Physics-inspired machine learning of localized intensive properties. Chem. Sci. 14, 4913–4922 (2023).
pubmed: 37181767
pmcid: 10171074
doi: 10.1039/D3SC00841J
Schweidtmann, A. M. et al. Physical pooling functions in graph neural networks for molecular property prediction. Comput. Chem. Eng. 172, 108202 (2023).
doi: 10.1016/j.compchemeng.2023.108202
Buterez, D., Janet, J. P., Kiddle, S. J., Oglic, D. & Liò, P. Modelling local and general quantum mechanical properties with attention-based pooling. Commun. Chem. 6, 262 (2023).
pubmed: 38030692
pmcid: 10686994
doi: 10.1038/s42004-023-01045-7
Blum, L. C. & Reymond, J.-L. 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. J. Am. Chem. Soc. 131, 8732 (2009).
pubmed: 19505099
doi: 10.1021/ja902302h
Rupp, M., Tkatchenko, A., Müller, K.-R. & von Lilienfeld, O. A. Fast and accurate modeling of molecular atomization energies with machine learning. Phys. Rev. Lett. 108, 058301 (2012).
pubmed: 22400967
doi: 10.1103/PhysRevLett.108.058301
Montavon, G. et al. Machine learning of molecular electronic properties in chemical compound space. N. J. Phys. 15, 095003 (2013).
doi: 10.1088/1367-2630/15/9/095003
Hu*, W. et al. Strategies for pre-training graph neural networks. In International Conference on Learning Representations (2020).
Sterling, T. & Irwin, J. J. Zinc 15 – ligand discovery for everyone. J. Chem. Inf. Model. 55, 2324–2337 (2015). PMID: 26479676.
pubmed: 26479676
pmcid: 4658288
doi: 10.1021/acs.jcim.5b00559
Fare, C., Turcani, L. & Pyzer-Knapp, E. O. Powerful, transferable representations for molecules through intelligent task selection in deep multitask networks. Phys. Chem. Chem. Phys. 22, 13041–13048 (2020).
pubmed: 32478374
doi: 10.1039/D0CP02319A
Xu, Y., Ma, J., Liaw, A., Sheridan, R. P. & Svetnik, V. Demystifying multitask deep neural networks for quantitative structure–activity relationships. J. Chem. Inf. Model. 57, 2490–2504 (2017). PMID: 28872869.
pubmed: 28872869
doi: 10.1021/acs.jcim.7b00087
Pan, S. J. & Yang, Q. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 1345–1359 (2010).
doi: 10.1109/TKDE.2009.191
Zhuang, F. et al. A comprehensive survey on transfer learning. Proc. IEEE 109, 43–76 (2021).
doi: 10.1109/JPROC.2020.3004555
Xu, K., Hu, W., Leskovec, J. & Jegelka, S. How powerful are graph neural networks? In International Conference on Learning Representations (2019).
Lee, J. et al. Set transformer: a framework for attention-based permutation-invariant neural networks. In Proc. 36th International Conference on Machine Learning 3744–3753 (2019).
Zaheer, M. et al. Deep sets. In Proc. 31st International Conference on Neural Information Processing Systems, NIPS’17 3394–3404 (Curran Associates Inc., Red Hook, NY, USA, 2017).
Kipf, T. N. & Welling, M. Variational graph auto-encoders. In NIPS Workshop on Bayesian Deep Learning (2016).
Weininger, D. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. J. Chem. Inf. Comput. Sci. 28, 31–36 (1988).
doi: 10.1021/ci00057a005
Kusner, M. J., Paige, B. & Hernández-Lobato, J. M. Grammar variational autoencoder. In Proc. 34th International Conference on Machine Learning, ICML’17, Vol. 70, 1945–1954 (JMLR.org, 2017).
Jin, W., Barzilay, R. & Jaakkola, T. Junction tree variational autoencoder for molecular graph generation. In Proc. 35th International Conference on Machine Learning, Proc. Machine Learning Research, Vol. 80 (eds Dy, J. & Krause, A.) 2323–2332 (PMLR, 2018).
Fare, C., Fenner, P. & Pyzer-Knapp, E. O. A principled method for the creation of synthetic multi-fidelity data sets. Preprint at https://arxiv.org/abs/2208.05667 (2022).
Buterez, D., Janet, J. P., Kiddle, S. J., Oglic, D. & Liò, P. Transfer learning with graph neural networks for improved molecular property prediction in the multi-fidelity setting. https://doi.org/10.5281/zenodo.10423965 (2023). Repository name: multi-fidelity-gnns-for-drug-discovery-and-quantum-mechanics.