Multimodal deep learning using on-chip diffractive optics with in situ training capability.
Journal
Nature communications
ISSN: 2041-1723
Titre abrégé: Nat Commun
Pays: England
ID NLM: 101528555
Informations de publication
Date de publication:
23 Jul 2024
23 Jul 2024
Historique:
received:
31
01
2024
accepted:
18
07
2024
medline:
24
7
2024
pubmed:
24
7
2024
entrez:
23
7
2024
Statut:
epublish
Résumé
Multimodal deep learning plays a pivotal role in supporting the processing and learning of diverse data types within the realm of artificial intelligence generated content (AIGC). However, most photonic neuromorphic processors for deep learning can only handle a single data modality (either vision or audio) due to the lack of abundant parameter training in optical domain. Here, we propose and demonstrate a trainable diffractive optical neural network (TDONN) chip based on on-chip diffractive optics with massive tunable elements to address these constraints. The TDONN chip includes one input layer, five hidden layers, and one output layer, and only one forward propagation is required to obtain the inference results without frequent optical-electrical conversion. The customized stochastic gradient descent algorithm and the drop-out mechanism are developed for photonic neurons to realize in situ training and fast convergence in the optical domain. The TDONN chip achieves a potential throughput of 217.6 tera-operations per second (TOPS) with high computing density (447.7 TOPS/mm
Identifiants
pubmed: 39043669
doi: 10.1038/s41467-024-50677-3
pii: 10.1038/s41467-024-50677-3
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
6189Subventions
Organisme : National Natural Science Foundation of China (National Science Foundation of China)
ID : U21A20511
Informations de copyright
© 2024. The Author(s).
Références
Huang, H. et al. ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model. Int. J. Oral Sci. 15, 29 (2023).
pubmed: 37507396
pmcid: 10382494
doi: 10.1038/s41368-023-00239-y
Mesko, B. The impact of multimodal large language models on health care’s future. J. Med. Internet Res. 25, e52865 (2023).
pubmed: 37917126
pmcid: 10654899
doi: 10.2196/52865
Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259–265 (2023).
pubmed: 37045921
doi: 10.1038/s41586-023-05881-4
Wang, X. et al. Large-scale multi-modal pre-trained models: a comprehensive survey. Mach. Intell. Res. 20, 447–482 (2023).
doi: 10.1007/s11633-022-1410-8
Waldrop, M. M. The chips are down for Moore’s law. Nature 530, 144–147 (2016).
pubmed: 26863965
doi: 10.1038/530144a
Huang, C. et al. Prospects and applications of photonic neural networks. Adv. Phys. X 7, 1981155 (2022).
Wu, J. et al. Analog optical computing for artificial intelligence. Engineering 10, 133–145 (2022).
doi: 10.1016/j.eng.2021.06.021
Xu, P. & Zhou, Z. Silicon-based optoelectronics for general-purpose matrix computation: a review. Adv. Photonics 4, 044001 (2022).
doi: 10.1117/1.AP.4.4.044001
Zhou, H. L. et al. Photonic matrix multiplication lights up photonic accelerator and beyond. Light Sci. Appl. 11, 1–21 (2022).
doi: 10.1038/s41377-022-00717-8
Bai, Y. et al. Photonic multiplexing techniques for neuromorphic computing. Nanophotonics 12, 795–817 (2023).
doi: 10.1515/nanoph-2022-0485
Xu, X. et al. Neuromorphic computing based on wavelength-division multiplexing. IEEE J. Sel. Top. Quantum Electron 29, 1–12 (2023).
doi: 10.1109/JSTQE.2022.3203159
Clements, W. R., Humphreys, P. C., Metcalf, B. J., Kolthammer, W. S. & Walmsley, I. A. Optimal design for universal multiport interferometers. Optica 3, 1460–1465 (2016).
doi: 10.1364/OPTICA.3.001460
Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photonics 11, 441–446 (2017).
doi: 10.1038/nphoton.2017.93
Harris, N. C. et al. Linear programmable nanophotonic processors. Optica 5, 1623–1631 (2018).
doi: 10.1364/OPTICA.5.001623
Zhou, H. et al. Self-configuring and reconfigurable silicon photonic signal processor. ACS Photonics 7, 792–799 (2020).
doi: 10.1021/acsphotonics.9b01673
Zhou, H. et al. Chip-scale optical matrix computation for PageRank algorithm. IEEE J. Sel. Top. Quantum Electron. 26, 1–10 (2020).
Tian Y., et al. Scalable and compact photonic neural chip with low learning-capability-loss. Nanophotonics 11, 329–344 (2022).
Zhang, H. et al. An optical neural chip for implementing complex-valued neural network. Nat. Commun. 12, 457 (2021).
pubmed: 33469031
pmcid: 7815828
doi: 10.1038/s41467-020-20719-7
Mourgias-Alexandris, G. et al. Noise-resilient and high-speed deep learning with coherent silicon photonics. Nat. Commun. 13, 5572 (2022).
pubmed: 36151214
pmcid: 9508134
doi: 10.1038/s41467-022-33259-z
Yang, L., Ji, R., Zhang, L., Ding, J. & Xu, Q. On-chip CMOS-compatible optical signal processor. Opt. Express 20, 13560–13565 (2012).
pubmed: 22714383
doi: 10.1364/OE.20.013560
Tait, A. N. et al. Neuromorphic photonic networks using silicon photonic weight banks. Sci. Rep. 7, 7430 (2017).
pubmed: 28784997
pmcid: 5547135
doi: 10.1038/s41598-017-07754-z
Tait, A. N. et al. Feedback control for microring weight banks. Opt. Express 26, 26422–26443 (2018).
pubmed: 30469730
doi: 10.1364/OE.26.026422
Huang, C. R. et al. A silicon photonic-electronic neural network for fibre nonlinearity compensation. Nat. Electron. 4, 837–844 (2021).
doi: 10.1038/s41928-021-00661-2
Jiang, Y., Zhang, W. J., Yang, F. & He, Z. Y. Photonic convolution neural network based on interleaved time-wavelength modulation. J. Lightwave Technol. 39, 4592–4600 (2021).
doi: 10.1109/JLT.2021.3076070
Cheng, J. et al. A small microring array that performs large complex-valued matrix-vector multiplication. Front. Optoelectron 15, 15 (2022).
pubmed: 36637556
pmcid: 9756268
doi: 10.1007/s12200-022-00009-4
Filipovich, M. J. et al. Silicon photonic architecture for training deep neural networks with direct feedback alignment. Optica 9, 1323–1332 (2022).
doi: 10.1364/OPTICA.475493
Ohno S., Tang R., Toprasertpong K., Takagi S., Takenaka M. Si microring resonator crossbar array for on-chip inference and training of the optical neural network. ACS Photonics 9, 2614–2622 (2022).
Xu, S., Wang, J., Yi, S. & Zou, W. High-order tensor flow processing using integrated photonic circuits. Nat. Commun. 13, 7970 (2022).
pubmed: 36577748
pmcid: 9797566
doi: 10.1038/s41467-022-35723-2
Zhang, W. et al. Silicon microring synapses enable photonic deep learning beyond 9-bit precision. Optica 9, 579–584 (2022).
doi: 10.1364/OPTICA.446100
Cheng, J. et al. Self-calibrating microring synapse with dual-wavelength synchronization. Photonics Res. 11, 347–356 (2023).
doi: 10.1364/PRJ.478370
Wang, B., Lima, T. F. D., Shastri, B. J., Prucnal, P. R. & Huang, C. Multi-wavelength photonic neuromorphic computing for intra and inter-channel distortion compensations in WDM optical communication systems. IEEE J. Sel. Top. Quantum Electron. 29, 1–12 (2023).
Zhang, W. et al. Broadband physical layer cognitive radio with an integrated photonic processor for blind source separation. Nat. Commun. 14, 1107 (2023).
pubmed: 36849533
pmcid: 9971366
doi: 10.1038/s41467-023-36814-4
Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).
pubmed: 31068721
pmcid: 6522354
doi: 10.1038/s41586-019-1157-8
Feldmann, J. et al. Parallel convolutional processing using an integrated photonic tensor core. Nature 589, 52–58 (2021).
pubmed: 33408373
doi: 10.1038/s41586-020-03070-1
Dong, B. et al. Higher-dimensional processing using a photonic tensor core with continuous-time data. Nat. Photonics 17, 1080–1088 (2023).
doi: 10.1038/s41566-023-01313-x
Wei, M. et al. Electrically programmable phase-change photonic memory for optical neural networks with nanoseconds in situ training capability. Adv. Photonics 5, 046004 (2023).
doi: 10.1117/1.AP.5.4.046004
Xu, X. et al. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature 589, 44–51 (2021).
pubmed: 33408378
doi: 10.1038/s41586-020-03063-0
Bai, B. et al. Microcomb-based integrated photonic processing unit. Nat. Commun. 14, 66 (2023).
pubmed: 36604409
pmcid: 9814295
doi: 10.1038/s41467-022-35506-9
McMahon P. L. The physics of optical computing. Nat. Rev. Phys. 5, 717–734 (2023).
Cheng, J. et al. Human emotion recognition with a microcomb-enabled integrated optical neural network. Nanophotonics 12, 3883–3894 (2023).
doi: 10.1515/nanoph-2023-0298
LeCun Y., Cortes C. & Burges C. J. C. The MNIST dataset of handwritten digits. http://yann.lecun.com/exdb/mnist/ (1998).
Xiao H., Rasul K., Vollgraf R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Preprint at https://ui.adsabs.harvard.edu/abs/2017arXiv170807747X (2017).
Fu, T. et al. On-chip photonic diffractive optical neural network based on a spatial domain electromagnetic propagation model. Opt. Express 29, 31924–31940 (2021).
pubmed: 34615274
doi: 10.1364/OE.435183
Fu, T. et al. Photonic machine learning with on-chip diffractive optics. Nat. Commun. 14, 70 (2023).
pubmed: 36604423
pmcid: 9814266
doi: 10.1038/s41467-022-35772-7
Huang, Y., Fu, T., Huang, H., Yang, S. & Chen, H. Sophisticated deep learning with on-chip optical diffractive tensor processing. Photonics Res. 11, 1125–1138 (2023).
doi: 10.1364/PRJ.484662
Mandel, L. & Wolf, E. Some properties of coherent light. J. Opt. Soc. Am. 51, 815–819 (1961).
doi: 10.1364/JOSA.51.000815
Pannous. The Spoken_numbers_pcm dataset of spoken numbers. https://github.com/pannous/tensorflow-speech-recognition (2018).
Williamson, I. A. D. et al. Reprogrammable electro-optic nonlinear activation functions for optical neural networks. IEEE J. Sel. Top. Quantum Electron. 26, 12 (2020).
doi: 10.1109/JSTQE.2019.2930455
Wu, B., Li, H., Tong, W., Dong, J. & Zhang, X. Low-threshold all-optical nonlinear activation function based on a Ge/Si hybrid structure in a microring resonator. Opt. Mater. Express 12, 970–980 (2022).
doi: 10.1364/OME.447330
Li, H., Wu, B., Tong, W., Dong, J. & Zhang, X. All-optical nonlinear activation function based on germanium silicon hybrid asymmetric coupler. IEEE J. Sel. Top. Quantum Electron. 29, 1–6 (2023).
Ríos, C. et al. Ultra-compact nonvolatile phase shifter based on electrically reprogrammable transparent phase change materials. PhotoniX 3, 26 (2022).
doi: 10.1186/s43074-022-00070-4
Tan, J. Y. S. et al. Monadic Pavlovian associative learning in a backpropagation-free photonic network. Optica 9, 792–802 (2022).
doi: 10.1364/OPTICA.455864
Chen, X. et al. Neuromorphic photonic memory devices using ultrafast, non-volatile phase-change materials. Adv. Mater. 35, 2203909 (2023).
doi: 10.1002/adma.202203909
Sun, C. et al. Single-chip microprocessor that communicates directly using light. Nature 528, 534–538 (2015).
pubmed: 26701054
doi: 10.1038/nature16454
Atabaki, A. H. et al. Integrating photonics with silicon nanoelectronics for the next generation of systems on a chip. Nature 556, 349–354 (2018).
pubmed: 29670262
doi: 10.1038/s41586-018-0028-z
Tan, M., Wang, Y., Wang, K. X., Yu, Y. & Zhang, X. Circuit-level convergence of electronics and photonics: basic concepts and recent advances. Front. Optoelectron 15, 16 (2022).
pubmed: 36637580
pmcid: 9756227
doi: 10.1007/s12200-022-00013-8
Tan, M. et al. Co-packaged optics (CPO): status, challenges, and solutions. Front. Optoelectron. 16, 1 (2023).
pubmed: 36939942
pmcid: 10027985
doi: 10.1007/s12200-022-00055-y
Zhang, J., Wu, B., Cheng, J., Dong, J. & Zhang, X. Compact, efficient, and scalable nanobeam core for photonic matrix-vector multiplication. Optica 11, 190–196 (2024).
doi: 10.1364/OPTICA.506603