Inference in artificial intelligence with deep optics and photonics.
Journal
Nature
ISSN: 1476-4687
Titre abrégé: Nature
Pays: England
ID NLM: 0410462
Informations de publication
Date de publication:
12 2020
12 2020
Historique:
received:
28
11
2019
accepted:
20
08
2020
entrez:
3
12
2020
pubmed:
4
12
2020
medline:
4
12
2020
Statut:
ppublish
Résumé
Artificial intelligence tasks across numerous applications require accelerators for fast and low-power execution. Optical computing systems may be able to meet these domain-specific needs but, despite half a century of research, general-purpose optical computing systems have yet to mature into a practical technology. Artificial intelligence inference, however, especially for visual computing applications, may offer opportunities for inference based on optical and photonic systems. In this Perspective, we review recent work on optical computing for artificial intelligence applications and discuss its promise and challenges.
Identifiants
pubmed: 33268862
doi: 10.1038/s41586-020-2973-6
pii: 10.1038/s41586-020-2973-6
doi:
Types de publication
Journal Article
Review
Research Support, U.S. Gov't, Non-P.H.S.
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
39-47Subventions
Organisme : European Research Council
Pays : International
Références
LeCun, Y. et al. Handwritten digit recognition with a back-propagation network. In Advances in Neural Information Processing Systems 2 (NIPS 1989) (ed. Touretzky, D. S.) 396–404 (1990).
Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS 2012) (eds Pereira, F. et al.) 1097–1105 (2012).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
pubmed: 26017442
Miller, D. A. B. Waves, modes, communications, and optics: a tutorial. Adv. Opt. Photonics 11, 679–825 (2019).
Brunner, D., Soriano, M. C., Mirasso, C. R. & Fischer, I. Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun. 4, 1364 (2013).
pubmed: 23322052
pmcid: 3562454
Goodman, J. W., Leonberger, F. J., Kung, S.-Y. & Athale, R. A. Optical interconnections for VLSI systems. Proc. IEEE 72, 850–866 (1984). The first paper to provide a substantial analysis and reasons for the use of optics in interconnection (rather than for logic) in digital systems.
Miller, D. A. B. Rationale and challenges for optical interconnects to electronic chips. Proc. IEEE 88, 728–749 (2000).
Miller, D. A. B. Attojoule optoelectronics for low-energy information processing and communications. J. Lightwave Technol. 35, 346–396 (2017).
Miller, D. A. B. Are optical transistors the logical next step? Nat. Photon. 4, 3–5 (2010).
Athale, R. & Psaltis, D. Optical computing: past and future. Opt. Photon. News 27, 32–39 (2016).
Goodman, J. W. Introduction to Fourier Optics (Roberts and Co, 2005).
Liutkus, A. et al. Imaging with nature: compressive imaging using a multiply scattering medium. Sci. Rep. 4, 5552 (2014).
pubmed: 25005695
pmcid: 4087920
Saade, A. et al. Random projections through multiple optical scattering: approximating kernels at the speed of light. In 2016 IEEE Intl Conf. Acoustics, Speech and Signal Processing (ICASSP) 6215–6219 (IEEE, 2016).
Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018). An optical implementation using multiple optimized layers for all-optical image classification.
pubmed: 30049787
Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical–electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 12324 (2018). An optical implementation of a single CNN layer demonstrated for hybrid optical–electronic image classification.
pubmed: 30120316
pmcid: 6098044
Rosenblatt, F. The Perceptron, A Perceiving and Recognizing Automaton Report no. 85-460-1 (Project Para, Cornell Aeronautical Laboratory, 1957).
Hebb, D. O. The Organization of Behavior (Wiley, 1949).
Widrow, B. & Hoff, M. E. Adaptive switching circuits. In 1960 IRE WESCON Convention Record 96–104 (Institute of Radio Engineers, 1960).
Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl Acad. Sci. USA 79, 2554–2558 (1982).
pubmed: 6953413
Carpenter, G. A. & Grossberg, S. A massively parallel architecture for a self-organizing neural pattern recognition machine. Comput. Vis. Graph. Image Process. 37, 54–115 (1987).
Kohonen, T. Self-organized formation of topologically correct feature maps. Biol. Cybern. 43, 59–69 (1982).
Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).
Mead, C. Neuromorphic electronic systems. Proc. IEEE 78, 1629–1636 (1990).
Farhat, N. H., Psaltis, D., Prata, A. & Paek, E. Optical implementation of the Hopfield model. Appl. Opt. 24, 1469–1475 (1985). Optical implementation of content-addressable associative memory based on the Hopfield model for neural networks and on the addition of nonlinear iterative feedback to a vector–matrix multiplier.
pubmed: 18223740
Denz, C. Optical Neural Networks (Springer Science & Business Media, 2013).
Psaltis, D., Brady, D., Gu, X.-G. & Lin, S. Holography in artificial neural networks. Nature 343, 325–330 (1990). Introduction of nonlinear photorefractive crystals for optical computing.
pubmed: 2300184
Li, H.-Y. S., Qiao, Y. & Psaltis, D. Optical network for real-time face recognition. Appl. Opt. 32, 5026–5035 (1993).
pubmed: 20856307
Miller, D. A. B. Self-configuring universal linear optical component. Photon. Res. 1, 1–15 (2013). Proof that arbitrary linear operations such as singular value decompositions can be performed in optics—not just Fourier transforms and convolutions as in early optical computing.
Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441 (2017). A silicon photonic neural network using meshes of MZIs for vowel recognition.
Fang, M. Y.-S., Manipatruni, S., Wierzynski, C., Khosrowshahi, A. & DeWeese, M. R. Design of optical neural networks with component imprecisions. Opt. Express 27, 14009–14029 (2019).
pubmed: 31163856
Wilkes, C. M. et al. 60 dB high-extinction auto-configured Mach–Zehnder interferometer. Opt. Lett. 41, 5318–5321 (2016).
pubmed: 27842122
Hughes, T. W., Minkov, M., Shi, Y. & Fan, S. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018).
Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019). A photonic circuit that exploits wavelength division multiplexing techniques for pattern recognition directly in the optical domain.
pubmed: 31068721
pmcid: 6522354
Tait, A. N. et al. Neuromorphic photonic networks using silicon photonic weight banks. Sci. Rep. 7, 7430 (2017).
pubmed: 28784997
pmcid: 5547135
Huang, C. et al. Giant enhancement in signal contrast using integrated all-optical nonlinear thresholder. In 2019 Optical Fiber Communications Conference and Exhibition (OFC) 415–417 (IEEE, 2019).
Nahmias, M. A., Shastri, B. J., Tait, A. N. & Prucnal, P. R. A leaky integrate-and-fire laser neuron for ultrafast cognitive computing. IEEE J. Sel. Top. Quantum Electron. 19, 1800212 (2013).
Amin, R. et al. ITO-based electro-absorption modulator for photonic neural activation function. APL Mater. 7, 081112 (2019).
Williamson, I. A. D. et al. Reprogrammable electro-optic nonlinear activation functions for optical neural networks. IEEE J. Sel. Top. Quantum Electron. 26, 7700412 (2020).
Miller, D. A. B. Novel analog self-electrooptic-effect devices. IEEE J. Quantum Electron. 29, 678–698 (1993).
Srinivasan, S. A. et al. High absorption contrast quantum confined stark effect in ultra-thin Ge/SiGe quantum well stacks grown on Si. IEEE J. Quantum Electron. 56, 5200207 (2020).
Ferreira de Lima, T., Shastri, B. J., Tait, A. N., Nahmias, M. A. & Prucnal, P. R. Progress in neuromorphic photonics. Nanophotonics 6, 577–599 (2017).
Nahmias, M. A. et al. Photonic multiply–accumulate operations for neural networks. IEEE J. Sel. Top. Quantum Electron. 26, 7701518 (2020). A review article on the state-of-the-art of photonic MACs along with detailed characterizations and comparisons of the performance of photonic and comparable electronic hardware.
Gupta, S., Agrawal, A., Gopalakrishnan, K. & Narayanan, P. Deep learning with limited numerical precision. In Proc. 32nd Intl Conf. Machine Learning (eds Bach, F. & Blei, D.) 1737–1746 (PMLR, 2015).
Hamerly, R., Bernstein, L., Sludds, A., Soljačić, M. & Englund, D. Large-scale optical neural networks based on photoelectric multiplication. Phys. Rev. X 9, 021032 (2019).
Lugt, A. V. Signal detection by complex spatial filtering. IEEE Trans. Inf. Theory 10, 139–145 (1964). The introduction of optical correlators.
Gregory, D. A. Real-time pattern recognition using a modified liquid crystal television in a coherent optical correlator. Appl. Opt. 25, 467–469 (1986).
pubmed: 20445697
Manzur, T., Zeller, J. & Serati, S. Optical-correlator-based target detection, recognition, classification, and tracking. Appl. Opt. 51, 4976–4983 (2012).
pubmed: 22858935
Javidi, B., Li, J. & Tang, Q. Optical implementation of neural networks for face recognition by the use of nonlinear joint transform correlators. Appl. Opt. 34, 3950–3962 (1995).
pubmed: 21052218
Koppal, S. J., Gkioulekas, I., Zickler, T. & Barrows, G. L. Wide-angle micro sensors for vision on a tight budget. In 2011 IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2011) 361–368 (IEEE, 2011).
Hughes, T. W., Williamson, I. A. D., Minkov, M. & Fan, S. Wave physics as an analog recurrent neural network. Sci. Adv. 5, eaay6946 (2019).
pubmed: 31903420
pmcid: 6924985
Duarte, M. F. et al. Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25, 83–91 (2008).
Moretti, C. & Gigan, S. Readout of fluorescence functional signals through highly scattering tissue. Nat. Photonics 14, 361–364 (2020).
Rahmani, B., Loterie, D., Konstantinou, G., Psaltis, D. & Moser, C. Multimode optical fiber transmission with a deep learning network. Light Sci. Appl. 7, 69 (2018).
pubmed: 30302240
pmcid: 6168552
Caramazza, P., Moran, O., Murray-Smith, R. & Faccio, D. Transmission of natural scene images through a multimode fibre. Nat. Commun. 10, 2029 (2019).
pubmed: 31048712
pmcid: 6497636
Li, Y., Xue, Y. & Tian, L. Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media. Optica 5, 1181–1190 (2018).
Horisaki, R., Takagi, R. & Tanida, J. Learning-based imaging through scattering media. Opt. Express 24, 13738–13743 (2016).
pubmed: 27410537
Ando, T., Horisaki, R. & Tanida, J. Speckle-learning-based object recognition through scattering media. Opt. Express 23, 33902–33910 (2015).
pubmed: 26832049
Mahoney, M. W. Randomized Algorithms for Matrices and Data (Now Publishers, 2011).
Dong, J., Rafayelyan, M., Krzakala, F. & Gigan, S. Optical reservoir computing using multiple light scattering for chaotic systems prediction. IEEE J. Sel. Top. Quantum Electron. 26, 7701012 (2019).
Gupta, S., Gribonval, R., Daudet, L. & Dokmanić, I. Don’t take it lightly: phasing optical random projections with unknown operators. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (eds Wallach, H. et al.) 14855–14865 (2019).
Marshall, J. & Oberwinkler, J. The colourful world of the mantis shrimp. Nature 401, 873–874 (1999).
pubmed: 10553902
Thoen, H. T., How, M. J., Chiou, T.-H. & Marshall, J. A different form of color vision in mantis shrimp. Science 343, 411–413 (2014).
pubmed: 24458639
Wetzstein, G., Ihrke, I., Lanman, D. & Heidrich, W. Computational plenoptic imaging. Comput. Graph. Forum 30, 2397–2426 (2011).
Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006).
pubmed: 16873662
Sitzmann, V. et al. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37, 114 (2018). The first demonstration of end-to-end optimization of optics and image processing for a computational camera design with computer vision applications.
Chakrabarti, A. Learning sensor multiplexing design through back-propagation. In Advances in Neural Information Processing Systems 29 (NIPS 2016) (eds Lee, D. D. et al.) 3081–3089 (2016).
Martel, J. N. P., Muller, L. K., Carey, S., Dudek, P. & Wetzstein, G. Neural sensors: learning pixel exposures for HDR imaging and video compressive sensing with programmable sensors. IEEE Trans. Pattern Anal. Mach. Intell. 42, 1642–1653 (2020).
pubmed: 32305899
Horstmeyer, R., Chen, R. Y., Kappes, B. & Judkewitz, B. Convolutional neural networks that teach microscopes how to image. Preprint at https://arxiv.org/abs/1709.07223 (2017).
Marco, J. et al. DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Trans. Graph. 36, 219 (2017).
Su, S., Heide, F., Wetzstein, G. & Heidrich, W. Deep end-to-end time-of-flight imaging. In 2018 IEEE Conf. Computer Vision and Pattern Recognition (CVPR) 6383–6392 (IEEE, 2018).
Kellman, M., Bostan, E., Repina, N. & Waller, L. Physics-based learned design: optimized coded-illumination for quantitative phase imaging. IEEE Trans. Comput. Imaging 5, 344–353 (2019).
Sinha, A., Lee, J., Li, S. & Barbastathis, G. Lensless computational imaging through deep learning. Optica 4, 1117–1125 (2017).
Metzler, C. A., Ikoma, H., Peng, Y. & Wetzstein, G. Deep optics for single-shot high-dynamic-range imaging. In 2020 IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR) 1372–1382 (IEEE, 2020).
Luo, Y. et al. Design of task-specific optical systems using broadband diffractive neural networks. Light Sci. Appl. 8, 112 (2019).
pubmed: 31814969
pmcid: 6885516
Haim, H., Elmalem, S., Giryes, R., Bronstein, A. M. & Marom, E. Depth estimation from a single image using deep learned phase coded mask. IEEE Trans. Comput. Imaging 4, 298–310 (2018).
Chang, J. & Wetzstein, G. Deep optics for monocular depth estimation and 3D object detection. In 2019 IEEE/CVF Intl Conf. Computer Vision (ICCV) 10192–10211 (IEEE, 2019).
Wu, Y., Boominathan, V., Chen, H., Sankaranarayanan, A. & Veeraraghavan, A. Phasecam3D—learning phase masks for passive single view depth estimation. In 2019 IEEE Intl Conf. Computational Photography (ICCP) 19–30 (IEEE, 2019).
Bertero, M. & Boccacci, P. Introduction to Inverse Problems in Imaging (CRC Press, 1998).
Barbastathis, G., Ozcan, A. & Situ, G. On the use of deep learning for computational imaging. Optica 6, 921–943 (2019).
Rivenson, Y. et al. Deep learning microscopy. Optica 4, 1437–1443 (2017).
Wu, Y. et al. Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery. Optica 5, 704–710 (2018).
Nehme, E. & Weiss, L. E., Michaeli, T. & Shechtman, Y. Deep-storm: super-resolution single-molecule microscopy by deep learning. Optica 5, 458–464 (2018).
Ouyang, W., Aristov, A., Lelek, M., Hao, X. & Zimmer, C. Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018).
pubmed: 29658943
Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803 (2018).
pubmed: 29656897
pmcid: 6309178
Wu, Y. et al. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nat. Methods 16, 1323–1331 (2019).
pubmed: 31686039
Wang, H. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16, 103–110 (2019).
pubmed: 30559434
Rivenson, Y., Zhang, Y., Günaydın, H., Teng, D. & Ozcan, A. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci. Appl. 7, 17141 (2018).
pubmed: 30839514
pmcid: 6060068
Boyd, N., Jonas, E., Babcock, H. & Recht, B. DeepLoco: Fast 3D localization microscopy using neural networks. Preprint at https://doi.org/10.1101/267096 (2018).
Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090 (2018).
Nehme, E. et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning. Nat. Methods 17, 734–740 (2020). An end-to-end optimization approach for point spread function engineering and neural-network-based locations for 3D fluorescence superresolution microscopy.
pubmed: 32541853
Liu, T. et al. Deep learning-based super-resolution in coherent imaging systems. Sci. Rep. 9, 3926 (2019).
pubmed: 30850721
pmcid: 6408569
Zhang, H. et al. High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network. Biomed. Opt. Express 10, 1044–1063 (2019).
pubmed: 30891329
pmcid: 6420277
Escudero, M. C. et al. Digitally stained confocal microscopy through deep learning. In Proc. 2nd Intl Conf. Medical Imaging with Deep Learning (eds Cardoso, M. J. et al.) 121–129 (PMLR, 2019).
Rivenson, Y. et al. Deep learning enhanced mobile-phone microscopy. ACS Photonics 5, 2354–2364 (2018).
Goy, A., Arthur, K., Li, S. & Barbastathis, G. Low photon count phase retrieval using deep learning. Phys. Rev. Lett. 121, 243902 (2018).
pubmed: 30608745
Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat. Biomed. Eng. 3, 466–477 (2019).
pubmed: 31142829
Wu, Y. et al. Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram. Light Sci. Appl. 8, 25 (2019).
pubmed: 30854197
pmcid: 6401162
Rivenson, Y. et al. PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning. Light Sci. Appl. 8, 23 (2019).
pubmed: 30728961
pmcid: 6363787
Mengu, D., Luo, Y., Rivenson, Y. & Ozcan, A. Analysis of diffractive optical neural networks and their integration with electronic neural networks. IEEE J. Sel. Top. Quantum Electron. 26, 3700114 (2019).
pubmed: 33223801
Dagenais, M., Sharfin, W. F. & Seymour, R. J. Optical digital matrix multiplication apparatus. EU patent EP0330710A1 (1988).