Optogenetics inspired transition metal dichalcogenide neuristors for in-memory deep recurrent neural networks.
Journal
Nature communications
ISSN: 2041-1723
Titre abrégé: Nat Commun
Pays: England
ID NLM: 101528555
Informations de publication
Date de publication:
25 06 2020
25 06 2020
Historique:
received:
04
12
2019
accepted:
04
06
2020
entrez:
27
6
2020
pubmed:
27
6
2020
medline:
27
6
2020
Statut:
epublish
Résumé
Shallow feed-forward networks are incapable of addressing complex tasks such as natural language processing that require learning of temporal signals. To address these requirements, we need deep neuromorphic architectures with recurrent connections such as deep recurrent neural networks. However, the training of such networks demand very high precision of weights, excellent conductance linearity and low write-noise- not satisfied by current memristive implementations. Inspired from optogenetics, here we report a neuromorphic computing platform comprised of photo-excitable neuristors capable of in-memory computations across 980 addressable states with a high signal-to-noise ratio of 77. The large linear dynamic range, low write noise and selective excitability allows high fidelity opto-electronic transfer of weights with a two-shot write scheme, while electrical in-memory inference provides energy efficiency. This method enables implementing a memristive deep recurrent neural network with twelve trainable layers with more than a million parameters to recognize spoken commands with >90% accuracy.
Identifiants
pubmed: 32587241
doi: 10.1038/s41467-020-16985-0
pii: 10.1038/s41467-020-16985-0
pmc: PMC7316775
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
3211Références
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436 (2015).
pubmed: 26017442
pmcid: 26017442
Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. Deepface: closing the gap to human-level performance in face verification. in Proc IEEE Conference on Computer Vision and Pattern Recognition. 1701–1708 (IEEE, 2014).
Merolla, P. A. et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673 (2014).
pubmed: 25104385
Park, H. et al. Flexible neuromorphic electronics for computing, soft robotics, and neuroprosthetics. Adv. Mater. 32, 1903558 (2019).
Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).
pubmed: 31308498
Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. Nature 577, 641–646 (2020).
pubmed: 31996818
Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2015).
pubmed: 25951284
Wang, Z. et al. Resistive switching materials for information processing. Nat. Rev. Mater. 5, 173–195 (2020).
Burr, G. W. et al. Neuromorphic computing using non-volatile memory. Adv. Phys. X 2, 89–124 (2017).
Park, D. S. et al. Specaugment: a simple data augmentation method for automatic speech recognition. Proc. Interspeech 2613–2617 (2019).
Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. Proc. NAACL 4171–4186 (2019).
Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208 (2019).
pubmed: 31068721
pmcid: 6522354
Cheng, Z., Ríos, C., Pernice, W. H. P., Wright, C. D. & Bhaskaran, H. On-chip photonic synapse. Sci. Adv. 3, e1700160 (2017).
pubmed: 28959725
pmcid: 5617375
Häusser, M. Optogenetics: the age of light. Nat. Methods 11, 1012 (2014).
pubmed: 25264778
Boyden, E. S., Zhang, F., Bamberg, E., Nagel, G. & Deisseroth, K. Millisecond-timescale, genetically targeted optical control of neural activity. Nat. Neurosci. 8, 1263 (2005).
pubmed: 16116447
Deisseroth, K. Optogenetics. Nat. Methods 8, 26–29 (2011).
pubmed: 21191368
Honjo, K., Hwang, R. Y. & Tracey, W. D. Jr Optogenetic manipulation of neural circuits and behavior in Drosophila larvae. Nat. Protoc. 7, 1470 (2012).
pubmed: 22790083
pmcid: 3751580
Burguière, E., Monteiro, P., Feng, G. & Graybiel, A. M. Optogenetic stimulation of lateral orbitofronto-striatal pathway suppresses compulsive behaviors. Science 340, 1243–1246 (2013).
pubmed: 23744950
Luo, Z.-D. et al. Artificial optoelectronic synapses based on ferroelectric field-effect enabled 2D transition metal dichalcogenide memristive transistors. ACS Nano 14, 746–754 (2020).
pubmed: 31887010
Park, H. et al. Retina‐inspired carbon nitride‐based photonic synapses for selective detection of UV light. Adv. Mater. 32, 1906899 (2020).
Diehl, P. U. & Cook, M. Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Front. Comput. Neurosci. https://doi.org/10.3389/fncom.2015.00099 (2015).
Prezioso, M. et al. Spike-timing-dependent plasticity learning of coincidence detection with passively integrated memristive circuits. Nat. Commun. 9, 1–8 (2018).
Cavaccini, A. et al. Serotonergic signaling controls input-specific synaptic plasticity at striatal circuits. Neuron 98, 801–816 (2018).
pubmed: 29706583
González-Rueda, A., Pedrosa, V., Feord, R. C., Clopath, C. & Paulsen, O. Activity-dependent downscaling of subthreshold synaptic inputs during slow-wave-sleep-like activity in vivo. Neuron 97, 1244–1252 (2018).
pubmed: 29503184
pmcid: 5873548
John, R. A. et al. Ultralow power dual-gated subthreshold oxide neuristors: an enabler for higher order neuronal temporal correlations. ACS Nano 12, 11263–11273 (2018).
pubmed: 30395439
Liu, F. et al. Highly sensitive detection of polarized light using anisotropic 2D ReS2. Adv. Funct. Mater. 26, 1169–1177 (2016).
Tiwari, N. et al. Indium tungsten oxide thin films for flexible high-performance transistors and neuromorphic electronics. ACS Appl. Mater. Interfaces 10, 30506–30513 (2018).
pubmed: 30129368
Burr, G. W. et al. Experimental demonstration and tolerancing of a large-scale neural network (165 000 synapses) using phase-change memory as the synaptic weight element. IEEE Trans. Electron Devices 62, 3498–3507 (2015).
Kim, S., Hasler, J. & George, S. Integrated floating-gate programming environment for system-level ICs. IEEE Trans. Very Large Scale Integr. Syst. 24, 2244–2252 (2015).
Cai, F. et al. A fully integrated reprogrammable memristor–CMOS system for efficient multiply–accumulate operations. Nat. Electron 2, 290–299 (2019).
Fuller, E. J. et al. Li‐ion synaptic transistor for low power analog computing. Adv. Mater. 29, 1604310 (2017).
Lopez-Sanchez, O., Lembke, D., Kayci, M., Radenovic, A. & Kis, A. Ultrasensitive photodetectors based on monolayer MoS 2. Nat. Nanotechnol. 8, 497–501 (2013).
pubmed: 23748194
Fang, H. & Hu, W. Photogating in low dimensional photodetectors. Adv. Sci. 4, 1700323 (2017).
Jeon, S. et al. Gated three-terminal device architecture to eliminate persistent photoconductivity in oxide semiconductor photosensor arrays. Nat. Mater. 11, 301–305 (2012).
pubmed: 22367002
Aly, M. M. S. et al. The N3XT approach to energy-efficient abundant-data computing. Proc. IEEE 107, 19–48 (2019).
Courbariaux, M., Bengio, Y. & David, J.-P. Binaryconnect: Training deep neural networks with binary weights during propagations. In Proc. Advances in Neural Information Processing Systems 28 (NIPS 2015). 3123–3131 (Curran Associates, 2015).
Lin, X., Zhao, C. & Pan, W. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems 30 (NIPS 2017). 345–353 (Curran Associates, 2017).
Rastegari, M., Ordonez, V., Redmon, J. & Farhadi, A. Xnor-net: Imagenet classification using binary convolutional neural networks. in Proc. Computer Vision – ECCV 2016: 14th European Conference. 525–542 (Springer, 2016).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
pubmed: 9377276
Gokmen, T., Rasch, M. & Haensch, W. Training LSTM networks with resistive cross-point devices. Front. Neurosci. 12, 745 (2018).
pubmed: 30405334
pmcid: 6207602
van de Burgt, Y. et al. A non-volatile organic electrochemical device as a low-voltage artificial synapse for neuromorphic computing. Nat. Mater. 16, 414–418 (2017).
pubmed: 28218920
Fuller, E. J. et al. Li-ion synaptic transistor for low power analog computing. Adv. Mater. 10.1002/adma.201604310 (2016).
LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR), Conference Track Proceedings (2015).
Basu, A. & Hasler, P. E. A fully integrated architecture for fast and accurate programming of floating gates over six decades of current. IEEE Trans. Very Large Scale Integr. Syst. 19, 953–962 (2010).
Basu, A. et al. A floating-gate-based field-programmable analog array. IEEE J. Solid State Circuits 45, 1781–1794 (2010).
Bayat, F. M. et al. Model-based high-precision tuning of NOR flash memory cells for analog computing applications. in 2016 74th Annual Device Research Conference (DRC). 1–2 (IEEE, 2016).
Fuller, E. J. et al. Parallel programming of an ionic floating-gate memory array for scalable neuromorphic computing. Science 364, 570–574 (2019).
pubmed: 31023890
Warden, P. Speech commands: a public dataset for single-word speech recognition. http://download.tensorflow.org/data/speech_commands_v0. Vol. 1 (2017).
Sun, J. et al. Optoelectronic synapse based on igzo‐alkylated graphene oxide hybrid structure. Adv. Funct. Mater. 28, 1804397 (2018).
Lv, Z. et al. Mimicking neuroplasticity in a hybrid biopolymer transistor by dual modes modulation. Adv. Funct. Mater. 29, 1902374 (2019).
Seo, S. et al. Artificial optic-neural synapse for colored and color-mixed pattern recognition. Nat. Commun. 9, 1–8 (2018).
Jin, G. et al. Atomically thin three-dimensional membranes of van der Waals semiconductors by wafer-scale growth. Sci. Adv. 5, eaaw3180 (2019).
pubmed: 31360767
pmcid: 6660212
Shim, J. et al. Controlled crack propagation for atomic precision handling of wafer-scale two-dimensional materials. Science 362, 665–670 (2018).
pubmed: 30309906
Zhou, J. et al. A library of atomically thin metal chalcogenides. Nature 556, 355–359 (2018).
pubmed: 29670263