Neural encoding with unsupervised spiking convolutional neural network.
Journal
Communications biology
ISSN: 2399-3642
Titre abrégé: Commun Biol
Pays: England
ID NLM: 101719179
Informations de publication
Date de publication:
28 08 2023
28 08 2023
Historique:
received:
06
02
2023
accepted:
18
08
2023
medline:
31
8
2023
pubmed:
29
8
2023
entrez:
28
8
2023
Statut:
epublish
Résumé
Accurately predicting the brain responses to various stimuli poses a significant challenge in neuroscience. Despite recent breakthroughs in neural encoding using convolutional neural networks (CNNs) in fMRI studies, there remain critical gaps between the computational rules of traditional artificial neurons and real biological neurons. To address this issue, a spiking CNN (SCNN)-based framework is presented in this study to achieve neural encoding in a more biologically plausible manner. The framework utilizes unsupervised SCNN to extract visual features of image stimuli and employs a receptive field-based regression algorithm to predict fMRI responses from the SCNN features. Experimental results on handwritten characters, handwritten digits and natural images demonstrate that the proposed approach can achieve remarkably good encoding performance and can be utilized for "brain reading" tasks such as image reconstruction and identification. This work suggests that SNN can serve as a promising tool for neural encoding.
Identifiants
pubmed: 37640808
doi: 10.1038/s42003-023-05257-4
pii: 10.1038/s42003-023-05257-4
pmc: PMC10462614
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
880Informations de copyright
© 2023. Springer Nature Limited.
Références
Kay, K. N., Naselaris, T., Prenger, R. J. & Gallant, J. L. Identifying natural images from human brain activity. Nature 452, 352–355 (2008).
pubmed: 18322462
pmcid: 3556484
Güçlü, U. & van Gerven, M. A. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35, 10005–10014 (2015).
pubmed: 26157000
pmcid: 6605414
Nishimoto, S. et al. Reconstructing visual experiences from brain activity evoked by natural movies. Curr. Biol. 21, 1641–1646 (2011).
pubmed: 21945275
pmcid: 3326357
Wen, H. et al. Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision. Cereb. Cortex 28, 4136–4160 (2018).
pubmed: 29059288
Naselaris, T., Kay, K. N., Nishimoto, S. & Gallant, J. L. Encoding and decoding in fMRI. NeuroImage 56, 400–410 (2011).
pubmed: 20691790
Wu, M. C. K., David, S. V. & Gallant, J. L. Complete functional characterization of sensory neurons by system identification. Annu. Rev. Neurosci. 29, 477–505 (2006).
pubmed: 16776594
Adelson, E. H. & Bergen, J. R. Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Am. A 2, 284–299 (1985).
pubmed: 3973762
Jones, J. P. & Palmer, L. A. An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. J. Neurophysiol. 58, 1233–1258 (1987).
pubmed: 3437332
Carandini, M. et al. Do we know what the early visual system does? J. Neurosci. 25, 10577–10597 (2005).
pubmed: 16291931
pmcid: 6725861
Khaligh-Razavi, S. M. & Kriegeskorte, N. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014).
pubmed: 25375136
pmcid: 4222664
Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A. & Oliva, A. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Sci. Rep. 6, 27755 (2016).
pubmed: 27282108
pmcid: 4901271
Kriegeskorte, N. & Kievit, R. A. Representational geometry: integrating cognition, computation, and the brain. Trends Cogn. Sci. 17, 401–412 (2013).
pubmed: 23876494
pmcid: 3730178
Allen, E. J. et al. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci. 25, 116–126 (2022).
pubmed: 34916659
Khosla, M., Ngo, G. H., Jamison, K., Kuceyeski, A. & Sabuncu, M. R. Cortical response to naturalistic stimuli is largely predictable with deep neural networks. Sci. Adv. 7, eabe7547 (2021).
pubmed: 34049888
pmcid: 8163078
Xu, Y. & Vaziri-Pashkam, M. Limits to visual representational correspondence between convolutional neural networks and the human brain. Nat. Commun. 12, 2065 (2021).
pubmed: 33824315
pmcid: 8024324
Maass, W. Networks of spiking neurons: the third generation of neural network models. Neural Netw. 10, 1659–1671 (1997).
Gerstner, W., Kempter, R., van Hemmen, J. L. & Wagner, H. A neuronal learning rule for sub-millisecond temporal coding. Nature 383, 76–78 (1996).
pubmed: 8779718
Bi, G.-Q. & Poo, M.-M. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18, 10464 (1998).
pubmed: 9852584
pmcid: 6793365
Huang, S. et al. Associative Hebbian synaptic plasticity in primate visual cortex. J. Neurosci. 34, 7575–7579 (2014).
pubmed: 24872561
pmcid: 4035519
McMahon, DavidB. T. & Leopold, DavidA. Stimulus timing-dependent plasticity in high-level vision. Curr. Biol. 22, 332–337 (2012).
pubmed: 22305750
pmcid: 4342232
Meliza, C. D. & Dan, Y. Receptive-field modification in rat visual cortex induced by paired visual stimulation and single-cell spiking. Neuron 49, 183–189 (2006).
pubmed: 16423693
Diehl, P. & Cook, M. Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Front. Comput. Neurosci. https://doi.org/10.3389/fncom.2015.00099 (2015).
Kheradpisheh, S. R., Ganjtabesh, M., Thorpe, S. J. & Masquelier, T. STDP-based spiking deep convolutional neural networks for object recognition. Neural Netw. 99, 56–67 (2018).
pubmed: 29328958
Mozafari, M., Ganjtabesh, M., Nowzari-Dalini, A., Thorpe, S. J. & Masquelier, T. Bio-inspired digit recognition using reward-modulated spike-timing-dependent plasticity in deep convolutional networks. Pattern Recognit. 94, 87–95 (2019).
Schoenmakers, S., Barth, M., Heskes, T. & van Gerven, M. Linear reconstruction of perceived images from human brain activity. Neuroimage 83, 951–961 (2013).
pubmed: 23886984
Van Gerven, M. A., De Lange, F. P. & Heskes, T. Neural decoding with hierarchical generative models. Neural Comput. 22, 3127–3142 (2010).
pubmed: 20858128
Horikawa, T. & Kamitani, Y. Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8, 15037 (2017).
pubmed: 28530228
pmcid: 5458127
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. International Conference on Learning Representations. https://doi.org/10.48550/arXiv.1412.6980 (2015).
Seeliger, K. et al. End-to-end neural system identification with neural information flow. PLoS Comput. Biol. 17, e1008558 (2021).
pubmed: 33539366
pmcid: 7888598
Zhou, W., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).
Miyawaki, Y. et al. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 60, 915–929 (2008).
pubmed: 19081384
Wang, W., Arora, R., Livescu, K. & Bilmes, J. On deep multi-view representation learning. Proc. 32nd Int. Conf. Mach. Learn. 37, 1083–1092 (2015).
Du, C., Du, C., Huang, L. & He, H. Reconstructing perceived images from human brain activities with bayesian deep multiview learning. IEEE Trans. Neural Netw. Learn. Syst. 30, 2310–2323 (2019).
pubmed: 30561354
Seeliger, K., Güçlü, U., Ambrogioni, L., Güçlütürk, Y. & van Gerven, M. A. J. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage 181, 775–785 (2018).
pubmed: 30031932
Victor, J. D., Purpura, K., Katz, E. & Mao, B. Population encoding of spatial frequency, orientation, and color in macaque V1. J. Neurophysiol. 72, 2151–2166 (1994).
pubmed: 7884450
Dumoulin, S. O. & Wandell, B. A. Population receptive field estimates in human visual cortex. NeuroImage 39, 647–660 (2008).
pubmed: 17977024
Gao, X., Wang, Y., Chen, X. & Gao, S. Interface, interaction, and intelligence in generalized brain-computer interfaces. Trends Cogn. Sci. 25, 671–684 (2021).
pubmed: 34116918
Ren, Z. et al. Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning. NeuroImage 228, 117602 (2021).
pubmed: 33395572
Wang, C. et al. Reconstructing rapid natural vision with fMRI-conditional video generative adversarial network. Cerebral Cortex https://doi.org/10.1093/cercor/bhab498 (2022).
Wu, Y., Deng, L., Li, G., Zhu, J. & Shi, L. Spatio-temporal backpropagation for training high-performance spiking neural networks. Front. Neurosci. 12, 331 (2018).
pubmed: 29875621
pmcid: 5974215
Izhikevich, E. M. Simple model of spiking neurons. IEEE Trans. Neural Netw. 14, 1569–1572 (2003).
pubmed: 18244602
Enroth-Cugell, C. & Robson, J. G. The contrast sensitivity of retinal ganglion cells of the cat. J. Physiol. 187, 517–552 (1966).
pubmed: 16783910
pmcid: 1395960
McMahon, M. J., Packer, O. S. & Dacey, D. M. The classical receptive field surround of primate parasol ganglion cells is mediated primarily by a non-GABAergic pathway. J. Neurosci. 24, 3736–3745 (2004).
pubmed: 15084653
pmcid: 6729348
Gautrais, J. & Thorpe, S. Rate coding versus temporal order coding: a theoretical approach. Biosystems 48, 57–65 (1998).
pubmed: 9886632
Mozafari, M., Ganjtabesh, M., Nowzari-Dalini, A. & Masquelier, T. SpykeTorch: efficient simulation of convolutional spiking neural networks with at most one spike per neuron. Front. Neurosci. https://doi.org/10.3389/fnins.2019.00625 (2019).
Du, C., Du, C., Huang, L. & He, H. Conditional generative neural decoding with structured CNN feature prediction. Proc. AAAI Conf. Artif. Intell. 34, 2629–2636 (2020).
Van der Maaten, L. A new benchmark dataset for handwritten character recognition. Tilburg Univ. 2–5 (2009).
Friston, K. J. et al. Statistical parametric maps in functional imaging: a general linear approach. Hum. Brain Mapp. 2, 189–210 (1994).
Han, K. et al. Variational autoencoder: an unsupervised model for encoding and decoding fMRI activity in visual cortex. NeuroImage 198, 125–136 (2019).
pubmed: 31103784
Kay, K. N., Winawer, J., Mezer, A. & Wandell, B. A. Compressive spatial summation in human visual cortex. J. Neurophysiol. 110, 481–494 (2013).
pubmed: 23615546
pmcid: 3727075
Lage-Castellanos, A., Valente, G., Formisano, E. & De Martino, F. Methods for computing the maximum performance of computational models of fMRI responses. PLoS Comput. Biol. 15, e1006397 (2019).
pubmed: 30849071
pmcid: 6426260
Deng, J. et al. Imagenet: a large-scale hierarchical image database. IEEE Conf. Comput. Vis. Pattern Recognit. https://doi.org/10.1109/CVPR.2009.5206848 (2009).