A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence.
Journal
Nature neuroscience
ISSN: 1546-1726
Titre abrégé: Nat Neurosci
Pays: United States
ID NLM: 9809671
Informations de publication
Date de publication:
01 2022
01 2022
Historique:
received:
01
03
2021
accepted:
12
10
2021
pubmed:
18
12
2021
medline:
7
4
2022
entrez:
17
12
2021
Statut:
ppublish
Résumé
Extensive sampling of neural activity during rich cognitive phenomena is critical for robust understanding of brain function. Here we present the Natural Scenes Dataset (NSD), in which high-resolution functional magnetic resonance imaging responses to tens of thousands of richly annotated natural scenes were measured while participants performed a continuous recognition task. To optimize data quality, we developed and applied novel estimation and denoising techniques. Simple visual inspections of the NSD data reveal clear representational transformations along the ventral visual pathway. Further exemplifying the inferential power of the dataset, we used NSD to build and train deep neural network models that predict brain activity more accurately than state-of-the-art models from computer vision. NSD also includes substantial resting-state and diffusion data, enabling network neuroscience perspectives to constrain and enhance models of perception and memory. Given its unprecedented scale, quality and breadth, NSD opens new avenues of inquiry in cognitive neuroscience and artificial intelligence.
Identifiants
pubmed: 34916659
doi: 10.1038/s41593-021-00962-x
pii: 10.1038/s41593-021-00962-x
doi:
Types de publication
Journal Article
Research Support, N.I.H., Extramural
Research Support, N.I.H., Intramural
Research Support, U.S. Gov't, Non-P.H.S.
Langues
eng
Sous-ensembles de citation
IM
Pagination
116-126Subventions
Organisme : NIBIB NIH HHS
ID : P41 EB015894
Pays : United States
Organisme : NINDS NIH HHS
ID : P30 NS076408
Pays : United States
Organisme : NCRR NIH HHS
ID : S10 RR026783
Pays : United States
Organisme : NIH HHS
ID : S10 OD017974
Pays : United States
Organisme : Intramural NIH HHS
ID : ZIA MH002909
Pays : United States
Organisme : NIBIB NIH HHS
ID : R01 EB030896
Pays : United States
Organisme : NIBIB NIH HHS
ID : R01 EB029272
Pays : United States
Commentaires et corrections
Type : CommentIn
Informations de copyright
© 2021. The Author(s), under exclusive licence to Springer Nature America, Inc.
Références
de Vries, S. E. J. et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nat. Neurosci. 23, 138–151 (2020).
pubmed: 31844315
doi: 10.1038/s41593-019-0550-9
Siegle, J. H. et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature 592, 86–92 (2021).
Stringer, C., Pachitariu, M., Steinmetz, N., Carandini, M. & Harris, K. D. High-dimensional geometry of population responses in visual cortex. Nature 571, 361–365 (2019).
pubmed: 31243367
pmcid: 6642054
doi: 10.1038/s41586-019-1346-5
Markram, H. et al. Reconstruction and simulation of neocortical microcircuitry. Cell 163, 456–492 (2015).
pubmed: 26451489
doi: 10.1016/j.cell.2015.09.029
Van Essen, D. C. et al. The WU-Minn human connectome project: an overview. Neuroimage 80, 62–79 (2013).
pubmed: 23684880
doi: 10.1016/j.neuroimage.2013.05.041
Zheng, Z. et al. A complete electron microscopy volume of the brain of adult Drosophila melanogaster. Cell 174, 730–743 (2018).
pubmed: 30033368
pmcid: 6063995
doi: 10.1016/j.cell.2018.06.019
Van Essen, D. C. et al. Mapping visual cortex in monkeys and humans using surface-based atlases. Vis. Res. 41, 1359–1378 (2001).
pubmed: 11322980
doi: 10.1016/S0042-6989(01)00045-1
Grill-Spector, K. & Malach, R. The human visual cortex. Annu. Rev. Neurosci. 27, 649–677 (2004).
pubmed: 15217346
doi: 10.1146/annurev.neuro.27.070203.144220
Wheeler, M. E., Petersen, S. E. & Buckner, R. L. Memory’s echo: vivid remembering reactivates sensory-specific cortex. Proc. Natl Acad. Sci. USA 97, 11125–11129 (2000).
pubmed: 11005879
pmcid: 27159
doi: 10.1073/pnas.97.20.11125
Breedlove, J. L., St-Yves, G., Olman, C. A. & Naselaris, T. Generative feedback explains distinct brain activity codes for seen and mental images. Curr. Biol. 30, 2211–2224 (2020).
pubmed: 32359428
doi: 10.1016/j.cub.2020.04.014
Kay, K. N., Weiner, K. S. & Grill-Spector, K. Attention reduces spatial uncertainty in human ventral temporal cortex. Curr. Biol. 25, 595–600 (2015).
pubmed: 25702580
pmcid: 4348205
doi: 10.1016/j.cub.2014.12.050
Huth, A. G., Nishimoto, S., Vu, A. T. & Gallant, J. L. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76, 1210–1224 (2012).
pubmed: 23259955
pmcid: 3556488
doi: 10.1016/j.neuron.2012.10.014
Krizhevsky, A. Learning Multiple Layers of Features from Tiny Images. https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf (University of Toronto, 2009).
Lin, T.-Y. et al. Microsoft COCO: Common Objects in Context. European Conference on Computer Vision. https://link.springer.com/chapter/10.1007/978-3-319-10602-1_48 , 740–755 (Springer, 2014).
Güçlü, U. & van Gerven, M. A. J. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35, 10005–10014 (2015).
pubmed: 26157000
pmcid: 6605414
doi: 10.1523/JNEUROSCI.5023-14.2015
Khaligh-Razavi, S.-M. & Kriegeskorte, N. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014).
pubmed: 25375136
pmcid: 4222664
doi: 10.1371/journal.pcbi.1003915
Seeliger, K. et al. End-to-end neural system identification with neural information flow. PLoS Comput. Biol. 17, e1008558 (2021).
pubmed: 33539366
pmcid: 7888598
doi: 10.1371/journal.pcbi.1008558
Stansbury, D. E., Naselaris, T. & Gallant, J. L. Natural scene statistics account for the representation of scene categories in human visual cortex. Neuron 79, 1025–1034 (2013).
pubmed: 23932491
pmcid: 5464350
doi: 10.1016/j.neuron.2013.06.034
St-Yves, G. & Naselaris, T. The feature-weighted receptive field: an interpretable encoding model for complex feature spaces. Neuroimage 180, 188–202 (2018).
Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl Acad. Sci. USA 111, 8619–8624 (2014).
pubmed: 24812127
pmcid: 4060707
doi: 10.1073/pnas.1403112111
Naselaris, T. et al. Cognitive computational neuroscience: a new conference for an emerging discipline. Trends Cogn. Sci. 22, 365–367 (2018).
pubmed: 29500078
pmcid: 5911192
doi: 10.1016/j.tics.2018.02.008
Chang, N. et al. BOLD5000, a public fMRI dataset while viewing 5000 visual images. Sci. Data 6, 49 (2019).
pubmed: 31061383
pmcid: 6502931
doi: 10.1038/s41597-019-0052-3
Horikawa, T. & Kamitani, Y. Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8, 15037 (2017).
pubmed: 28530228
pmcid: 5458127
doi: 10.1038/ncomms15037
Kay, K. N., Naselaris, T., Prenger, R. J. & Gallant, J. L. Identifying natural images from human brain activity. Nature 452, 352–355 (2008).
pubmed: 18322462
pmcid: 3556484
doi: 10.1038/nature06713
Triantafyllou, C. et al. Comparison of physiological noise at 1.5 T, 3 T and 7 T and optimization of fMRI acquisition parameters. Neuroimage 26, 243–250 (2005).
pubmed: 15862224
doi: 10.1016/j.neuroimage.2005.01.007
Brady, T. F., Konkle, T., Alvarez, G. A. & Oliva, A. Visual long-term memory has a massive storage capacity for object details. Proc. Natl Acad. Sci. USA 105, 14325–14329 (2008).
pubmed: 18787113
pmcid: 2533687
doi: 10.1073/pnas.0803390105
Haxby, J. V., Guntupalli, J. S., Nastase, S. A. & Feilong, M. Hyperalignment: modeling shared information encoded in idiosyncratic cortical topographies. eLife 9, e56601 (2020).
pubmed: 32484439
pmcid: 7266639
doi: 10.7554/eLife.56601
Power, J. D., Lynch, C. J., Adeyemo, B. & Petersen, S. E. A critical, event-related appraisal of denoising in resting-state fMRI studies. Cereb. Cortex 30, 5544–5559 (2020).
pubmed: 32494823
doi: 10.1093/cercor/bhaa139
Roth, Z. N., Ryoo, M. & Merriam, E. P. Task-related activity in human visual cortex. PLoS Biol. 18, e3000921 (2020).
pubmed: 33156829
pmcid: 7673548
doi: 10.1371/journal.pbio.3000921
Benson, N. C. et al. The human connectome project 7 Tesla retinotopy dataset: description and population receptive field analysis. J. Vis. 18, 23 (2018).
Stigliani, A., Weiner, K. S. & Grill-Spector, K. Temporal processing capacity in high-level visual cortex is domain specific. J. Neurosci. 35, 12412–12424 (2015).
pubmed: 26354910
pmcid: 4563034
doi: 10.1523/JNEUROSCI.4822-14.2015
Kay, K. et al. A critical assessment of data quality and venous effects in sub-millimeter fMRI. Neuroimage 189, 847–869 (2019).
pubmed: 30731246
doi: 10.1016/j.neuroimage.2019.02.006
Gordon, E. M. et al. Precision functional mapping of individual human brains. Neuron 95, 791–807 (2017).
pubmed: 28757305
pmcid: 5576360
doi: 10.1016/j.neuron.2017.07.011
Kang, X., Yund, E. W., Herron, T. J. & Woods, D. L. Improving the resolution of functional brain imaging: analyzing functional data in anatomical space. Magn. Reson. Imaging 25, 1070–1078 (2007).
pubmed: 17707169
doi: 10.1016/j.mri.2006.12.005
Kay, K. N., Rokem, A., Winawer, J., Dougherty, R. F. & Wandell, B. GLMdenoise: a fast, automated technique for denoising task-based fMRI data. Front. Neurosci. 7, 247 (2013).
pubmed: 24381539
pmcid: 3865440
doi: 10.3389/fnins.2013.00247
Rokem, A. & Kay, K. Fractional ridge regression: a fast, interpretable reparameterization of ridge regression. Gigascience 9, giaa133 (2020).
Albrecht, D. G. & Hamilton, D. B. Striate cortex of monkey and cat: contrast response function. J. Neurophysiol. 48, 217–237 (1982).
pubmed: 7119846
doi: 10.1152/jn.1982.48.1.217
Wagner, A. D., Shannon, B. J., Kahn, I. & Buckner, R. L. Parietal lobe contributions to episodic memory retrieval. Trends Cogn. Sci. 9, 445–453 (2005).
pubmed: 16054861
doi: 10.1016/j.tics.2005.07.001
Spaniol, J. et al. Event-related fMRI studies of episodic encoding and retrieval: meta-analyses using activation likelihood estimation. Neuropsychologia 47, 1765–1779 (2009).
pubmed: 19428409
doi: 10.1016/j.neuropsychologia.2009.02.028
Gonzalez-Castillo, J. et al. Whole-brain, time-locked activation with simple tasks revealed using massive averaging and model-free analysis. Proc. Natl Acad. Sci. USA 109, 5487–5492 (2012).
pubmed: 22431587
pmcid: 3325687
doi: 10.1073/pnas.1121049109
Maaten, Lvander & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).
Connolly, A. C. et al. The representation of biological classes in the human brain. J. Neurosci. 32, 2608–2618 (2012).
pubmed: 22357845
pmcid: 3532035
doi: 10.1523/JNEUROSCI.5547-11.2012
Naselaris, T., Stansbury, D. E. & Gallant, J. L. Cortical representation of animate and inanimate objects in complex natural scenes. J. Physiol. Paris 106, 239–249 (2012).
pubmed: 22472178
pmcid: 3407302
doi: 10.1016/j.jphysparis.2012.02.001
Long, B., Yu, C.-P. & Konkle, T. Mid-level visual features underlie the high-level categorical organization of the ventral stream. Proc. Natl Acad. Sci. USA 115, E9015–E9024 (2018).
pubmed: 30171168
pmcid: 6156638
doi: 10.1073/pnas.1719616115
Henriksson, L., Khaligh-Razavi, S.-M., Kay, K. & Kriegeskorte, N. Visual representations are dominated by intrinsic fluctuations correlated between areas. Neuroimage 114, 275–286 (2015).
pubmed: 25896934
doi: 10.1016/j.neuroimage.2015.04.026
Naselaris, T., Kay, K. N., Nishimoto, S. & Gallant, J. L. Encoding and decoding in fMRI. Neuroimage 56, 400–410 (2011).
pubmed: 20691790
doi: 10.1016/j.neuroimage.2010.07.073
Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25 https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html , 1097–1105 (2012).
Cadena, S. A. et al. Deep convolutional models improve predictions of macaque V1 responses to natural images. PLoS Comput. Biol. 15, e1006897 (2019).
pubmed: 31013278
pmcid: 6499433
doi: 10.1371/journal.pcbi.1006897
Wang, A., Tarr, M. & Wehbe, L. Neural Taskonomy: Inferring the Similarity of Task-Derived Representations from Brain Activity. In Advances in Neural Information Processing Systems 32 https://papers.nips.cc/paper/2019/hash/f490c742cd8318b8ee6dca10af2a163f-Abstract.html , 15475–15485 (2019).
Sinz, F. H., Pitkow, X., Reimer, J., Bethge, M. & Tolias, A. S. Engineering a less artificial intelligence. Neuron 103, 967–979 (2019).
pubmed: 31557461
doi: 10.1016/j.neuron.2019.08.034
Aliko, S., Huang, J., Gheorghiu, F., Meliss, S. & Skipper, J. I. A naturalistic neuroimaging database for understanding the brain using ecological stimuli. Sci. Data 7, 347 (2020).
pubmed: 33051448
pmcid: 7555491
doi: 10.1038/s41597-020-00680-2
Nastase, S. A., Liu, Y.-F., Hillman, H., Norman, K. A. & Hasson, U. Leveraging shared connectivity to aggregate heterogeneous datasets into a common response space. Neuroimage 217, 116865 (2020).
pubmed: 32325212
doi: 10.1016/j.neuroimage.2020.116865
Taylor, J. R. et al. The cambridge centre for ageing and neuroscience (Cam-CAN) data repository: structural and functional MRI, MEG, and cognitive data from a cross-sectional adult lifespan sample. Neuroimage 144, 262–269 (2017).
pubmed: 26375206
doi: 10.1016/j.neuroimage.2015.09.018
Bellec, P. & Boyle, J. A. Bridging the gap between perception and action: the case for neuroimaging, AI and video games. Preprint at https://psyarxiv.com/3epws (2019).
Pinho, A. L. et al. Individual Brain Charting, a high-resolution fMRI dataset for cognitive mapping. Sci. Data 5, 180105 (2018).
pubmed: 29893753
pmcid: 5996851
doi: 10.1038/sdata.2018.105
Poldrack, R. A. et al. Long-term neural and physiological phenotyping of a single human. Nat. Commun. 6, 8885 (2015).
pubmed: 26648521
doi: 10.1038/ncomms9885
Seeliger, K., Sommers, R. P., Güçlü, U., Bosch, S. E. & van Gerven, M. A. J. A large single-participant fMRI dataset for probing brain responses to naturalistic stimuli in space and time. Preprint at https://www.biorxiv.org/content/10.1101/687681v1 (2019).
Naselaris, T., Allen, E. & Kay, K. Extensive sampling for complete models of individual brains. Curr. Opin. Behav. Sci. 40, 45–51 (2021).
doi: 10.1016/j.cobeha.2020.12.008
Polimeni, J. R., Renvall, V., Zaretskaya, N. & Fischl, B. Analysis strategies for high-resolution UHF-fMRI data. Neuroimage 168, 296–320 (2018).
pubmed: 28461062
doi: 10.1016/j.neuroimage.2017.04.053
Harms, M. P. et al. Extending the Human Connectome Project across ages: imaging protocols for the Lifespan Development and Aging projects. Neuroimage 183, 972–984 (2018).
pubmed: 30261308
doi: 10.1016/j.neuroimage.2018.09.060
Power, J. D. et al. Customized head molds reduce motion during resting state fMRI scans. Neuroimage 189, 141–149 (2019).
pubmed: 30639840
doi: 10.1016/j.neuroimage.2019.01.016
Brainard, D. H. The Psychophysics Toolbox. Spat. Vis. 10, 433–436 (1997).
pubmed: 9176952
doi: 10.1163/156856897X00357
Pelli, D. G. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat. Vis. 10, 437–442 (1997).
pubmed: 9176953
doi: 10.1163/156856897X00366
Caesar, H., Uijlings, J. & Ferrari, V. COCO-Stuff: Thing and Stuff classes in context. In IEEE/CVF Conf. Computer Vision and Pattern Recognition https://doi.ieeecomputersociety.org/10.1109/CVPR.2018.00132 1209–1218 (2018).
Schira, M. M., Tyler, C. W., Breakspear, M. & Spehar, B. The foveal confluence in human visual cortex. J. Neurosci. 29, 9050–9058 (2009).
pubmed: 19605642
pmcid: 6665445
doi: 10.1523/JNEUROSCI.1760-09.2009
Shahid, A., Wilkinson, K., Marcu, S. & Shapiro, C. M. Stanford Sleepiness Scale (SSS). In: STOP, THAT and One Hundred Other Sleep Scales (eds. Shahid, A., Wilkinson, K., Marcu, S. & Shapiro, C. M.) 369–370 (Springer, 2012).
Marks, D. F. Visual imagery differences in the recall of pictures. Br. J. Psychol. 64, 17–24 (1973).
pubmed: 4742442
doi: 10.1111/j.2044-8295.1973.tb01322.x
Torgesen, J. K., Wagner, R. & Rashotte, C. TOWRE-2: Test of Word Reading Efficiency (Pearson, 2012).
Duchaine, B. & Nakayama, K. The Cambridge Face Memory Test: results for neurologically intact individuals and an investigation of its validity using inverted face stimuli and prosopagnosic participants. Neuropsychologia 44, 576–585 (2006).
pubmed: 16169565
doi: 10.1016/j.neuropsychologia.2005.07.001
Tardif, J., Watson, M., Giaschi, D. & Gosselin, F. Measuring the contrast sensitivity function in just three clicks. J. Vis. 16, 966–966 (2016).
doi: 10.1167/16.12.966
Arora, S., Liang, Y. & Ma, T. A simple but tough-to-beat baseline for sentence embeddings. https://openreview.net/pdf?id=SyK00v5xx (2017).
Kriegeskorte, N. & Mur, M. Inverse MDS: inferring dissimilarity structure from multiple item arrangements. Front. Psychol. 3, 245 (2012).
pubmed: 22848204
pmcid: 3404552
doi: 10.3389/fpsyg.2012.00245
Kay, K., Jamison, K. W., Zhang, R.-Y. & Uğurbil, K. A temporal decomposition method for identifying venous effects in task-based fMRI. Nat. Methods 17, 1033–1039 (2020).
pubmed: 32895538
pmcid: 7721302
doi: 10.1038/s41592-020-0941-6
Avants, B. B. et al. A reproducible evaluation of ANTs similarity metric performance in brain image registration. Neuroimage 54, 2033–2044 (2011).
pubmed: 20851191
doi: 10.1016/j.neuroimage.2010.09.025
Yushkevich, P. A. et al. User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage 31, 1116–1128 (2006).
pubmed: 16545965
doi: 10.1016/j.neuroimage.2006.01.015
Esteban, O. et al. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nat. Methods 16, 111–116 (2019).
pubmed: 30532080
doi: 10.1038/s41592-018-0235-4
Power, J. D., Barnes, K. A., Snyder, A. Z., Schlaggar, B. L. & Petersen, S. E. Spurious but systematic correlations in functional connectivity MRI networks arise from subject motion. Neuroimage 59, 2142–2154 (2012).
pubmed: 22019881
doi: 10.1016/j.neuroimage.2011.10.018
Handwerker, D. A., Gonzalez-Castillo, J., D’Esposito, M. & Bandettini, P. A. The continuing challenge of understanding and modeling hemodynamic variation in fMRI. Neuroimage 62, 1017–1023 (2012).
pubmed: 22366081
doi: 10.1016/j.neuroimage.2012.02.015
Hoerl, A. E. & Kennard, R. W. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12, 55–67 (1970).
doi: 10.1080/00401706.1970.10488634
Kay, K. N., Winawer, J., Mezer, A. & Wandell, B. Compressive spatial summation in human visual cortex. J. Neurophysiol. 110, 481–494 (2013).
pubmed: 23615546
pmcid: 3727075
doi: 10.1152/jn.00105.2013
Lage-Castellanos, A., Valente, G., Formisano, E. & De Martino, F. Methods for computing the maximum performance of computational models of fMRI responses. PLoS Comput. Biol. 15, e1006397 (2019).
pubmed: 30849071
pmcid: 6426260
doi: 10.1371/journal.pcbi.1006397
Biswal, B., Yetkin, F. Z., Haughton, V. M. & Hyde, J. S. Functional connectivity in the motor cortex of resting human brain using echo-planar MRI. Magn. Reson. Med. 34, 537–541 (1995).
pubmed: 8524021
doi: 10.1002/mrm.1910340409
Nili, H. et al. A toolbox for representational similarity analysis. PLoS Comput. Biol. 10, e1003553 (2014).
pubmed: 24743308
pmcid: 3990488
doi: 10.1371/journal.pcbi.1003553
Kriegeskorte, N., Mur, M. & Bandettini, P. Representational similarity analysis—connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).
pubmed: 19104670
pmcid: 2605405
Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
Gorgolewski, K. J. et al. The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Sci. Data 3, 1–9 (2016).
doi: 10.1038/sdata.2016.44
Cichy, R. M., Roig, G. & Oliva, A. The Algonauts Project. Nat. Mach. Intell. 1, 613 (2019).
doi: 10.1038/s42256-019-0127-z