Computational processes of simultaneous learning of stochasticity and volatility in humans.
Journal
Nature communications
ISSN: 2041-1723
Titre abrégé: Nat Commun
Pays: England
ID NLM: 101528555
Informations de publication
Date de publication:
21 Oct 2024
21 Oct 2024
Historique:
received:
21
07
2023
accepted:
10
10
2024
medline:
22
10
2024
pubmed:
22
10
2024
entrez:
21
10
2024
Statut:
epublish
Résumé
Making adaptive decisions requires predicting outcomes, and this in turn requires adapting to uncertain environments. This study explores computational challenges in distinguishing two types of noise influencing predictions: volatility and stochasticity. Volatility refers to diffusion noise in latent causes, requiring a higher learning rate, while stochasticity introduces moment-to-moment observation noise and reduces learning rate. Dissociating these effects is challenging as both increase the variance of observations. Previous research examined these factors mostly separately, but it remains unclear whether and how humans dissociate them when they are played off against one another. In two large-scale experiments, through a behavioral prediction task and computational modeling, we report evidence of humans dissociating volatility and stochasticity solely based on their observations. We observed contrasting effects of volatility and stochasticity on learning rates, consistent with statistical principles. These results are consistent with a computational model that estimates volatility and stochasticity by balancing their dueling effects.
Identifiants
pubmed: 39433765
doi: 10.1038/s41467-024-53459-z
pii: 10.1038/s41467-024-53459-z
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
9073Subventions
Organisme : U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (NIMH)
ID : R21MH134217
Organisme : National Science Foundation (NSF)
ID : IIS-1822571
Informations de copyright
© 2024. The Author(s).
Références
Dayan, P. & Long, T. Statistical Models of Conditioning. 10, 117–123 (1998).
Dayan, P., Kakade, S. & Montague, P. R. Learning and selective attention. Nat. Neurosci. 3, 1218–1223 (2000).
pubmed: 11127841
doi: 10.1038/81504
Courville, A. C., Daw, N. D. & Touretzky, D. S. Bayesian theories of conditioning in a changing world. Trends Cognit. Sci. 10, 294–300 (2006).
doi: 10.1016/j.tics.2006.05.004
Daunizeau, J. et al. Observing the observer (I): meta-bayesian models of learning and decision-making. PLoS ONE 5, e15554 (2010).
pubmed: 21179480
pmcid: 3001878
doi: 10.1371/journal.pone.0015554
Gershman, S. J., Blei, D. M. & Niv, Y. Context, learning, and extinction. Psychol. Rev. 117, 197–209 (2010).
pubmed: 20063968
doi: 10.1037/a0017808
Behrens, T. E. J., Woolrich, M. W., Walton, M. E. & Rushworth, M. F. S. Learning the value of information in an uncertain world. Nat. Neurosci. 10, 1214–1221 (2007).
pubmed: 17676057
doi: 10.1038/nn1954
Nassar, M. R. et al. Rational regulation of learning dynamics by pupil-linked arousal systems. Nat. Neurosci. 15, 1040–1046 (2012).
pubmed: 22660479
pmcid: 3386464
doi: 10.1038/nn.3130
Iglesias, S. et al. Hierarchical prediction errors in midbrain and basal forebrain during sensory learning. Neuron 80, 519–530 (2013).
pubmed: 24139048
doi: 10.1016/j.neuron.2013.09.009
McGuire, J. T., Nassar, M. R., Gold, J. I. & Kable, J. W. Functionally dissociable influences on learning rate in a dynamic environment. Neuron 84, 870–881 (2014).
pubmed: 25459409
pmcid: 4437663
doi: 10.1016/j.neuron.2014.10.013
Soltani, A. & Izquierdo, A. Adaptive learning under expected and unexpected uncertainty. Nat. Rev. Neurosci. 20, 635–644 (2019).
pubmed: 31147631
pmcid: 6752962
doi: 10.1038/s41583-019-0180-y
Mathys, C., Daunizeau, J., Friston, K. J. & Stephan, K. E. A bayesian foundation for individual learning under uncertainty. Front. Hum. Neurosci. 5, 39 (2011).
pubmed: 21629826
pmcid: 3096853
doi: 10.3389/fnhum.2011.00039
Browning, M., Behrens, T. E., Jocham, G., O’Reilly, J. X. & Bishop, S. J. Anxious individuals have difficulty learning the causal statistics of aversive environments. Nat. Neurosci. 18, 590–596 (2015).
pubmed: 25730669
pmcid: 4644067
doi: 10.1038/nn.3961
Piray, P. & Daw, N. D. A model for learning based on the joint estimation of stochasticity and volatility. Nat. Commun. 12, 6587 (2021).
pubmed: 34782597
pmcid: 8592992
doi: 10.1038/s41467-021-26731-9
Piray, P. & Daw, N. D. A simple model for learning in volatile environments. PLoS Comput. Biol. 16, e1007963 (2020).
pubmed: 32609755
pmcid: 7329063
doi: 10.1371/journal.pcbi.1007963
Pulcu, E. & Browning, M. The misestimation of uncertainty in affective disorders. Trends Cognit. Sci. 23, 865–875 (2019).
doi: 10.1016/j.tics.2019.07.007
Brazil, I. A., Mathys, C. D., Popma, A., Hoppenbrouwers, S. S. & Cohn, M. D. Representational uncertainty in the brain during threat conditioning and the link with psychopathic traits. Biol. Psychiatry Cognit. Neurosci. Neuroimaging 2, 689–695 (2017).
doi: 10.1016/j.bpsc.2017.04.005
Cole, D. M. et al. Atypical processing of uncertainty in individuals at risk for psychosis. NeuroImage: Clin. 26, 102239 (2020).
pubmed: 32182575
doi: 10.1016/j.nicl.2020.102239
Deserno, L. et al. Volatility estimates increase choice switching and relate to prefrontal activity in Schizophrenia. Biol. Psychiatry. Cognit. Neurosci. Neuroimaging 5, 173–183 (2020).
doi: 10.1016/j.bpsc.2019.10.007
Diaconescu, A. O., Wellstein, K. V., Kasper, L., Mathys, C. & Stephan, K. E. Hierarchical Bayesian models of social inference for probing persecutory delusional ideation. J. Abnorm. Psychol. 129, 556–569 (2020).
pubmed: 32757600
doi: 10.1037/abn0000500
Farashahi, S. et al. Metaplasticity as a NEURAL SUBSTRATE FOR ADAPTIVE LEARNING AND CHOICE UNDER UNCERTAINty. Neuron 94, 401–414.e6 (2017).
pubmed: 28426971
pmcid: 5515734
doi: 10.1016/j.neuron.2017.03.044
Gagne, C., Dayan, P. & Bishop, S. J. When planning to survive goes wrong: predicting the future and replaying the past in anxiety and PTSD. Curr. Opin. Behav. Sci. 24, 89–95 (2018).
doi: 10.1016/j.cobeha.2018.03.013
Katthagen, T. et al. Modeling subjective relevance in schizophrenia and its relation to aberrant salience. PLoS Comput. Biol. 14, e1006319 (2018).
pubmed: 30096179
pmcid: 6105009
doi: 10.1371/journal.pcbi.1006319
Lawson, R. P., Mathys, C. & Rees, G. Adults with autism overestimate the volatility of the sensory environment. Nat. Neurosci. 20, 1293–1299 (2017).
pubmed: 28758996
pmcid: 5578436
doi: 10.1038/nn.4615
Paliwal, S. et al. Subjective estimates of uncertainty during gambling and impulsivity after subthalamic deep brain stimulation for Parkinson’s disease. Sci. Rep. 9, 14795 (2019).
pubmed: 31616015
pmcid: 6794275
doi: 10.1038/s41598-019-51164-2
Piray, P., Ly, V., Roelofs, K., Cools, R. & Toni, I. Emotionally aversive cues suppress neural systems underlying optimal learning in socially anxious individuals. J. Neurosci. 39, 1445–1456 (2019).
pubmed: 30559152
pmcid: 6381249
doi: 10.1523/JNEUROSCI.1394-18.2018
Powers, A. R., Mathys, C. & Corlett, P. R. Pavlovian conditioning-induced hallucinations result from overweighting of perceptual priors. Science 357, 596–600 (2017).
pubmed: 28798131
pmcid: 5802347
doi: 10.1126/science.aan3458
Diaconescu, A. O. et al. Inferring on the Intentions of Others by Hierarchical Bayesian Learning. PLoS Comput. Biol 10, e1003810 (2014).
Reed, E. J. et al. Paranoia as a deficit in non-social belief updating. eLife 9, e56345 (2020).
Nassar, M. R., Wilson, R. C., Heasly, B. & Gold, J. I. An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J. Neurosci. 30, 12366–12378 (2010).
pubmed: 20844132
pmcid: 2945906
doi: 10.1523/JNEUROSCI.0822-10.2010
Lee, S., Gold, J. I. & Kable, J. W. The human as delta-rule learner. PsycNET. Decis. 7, 55–66 (2020).
doi: 10.1037/dec0000112
Mackintosh, N. J. A theory of attention: variations in the associability of stimuli with reinforcement. Psychol. Rev. 82, 276–298 (1975).
doi: 10.1037/h0076778
Pearce, J. M. & Hall, G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 87, 532–552 (1980).
pubmed: 7443916
doi: 10.1037/0033-295X.87.6.532
Hall, G. & Pearce, J. M. Restoring the associability of a Pre-Exposed CS by a Surprising Event. Q. J. Exp. Psychol. Sect. B 34, 127–140 (1982).
doi: 10.1080/14640748208400881
Holland, P. C. & Gallagher, M. Amygdala circuitry in attentional and representational processes. Trends Cognit. Sci. 3, 65–73 (1999).
doi: 10.1016/S1364-6613(98)01271-6
Holland, P. C. & Schiffino, F. L. Mini-review: prediction errors, attention and associative learning. Neurobiol. Learn. Mem. 131, 207–215 (2016).
pubmed: 26948122
pmcid: 4862921
doi: 10.1016/j.nlm.2016.02.014
Costa, V. D., Dal Monte, O., Lucas, D. R., Murray, E. A. & Averbeck, B. B. Amygdala and ventral striatum make distinct contributions to reinforcement learning. Neuron 92, 505–517 (2016).
pubmed: 27720488
pmcid: 5074688
doi: 10.1016/j.neuron.2016.09.025
Piray, P., Dezfouli, A., Heskes, T., Frank, M. J. & Daw, N. D. Hierarchical Bayesian inference for concurrent model fitting and comparison for group studies. PLOS Comput. Biol. 15, e1007043 (2019).
pubmed: 31211783
pmcid: 6581260
doi: 10.1371/journal.pcbi.1007043
Gibbon, J., Farrell, L., Locurto, C. M., Duncan, H. J. & Terrace, H. S. Partial reinforcement in autoshaping with pigeons. Anim. Learn. Behav. 8, 45–59 (1980).
doi: 10.3758/BF03209729
Nassar, M. R. et al. Age differences in learning emerge from an insufficient representation of uncertainty in older adults. Nat Commun 7, 11609 (2016).
Kalman, R. E. A new approach to linear filtering and prediction problems. Trans. ASME J. Basic Eng. 82, 35–45 (1960).
doi: 10.1115/1.3662552
Doucet, A., de Freitas, N., Murphy, K. P. & Russell, S. J. Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks. In Proc. 16th Conference on Uncertainty in Artificial Intelligence 176–183 (Morgan Kaufmann Publishers Inc., 2000).
Doucet, A. & Johansen, A. M. A tutorial on particle filtering and smoothing: Fifteen years later. in Handbook of Nonlinear Filtering (eds. Crisan, D. & Rozovskii, B.) (Oxford University Press, 2011).
Gelbart, M. A., Snoek, J. & Adams, R. P. Bayesian optimization with unknown constraints. in Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence 250259 (AUAI Press, Arlington, Virginia, USA, 2014)
Snoek, J., Larochelle, H. & Adams, R. P. Practical bayesian optimization of machine learning algorithms. https://doi.org/10.48550/arXiv.1206.2944 (2012).
d’Acremont, M. & Bossaerts, P. Neural mechanisms behind identification of Leptokurtic Noise and Adaptive Behavioral Response. Cereb. Cortex 26, 1818–1830 (2016).
pubmed: 26850528
pmcid: 4785960
doi: 10.1093/cercor/bhw013
Sutton, R. S. Adapting bias by gradient descent: an incremental version of delta-bar-delta. In Proc. tenth national conference on Artificial intelligence 171–176 (AAAI Press, San Jose, 1992).
Findling, C., Skvortsova, V., Dromnelle, R., Palminteri, S. & Wyart, V. Computational noise in reward-guided learning drives behavioral variability in volatile environments. Nat. Neurosci. 22, 2066–2077 (2019).
pubmed: 31659343
doi: 10.1038/s41593-019-0518-9
Diederen, K. M. J. & Schultz, W. Scaling prediction errors to reward variability benefits error-driven learning in humans. J. Neurophysiol. 114, 1628–1640 (2015).
pubmed: 26180123
pmcid: 4563025
doi: 10.1152/jn.00483.2015
Hartley, C. A. & Phelps, E. A. Anxiety and decision-making. Biol. Psychiatry 72, 113–118 (2012).
pubmed: 22325982
doi: 10.1016/j.biopsych.2011.12.027
Huys, Q. J. M., Daw, N. D. & Dayan, P. Depression: a decision-theoretic analysis. Annu. Rev. Neurosci. 38, 1–23 (2015).
pubmed: 25705929
doi: 10.1146/annurev-neuro-071714-033928
Paulus, M. P. & Yu, A. J. Emotion and decision-making: affect-driven belief systems in anxiety and depression. Trends Cognit. Sci. 16, 476–483 (2012).
doi: 10.1016/j.tics.2012.07.009
Aylward, J. et al. Altered learning under uncertainty in unmedicated mood and anxiety disorders. Nat. Hum. Behav. 3, 1116–1123 (2019).
pubmed: 31209369
pmcid: 6790140
doi: 10.1038/s41562-019-0628-0
Huang, H., Thompson, W. & Paulus, M. P. Computational dysfunctions in anxiety: failure to differentiate signal from noise. Biol. Psychiatry 82, 440–446 (2017).
pubmed: 28838468
pmcid: 5576575
doi: 10.1016/j.biopsych.2017.07.007
de Berker, A. O. et al. Computations of uncertainty mediate acute stress responses in humans. Nat. Commun. 7, 10996 (2016).
pubmed: 27020312
pmcid: 4820542
doi: 10.1038/ncomms10996
Khorsand, P. & Soltani, A. Optimal structure of metaplasticity for adaptive learning. PLoS Comput. Biol. 13, e1005630 (2017).
pubmed: 28658247
pmcid: 5509349
doi: 10.1371/journal.pcbi.1005630
Pulcu, E. & Browning, M. Humans adapt rationally to approximate estimates of uncertainty. 2023.11.26.568699 Preprint at https://doi.org/10.1101/2023.11.26.568699 (2023).
Andrieu, C., Doucet, A. & Holenstein, R. Particle Markov chain Monte Carlo methods. J. R. Stat. Soc. Ser. B 72, 269–342 (2010).
doi: 10.1111/j.1467-9868.2009.00736.x
Lindsten, F., Jordan, M. I. & Schön, T. B. Particle gibbs with ancestor sampling. J. Mach. Learn. Res. 15, 2145–2184 (2014).
Lee, J. K., Rouault, M. & Wyart, V. Adaptive tuning of human learning and choice variability to unexpected uncertainty. Sci. Adv. 9, eadd0501 (2023).
pubmed: 36989365
pmcid: 10058239
doi: 10.1126/sciadv.add0501
FeldmanHall, O. & Nassar, M. R. The computational challenge of social learning. Trends Cognit. Sci. 25, 1045–1057 (2021).
doi: 10.1016/j.tics.2021.09.002
Mahmoodi, A., Luo, S., Harbison, C., Piray, P. & Rushworth, M. F. S. Human hippocampus and dorsomedial prefrontal cortex infer and update latent causes during social interaction. Neuron 112, 1–14 (2024).
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G. & Niv, Y. Orbitofrontal cortex as a cognitive map of task space. Neuron 81, 267–279 (2014).
pubmed: 24462094
pmcid: 4001869
doi: 10.1016/j.neuron.2013.11.005
Niv, Y. Learning task-state representations. Nat. Neurosci. 22, 1544–1553 (2019).
pubmed: 31551597
pmcid: 7241310
doi: 10.1038/s41593-019-0470-8
Gershman, S. J. A unifying probabilistic view of associative learning. PLoS Comput. Biol. 11, e1004567 (2015).
Piray, P. & Daw, N. D. Linear reinforcement learning in planning, grid fields, and cognitive control. Nat. Commun. 12, 4942 (2021).
pubmed: 34400622
pmcid: 8368103
doi: 10.1038/s41467-021-25123-3
Hunt, L. T. et al. Formalizing planning and information search in naturalistic decision-making. Nat. Neurosci. 24, 1051–1064 (2021).
pubmed: 34155400
doi: 10.1038/s41593-021-00866-w
Collins, A. G. E. & Cockburn, J. Beyond dichotomies in reinforcement learning. Nat. Rev. Neurosci. 21, 576–586 (2020).
pubmed: 32873936
pmcid: 7800310
doi: 10.1038/s41583-020-0355-6
Zorowitz, S., Bennett, D. & jbransonbyers. nivlab/nivturk: Prolific v1.2. Zenodo https://doi.org/10.5281/zenodo.6609218 (2022).
Douc, R. & Cappe, O. Comparison of resampling schemes for particle filtering. In ISPA 2005. Proc. 4th International Symposium on Image and Signal Processing and Analysis, 2005. 64–69. https://doi.org/10.1109/ISPA.2005.195385 (2005).
Piray, P. Computational processes of simultaneous learning of stochasticity and volatility in humans: code and data. Zenodo https://doi.org/10.5281/zenodo.13840905 (2024).