Prediction-error-dependent processing of immediate and delayed positive feedback.
FRN
Feedback delay
Prediction error
Reinforcement learning
RewP
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
27 Apr 2024
27 Apr 2024
Historique:
received:
05
12
2023
accepted:
22
04
2024
medline:
28
4
2024
pubmed:
28
4
2024
entrez:
27
4
2024
Statut:
epublish
Résumé
Learning often involves trial-and-error, i.e. repeating behaviours that lead to desired outcomes, and adjusting behaviour when outcomes do not meet our expectations and thus lead to prediction errors (PEs). PEs have been shown to be reflected in the reward positivity (RewP), an event-related potential (ERP) component between 200 and 350 ms after performance feedback which is linked to striatal processing and assessed via electroencephalography (EEG). Here we show that this is also true for delayed feedback processing, for which a critical role of the hippocampus has been suggested. We found a general reduction of the RewP for delayed feedback, but the PE was similarly reflected in the RewP and the later P300 for immediate and delayed positive feedback, while no effect was found for negative feedback. Our results suggest that, despite processing differences between immediate and delayed feedback, positive PEs drive feedback processing and learning irrespective of delay.
Identifiants
pubmed: 38678065
doi: 10.1038/s41598-024-60328-8
pii: 10.1038/s41598-024-60328-8
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
9674Informations de copyright
© 2024. The Author(s).
Références
Hajcak, G., Moser, J. S., Holroyd, C. B. & Simons, R. F. It’s worse than you thought: The feedback negativity and violations of reward prediction in gambling tasks. Psychophysiology 44, 905–912. https://doi.org/10.1111/j.1469-8986.2007.00567.x (2007).
doi: 10.1111/j.1469-8986.2007.00567.x
pubmed: 17666029
Höltje, G. & Mecklinger, A. Electrophysiological reward signals predict episodic memory for immediate and delayed positive feedback events. Brain Res. 1701, 64–74. https://doi.org/10.1016/j.brainres.2018.07.011 (2018).
doi: 10.1016/j.brainres.2018.07.011
pubmed: 30017716
Holroyd, C. B., Larsen, J. T. & Cohen, J. D. Context dependence of the event-related brain potential associated with reward and punishment. Psychophysiology 41, 245–253. https://doi.org/10.1111/j.1469-8986.2004.00152.x (2004).
doi: 10.1111/j.1469-8986.2004.00152.x
pubmed: 15032989
Delgado, M. R. Reward-related responses in the human striatum. Ann. N. Y. Acad. Sci. 1104, 70–88. https://doi.org/10.1196/annals.1390.002 (2007).
doi: 10.1196/annals.1390.002
pubmed: 17344522
Walsh, M. M. & Anderson, J. R. Learning from experience: Event-related potential correlates of reward processing, neural adaptation, and behavioral choice. Neurosci. Biobehav. Rev. 36, 1870–1884. https://doi.org/10.1016/j.neubiorev.2012.05.008 (2012).
doi: 10.1016/j.neubiorev.2012.05.008
pubmed: 22683741
pmcid: 3432149
Schultz, W. & Dickinson, A. Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23, 473–500. https://doi.org/10.1146/annurev.neuro.23.1.473 (2000).
doi: 10.1146/annurev.neuro.23.1.473
pubmed: 10845072
Zaghloul, K. A. et al. Human substantia Nigra neurons encode unexpected financial rewards. Science 323, 1496–1499. https://doi.org/10.1126/science.1167342 (2009).
doi: 10.1126/science.1167342
pubmed: 19286561
pmcid: 2839450
Björklund, A. & Dunnett, S. B. Dopamine neuron systems in the brain: An update. Trends Neurosci. 30, 194–202. https://doi.org/10.1016/j.tins.2007.03.006 (2007).
doi: 10.1016/j.tins.2007.03.006
pubmed: 17408759
Haber, S. N. & Knutson, B. The reward circuit: Linking primate anatomy and human imaging. Neuropsychopharmacology 35, 4–26. https://doi.org/10.1038/npp.2009.129 (2010).
doi: 10.1038/npp.2009.129
pubmed: 19812543
Foerde, K., Race, E., Verfaellie, M. & Shohamy, D. A role for the medial temporal lobe in feedback-driven learning: evidence from amnesia. J. Neurosci. 33, 5698–5704. https://doi.org/10.1523/JNEUROSCI.5217-12.2013 (2013).
doi: 10.1523/JNEUROSCI.5217-12.2013
pubmed: 23536083
pmcid: 3865542
Foerde, K. & Shohamy, D. The role of the basal ganglia in learning and memory: insight from Parkinson’s disease. Neurobiol. Learn. Mem. 96, 624–636. https://doi.org/10.1016/j.nlm.2011.08.006 (2011).
doi: 10.1016/j.nlm.2011.08.006
pubmed: 21945835
pmcid: 3772079
Jocham, G. et al. Reward-guided learning with and without causal attribution. Neuron 90, 177–190. https://doi.org/10.1016/j.neuron.2016.02.018 (2016).
doi: 10.1016/j.neuron.2016.02.018
pubmed: 26971947
pmcid: 4826429
Lighthall, N. R., Pearson, J. M., Huettel, S. A. & Cabeza, R. Feedback-based learning in aging: Contributions and trajectories of change in striatal and hippocampal systems. J. Neurosci. 38, 8453–8462. https://doi.org/10.1523/JNEUROSCI.0769-18.2018 (2018).
doi: 10.1523/JNEUROSCI.0769-18.2018
pubmed: 30120208
pmcid: 6158690
Hinneberg, B. M. & Hegele, M. Acting in temporal contexts: On the behavioral and neurophysiological consequences of feedback delays. Neuroscience 486, 91–102. https://doi.org/10.1016/j.neuroscience.2021.06.028 (2022).
doi: 10.1016/j.neuroscience.2021.06.028
pubmed: 34175422
Miltner, W. H., Braun, C. H. & Coles, M. G. Event-related brain potentials following incorrect feedback in a time-estimation task: Evidence for a “generic” neural system for error detection. J. Cognit. Neurosci. 9, 788–798. https://doi.org/10.1162/jocn.1997.9.6.788 (1997).
doi: 10.1162/jocn.1997.9.6.788
Krigolson, O. E. Event-related brain potentials and the study of reward processing: Methodological considerations. Int. J. Psychophysiol. 132, 175–183. https://doi.org/10.1016/j.ijpsycho.2017.11.007 (2018).
doi: 10.1016/j.ijpsycho.2017.11.007
pubmed: 29154804
Proudfit, G. H. The reward positivity: From basic research on reward to a biomarker for depression. Psychophysiology 52, 449–459. https://doi.org/10.1111/psyp.12370 (2015).
doi: 10.1111/psyp.12370
pubmed: 25327938
Baker, T. E. & Holroyd, C. B. Which way do I go? Neural activation in response to feedback and spatial processing in a virtual T-maze. Cereb. Cortex 19, 1708–1722. https://doi.org/10.1093/cercor/bhn223 (2009).
doi: 10.1093/cercor/bhn223
pubmed: 19073622
Holroyd, C. B., Krigolson, O. E. & Lee, S. Reward positivity elicited by predictive cues. NeuroReport 22, 249–252. https://doi.org/10.1097/WNR.0b013e328345441d (2011).
doi: 10.1097/WNR.0b013e328345441d
pubmed: 21386699
Holroyd, C. B. & Coles, M. G. H. The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 109, 679–709. https://doi.org/10.1037/0033-295X.109.4.679 (2002).
doi: 10.1037/0033-295X.109.4.679
pubmed: 12374324
Bellebaum, C. & Daum, I. Learning-related changes in reward expectancy are reflected in the feedback-related negativity. Eur. J. Neurosci. 27, 1823–1835. https://doi.org/10.1111/j.1460-9568.2008.06138.x (2008).
doi: 10.1111/j.1460-9568.2008.06138.x
pubmed: 18380674
Holroyd, C. B., Krigolson, O. E., Baker, R., Lee, S. & Gibson, J. When is an error not a prediction error? An electrophysiological investigation. Cognit. Affect. Behav. Neurosci. 9, 59–70. https://doi.org/10.3758/CABN.9.1.59 (2009).
doi: 10.3758/CABN.9.1.59
Sambrook, T. D. & Goslin, J. A neural reward prediction error revealed by a meta-analysis of ERPs using great grand averages. Psychol. Bull. 141, 213–235. https://doi.org/10.1037/bul0000006 (2015).
doi: 10.1037/bul0000006
pubmed: 25495239
Burnside, R., Fischer, A. G. & Ullsperger, M. The feedback-related negativity indexes prediction error in active but not observational learning. Psychophysiology 56, e13389. https://doi.org/10.1111/psyp.13389 (2019).
doi: 10.1111/psyp.13389
pubmed: 31054155
Fischer, A. G. & Ullsperger, M. Real and fictive outcomes are processed differently but converge on a common adaptive mechanism. Neuron 79, 1243–1255. https://doi.org/10.1016/j.neuron.2013.07.006 (2013).
doi: 10.1016/j.neuron.2013.07.006
pubmed: 24050408
Höltje, G. & Mecklinger, A. Feedback timing modulates interactions between feedback processing and memory encoding: Evidence from event-related potentials. Cognit. Affect. Behav. Neurosci. 20, 250–264. https://doi.org/10.3758/s13415-019-00765-5 (2020).
doi: 10.3758/s13415-019-00765-5
Hoy, C. W., Steiner, S. C. & Knight, R. T. Single-trial modeling separates multiple overlapping prediction errors during reward processing in human EEG. Commun. Biol. 4, 910. https://doi.org/10.1038/s42003-021-02426-1 (2021).
doi: 10.1038/s42003-021-02426-1
pubmed: 34302057
pmcid: 8302587
Becker, M. P. I., Nitsch, A. M., Miltner, W. H. R. & Straube, T. A single-trial estimation of the feedback-related negativity and its relation to BOLD responses in a time-estimation task. J. Neurosci. 34, 3005–3012. https://doi.org/10.1523/JNEUROSCI.3684-13.2014 (2014).
doi: 10.1523/JNEUROSCI.3684-13.2014
pubmed: 24553940
pmcid: 6608516
Gehring, W. J. & Willoughby, A. R. The medial frontal cortex and the rapid processing of monetary gains and losses. Science 295, 2279–2282. https://doi.org/10.1126/science.1066893 (2002).
doi: 10.1126/science.1066893
pubmed: 11910116
Foti, D., Weinberg, A., Dien, J. & Hajcak, G. Event-related potential activity in the basal ganglia differentiates rewards from nonrewards: Temporospatial principal components analysis and source localization of the feedback negativity. Hum. Brain Mapp. 32, 2207–2216. https://doi.org/10.1002/hbm.21182 (2011).
doi: 10.1002/hbm.21182
pubmed: 21305664
pmcid: 6870417
Weismüller, B. & Bellebaum, C. Expectancy affects the feedback-related negativity (FRN) for delayed feedback in probabilistic learning. Psychophysiology 53, 1739–1750. https://doi.org/10.1111/psyp.12738 (2016).
doi: 10.1111/psyp.12738
pubmed: 27565454
Weismüller, B., Kullmann, J., Hoenen, M. & Bellebaum, C. Effects of feedback delay and agency on feedback-locked beta and theta power during reinforcement learning. Psychophysiology 56, e13428. https://doi.org/10.1111/psyp.13428 (2019).
doi: 10.1111/psyp.13428
pubmed: 31245849
Peterburs, J., Kobza, S. & Bellebaum, C. Feedback delay gradually affects amplitude and valence specificity of the feedback-related negativity (FRN). Psychophysiology 53, 209–215. https://doi.org/10.1111/psyp.12560 (2016).
doi: 10.1111/psyp.12560
pubmed: 26459164
Weinberg, A., Luhmann, C. C., Bress, J. N. & Hajcak, G. Better late than never? The effect of feedback delay on ERP indices of reward processing. Cognit. Affect. Behav. Neurosci. 12, 671–677. https://doi.org/10.3758/s13415-012-0104-z (2012).
doi: 10.3758/s13415-012-0104-z
Arbel, Y., Hong, L., Baker, T. E. & Holroyd, C. B. It’s all about timing: An electrophysiological examination of feedback-based learning with immediate and delayed feedback. Neuropsychologia 99, 179–186. https://doi.org/10.1016/j.neuropsychologia.2017.03.003 (2017).
doi: 10.1016/j.neuropsychologia.2017.03.003
pubmed: 28284986
Kimura, K. & Kimura, M. Temporal prediction restores the evaluative processing of delayed action feedback. NeuroReport 27, 1061–1067. https://doi.org/10.1097/WNR.0000000000000657 (2016).
doi: 10.1097/WNR.0000000000000657
pubmed: 27513200
Frank, M. J., Seeberger, L. C. & O’reilly, R. C. By carrot or by stick: Cognitive reinforcement learning in parkinsonism. Science 306, 1940–1943. https://doi.org/10.1126/science.1102941 (2004).
doi: 10.1126/science.1102941
pubmed: 15528409
Weismüller, B. et al. Effects of feedback delay on learning from positive and negative feedback in patients with Parkinson’s disease off medication. Neuropsychologia 117, 46–54. https://doi.org/10.1016/j.neuropsychologia.2018.05.010 (2018).
doi: 10.1016/j.neuropsychologia.2018.05.010
pubmed: 29758227
Dickerson, K. C., Li, J. & Delgado, M. R. Parallel contributions of distinct human memory systems during probabilistic learning. NeuroImage 55, 266–276. https://doi.org/10.1016/j.neuroimage.2010.10.080 (2011).
doi: 10.1016/j.neuroimage.2010.10.080
pubmed: 21056678
Kirsch, F., Kirschner, H., Fischer, A. G., Klein, T. A. & Ullsperger, M. Disentangling performance-monitoring signals encoded in feedback-related EEG dynamics. NeuroImage. 257, 119322. https://doi.org/10.1016/j.neuroimage.2022.119322 (2022).
doi: 10.1016/j.neuroimage.2022.119322
pubmed: 35577025
Albrecht, C. & Bellebaum, C. Disentangling effects of expectancy, accuracy, and empathy on the processing of observed actions. Psychophysiology 58, 13883. https://doi.org/10.1111/psyp.13883 (2021).
doi: 10.1111/psyp.13883
Wang, J., Chen, J., Lei, Y. & Li, P. P300, not feedback error-related negativity, manifests the waiting cost of receiving reward information. NeuroReport 25, 1044–1048. https://doi.org/10.1097/WNR.0000000000000226 (2014).
doi: 10.1097/WNR.0000000000000226
pubmed: 25004079
Stewardson, H. J. & Sambrook, T. D. Evidence for parietal reward prediction errors using great grand average meta-analysis. Int. J. Psychophysiol. 152, 81–86. https://doi.org/10.1016/j.ijpsycho.2020.03.002 (2020).
doi: 10.1016/j.ijpsycho.2020.03.002
pubmed: 32272127
Palminteri, S., Lefebvre, G., Kilford, E. J. & Blakemore, S.-J. Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing. PLOS Comput. Biol. 13, e1005684. https://doi.org/10.1371/journal.pcbi.1005684 (2017).
doi: 10.1371/journal.pcbi.1005684
pubmed: 28800597
pmcid: 5568446
Nieuwenhuis, S., Aston-Jones, G. & Cohen, J. D. Decision making, the P3, and the locus coeruleus-norepinephrine system. Psychol. Bull. 131, 510–532. https://doi.org/10.1037/0033-2909.131.4.510 (2005).
doi: 10.1037/0033-2909.131.4.510
pubmed: 16060800
Yeung, N. & Sanfey, A. G. Independent coding of reward magnitude and valence in the human brain. J. Neurosci. 24, 6258–6264. https://doi.org/10.1523/JNEUROSCI.4537-03.2004 (2004).
doi: 10.1523/JNEUROSCI.4537-03.2004
pubmed: 15254080
pmcid: 6729539
Palminteri, S. Choice-confirmation bias and gradual perseveration in human reinforcement learning. Behav. Neurosci. 137, 78–88. https://doi.org/10.1037/bne0000541 (2023).
doi: 10.1037/bne0000541
pubmed: 36395020
Polich, J. Updating P300: An integrative theory of P3a and P3b. Clin. Neurophysiol. 118, 2128–2148. https://doi.org/10.1016/j.clinph.2007.04.019 (2007).
doi: 10.1016/j.clinph.2007.04.019
pubmed: 17573239
pmcid: 2715154
Jepma, M. et al. Catecholaminergic regulation of learning rate in a dynamic environment. PLOS Comput. Biol. 12, e1005171. https://doi.org/10.1371/journal.pcbi.1005171 (2016).
doi: 10.1371/journal.pcbi.1005171
pubmed: 27792728
pmcid: 5085041
Cavanagh, J. F. Cortical delta activity reflects reward prediction error and related behavioral adjustments, but at different times. NeuroImage 110, 205–216. https://doi.org/10.1016/j.neuroimage.2015.02.007 (2015).
doi: 10.1016/j.neuroimage.2015.02.007
pubmed: 25676913
Wurm, F. et al. Task learnability modulates surprise but not valence processing for reinforcement learning in probabilistic choice tasks. J. Cognit. Neurosci. 34, 34–53. https://doi.org/10.1162/jocn_a_01777 (2021).
doi: 10.1162/jocn_a_01777
Sambrook, T. D. & Goslin, J. Principal components analysis of reward prediction errors in a reinforcement learning task. NeuroImage 124, 276–286. https://doi.org/10.1016/j.neuroimage.2015.07.032 (2016).
doi: 10.1016/j.neuroimage.2015.07.032
pubmed: 26196667
Bellebaum, C., Polezzi, D. & Daum, I. It is less than you expected: the feedback-related negativity reflects violations of reward magnitude expectations. Neuropsychologia 48, 3343–3350. https://doi.org/10.1016/j.neuropsychologia.2010.07.023 (2010).
doi: 10.1016/j.neuropsychologia.2010.07.023
pubmed: 20655319
Volpert-Esmond, H. I., Page-Gould, E. & Bartholow, B. D. Using multilevel models for the analysis of event-related potentials. Int. J. Psychophysiol. 162, 145–156. https://doi.org/10.1016/j.ijpsycho.2021.02.006 (2021).
doi: 10.1016/j.ijpsycho.2021.02.006
pubmed: 33600841
pmcid: 8050933
Green, P. & MacLeod, C. J. SIMR: An R package for power analysis of generalized linear mixed models by simulation. Methods Ecol. Evol. 7, 493–498. https://doi.org/10.1111/2041-210X.12504 (2016).
doi: 10.1111/2041-210X.12504
Sutton, R. S. & Barto, A. Reinforcement Learning (The MIT Press, 2018).
Lefebvre, G., Lebreton, M., Meyniel, F., Bourgeois-Gironde, S. & Palminteri, S. Behavioural and neural characterization of optimistic reinforcement learning. Nat. Hum. Behav. 1, 0067. https://doi.org/10.1038/s41562-017-0067 (2017).
doi: 10.1038/s41562-017-0067
Frank, M. J., Moustafa, A. A., Haughey, H. M., Curran, T. & Hutchison, K. E. Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc. Natl. Acad. Sci. 104, 16311–16316. https://doi.org/10.1073/pnas.0706111104 (2007).
doi: 10.1073/pnas.0706111104
pubmed: 17913879
pmcid: 2042203
Chatrian, G. E., Lettich, E. & Nelson, P. L. Modified nomenclature for the “10%” electrode system. J. Clin. Neurophysiol. 5, 183–186 (1988).
doi: 10.1097/00004691-198804000-00005
pubmed: 3250964
Gratton, G., Coles, M. G. H. & Donchin, E. A new method for off-line removal of ocular artifact. Electroencephalogr Clin Neurophysiol 55, 468–484. https://doi.org/10.1016/0013-4694(83)90135-9 (1983).
doi: 10.1016/0013-4694(83)90135-9
pubmed: 6187540
Bates, D., Mächler, M., Bolker, B. & Walker, S. Fitting linear mixed-effects models using lme4. J. Stat. Softw. 67, 1–48. https://doi.org/10.18637/jss.v067.i01 (2015).
doi: 10.18637/jss.v067.i01
Kuznetsova, A., Brockhoff, P. B. & Christensen, R. H. B. lmerTest Package: Tests in linear mixed effects models. J. Stat. Softw. https://doi.org/10.18637/jss.v082.i13 (2017).
doi: 10.18637/jss.v082.i13
Voeten, C. Using ‘buildmer’ to automatically find & compare maximal (mixed) models (2020).
Bates, D., Kliegl, R., Vasishth, S., & Baayen, H. Parsimonious mixed models (2015).
Sambrook, T. D. & Goslin, J. Mediofrontal event-related potentials in response to positive, negative and unsigned prediction errors. Neuropsychologia 61, 1–10. https://doi.org/10.1016/j.neuropsychologia.2014.06.004 (2014).
doi: 10.1016/j.neuropsychologia.2014.06.004
pubmed: 24946315