Pavlovian impatience: The anticipation of immediate rewards increases approach behaviour.
Delay discounting
Intertemporal choice
Motivational bias
Pavlovian bias
Present bias
Reinforcement learning
Journal
Cognitive, affective & behavioral neuroscience
ISSN: 1531-135X
Titre abrégé: Cogn Affect Behav Neurosci
Pays: United States
ID NLM: 101083946
Informations de publication
Date de publication:
28 Oct 2024
28 Oct 2024
Historique:
accepted:
03
10
2024
medline:
29
10
2024
pubmed:
29
10
2024
entrez:
29
10
2024
Statut:
aheadofprint
Résumé
People often exhibit intertemporal impatience by choosing immediate small over delayed larger rewards, which has been implicated across maladaptive behaviours and mental health symptoms. In this preregistered study, we tested the role of an intertemporal Pavlovian bias as possible psychological mechanism driving the temptation posed by immediate rewards. Concretely, we hypothesized that the anticipation of immediate rewards (compared with preference-matched delayed rewards) enhances goal-directed approach behaviour but interferes with goal-directed inhibition. Such a mechanism could contribute to the difficulty to inhibit ourselves in the face of immediate rewards (e.g., a drug), at the cost of long-term (e.g., health) goals. A sample of 184 participants completed a newly developed reinforcement learning go/no-go task with four trial types: Go to win immediate reward; Go to win delayed reward; No-go to win immediate reward; and No-go to win delayed reward trials. Go responding was increased in trials in which an immediate reward was available compared with trials in which a preference-matched delayed reward was available. Computational models showed that on average, this behavioural pattern was best captured by a cue-response bias reflecting a stronger elicitation of go responses upon presentation of an immediate (versus delayed) reward cue. The results of this study support the role of an intertemporal Pavlovian bias as a psychological mechanism contributing to impatient intertemporal choice.
Identifiants
pubmed: 39467981
doi: 10.3758/s13415-024-01236-2
pii: 10.3758/s13415-024-01236-2
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2024. The Author(s).
Références
Albrecht, M. A., Waltz, J. A., Cavanagh, J. F., Frank, M. J., & Gold, J. M. (2016). Reduction of Pavlovian bias in schizophrenia: Enhanced effects in clozapine-administered patients. PLoS ONE, 11(4), 1–23. https://doi.org/10.1371/journal.pone.0152781
doi: 10.1371/journal.pone.0152781
Algermissen, J., & den Ouden, H. E. M. (2023). Goal-directed recruitment of Pavlovian biases through selective visual attention. Journal of Experimental Psychology: General. https://doi.org/10.1037/xge0001425
doi: 10.1037/xge0001425
pubmed: 37199975
Algermissen, J., Swart, J. C., Scheeringa, R., Cools, R., & Den Ouden, H. E. M. (2022). Striatal BOLD and midfrontal theta power express motivation for action. Cerebral Cortex, 32(14), 2924–2942. https://doi.org/10.1093/cercor/bhab391
doi: 10.1093/cercor/bhab391
pubmed: 34849626
Amlung, M., Marsden, E., Holshausen, K., Morris, V., Patel, H., Vedelago, L., Naish, K. R., Reed, D. D., & McCabe, R. E. (2019). Delay discounting as a transdiagnostic process in psychiatric disorders: A meta-analysis. JAMA Psychiatry, 76(11), 1176–1186. https://doi.org/10.1001/jamapsychiatry.2019.2102
doi: 10.1001/jamapsychiatry.2019.2102
pubmed: 31461131
pmcid: 6714026
Amlung, M., Petker, T., Jackson, J., Balodis, I., & Mackillop, J. (2016). Steep discounting of delayed monetary and food rewards in obesity: A meta-analysis. Psychological Medicine, 46(11), 2423–2434. https://doi.org/10.1017/S0033291716000866
doi: 10.1017/S0033291716000866
pubmed: 27299672
Appelhans, B. M., Tangney, C. C., French, S. A., Crane, M. M., & Wang, Y. (2019). Delay discounting and household food purchasing decisions: The SHoPPER study. Health Psychology, 38(4), 334–342. https://doi.org/10.1037/hea0000727.Delay
doi: 10.1037/hea0000727.Delay
pubmed: 30896220
pmcid: 6430149
Barlow, P., Reeves, A., McKee, M., Galea, G., & Stuckler, D. (2016). Unhealthy diets, obesity and time discounting: A systemetic literature review and network analysis. Obesity Reviews, 17, 810–819. https://doi.org/10.1111/obr.12431
doi: 10.1111/obr.12431
pubmed: 27256685
pmcid: 4988386
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001
doi: 10.1016/j.jml.2012.11.001
Benhabib, J., Bisin, A., & Schotter, A. (2010). Present-bias, quasi-hyperbolic discounting, and fixed costs. Games and Economic Behavior, 69(2), 205–223. https://doi.org/10.1016/j.geb.2009.11.003
doi: 10.1016/j.geb.2009.11.003
Bettman, J. R., Luce, M. F., & Payne, J. W. (1998). Constructive consumer choice processes. Journal of Consumer Research, 25(3), 187–217. https://doi.org/10.1086/209535
doi: 10.1086/209535
Burghoorn, F., Heuvelmans, V. R., Scheres, A., Roelofs, K., & Figner, B. (2024). Pavlovian-to-instrumental transfer in intertemporal choice. Judgment and Decision Making, 19(e3). https://doi.org/10.1017/jdm.2023.42
Bürkner, P.-C. (2018). Advanced Bayesian multilevel modeling with the R package brms (Version 2.17.0) [Computer software]. The R Journal, 10(1), 395–411. https://doi.org/10.32614/RJ-2018-017
Cavanagh, J. F., Eisenberg, I., Guitart-Masip, M., Huys, Q., & Frank, M. J. (2013). Frontal theta overrides Pavlovian learning biases. Journal of Neuroscience, 33(19), 8541–8548. https://doi.org/10.1523/JNEUROSCI.5754-12.2013
doi: 10.1523/JNEUROSCI.5754-12.2013
pubmed: 23658191
Chabris, C. F., Laibson, D., Morris, C. L., Schuldt, J. P., & Taubinsky, D. (2008). Individual laboratory-measured discount rates predict field behavior. Journal of Risk and Uncertainty, 37(2–3), 237–269. https://doi.org/10.1038/nature08365.Reconstructing
doi: 10.1038/nature08365.Reconstructing
pubmed: 19412359
pmcid: 2676104
Csifcsák, G., Melsæter, E., & Mittner, M. (2020). Intermittent absence of control during reinforcement learning interferes with Pavlovian bias in action selection. Journal of Cognitive Neuroscience, 32(4), 646–663. https://doi.org/10.1162/jocn_a_01515
doi: 10.1162/jocn_a_01515
pubmed: 31851595
Dayan, P., & Berridge, K. C. (2014). Model-based and model-free Pavlovian reward learning: Revaluation, revision, and revelation. Cognitive, Affective and Behavioral Neuroscience, 14(2), 473–492. https://doi.org/10.3758/s13415-014-0277-8
doi: 10.3758/s13415-014-0277-8
pubmed: 24647659
Dayan, P., Niv, Y., Seymour, B., & Daw, D. N. (2006). The misbehavior of value and the discipline of the will. Neural Networks, 19(8), 1153–1160. https://doi.org/10.1016/j.neunet.2006.03.002
doi: 10.1016/j.neunet.2006.03.002
pubmed: 16938432
de Leeuw, J. R., & Gilbert, R. A. (2023). jsPsych: Enabling an open-source collaborative ecosystem of behavioral experiments. Journal of Open Source Software, 8(2022), 10–13. https://doi.org/10.21105/joss.05351
Fenneman, J., Frankenhuis, W. E., & Todd, P. M. (2022). In which environments is impulsive behavior adaptive? A cross-discipline review and integration of formal models. Psychological Bulletin, 148(7–8), 555–587. https://doi.org/10.1037/bul0000375
doi: 10.1037/bul0000375
Figner, B., Knoch, D., Johnson, E. J., Krosch, A. R., Lisanby, S. H., Fehr, E., & Weber, E. U. (2010). Lateral prefrontal cortex and self-control in intertemporal choice. Nature Neuroscience, 13(5), 538–539. https://doi.org/10.1038/nn.2516
doi: 10.1038/nn.2516
pubmed: 20348919
Garbusow, M., Ebrahimi, C., Riemerschmid, C., Daldrup, L., Rothkirch, M., Chen, K., Chen, H., Belanger, M. J., Hentschel, A., Smolka, M. N., Heinz, A., Pilhatsch, M., & Rapp, M. A. (2022). Pavlovian-To-Instrumental transfer across mental disorders: A review. Neuropsychobiology, 81(5), 418–437. https://doi.org/10.1159/000525579
doi: 10.1159/000525579
pubmed: 35843212
Garofalo, S., & di Pellegrino, G. (2015). Individual differences in the influence of task-irrelevant Pavlovian cues on human behavior. Frontiers in Behavioral Neuroscience, 9, Article 163. https://doi.org/10.3389/fnbeh.2015.00163
Gladwin, T. E., & Figner, B. (2014). ‘Hot’ cognition and dual systems: Introduction, criticism and ways forward. In E. A. Wilhems & V. F. Reyna (Eds.), Neuroeconomics, Judgment and Decision Making. Psychology Press.
Gladwin, T. E., Figner, B., Crone, E. A., & Wiers, R. W. (2011). Addiction, adolescence, and the integration of control and motivation. Developmental Cognitive Neuroscience, 1(4), 364–376. https://doi.org/10.1016/j.dcn.2011.06.008
doi: 10.1016/j.dcn.2011.06.008
pubmed: 22436562
pmcid: 6987561
Grether, D. M., & Plott, C. R. (1979). Economic theory of choice and the preference reversal phenomenon. The American Economic Review, 69(4), 623–638. https://doi.org/10.1017/cbo9780511618031.006
doi: 10.1017/cbo9780511618031.006
Guitart-Masip, M., Chowdhury, R., Sharot, T., Dayan, P., Duzel, E., & Dolan, R. J. (2012a). Action controls dopaminergic enhancement of reward representations. Proceedings of the National Academy of Sciences of the United States of America, 109(19), 7511–7516. https://doi.org/10.1073/pnas.1202229109
doi: 10.1073/pnas.1202229109
pubmed: 22529363
pmcid: 3358848
Guitart-Masip, M., Fuentemilla, L., Bach, D. R., Huys, Q. J. M., Dayan, P., Dolan, R. J., & Duzel, E. (2011). Action dominates valence in anticipatory representations in the human striatum and dopaminergic midbrain. Journal of Neuroscience, 31(21), 7867–7875. https://doi.org/10.1523/JNEUROSCI.6376-10.2011
doi: 10.1523/JNEUROSCI.6376-10.2011
pubmed: 21613500
Guitart-Masip, M., Huys, Q. J. M., Fuentemilla, L., Dayan, P., Duzel, E., & Dolan, R. J. (2012b). Go and no-go learning in reward and punishment: Interactions between affect and effect. NeuroImage, 62, 154–166. https://doi.org/10.1016/j.neuroimage.2012.04.024
doi: 10.1016/j.neuroimage.2012.04.024
pubmed: 22548809
Hazy, T. E., Frank, M. J., & O’Reilly, R. C. (2006). Banishing the homunculus: Making working memory work. Neuroscience, 139(1), 105–118. https://doi.org/10.1016/j.neuroscience.2005.04.067
doi: 10.1016/j.neuroscience.2005.04.067
pubmed: 16343792
Huys, Q. J. M., Cools, R., Gölzer, M., Friedel, E., Heinz, A., Dolan, R. J., & Dayan, P. (2011). Disentangling the roles of approach, activation and valence in instrumental and pavlovian responding. PLoS Computational Biology, 7(4), Article e1002028. https://doi.org/10.1371/journal.pcbi.1002028
Huys, Q. J. M., Eshel, N., O’Nions, E., Sheridan, L., Dayan, P., & Roiser, J. P. (2012). Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Computational Biology, 8(3), e1002410. https://doi.org/10.1371/journal.pcbi.1002410
doi: 10.1371/journal.pcbi.1002410
pubmed: 22412360
pmcid: 3297555
Huys, Q. J. M., Gölzer, M., Friedel, E., Heinz, A., Cools, R., Dayan, P., & Dolan, R. J. (2016). The specificity of Pavlovian regulation is associated with recovery from depression. Psychological Medicine, 46(5), 1027–1035. https://doi.org/10.1017/S0033291715002597
doi: 10.1017/S0033291715002597
pubmed: 26841896
pmcid: 4825095
Johnson, J. G., & Busemeyer, J. R. (2005). A dynamic, stochastic, computational model of preference reversal phenomena. Psychological Review, 112(4), 841–861. https://doi.org/10.1037/0033-295X.112.4.841
doi: 10.1037/0033-295X.112.4.841
pubmed: 16262470
Kirby, K. N., Petry, N. M., & Bickel, W. K. (1999). Heroin addicts have higher discount rates for delayed rewards than non-drug-using controls. Journal of Experimental Psychology: General, 128(1), 78–87. https://doi.org/10.1037//0096-3445.128.1.78
doi: 10.1037/0096-3445.128.1.78
pubmed: 10100392
Kvam, P. D., & Busemeyer, J. R. (2020). A distributional and dynamic theory of pricing and preference. Psychological Review, 127(6), 1053–1078. https://doi.org/10.1037/rev0000215
doi: 10.1037/rev0000215
pubmed: 32463254
pmcid: 8407094
Lempert, K. M., Steinglass, J. E., Pinto, A., Kable, J. W., & Simpson, H. B. (2018). Can delay discounting deliver on the promise of RDoC? Psychological Medicine, 49(2), 190–199. https://doi.org/10.1017/S0033291718001770
doi: 10.1017/S0033291718001770
pubmed: 30070191
Lenth, R. (2019). emmeans: Estimated marginal means. (Version 1.8.1.1). [Computer software]. https://cran.r-project.org/package=emmeans . Accessed 7 Aug 2022.
Levin, M. E., Haeger, J., Ong, C. W., & Twohig, M. P. (2018). An examination of the transdiagnostic role of delay discounting in psychological inflexibility and mental health problems. Psychological Record, 68(2), 201–210. https://doi.org/10.1007/s40732-018-0281-4
doi: 10.1007/s40732-018-0281-4
Levitt, E. E., Oshri, A., Amlung, M., Ray, L. A., Sanchez-Roige, S., Palmer, A. A., & MacKillop, J. (2022). Evaluation of delay discounting as a transdiagnostic research domain criteria indicator in 1388 general community adults. Psychological Medicine https://doi.org/10.1017/S0033291721005110
Lichtenstein, S., & Slovic, P. (1971). Reversals of preference between bids and choices in gambling decisions. Journal of Experimental Psychology, 89(1), 46–55. https://doi.org/10.1037/h0031207
doi: 10.1037/h0031207
Loewenstein, G., & O’Donoghue, T. (2004). Animal Spirits: Affective and Deliberative Processes in Economic Behavior.
Luo, S., Ainslie, G., Giragosian, L., & Monterosso, J. R. (2009). Behavioral and neural evidence of incentive bias for immediate rewards relative to preference-matched delayed rewards. Journal of Neuroscience, 29(47), 14820–14827. https://doi.org/10.1523/JNEUROSCI.4261-09.2009
doi: 10.1523/JNEUROSCI.4261-09.2009
pubmed: 19940177
Mazur, J. E. (1987). An adjusting procedure for studying delayed reinforcement. In M. L. Commons, J. E. Mazur, J. A. Nevin, & H. Rachlin (Eds.), Quantitative analyses of behavior: Vol 5. The effect of delay and of intervening events on reinforcement. (pp. 55–73). Lawrence Erlbaum Associates.
Meier, S., & Sprenger, C. (2010). Present-biased preferences and credit card borrowing. American Economic Journal: Applied Economics, 2(1), 193–210. https://doi.org/10.1257/app.2.1.193
doi: 10.1257/app.2.1.193
Metcalfe, J., & Mischel, W. (1999). A hot/cool-system analysis of delay of gratification: Dynamics of willpower. Psychological Review, 106(1), 3–19. https://doi.org/10.1037/0033-295X.106.1.3
doi: 10.1037/0033-295X.106.1.3
pubmed: 10197361
Millner, A. J., Den Ouden, H. E. M., Gershman, S. J., Glenn, C. R., Kearns, J. C., Bornstein, A. M., Marx, B. P., Keane, T. M., & Nock, M. K. (2019). Suicidal thoughts and behaviors are associated with an increased decision-making bias for active responses to escape aversive states. Journal of Abnormal Psychology, 128(2), 106–118. https://doi.org/10.1037/abn0000395
doi: 10.1037/abn0000395
pubmed: 30589305
Mkrtchian, A., Aylward, J., Dayan, P., Roiser, J. P., & Robinson, O. J. (2017). Modeling avoidance in mood and anxiety disorders using reinforcement learning. Biological Psychiatry, 82(7), 532–539. https://doi.org/10.1016/j.biopsych.2017.01.017
doi: 10.1016/j.biopsych.2017.01.017
pubmed: 28343697
pmcid: 5598542
Montagnese, M., Knolle, F., Haarsma, J., Griffin, J. D., Richards, A., Vertes, P. E., Kiddle, B., Fletcher, P. C., Jones, P. B., Owen, M. J., Fonagy, P., Bullmore, E. T., Dolan, R. J., Moutoussis, M., Goodyer, I. M., & Murray, G. K. (2020). Reinforcement learning as an intermediate phenotype in psychosis? Deficits sensitive to illness stage but not associated with polygenic risk of schizophrenia in the general population. Schizophrenia Research, 222, 389–396. https://doi.org/10.1016/j.schres.2020.04.022
doi: 10.1016/j.schres.2020.04.022
pubmed: 32389614
Monterosso, J. R., Ainslle, G., Xu, J., Cordova, X., Domier, C. P., & London, E. D. (2007). Frontoparietal cortical activity of methamphetamine-dependent and comparison subjects performing a delay discounting task. Human Brain Mapping, 28(5), 383–393. https://doi.org/10.1002/hbm.20281
doi: 10.1002/hbm.20281
pubmed: 16944492
Mullen, K. M., Ardia, D., Gil, D. L., Windover, D., & Cline, J. (2011). DEoptim: An R package for global optimization by differential evolution (Version 2.2.8) [Computer Software]. Journal of Statistical Software, 40(6), 1–26. https://doi.org/10.18637/jss.v040.i06
Nord, C. L., Lawson, R. P., Huys, Q. J. M., Pilling, S., & Roiser, J. P. (2018). Depression is associated with enhanced aversive Pavlovian control over instrumental behaviour. Scientific Reports, 8(1). https://doi.org/10.1038/s41598-018-30828-5
O’Connor, D. A., Janet, R., Guigon, V., Belle, A., Vincent, B. T., Bromberg, U., Peters, J., Corgnet, B., & Dreher, J. C. (2021). Rewards that are near increase impulsive action. iScience, 24(4), 102292. https://doi.org/10.1016/j.isci.2021.102292
Palminteri, S., Wyart, V., & Koechlin, E. (2017). The importance of falsification in computational cognitive modeling. Trends in Cognitive Sciences, 21(6), 425–433. https://doi.org/10.1016/j.tics.2017.03.011
doi: 10.1016/j.tics.2017.03.011
pubmed: 28476348
Peterburs, J., Albrecht, C., & Bellebaum, C. (2022). The impact of social anxiety on feedback-based go and nogo learning. Psychological Research Psychologische Forschung, 86(1), 110–124. https://doi.org/10.1007/s00426-021-01479-5
doi: 10.1007/s00426-021-01479-5
pubmed: 33527222
R Core Team. (2022). R: A language and environment for statistical computing. (Version 4.2.1) [Computer software].
Rozental, A., Forsell, E., Svensson, A., Forsström, D., Andersson, G., & Carlbring, P. (2014). Psychometric evaluation of the Swedish version of the pure procrastination scale, the irrational procrastination scale, and the susceptibility to temptation scale in a clinical population. BMC Psychology, 2(1), 1–12. https://doi.org/10.1186/s40359-014-0054-z
doi: 10.1186/s40359-014-0054-z
Schad, D. J., Rapp, M. A., Garbusow, M., Nebe, S., Sebold, M., Obst, E., Sommer, C., Deserno, L., Rabovsky, M., Friedel, E., Romanczuk-Seiferth, N., Wittchen, H. U., Zimmermann, U. S., Walter, H., Sterzer, P., Smolka, M. N., Schlagenhauf, F., Heinz, A., Dayan, P., & Huys, Q. J. M. (2020). Dissociating neural learning signals in human sign- and goal-trackers. Nature Human Behaviour, 4(2), 201–214. https://doi.org/10.1038/s41562-019-0765-5
doi: 10.1038/s41562-019-0765-5
pubmed: 31712764
Scholten, H., Scheres, A., de Water, E., Graf, U., Granic, I., & Luijten, M. (2019). Behavioral trainings and manipulations to reduce delay discounting: A systematic review. Psychonomic Bulletin and Review, 26(6), 1803–1849. https://doi.org/10.3758/s13423-019-01629-2
doi: 10.3758/s13423-019-01629-2
pubmed: 31270766
Scholz, V., Hook, R. W., Kandroodi, M. R., Algermissen, J., Ioannidis, K., Christmas, D., Valle, S., Robbins, T. W., Grant, J. E., Chamberlain, S. R., & den Ouden, H. E. M. (2022). Cortical dopamine reduces the impact of motivational biases governing automated behaviour. Neuropsychopharmacology, 47, 1503–1512. https://doi.org/10.1038/s41386-022-01291-8
doi: 10.1038/s41386-022-01291-8
pubmed: 35260787
pmcid: 9206002
Slovic, P. (1995). The construction of preference. American Psychologist, 50(5), 364–371. https://doi.org/10.1017/CBO9780511803475.028
doi: 10.1017/CBO9780511803475.028
Steel, P. (2010). Arousal, avoidant and decisional procrastinators: Do they exist? Personality and Individual Differences, 48(8), 926–934. https://doi.org/10.1016/j.paid.2010.02.025
doi: 10.1016/j.paid.2010.02.025
Steingroever, H., Wetzels, R., & Wagenmakers, E.-J. (2014). Absolute performance of reinforcement learning models for the Iowa Gambling Task. Decision, 1, 161–183.
doi: 10.1037/dec0000005
Swart, J. C., Cook, J. L., Geurts, D. E., Frank, M. J., Cools, R., & den Ouden, H. E. M. (2017). Catecholaminergic challenge uncovers distinct Pavlovian and instrumental mechanisms of motivated (in)action. eLife, 6, e22169. https://doi.org/10.7554/eLife.22169.001
doi: 10.7554/eLife.22169.001
pubmed: 28504638
pmcid: 5432212
Swart, J. C., Frank, M. J., Määttä, J. I., Jensen, O., Cools, R., & den Ouden, H. E. M. (2018). Frontal network dynamics reflect neurocomputational mechanisms for reducing maladaptive biases in motivated action. PLoS Biology, 16(10), 1–25. https://doi.org/10.1371/journal.pbio.2005979
doi: 10.1371/journal.pbio.2005979
Tversky, B. A., Slovic, P., & Kahneman, D. (1990). The Causes of Preference Reversal., 80(1), 204–217.
van Nuland, A. J., Helmich, R. C., Dirkx, M. F., Zach, H., Toni, I., Cools, R., & Den Ouden, H. E. M. (2020). Effects of dopamine on reinforcement learning in Parkinson’s disease depend on motor phenotype. Brain. https://doi.org/10.1093/brain/awaa335
Warren, C., Mcgraw, A. P., & Van Boven, L. (2011). Values and preferences: Defining preference construction. Wiley Interdisciplinary Reviews: Cognitive Science, 2(2), 193–205. https://doi.org/10.1002/wcs.98
doi: 10.1002/wcs.98
pubmed: 26302010
Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. (Version 3.3.6) [Computer software]. New York, NY: Spinger-Verlag.
Wilson, R. C., & Collins, A. G. E. (2019). Ten simple rules for the computational modeling of behavioral data. eLife, 8. https://doi.org/10.7554/eLife.49547
Yarkoni, T. (2020). The generalizability crisis. Behavioral and Brain Sciences, 1–37. https://doi.org/10.1017/S0140525X20001685