Reinforcement learning of adaptive control strategies.
Journal
Communications psychology
ISSN: 2731-9121
Titre abrégé: Commun Psychol
Pays: England
ID NLM: 9918716686206676
Informations de publication
Date de publication:
12 Jan 2024
12 Jan 2024
Historique:
received:
04
05
2023
accepted:
02
01
2024
medline:
7
9
2024
pubmed:
7
9
2024
entrez:
6
9
2024
Statut:
epublish
Résumé
Humans can up- or downregulate the degree to which they rely on task information for goal-directed behaviour, a process often referred to as cognitive control. Adjustments in cognitive control are traditionally studied in response to experienced or expected task-rule conflict. However, recent theories suggest that people can also learn to adapt control settings through reinforcement. Across three preregistered task switching experiments (n = 415), we selectively rewarded correct performance on trials with either more (incongruent) or less (congruent) task-rule conflict. Results confirmed the hypothesis that people rewarded more on incongruent trials showed smaller task-rule congruency effects, thus optimally adapting their control settings to the reward scheme. Using drift diffusion modelling, we further show that this reinforcement of cognitive control may occur through conflict-dependent within-trial adjustments of response thresholds after conflict detection. Together, our findings suggest that, while people remain more efficient at learning stimulus-response associations through reinforcement, they can similarly learn cognitive control strategies through reinforcement.
Identifiants
pubmed: 39242891
doi: 10.1038/s44271-024-00055-y
pii: 10.1038/s44271-024-00055-y
doi:
Types de publication
Journal Article
Langues
eng
Pagination
8Subventions
Organisme : Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)
ID : 11C2322N
Organisme : Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)
ID : 11H5619N
Informations de copyright
© 2024. The Author(s).
Références
Diamond, A. Executive functions. Annu. Rev. Psychol. 64, 135–168 (2013).
pubmed: 23020641
doi: 10.1146/annurev-psych-113011-143750
Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S. & Cohen, J. D. Conflict monitoring and cognitive control. Psychol. Rev. 108, 624–652 (2001).
pubmed: 11488380
doi: 10.1037/0033-295X.108.3.624
Braver, T. S. The variable nature of cognitive control: a dual mechanisms framework. Trends Cogn. Sci. 16, 106–113 (2012).
pubmed: 22245618
pmcid: 3289517
doi: 10.1016/j.tics.2011.12.010
Verguts, T. & Notebaert, W. Adaptation by binding: a learning account of cognitive control. Trends Cogn. Sci. 13, 252–257 (2009).
pubmed: 19428288
doi: 10.1016/j.tics.2009.02.007
Cohen, J. D., Dunbar, K. & McClelland, J. L. On the control of automatic processes: a parallel distributed processing account of the Stroop effect. Psychol. Rev. 97, 332–361 (1990).
pubmed: 2200075
doi: 10.1037/0033-295X.97.3.332
Musslick, S. & Cohen, J. D. Rationalizing constraints on the capacity for cognitive control. Trends Cogn. Sci. 25, 757–775 (2021).
pubmed: 34332856
doi: 10.1016/j.tics.2021.06.001
Abrahamse, E., Braem, S., Notebaert, W. & Verguts, T. Grounding cognitive control in associative learning. Psychol. Bull. 142, 693–728 (2016).
pubmed: 27148628
doi: 10.1037/bul0000047
Braem, S. & Egner, T. Getting a grip on cognitive flexibility. Curr. Dir. Psychol. Sci. 27, 470–476 (2018).
pubmed: 30555214
pmcid: 6291219
doi: 10.1177/0963721418787475
Chiu, Y.-C. & Egner, T. Cortical and subcortical contributions to context-control learning. Neurosci. Biobehav. Rev. 99, 33–41 (2019).
pubmed: 30685484
pmcid: 6399056
doi: 10.1016/j.neubiorev.2019.01.019
Doebel, S. Rethinking executive function and its development. Perspect. Psychol. Sci. 15, 942–956 (2020).
pubmed: 32348707
doi: 10.1177/1745691620904771
Logan, G. D. Toward an instance theory of automatization. Psychol. Rev. 95, 492–527 (1988).
doi: 10.1037/0033-295X.95.4.492
Verbruggen, F., McLaren, I. P. L. & Chambers, C. D. Banishing the control homunculi in studies of action control and behavior change. Perspect. Psychol. Sci. 9, 497–524 (2014).
pubmed: 25419227
pmcid: 4232338
doi: 10.1177/1745691614526414
Simen, P., Cohen, J. D. & Holmes, P. Rapid decision threshold modulation by reward rate in a neural network. Neural Netw. 19, 1013–1026 (2006).
pubmed: 16987636
pmcid: 1808344
doi: 10.1016/j.neunet.2006.05.038
Neisser, U. Cognitive Psychology (Psychology Press, 2014). https://doi.org/10.4324/9781315736174 .
Logan, G. D. & Zbrodoff, N. J. When it helps to be misled: Facilitative effects of increasing the frequency of conflicting stimuli in a Stroop-like task. Mem. Cognit. 7, 166–174 (1979).
doi: 10.3758/BF03197535
Tzelgov, J., Henik, A. & Berger, J. Controlling Stroop effects by manipulating expectations for color words. Mem. Cognit. 20, 727–735 (1992).
pubmed: 1435275
doi: 10.3758/BF03202722
Braem, S. et al. Measuring adaptive control in conflict tasks. Trends Cogn. Sci. 23, 769–783 (2019).
pubmed: 31331794
pmcid: 6699878
doi: 10.1016/j.tics.2019.07.002
Bugg, J. M. & Crump, M. J. C. In support of a distinction between voluntary and stimulus-driven control: a review of the literature on proportion congruent effects. Front. Psychol. 3, 367 (2012).
pubmed: 23060836
pmcid: 3459019
doi: 10.3389/fpsyg.2012.00367
Botvinick, M. & Braver, T. Motivation and cognitive control: from behavior to neural mechanism. Annu. Rev. Psychol. 66, 83–113 (2015).
pubmed: 25251491
doi: 10.1146/annurev-psych-010814-015044
Diao, L., Li, W., Chang, W. & Ma, Q. Reward modulates unconsciously triggered adaptive control processes. IPerception 13, 204166952110738 (2022).
Etzel, J. A., Cole, M. W., Zacks, J. M., Kay, K. N. & Braver, T. S. Reward motivation enhances task coding in frontoparietal cortex. Cereb. Cortex 26, 1647–1659 (2016).
pubmed: 25601237
doi: 10.1093/cercor/bhu327
Hall-McMaster, S., Muhle-Karbe, P. S., Myers, N. E. & Stokes, M. G. Reward boosts neural coding of task rules to optimize cognitive flexibility. J. Neurosci. 39, 8549–8561 (2019).
pubmed: 31519820
pmcid: 6807286
doi: 10.1523/JNEUROSCI.0631-19.2019
Wisniewski, D., Reverberi, C., Momennejad, I., Kahnt, T. & Haynes, J.-D. The role of the parietal cortex in the representation of task-reward associations. J. Neurosci. 35, 12355–12365 (2015).
pubmed: 26354905
pmcid: 6605393
doi: 10.1523/JNEUROSCI.4882-14.2015
Schneider, D. W. Isolating a mediated route for response congruency effects in task switching. J. Exp. Psychol. Learn. Mem. Cogn. 41, 235–245 (2015).
pubmed: 25068859
doi: 10.1037/xlm0000049
Braem, S. Conditioning task switching behavior. Cognition 166, 272–276 (2017).
pubmed: 28595140
doi: 10.1016/j.cognition.2017.05.037
Ratcliff, R. & Rouder, J. N. Modeling response times for two-choice decisions. Psychol. Sci. 9, 347–356 (1998).
doi: 10.1111/1467-9280.00067
Ulrich, R., Schröter, H., Leuthold, H. & Birngruber, T. Automatic and controlled stimulus processing in conflict tasks: superimposed diffusion processes and delta functions. Cognit. Psychol. 78, 148–174 (2015).
pubmed: 25909766
doi: 10.1016/j.cogpsych.2015.02.005
de Leeuw, J. R. jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behav. Res. Methods 47, 1–12 (2015).
pubmed: 24683129
doi: 10.3758/s13428-014-0458-y
Carver, C. S. & White, T. L. Behavioral inhibition, behavioral activation, and affective responses to impending reward and punishment: The BIS/BAS Scales. J. Pers. Soc. Psychol. 67, 319–333 (1994).
doi: 10.1037/0022-3514.67.2.319
Bürkner, P.-C. brms: An R package for Bayesian multilevel models using Stan. J. Stat. Softw. 80, 1–28. (2017).
R Core Team. R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2022).
Makowski, D. & Chen, S. H. A. Indices of effect existence and significance in the bayesian framework. Front. Psychol. 10, 14 (2019).
doi: 10.3389/fpsyg.2019.02767
Bartoš F. & Maier, M. RoBMA: An R Package for Robust Bayesian Meta-Analyses. R package version 2.3.1 (2020).
Goschke, T. 14 Intentional Reconfiguration and. Control Cogn. Process. Atten. Perform. XVIII 18, 331 (2000).
Bartoš, F. et al. Bayesian model‐averaged meta‐analysis in medicine. Stat. Med. 40, 6743–6761 (2021).
pubmed: 34705280
pmcid: 9298250
doi: 10.1002/sim.9170
Gronau, Q. F. et al. A Bayesian model-averaged meta-analysis of the power pose effect with informed and default priors: the case of felt power. Compr. Results Soc. Psychol. 2, 123–138 (2017).
doi: 10.1080/23743603.2017.1326760
Gronau, Q. F., Heck, D. W., Berkhout, S. W., Haaf, J. M. & Wagenmakers, E.-J. A primer on bayesian model-averaged meta-analysis. Adv. Methods Pract. Psychol. Sci. 4, 251524592110312 (2021).
doi: 10.1177/25152459211031256
Meiran, N., Chorev, Z. & Sapir, A. Component processes in task switching. Cognit. Psychol. 41, 211–253 (2000).
pubmed: 11032657
doi: 10.1006/cogp.2000.0736
Monsell, S. Task switching. Trends Cogn. Sci. 7, 134–140 (2003).
pubmed: 12639695
doi: 10.1016/S1364-6613(03)00028-7
Ruthruff, E., Remington, R. W. & Johnston, J. C. Switching between simple cognitive tasks: the interaction of top-down and bottom-up factors. J. Exp. Psychol. Hum. Percept. Perform. 27, 1404–1419 (2001).
pubmed: 11766933
doi: 10.1037/0096-1523.27.6.1404
Eddelbuettel, D. Seamless R and C++ Integration with Rcpp (Springer New York, 2013). https://doi.org/10.1007/978-1-4614-6868-4 .
Eddelbuettel, D. & Balamuta, J. J. Extending R with C++: A Brief Introduction to Rcpp. Am. Stat. 72, 28–36 (2018).
doi: 10.1080/00031305.2017.1375990
Eddelbuettel, D. & François, R. Rcpp: Seamless R and C++ Integration. J. Stat. Softw. 40 (2011).
Bogacz, R., Brown, E., Moehlis, J., Holmes, P. & Cohen, J. D. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol. Rev. 113, 700–765 (2006).
pubmed: 17014301
doi: 10.1037/0033-295X.113.4.700
Ritz, H., Leng, X. & Shenhav, A. Cognitive control as a multivariate optimization problem. J. Cogn. Neurosci. 34, 569–591 (2022).
pubmed: 35061027
pmcid: 8939373
doi: 10.1162/jocn_a_01822
Yee, D. M., Leng, X., Shenhav, A. & Braver, T. S. Aversive motivation and cognitive control. Neurosci. Biobehav. Rev. 133, 104493 (2022).
pubmed: 34910931
doi: 10.1016/j.neubiorev.2021.12.016
Bustamante, L., Lieder, F., Musslick, S., Shenhav, A. & Cohen, J. Learning to overexert cognitive control in a stroop task. Cogn. Affect. Behav. Neurosci. 21, 453–471 (2021).
pubmed: 33409959
pmcid: 8208940
doi: 10.3758/s13415-020-00845-x
Manohar, S. G. et al. Reward pays the cost of noise reduction in motor and cognitive control. Curr. Biol. 25, 1707–1716 (2015).
pubmed: 26096975
pmcid: 4557747
doi: 10.1016/j.cub.2015.05.038
Corlazzoli, G., Desender, K. & Gevers, W. Feeling and deciding: subjective experiences rather than objective factors drive the decision to invest cognitive control. Cognition 240, 105587 (2023).
pubmed: 37597356
doi: 10.1016/j.cognition.2023.105587
Lerche, V. & Voss, A. Retest reliability of the parameters of the Ratcliff diffusion model. Psychol. Res. 81, 629–652 (2017).
pubmed: 27107855
doi: 10.1007/s00426-016-0770-5
Voss, A., Voss, J. & Lerche, V. Assessing cognitive processes with diffusion model analyses: a tutorial based on fast-dm-30. Front. Psychol. 6, 336 (2015).
pubmed: 25870575
pmcid: 4376117
doi: 10.3389/fpsyg.2015.00336
Churchland, A. K., Kiani, R. & Shadlen, M. N. Decision-making with multiple alternatives. Nat. Neurosci. 11, 693–702 (2008).
pubmed: 18488024
pmcid: 2453226
doi: 10.1038/nn.2123
Tajima, S., Drugowitsch, J., Patel, N. & Pouget, A. Optimal policy for multi-alternative decisions. Nat. Neurosci. 22, 1503–1511 (2019).
pubmed: 31384015
doi: 10.1038/s41593-019-0453-9
Tajima, S., Drugowitsch, J. & Pouget, A. Optimal policy for value-based decision-making. Nat. Commun. 7, 12400 (2016).
pubmed: 27535638
pmcid: 4992126
doi: 10.1038/ncomms12400
Cavanagh, J. F. et al. Subthalamic nucleus stimulation reverses mediofrontal influence over decision threshold. Nat. Neurosci. 14, 1462–1467 (2011).
pubmed: 21946325
pmcid: 3394226
doi: 10.1038/nn.2925
Fontanesi, L., Gluth, S., Spektor, M. S. & Rieskamp, J. A reinforcement learning diffusion decision model for value-based decisions. Psychon. Bull. Rev. 26, 1099–1121 (2019).
pubmed: 30924057
pmcid: 6820465
doi: 10.3758/s13423-018-1554-2
Frömer, R. & Shenhav, A. Filling the gaps: cognitive control as a critical lens for understanding mechanisms of value-based decision-making. Neurosci. Biobehav. Rev. 134, 104483 (2022).
pubmed: 34902441
doi: 10.1016/j.neubiorev.2021.12.006
Ratcliff, R. & Frank, M. J. Reinforcement-based decision making in corticostriatal circuits: mutual constraints by neurocomputational and diffusion models. Neural Comput. 24, 1186–1229 (2012).
pubmed: 22295983
doi: 10.1162/NECO_a_00270
Braem, S., Held, L., Shenhav, A. & Frömer, R. Learning how to reason and deciding when to decide. Behav. Brain Sci. 46, e115 (2023).
pubmed: 37462203
pmcid: 10597599
doi: 10.1017/S0140525X22003090
Bijleveld, E., Custers, R. & Aarts, H. Unconscious reward cues increase invested effort, but do not change speed–accuracy tradeoffs. Cognition 115, 330–335 (2010).
pubmed: 20089247
doi: 10.1016/j.cognition.2009.12.012
Krebs, R. M., Boehler, C. N. & Woldorff, M. G. The influence of reward associations on conflict processing in the Stroop task. Cognition 117, 341–347 (2010).
pubmed: 20864094
pmcid: 2967668
doi: 10.1016/j.cognition.2010.08.018
Armbruster, D. J. N., Ueltzhöffer, K., Basten, U. & Fiebach, C. J. Prefrontal cortical mechanisms underlying individual differences in cognitive flexibility and stability. J. Cogn. Neurosci. 24, 2385–2399 (2012).
pubmed: 22905818
doi: 10.1162/jocn_a_00286
Thorndike, E. L. The law of effect. Am. J. Psychol. 39, 212–222 (1927).
doi: 10.2307/1415413
Chen, J., Tan, L., Liu, L. & Wang, L. Reinforcement learning of irrelevant stimulus-response associations modulates cognitive control. J. Exp. Psychol. Learn. Mem. Cogn. 47, 1585–1598 (2021).
pubmed: 32324022
doi: 10.1037/xlm0000850
Mittelstädt, V., Ulrich, R., König, J., Hofbauer, K. & Mackenzie, I. G. The influence of reward in the Simon task: differences and similarities to the Stroop and Eriksen flanker tasks. Atten. Percept. Psychophys. 85, 949–959 (2023).
pubmed: 36316615
doi: 10.3758/s13414-022-02563-7
Prével, A., Krebs, R. M., Kukkonen, N. & Braem, S. Selective reinforcement of conflict processing in the Stroop task. PLoS ONE 16, e0255430 (2021).
pubmed: 34329341
pmcid: 8323904
doi: 10.1371/journal.pone.0255430
Yang, Q., Xing, J., Braem, S. & Pourtois, G. The selective use of punishments on congruent versus incongruent trials in the Stroop task. Neurobiol. Learn. Mem. 193, 107654 (2022).
pubmed: 35777632
doi: 10.1016/j.nlm.2022.107654
Bugg, J. M. Conflict-triggered top-down control: default mode, last resort, or no such thing? J. Exp. Psychol. Learn. Mem. Cogn. 40, 567–587 (2014).
pubmed: 24274385
doi: 10.1037/a0035032
Vallacher, R. R. & Wegner, D. M. What do people think they’re doing? Action identification and human behavior. Psychol. Rev. 94, 3–15 (1987).
doi: 10.1037/0033-295X.94.1.3
Braem, S., Chai, M., Held, L. K., & Xu, S. One cannot simply “be flexible”: Regulating control parameters requires learning. Current Opinion in Behavioral Sciences, in press. (2024).
Bogacz, R., Hu, P. T., Holmes, P. J. & Cohen, J. D. Do humans produce the speed–accuracy trade-off that maximizes reward rate? Q. J. Exp. Psychol. 63, 863–891 (2010).
doi: 10.1080/17470210903091643
Otto, A. R., Braem, S., Silvetti, M. & Vassena, E. Is the juice worth the squeeze? Learning the marginal value of mental effort over time. J. Exp. Psychol. Gen. 151, 2324–2341 (2022).
pubmed: 35389742
doi: 10.1037/xge0001208
Shenhav, A., Botvinick, M. M. & Cohen, J. D. The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron 79, 217–240 (2013).
pubmed: 23889930
pmcid: 3767969
doi: 10.1016/j.neuron.2013.07.007
Botvinick, M. et al. Reinforcement learning, fast and slow. Trends Cogn. Sci. 23, 408–422 (2019).
pubmed: 31003893
doi: 10.1016/j.tics.2019.02.006
Kool, W. & Botvinick, M. The intrinsic cost of cognitive control. Behav. Brain Sci. 36, 697–698 (2013).
pubmed: 24304795
doi: 10.1017/S0140525X1300109X
Held, L., Vermeylen, L., Dignath, D. & Braem, S. Selective reinforcement of conflict processing in task switching https://doi.org/10.17605/OSF.IO/QDK5T (2021).
Allen, M., Poggiali, D., Whitaker, K., Marshall, T. R. & Kievit, R. A. Raincloud plots: a multi-platform tool for robust data visualization. Wellcome Open Res. 4, 63 (2019).
pubmed: 31069261
doi: 10.12688/wellcomeopenres.15191.1