Measuring Cognitive Abilities in the Wild: Validating a Population-Scale Game-Based Cognitive Assessment.
Big data
Cognitive abilities
Crowdsourcing
Gamification
Stealth assessment
Journal
Cognitive science
ISSN: 1551-6709
Titre abrégé: Cogn Sci
Pays: United States
ID NLM: 7708195
Informations de publication
Date de publication:
Jun 2023
Jun 2023
Historique:
revised:
26
04
2023
received:
21
03
2022
accepted:
05
06
2023
medline:
26
6
2023
pubmed:
24
6
2023
entrez:
24
6
2023
Statut:
ppublish
Résumé
Rapid individual cognitive phenotyping holds the potential to revolutionize domains as wide-ranging as personalized learning, employment practices, and precision psychiatry. Going beyond limitations imposed by traditional lab-based experiments, new efforts have been underway toward greater ecological validity and participant diversity to capture the full range of individual differences in cognitive abilities and behaviors across the general population. Building on this, we developed Skill Lab, a novel game-based tool that simultaneously assesses a broad suite of cognitive abilities while providing an engaging narrative. Skill Lab consists of six mini-games as well as 14 established cognitive ability tasks. Using a popular citizen science platform (N = 10,725), we conducted a comprehensive validation in the wild of a game-based cognitive assessment suite. Based on the game and validation task data, we constructed reliable models to simultaneously predict eight cognitive abilities based on the users' in-game behavior. Follow-up validation tests revealed that the models can discriminate nuances contained within each separate cognitive ability as well as capture a shared main factor of generalized cognitive ability. Our game-based measures are five times faster to complete than the equivalent task-based measures and replicate previous findings on the decline of certain cognitive abilities with age in our large cross-sectional population sample (N = 6369). Taken together, our results demonstrate the feasibility of rapid in-the-wild systematic assessment of cognitive abilities as a promising first step toward population-scale benchmarking and individualized mental health diagnostics.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
e13308Subventions
Organisme : ERC
ID : H2020
Organisme : ERC
ID : 639560
Organisme : Templeton, Synakos
Organisme : Carlsberg Foundations
Informations de copyright
© 2023 The Authors. Cognitive Science published by Wiley Periodicals LLC on behalf of Cognitive Science Society (CSS).
Références
Baniqued, P. L., Lee, H., Voss, M. W., Basak, C., Cosman, J. D., DeSouza, S., Severson, J., Salthouse, T. A., & Kramer, A. F. (2013). Selling points: What cognitive abilities are tapped by casual video games? Acta Psychologica, 142(1), 74-86. https://doi.org/10.1016/j.actpsy.2012.11.009
Barkley, R. A. (2001). The executive functions and self-regulation: An evolutionary neuropsychological perspective. Neuropsychology Review, 11(1), 1-29.
Bauer, P. J. (2020). Expanding the reach of psychological science. Psychological Science, 31(1), 3-5. https://doi.org/10.1177/0956797619898664
Baune, B. T., Malhi, G. S., Morris, G., Outhred, T., Hamilton, A., Das, P., Bassett, D., Berk, M., Boyce, P., Lyndon, B., Mulder, R., Parker, G., & Singh, A. B. (2018). Cognition in depression: Can we THINC-it better? Journal of Affective Disorders, 225, 559-562. https://doi.org/10.1016/j.jad.2017.08.080
Beaujean, A. A., & Benson, N. F. (2019). Theoretically-consistent cognitive ability test development and score interpretation. Contemporary School Psychology, 23(2), 126-137. https://doi.org/10.1007/s40688-018-0182-1
Biggs, J., & Madnani, N. (2022). Introduction-Factor_analyzer 0.4.0 documentation. API Documentation. Retrieved from https://factor-analyzer.readthedocs.io/en/latest/factor_analyzer.html#module-factor_analyzer.factor_analyzer
Birnbaum, M. H. (2004). Human research and data collection via the internet. Annual Review of Psychology, 55(1), 803-832. https://doi.org/10.1146/annurev.psych.55.090902.141601
Bollen, K. A., & Bauldry, S. (2011). Three Cs in measurement models: Causal indicators, composite indicators, and covariates. Psychological Methods, 16(3), 265-284. PubMed. https://doi.org/10.1037/a0024448
Brown, H. R., Zeidman, P., Smittenaar, P., Adams, R. A., McNab, F., Rutledge, R. B., & Dolan, R. J. (2014). Crowdsourcing for cognitive science - The utility of smartphones. PLoS ONE, 9(7), e100662. https://doi.org/10.1371/journal.pone.0100662
Brown, L. A., Forbes, D., & McConnell, J. (2006). Limiting the use of verbal coding in the visual patterns test. Quarterly Journal of Experimental Psychology, 59(7), 1169-1176. https://doi.org/10.1080/17470210600665954
Burman, P. (1989). A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods. Biometrika, 76(3), 503-514. https://doi.org/10.1093/biomet/76.3.503
Buuren, S. van, & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in r. Journal of Statistical Software, 45(3), 1-67. https://doi.org/10.18637/jss.v045.i03
Carlson, S. M., Moses, L. J., & Claxton, L. J. (2004). Individual differences in executive functioning and theory of mind: An investigation of inhibitory control and planning ability. Journal of Experimental Child Psychology, 87(4), 299-319. https://doi.org/10.1016/j.jecp.2004.01.002
Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge University Press.
Castro-Alonso, J. C., & Atit, K. (2019). Different abilities controlled by visuospatial processing. In J. C. Castro-Alonso (Ed.), Visuospatial processing for education in health and natural sciences (pp. 23-51). Springer International Publishing. https://doi.org/10.1007/978-3-030-20969-8_2
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed). L. Erlbaum Associates.
Collins, D. W., & Kimura, D. (1997). A large sex difference on a two-dimensional mental rotation task. Behavioral Neuroscience, 111(4), 845. https://doi.org/10.1037/0735-7044.111.4.845
Coughlan, G., Coutrot, A., Khondoker, M., Minihane, A.-M., Spiers, H., & Hornberger, M. (2019). Toward personalized cognitive diagnostics of at-genetic-risk Alzheimer's disease. Proceedings of the National Academy of Sciences, 116(19), 9285-9292. https://doi.org/10.1073/pnas.1901600116
Coutrot, A., Silva, R., Manley, E., de Cothi, W., Sami, S., Bohbot, V. D., Wiener, J. M., Hölscher, C., Dalton, R. C., Hornberger, M., & Spiers, H. J. (2018). Global determinants of navigation ability. Current Biology, 28(17), 2861-2866e4. https://doi.org/10.1016/j.cub.2018.06.009
Danmarks Statistik. (2020). Befolkningspyramide, Retrieved from http://extranet.dst.dk/pyramide/pyramide.htm#!y=2018&v=2 [Data set]. http://extranet.dst.dk/pyramide/pyramide.htm#!y=2018&v=2
Davelaar, E. J., & Stevens, J. (2009). Sequential dependencies in the Eriksen flanker task: A direct comparison of two competing accounts. Psychonomic Bulletin & Review, 16(1), 121-126. https://doi.org/10.3758/PBR.16.1.121
Deary, I. J. (2011). Intelligence. Annual Review of Psychology, 63(1), 453-482. https://doi.org/10.1146/annurev-psych-120710-100353
Deary, I. J., Liewald, D., & Nissan, J. (2011). A free, easy-to-use, computer-based simple and four-choice reaction time programme: The Deary-Liewald reaction time task. Behavior Research Methods, 43(1), 258-268. https://doi.org/10.3758/s13428-010-0024-1
Dockery, C. A., Hueckel-Weng, R., Birbaumer, N., & Plewnia, C. (2009). Enhancement of planning ability by transcranial direct current stimulation. Journal of Neuroscience, 29(22), 7271-7277. https://doi.org/10.1523/JNEUROSCI.0065-09.2009
Drucker, S. M., Fisher, D., Sadana, R., Herron, J., & Schraefel, M. C. (2013). TouchViz: A case study comparing two interfaces for data analytics on tablets. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2301-2310. https://doi.org/10.1145/2470654.2481318
Fellows, R. P., Dahmen, J., Cook, D., & Schmitter-Edgecombe, M. (2017). Multicomponent analysis of a digital trail making test. Clinical Neuropsychologist, 31(1), 154-167. https://doi.org/10.1080/13854046.2016.1238510
Ganis, G., & Kievit, R. (2015). A new set of three-dimensional shapes for investigating mental rotation processes: Validation data and stimulus set. Journal of Open Psychology Data, 3(1), e3. https://doi.org/10.5334/jopd.ai
Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012). Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review, 19(5), 847-857. https://doi.org/10.3758/s13423-012-0296-9
Geurts, H. M., Corbett, B., & Solomon, M. (2009). The paradox of cognitive flexibility in autism. Trends in Cognitive Sciences, 13(2), 74-82. https://doi.org/10.1016/j.tics.2008.11.006
Groves, S. J., Douglas, K. M., & Porter, R. J. (2018). A systematic review of cognitive predictors of treatment outcome in major depression. Frontiers in Psychiatry, 9. https://doi.org/10.3389/fpsyt.2018.00382
Guest, O., & Martin, A. E. (2021). How computational modeling can force theory building in psychological science. Perspectives on Psychological Science, 16(4), 789-802.
Hagler, S., Jimison, H. B., & Pavel, M. (2014). Assessing executive function using a computer game: Computational modeling of cognitive processes. IEEE Journal of Biomedical and Health Informatics, 18(4), 1442-1452. https://doi.org/10.1109/JBHI.2014.2299793
Harnad, S. (1987). Psychophysical and cognitive aspects of categorical perception: A critical overview. In S. Harnad (Ed.), Categorical perception: The groundwork of cognition (pp. 1-52). Cambridge University Press. Retrieved from http://cogprints.org/1571/
Hawkins, G. E., Rae, B., Nesbitt, K. V., & Brown, S. D. (2013). Gamelike features might not improve data. Behavior Research Methods, 45(2), 301-318. https://doi.org/10.3758/s13428-012-0264-3
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3), 61-83. https://doi.org/10.1017/S0140525x0999152X
Hoerl, A., & Kennard, R. (1988). Ridge regression. In Encyclopedia of Statistical Sciences (Vol. 8, pp. 129-136). Wiley.
Hunt, L. T., Rutledge, R. B., Malalasekera, W. M. N., Kennerley, S. W., & Dolan, R. J. (2016). Approach-induced biases in human information sampling. PLoS Biology, 14(11), e2000638. https://doi.org/10.1371/journal.pbio.2000638
Jacobs, G. H. (1993). The distribution and nature of colour vision among the mammals. Biological Reviews, 68(3), 413-471. https://doi.org/10.1111/j.1469-185X.1993.tb00738.x
Jaeger, J., Berns, S., Uzelac, S., & Davis-Conway, S. (2006). Neurocognitive deficits and disability in major depressive disorder. Psychiatry Research, 145(1), 39-48. https://doi.org/10.1016/j.psychres.2005.11.011
Jennett, C., Furniss, D. J., Iacovides, I., Wiseman, S., Gould, S. J. J., & Cox, A. L. (2014). Exploring citizen psych-science and the motivations of errordiary volunteers. Human Computation, 1(2), 201-220, https://doi.org/10.15346/hc.v1i2.10
Jensen, A. R. (1998). The g factor: The science of mental ability. Praeger.
Kaller, C., Unterrainer, J., Kaiser, S., Weisbrod, M., Aschenbrenner, S., & Debelak, R. (2011). Manual. Tower of London-Freiburg Version. Vienna test system.
Kessels, R. P. C., van Zandvoort, M. J. E., Postma, A., Kappelle, L. J., & de Haan, E. H. F. (2000). The Corsi Block-Tapping Task: Standardization and normative data. Applied Neuropsychology, 7(4), 252-258. https://doi.org/10.1207/S15324826AN0704_8
Knopik, V. S., Neiderhiser, J. M., DeFries, J. C., & Plomin, R. (2017). Behavioral genetics (7th edition). Worth Publishers, Macmillan Learning.
Krikorian, R., Bartok, J., & Gay, N. (1994). Tower of London procedure: A standard method and developmental data. Journal of Clinical and Experimental Neuropsychology, 16(6), 840-850. https://doi.org/10.1080/01688639408402697
Leduc-McNiven, K., White, B., Zheng, H., McLeod, R. D., & Friesen, M. R. (2018). Serious games to assess mild cognitive impairment: ‘The game is the assessment’. Research and Review Insights, 2(1). https://doi.org/10.15761/RRI.1000128
Lee, H.-J., Yost, B. P., & Telch, M. J. (2009). Differential performance on the go/no-go task as a function of the autogenous-reactive taxonomy of obsessions: Findings from a non-treatment seeking sample. Behaviour Research and Therapy, 47(4), 294-300. https://doi.org/10.1016/j.brat.2009.01.002
Lezak, M. D., Howieson, D. B., Bigler, E. D., & Tranel, D. (2012). Neuropsychological assessment. Oxford University Press.
Lieberoth, A., Pedersen, M. K., Marin, A. C., & Sherson, J. F. (2014). Getting humans to do quantum optimization-User acquisition, engagement and early results from the citizen cyberscience game Quantum Moves. Human Computation, 1(2), 221-249. https://doi.org/10.15346/hc.v1i2.11
Lindenberger, U. (2014). Human cognitive aging: Corriger la fortune? Science, 346(6209), 572-578. https://doi.org/10.1126/science.1254403
Logan, G. D. (1985). Executive control of thought and action. Acta Psychologica, 60(2), 193-210. https://doi.org/10.1016/0001-6918(85)90055-1
Lu, J. G., Akinola, M., & Mason, M. F. (2017). “Switching On” creativity: Task switching can increase creativity by reducing cognitive fixation. Organizational Behavior and Human Decision Processes, 139, 63-75. https://doi.org/10.1016/j.obhdp.2017.01.005
Lumsden, J., Edwards, E. A., Lawrence, N. S., Coyle, D., & Munafò, M. R. (2016). Gamification of cognitive assessment and cognitive training: A systematic review of applications and efficacy. JMIR Serious Games, 4(2), e11. https://doi.org/10.2196/games.5888
Lumsden, J., Skinner, A., Woods, A. T., Lawrence, N. S., & Munafò, M. (2016). The effects of gamelike features and test location on cognitive test performance and participant enjoyment. PeerJ, 4, e2184. https://doi.org/10.7717/peerj.2184
Mackintosh, N. J. (1998). IQ and human intelligence. Oxford University Press.
Mayo, D. G. (2018). Statistical inference as severe testing. Cambridge: Cambridge University Press.
McNab, F., & Dolan, R. J. (2014). Dissociating distractor-filtering at encoding and during maintenance. Journal of Experimental Psychology: Human Perception and Performance, 40(3), 960-967. https://doi.org/10.1037/a0036013
McNab, F., Zeidman, P., Rutledge, R. B., Smittenaar, P., Brown, H. R., Adams, R. A., & Dolan, R. J. (2015). Age-related changes in working memory and the ability to ignore distraction. Proceedings of the National Academy of Sciences, 112(20), 6515-6518. https://doi.org/10.1073/pnas.1504162112
Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1), i-29. https://doi.org/10.1002/j.2333-8504.2003.tb01908.x
Monsell, S. (2003). Task switching. Trends in Cognitive Sciences, 7(3), 134-140. https://doi.org/10.1016/S1364-6613(03)00028-7
Muender, T., Gulani, S. A., Westendorf, L., Verish, C., Malaka, R., Shaer, O., & Cooper, S. (2019). Comparison of mouse and multi-touch for protein structure manipulation in a citizen science game interface. Journal of Science Communication, 18(1), A05. https://doi.org/10.22323/2.18010205
Newell, A. (1966). On the analysis of human problem solving protocols [Microform]. Distributed by ERIC Clearinghouse.
Newell, A., & Simon, H. A. (1972). Human problem solving. Prentice-Hall.
Papp, K. V., Snyder, P. J., Maruff, P., Bartkowiak, J., & Pietrzak, R. H. (2011). Detecting subtle changes in visuospatial executive function and learning in the amnestic variant of mild cognitive impairment. PLoS ONE, 6(7), e21688. https://doi.org/10.1371/journal.pone.0021688
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(85), 2825-2830.
Radford, J., Pilny, A., Reichelmann, A., Keegan, B., Welles, B. F., Hoye, J., Ognyanova, K., Meleis, W., & Lazer, D. (2016). Volunteer science: An online laboratory for experiments in social psychology. Social Psychology Quarterly, 79(4), 376-396. https://doi.org/10.1177/0190272516675866
Reinecke, K., & Gajos, K. Z. (2015). Labinthewild: Conducting large-scale online experiments with uncompensated samples. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW ’15, 1364-1378. https://doi.org/10.1145/2675133.2675246
Rodd, J. M., Vitello, S., Woollams, A. M., & Adank, P. (2015). Localising semantic and syntactic processing in spoken and written language comprehension: An activation likelihood estimation meta-analysis. Brain and Language, 141, 89-102. https://doi.org/10.1016/j.bandl.2014.11.012
Rutledge, R. B., Skandali, N., Dayan, P., & Dolan, R. J. (2014). A computational and neural model of momentary subjective well-being. Proceedings of the National Academy of Sciences, 111(33), 12252-12257. https://doi.org/10.1073/pnas.1407535111
Rutledge, R. B., Smittenaar, P., Zeidman, P., Brown, H. R., Adams, R. A., Lindenberger, U., Dayan, P., & Dolan, R. J. (2016). Risk taking for potential reward decreases across the lifespan. Current Biology, 26(12), 1634-1639. https://doi.org/10.1016/j.cub.2016.05.017
Sagarra, O., Gutiérrez-Roig, M., Bonhoure, I., & Perelló, J. (2016). Citizen science practices for computational social science research: The conceptualization of pop-up experiments. Frontiers in Physics, 3. https://doi.org/10.3389/fphy.2015.00093
Salthouse, T. A. (2011). What cognitive abilities are involved in trail-making performance? Intelligence, 39(4), 222-232.
Salthouse, T. A. (2019). Trajectories of normal cognitive aging. Psychology and Aging, 34(1), 17-24. https://doi.org/10.1037/pag0000288
Schmuckler, M. A. (2001). What is ecological validity? A dimensional analysis. Infancy, 2(4), 419-436. https://doi.org/10.1207/S15327078IN0204_02
Shute, V. J., Wang, L., Greiff, S., Zhao, W., & Moore, G. (2016). Measuring problem solving skills via stealth assessment in an engaging video game. Computers in Human Behavior, 63, 106-117. https://doi.org/10.1016/j.chb.2016.05.047
Sklearn.linear_model.ElasticNetCV. (2022). Scikit-Learn. Retrieved from https://scikit-learn/stable/modules/generated/sklearn.linear_model.ElasticNetCV.html
Smittenaar, P., Rutledge, R. B., Zeidman, P., Adams, R. A., Brown, H., Lewis, G., & Dolan, R. J. (2015). Proactive and reactive response inhibition across the lifespan. PLoS ONE, 10(10), e0140383. https://doi.org/10.1371/journal.pone.0140383
Teki, S., Kumar, S., & Griffiths, T. D. (2016). Large-scale analysis of auditory segregation behavior crowdsourced via a smartphone app. PLoS ONE, 11(4), e0153916. https://doi.org/10.1371/journal.pone.0153916
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267-288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
Treisman, A. (1977). Focused attention in the perception and retrieval of multidimensional stimuli. Perception & Psychophysics, 22(1), 1-11. https://doi.org/10.3758/BF03206074
Turkyılmaz, M. D., & Belgin, E. (2012). Reliability, validity, and adaptation of computerized revised token test in normal subjects. Journal of International Advanced Otology, 8, 103-112.
Ullman, S. (2000). High-level vision: Object recognition and visual cognition. MIT Press.
Valladares-Rodríguez, S., Pérez-Rodríguez, R., Anido-Rifón, L., & Fernández-Iglesias, M. (2016). Trends on the application of serious games to neuropsychological evaluation: A scoping review. Journal of Biomedical Informatics, 64, 296-319. https://doi.org/10.1016/j.jbi.2016.10.019
Verbruggen, F., & Logan, G. D. (2008). Automatic and controlled response inhibition: Associative learning in the go/no-go and stop-signal paradigms. Journal of Experimental Psychology: General, 137(4), 649-672. https://doi.org/10.1037/a0013170
Watson, D., Hancock, M., Mandryk, R. L., & Birk, M. (2013). Deconstructing the touch experience. Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces, 199-208. https://doi.org/10.1145/2512349.2512819
Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004), 686-688. https://doi.org/10.1126/science.1193147
Xue, J., Li, C., Quan, C., Lu, Y., Yue, J., & Zhang, C. (2017). Uncovering the cognitive processes underlying mental rotation: An eye-movement study. Scientific Reports, 7(1), Article 1. https://doi.org/10.1038/s41598-017-10683-6
Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), 301-320. https://doi.org/10.1111/j.1467-9868.2005.00503.x
Zysset, S., Müller, K., Lohmann, G., & von Cramon, D. Y. (2001). Color-word matching Stroop task: Separating interference and response conflict. NeuroImage, 13(1), 29-36. https://doi.org/10.1006/nimg.2000.0665