Implicit association tests: Stimuli validation from participant responses.
implicit association test
internal validity
stimulus validation
Journal
The British journal of social psychology
ISSN: 2044-8309
Titre abrégé: Br J Soc Psychol
Pays: England
ID NLM: 8105534
Informations de publication
Date de publication:
02 Nov 2023
02 Nov 2023
Historique:
revised:
11
09
2023
received:
22
04
2021
accepted:
28
09
2023
medline:
2
11
2023
pubmed:
2
11
2023
entrez:
2
11
2023
Statut:
aheadofprint
Résumé
The Implicit Association Test (IAT, Greenwald et al., J. Pers. Soc. Psychol., 74, 1998, 1464) is a popular instrument for measuring attitudes and (stereotypical) biases. Greenwald et al. (Behav. Res. Methods, 54, 2021, 1161) proposed a concrete method for validating IAT stimuli: appropriate stimuli should be familiar and easy to classify - translating to rapid (response times <800 ms) and accurate (error < 10%) participant responses. We conducted three analyses to explore the theoretical and practical utility of these proposed validation criteria. We first applied the proposed validation criteria to the data of 15 IATs that were available via Project Implicit. A bootstrap approach with 10,000 'experiments' of 100 participants showed that 5.85% of stimuli were reliably valid (i.e., we are more than 95% confident that a stimulus will also be valid in a new sample of 18- to 25-year-old US participants). Most stimuli (78.44%) could not be reliably validated, indicating a less than 5% certainty in the outcome of stimulus (in)validity for a new sample of participants. We then explored how stimulus validity differs across IATs. Results show that only some stimuli are consistently (in)valid. Most stimuli show between-IAT variances, which indicate that stimulus validity differs across IAT contexts. In the final analysis, we explored the effect of stimulus type (images, nouns, names, adjectives) on stimulus validity. Stimulus type was a significant predictor of stimulus validity. Although images attain the highest stimulus validity, raw data show large differences within stimulus types. Together, the results indicate a need for revised validation criteria. We finish with practical recommendations for stimulus selection and (post-hoc) stimulus validation.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Subventions
Organisme : Dutch Research Council (NL)
ID : 406.DI.19.059
Informations de copyright
© 2023 The Authors. British Journal of Social Psychology published by John Wiley & Sons Ltd on behalf of British Psychological Society.
Références
Allaire, J., Xie, Y., McPherson, J., Luraschi, J., Ushey, K., Atkins, A., Wickham, H., Cheng, J., Chang, W., & Iannone, R. (2022). Rmarkdown: Dynamic documents for r. https://github.com/rstudio/rmarkdown
Aust, F., & Barth, M. (2020). papaja: Create APA manuscripts with R Markdown. https://github.com/crsh/papaja
Axt, J. R., Feng, T. Y., & Bar-Anan, Y. (2021). The good and the bad: Are some attribute words better than others in the implicit association test? Behavior Research Methods, 53, 2512-2527. https://doi.org/10.3758/s13428-021-01592-8
Babchishin, K. M., Nunes, K. L., & Hermann, C. A. (2013). The validity of implicit association test (IAT) measures of sexual attraction to children: A meta-analysis. Archives of Sexual Behavior, 42(3), 487-499. https://doi.org/10.1007/s10508-012-0022-8
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://doi.org/10.18637/jss.v067.i01
Bickel, P. J., Götze, F., & van Zwet, W. R. (2012). Resampling fewer than n observations: Gains, losses, and remedies for losses. In S. van de Geer & M. Wegkamp (Eds.), Selected works of Willem van Zwet (pp. 267-297). Springer. https://doi.org/10.1007/978-1-4614-1314-1_17
Bluemke, M., & Friese, M. (2006). Do features of stimuli influence IAT effects? Journal of Experimental Social Psychology, 42(2), 163-176. https://doi.org/10.1016/j.jesp.2005.03.004
Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S., & Cohen, J. D. (2001). Conflict monitoring and cognitive control. Psychological Review, 108(3), 624-652. https://doi.org/10.1037/0033-295X.I08.3.624
Brendl, C. M., Markman, A. B., & Messner, C. (2001). How do indirect measures of evaluation work? Evaluating the inference of prejudice in the implicit association test. Journal of Personal and Social Psychology, 81(5), 760-773. https://doi.org/10.1037/0022-3514.81.5.760
Charlesworth, T. E. S., & Banaji, M. R. (2019). Patterns of implicit and explicit attitudes: I. Long-term change and stability from 2007 to 2016. Psychological Science, 30(2), 174-192. https://doi.org/10.1177/0956797618813087
Darling-Hammond, S., Michaels, E. K., Allen, A. M., Chae, D. H., Thomas, M. D., Nguyen, T. T., Mujahid, M. M., & Johnson, R. C. (2020). After “the China virus” went viral: Racially charged coronavirus coverage and trends in bias against Asian Americans. Health Education & Behavior, 47(6), 870-879. https://doi.org/10.1177/1090198120957949
De Houwer, J. (2001). A structural and process analysis of the implicit association test. Journal of Experimental Social Psychology, 37(6), 443-451. https://doi.org/10.1006/jesp.2000.1464
Fazio, R. H., Jackson, J. R., Dunton, B. C., & Williams, C. J. (1995). Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? Journal of Personality and Social Psychology, 69(6), 1013-1027. https://doi.org/10.1037/0022-3514.69.6.1013
Fisher, R. J. (1993). Social desirability bias and the validity of indirect questioning. Journal of Consumer Research, 20(2), 303-315. https://doi.org/10.1086/209351
Garimella, A., Banea, C., & Mihalcea, R. (2017). Demographic-aware word associations. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2285-2295. https://doi.org/10.18653/v1/D17-1242
Gast, A., & Rothermund, K. (2010). When old and frail is not the same: Dissociating category and stimulus effects in four implicit attitude measurement methods. Quarterly Journal of Experimental Psychology, 63(3), 479-498. https://doi.org/10.1080/17470210903049963
Gawronski, B. (2009). Ten frequently asked questions about implicit measures and their frequently supposed, but not entirely correct answers. Canadian Psychology/Psychologie Canadienne, 50(3), 141-150. https://doi.org/10.1037/a0013848
Greenwald, A. G., Brendl, M., Cai, H., Cvencek, D., Dovidio, J., Friese, M., Hahn, A., Hehman, E., Hofmann, W., Hughes, S., Hussey, I., Jordan, C. H., Jost, J., Kirby, T. A., Lai, C. K., Lang, J. W. B., Lindgren, K. P., Maison, D., Ostafin, B., … Wiers, R. (2020). The Implicit Association Test at age 20: What is known and what is not known about implicit bias. PsyArXiv. https://doi.org/10.31234/osf.io/bf97c
Greenwald, A. G., Brendl, M., Cai, H., Cvencek, D., Dovidio, J. F., Friese, M., Hahn, A., Hehman, E., Hofmann, W., Hughes, S., Hussey, I., Jordan, C., Kirby, T. A., Lai, C. K., Lang, J. W. B., Lindgren, K. P., Maison, D., Ostafin, B. D., Rae, J. R., … Wiers, R. W. (2021). Best research practices for using the implicit association test. Behavior Research Methods, 54, 1161-1180. https://doi.org/10.3758/s13428-021-01624-3
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74(6), 1464-1480. https://doi.org/10.1037/0022-3514.74.6.1464
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the implicit association test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85(2), 197-216. https://doi.org/10.1037/0022-3514.85.2.197
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009). Understanding and using the implicit association test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97(1), 17-41. https://doi.org/10.1037/a0015575
Grolemund, G., & Wickham, H. (2011). Dates and times made easy with lubridate. Journal of Statistical Software, 40(3), 1-25. https://www.jstatsoft.org/v40/i03/
Hehman, E., Calanchini, J., Flake, J. K., & Leitner, J. B. (2019). Establishing construct validity evidence for regional measures of explicit and implicit racial bias. Journal of Experimental Psychology: General, 148(6), 1022-1040. https://doi.org/10.1037/xge0000623
Heiphetz, L., Spelke, E. S., & Banaji, M. R. (2013). Patterns of implicit and explicit attitudes in children and adults: Tests in the domain of religion. Journal of Experimental Psychology. General, 142(3), 864-879. https://doi.org/10.1037/a0029714
Heitz, R. P. (2014). The speed-accuracy tradeoff: History, physiology, methodology, and behavior. Frontiers in Neuroscience, 8. https://www.frontiersin.org/articles/10.3389/fnins.2014.00150/full
Hester, J. (2020). Glue: Interpreted string literals. https://CRAN.R-project.org/package=glue
Hofmann, W., Gawronski, B., Gschwendner, T., Le, H., & Schmitt, M. (2005). A meta-analysis on the correlation between the implicit association test and explicit self-report measures. Personality and Social Psychology Bulletin, 31(10), 1369-1385. https://doi.org/10.1177/0146167205275613
Hope, R. M. (2013). Rmisc: Rmisc: Ryan miscellaneous. https://CRAN.R-project.org/package=Rmisc
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1-26. https://doi.org/10.18637/jss.v082.i13
MacLeod, C. M. (1991). Half a century of research on the Stroop effect: An integrative review. Psychonomic Bulletin, 109(2), 163-203. https://doi.org/10.1037/0033-2909.109.2.163
Miller, S. (2018). Mixed Effects Modeling Tips: Use a Fast Optimizer, but Perform Optimizer Checks [Blog]. In Steven V. Miller. http://svmiller.com/blog/2018/06/mixed-effects-models-optimizer-checks/
Müller, K. (2020). Here: A simpler way to find your files. https://CRAN.R-project.org/package=here
Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2005). Understanding and using the implicit association test: II. Method variables and construct validity. Personality and Social Psychology Bulletin, 31(2), 166-180. https://doi.org/10.1177/0146167204271418
Oswald, F. L., Mitchell, G., Blanton, H., Jaccard, J., & Tetlock, P. E. (2013). Predicting ethnic and racial discrimination: A meta-analysis of IAT criterion studies. Journal of Personality and Social Psychology, 105(2), 171-192. https://doi.org/10.1037/a0032734
Powell, M. J. D. (2009). The BOBYQA algorithm for bound constrained optimization without derivatives. 39.
R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
R Core Team. (2022). Foreign: Read data stored by ‘minitab’, ‘s’, ‘SAS’, ‘SPSS’, ‘stata’, ‘systat’, ‘weka’, ‘dBase’, … https://CRAN.R-project.org/package=foreign
Ratliff, K. A., & Smith, C. T. (2021, in press). Lessons from two decades of project implicit. In The Cambridge handbook of implicit bias and racism. Cambridge University Press.
Ravary, A., Baldwin, M. W., & Bartz, J. A. (2019). Shaping the body politic: Mass media fat-shaming affects implicit anti-fat attitudes. Personality and Social Psychology Bulletin, 45(11), 1580-1589. https://doi.org/10.1177/0146167219838550
Ridderinkhof, K. R., Wylie, S. A., van den Wildenberg, W. P. M., Bashore, T. R., & van der Molen, M. W. (2021). The arrow of time: Advancing insights into action control from the arrow version of the Eriksen flanker task. Attention, Perception, & Psychophysics, 83(2), 700-721. https://doi.org/10.3758/s13414-020-02167-z
Rothermund, K., Teige-Mocigemba, S., Gast, A., & Wentura, D. (2009). Minimizing the influence of recoding in the implicit association test: The recoding-free implicit association test (IAT-RF). Quarterly Journal of Experimental Psychology, 62(1), 84-98. https://doi.org/10.1080/17470210701822975
Schimmack, U. (2021). The implicit association test: A method in search of a construct. Perspectives on Psychological Science, 16(2), 396-414. https://doi.org/10.1177/1745691619863798
Steffens, M. (2005). Implicit and explicit attitudes towards lesbians and gay men. Journal of Homosexuality, 49(2), 39-66. https://doi.org/10.1300/J082v49n02_03
Steffens, M., & Plewe, I. (2001). Items' cross-category associations as a confounding factor in the implicit association test. Zeitschrift Für Experimentelle Psychologie: Organ Der Deutschen Gesellschaft Für Psychologie, 48, 123-134. https://doi.org/10.1026//0949-3946.48.2.123
Tello, N., Harika-Germaneau, G., Serra, W., Jaafari, N., & Chatard, A. (2020). Forecasting a fatal decision: Direct replication of the predictive validity of the suicide. Psychological Science, 31(1), 65-74. https://doi.org/10.1177/0956797619893062
van Ravenzwaaij, D., van der Maas, H. L. J., & Wagenmakers, E.-J. (2011). Does the name-race implicit association test measure racial prejudice? Experimental Psychology, 58(4), 271-277. https://doi.org/10.1027/1618-3169/a000093
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. https://doi.org/10.3389/fpsyg.2016.01832
Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag. https://ggplot2.tidyverse.org
Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M., Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., … Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4(43), 1686. https://doi.org/10.21105/joss.01686
Wickham, H., François, R., Henry, L., & Müller, K. (2021). Dplyr: A grammar of data manipulation. https://CRAN.R-project.org/package=dplyr
Wickham, H., & Hester, J. (2020). Readr: Read rectangular text data. https://CRAN.R-project.org/package=readr
Wickham, H., & Seidel, D. (2020). Scales: Scale functions for visualization. https://CRAN.R-project.org/package=scales
Wilke, C. O. (2020a). Cowplot: Streamlined plot theme and plot annotations for ‘ggplot2’. https://CRAN.R-project.org/package=cowplot
Wilke, C. O. (2020b). Ggtext: Improved text rendering support for ‘ggplot2’. https://CRAN.R-project.org/package=ggtext
Wolsiefer, K., Westfall, J., & Judd, C. M. (2017). Modeling stimulus variation in three common implicit attitude tasks. Behavior Research Methods, 49(4), 1193-1209. https://doi.org/10.3758/s13428-016-0779-0
Xie, Y. (2016). Bookdown: Authoring books and technical documents with R markdown. Hall/CRC. https://github.com/rstudio/bookdown
Xie, Y. (2022). Knitr: A general-purpose package for dynamic report generation in r. https://yihui.org/knitr/