Reclassifying guesses to increase signal-to-noise ratio in psychological experiments.
Accuracy
Guess
Psychological experiment
Reclassification
Signal-to-noise ratio
Journal
Behavior research methods
ISSN: 1554-3528
Titre abrégé: Behav Res Methods
Pays: United States
ID NLM: 101244316
Informations de publication
Date de publication:
Mar 2024
Mar 2024
Historique:
accepted:
02
06
2023
medline:
5
4
2024
pubmed:
10
7
2023
entrez:
10
7
2023
Statut:
ppublish
Résumé
This paper introduces a novel procedure that can increase the signal-to-noise ratio in psychological experiments that use accuracy as a selection variable for another dependent variable. This procedure relies on the fact that some correct responses result from guesses and reclassifies them as incorrect responses using a trial-by-trial reclassification evidence such as response time. It selects the optimal reclassification evidence criterion beyond which correct responses should be reclassified as incorrect responses. We show that the more difficult the task and the fewer the response alternatives, the more to be gained from this reclassification procedure. We illustrate the procedure on behavioral and ERP data from two different datasets (Caplette et al. NeuroImage 218, 116994, 2020; Faghel-Soubeyrand et al. Journal of Experimental Psychology: General 148, 1834-1841, 2019) using response time as reclassification evidence. In both cases, the reclassification procedure increased signal-to-noise ratio by more than 13%. Matlab and Python implementations of the reclassification procedure are openly available ( https://github.com/GroupeLaboGosselin/Reclassification ).
Identifiants
pubmed: 37428394
doi: 10.3758/s13428-023-02158-6
pii: 10.3758/s13428-023-02158-6
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
2452-2468Informations de copyright
© 2023. The Psychonomic Society, Inc.
Références
Berger, A., & Kiefer, M. (2021). Comparison of different response time outlier exclusion methods: A simulation study. Frontiers in Psychology, 12, 675558.
doi: 10.3389/fpsyg.2021.675558
pubmed: 34194371
pmcid: 8238084
Blais, C., Fiset, D., Roy, C., Saumure Régimbald, C., & Gosselin, F. (2017). Eye fixation patterns for categorizing static and dynamic facial expressions. Emotion, 17(7), 1107.
doi: 10.1037/emo0000283
pubmed: 28368152
Caplette, L., Ince, R. A. A., Jerbi, K., & Gosselin, F. (2020). Disentangling presentation and processing times in the brain. NeuroImage, 218, 116994.
doi: 10.1016/j.neuroimage.2020.116994
pubmed: 32474082
Chauvin, A., Worsley, K. J., Schyns, P. G., Arguin, M., & Gosselin, F. (2005). Accurate statistical tests for smooth classification images. Journal of Vision, 5(9), 1–1.
doi: 10.1167/5.9.1
Dupuis-Roy, N., Fortin, I., Fiset, D., & Gosselin, F. (2009). Uncovering gender discrimination cues in a realistic setting. Journal of Vision, 9(2), 10, 1–8.
Eckstein, M. K., Guerra-Carrillo, B., Singley, A. T. M., & Bunge, S. A. (2017). Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development? Developmental Cognitive Neuroscience, 25, 69–91.
doi: 10.1016/j.dcn.2016.11.001
pubmed: 27908561
Eriksen, C. W. (1988). A source of error in attempts to distinguish coactivation from separate activation in the perception of redundant targets. Perception & Psychophysics, 44, 191–193.
doi: 10.3758/BF03208712
Faghel-Soubeyrand, S., Dupuis-Roy, N., & Gosselin, F. (2019). Inducing the use of right eye enhances face–sex categorization performance. Journal of Experimental Psychology: General, 148(10), 1834–1841.
doi: 10.1037/xge0000542
pubmed: 30667259
Glickman, M. E., Gray, J. R., & Morales, C. J. (2005). Combining speed and accuracy to assess error-free cognitive processes. Psychometrika, 70(3), 405–425.
doi: 10.1007/s11336-002-0999-3
Gondan, M., & Heckel, A. (2008). Testing the race inequality: A simple correction procedure for fast guesses. Journal of Mathematical Psychology, 52(5), 322–325.
doi: 10.1016/j.jmp.2008.08.002
Guo, H., Rios, J. A., Haberman, S., Liu, O. L., Wang, J., & Paek, I. (2016). A new procedure for detection of students’ rapid guessing responses using response time. Applied Measurement in Education, 29, 173–183. https://doi.org/10.1080/08957347.2016.1171766
doi: 10.1080/08957347.2016.1171766
Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley.
Hautus, M. J., Macmillan, N. A., & Creelman, C. D. (2021). Detection theory: A user’s guide. Routledge.
Hollingworth, A., & Bahle, B. (2019). Eye tracking in visual search experiments. In Spatial learning and attention guidance (pp. 23–35). Humana.
Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S., & Baker, C. I. (2009). Circular analysis in systems neuroscience: The dangers of double dipping. Nature Neuroscience, 12(5), 535–540.
doi: 10.1038/nn.2303
pubmed: 19396166
pmcid: 2841687
Lee, Y. H., & Jia, Y. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-scale Assessments in Education, 2(8), 1–24. https://doi.org/10.1186/s40536-014-0008-1
doi: 10.1186/s40536-014-0008-1
Manning, C., Dakin, S. C., Tibber, M. S., & Pellicano, E. (2014). Averaging, not internal noise, limits the development of coherent motion processing. Developmental Cognitive Neuroscience, 10, 44–56.
doi: 10.1016/j.dcn.2014.07.004
pubmed: 25160679
pmcid: 4256063
Miller, J., & Lopes, A. (1991). Bias produced by fast guessing in distribution-based tests of race models. Perception & Psychophysics, 50(6), 584–590.
doi: 10.3758/BF03207544
Qian, M., Aguilar, M., Zachery, K. N., Privitera, C., Klein, S., Carney, T., & Nolte, L. W. (2009). Decision-level fusion of EEG and pupil features for single-trial visual detection analysis. IEEE Transactions on Biomedical Engineering, 56(7), 1929–1937.
doi: 10.1109/TBME.2009.2016670
pubmed: 19336285
Ratcliff, R., & Kang, I. (2021). Qualitative speed-accuracy tradeoff effects can be explained by a diffusion/fast-guess mixture model. Scientific Reports, 11(1), 15169.
Roy, Y., Banville, H., Albuquerque, I., Gramfort, A., Falk, T. H., & Faubert, J. (2019). Deep learning-based electroencephalography analysis: A systematic review. Journal of Neural Engineering, 16(5), 051001.
doi: 10.1088/1741-2552/ab260c
pubmed: 31151119
Schnipke, D. L. (1995). Assessing speededness in computer-based tests using item response times (Unpublished doctoral dissertation). Johns Hopkins University.
Sheldon, S. S., & Mathewson, K. E. (2022). To see, not to see or to see poorly: Perceptual quality and guess rate as a function of electroencephalography (EEG) brain activity in an orientation perception task. European Journal of Neuroscience, 55(11-12), 3154–3177.
Tukey, J. (1977). Exploratory data analysis. London: Pearson.
Uggeldahl, K., Jacobsen, C., Lundhede, T. H., & Olsen, S. B. (2016). Choice certainty in discrete choice experiments: Will eye tracking provide useful measures? Journal of Choice Modelling, 20, 35–48.
doi: 10.1016/j.jocm.2016.09.002
Wagner, A. D., Koutstaal, W., & Schacter, D. L. (1999). When encoding yields remembering: insights from event-related neuroimaging. Philosophical Transactions of the Royal Society B: Biological Sciences, 354, 1307–1324. https://doi.org/10.1098/rstb.1999.0481
doi: 10.1098/rstb.1999.0481
Watson, A. B., & Pelli, D. G. (1983). QUEST: A Bayesian adaptive psychometric method. Attention, Perception & Psychophysics, 33, 113–120. https://doi.org/10.3758/BF03202828
doi: 10.3758/BF03202828
Wise, S. L. (2019). An information-based approach to identifying rapid-guessing thresholds. Applied Measurement in Education, 32(4), 325–336. https://doi.org/10.1080/08957347.2019.1660350
doi: 10.1080/08957347.2019.1660350
Wise, S. L., & Ma, L. (2012). Setting response time thresholds for a CAT item pool: The normative threshold method. Paper presented at the annual meeting of the National Council on Measurement in Education, Vancouver, Canada.