Reliable but multi-dimensional cognitive demand in operating partially automated vehicles: implications for real-world automation research.
Automation
Cognitive demand
Performance
Psychophysiology
Real-world
Journal
Cognitive research: principles and implications
ISSN: 2365-7464
Titre abrégé: Cogn Res Princ Implic
Pays: England
ID NLM: 101697632
Informations de publication
Date de publication:
11 Sep 2024
11 Sep 2024
Historique:
received:
27
12
2023
accepted:
23
08
2024
medline:
11
9
2024
pubmed:
11
9
2024
entrez:
10
9
2024
Statut:
epublish
Résumé
The reliability of cognitive demand measures in controlled laboratory settings is well-documented; however, limited research has directly established their stability under real-life and high-stakes conditions, such as operating automated technology on actual highways. Partially automated vehicles have advanced to become an everyday mode of transportation, and research on driving these advanced vehicles requires reliable tools for evaluating the cognitive demand on motorists to sustain optimal engagement in the driving process. This study examined the reliability of five cognitive demand measures, while participants operated partially automated vehicles on real roads across four occasions. Seventy-one participants (aged 18-64 years) drove on actual highways while their heart rate, heart rate variability, electroencephalogram (EEG) alpha power, and behavioral performance on the Detection Response Task were measured simultaneously. Findings revealed that EEG alpha power had excellent test-retest reliability, heart rate and its variability were good, and Detection Response Task reaction time and hit-rate had moderate reliabilities. Thus, the current study addresses concerns regarding the reliability of these measures in assessing cognitive demand in real-world automation research, as acceptable test-retest reliabilities were found across all measures for drivers across occasions. Despite the high reliability of each measure, low intercorrelations among measures were observed, and internal consistency was better when cognitive demand was estimated as a multi-factorial construct. This suggests that they tap into different aspects of cognitive demand while operating automation in real life. The findings highlight that a combination of psychophysiological and behavioral methods can reliably capture multi-faceted cognitive demand in real-world automation research.
Identifiants
pubmed: 39256243
doi: 10.1186/s41235-024-00591-5
pii: 10.1186/s41235-024-00591-5
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
60Informations de copyright
© 2024. The Author(s).
Références
Alhanbali, S., Dawes, P., Millman, R. E., & Munro, K. J. (2019). Measures of listening effort are multi-dimensional. Ear and Hearing, 40(5), 1084.
doi: 10.1097/AUD.0000000000000697
pubmed: 30747742
Bazilinskyy, P., Kyriakidis, M., Dodou, D., & de Winter, J. (2019). When will most cars be able to drive fully automatically? Projections of 18,970 survey respondents. Transportation Research Part F: Traffic Psychology and Behaviour, 64, 184–195.
doi: 10.1016/j.trf.2019.05.008
Bergen, B., Medeiros-Ward, N., Wheeler, K., Drews, F., & Strayer, D. L. (2013). The crosstalk hypothesis: Language interferes with driving because of modality-specific mental simulation. Journal of Experimental Psychology. General, 142, 119–130.
doi: 10.1037/a0028428
pubmed: 22612769
Berntson, G. G., Quigley, K. S., & Lozano, D. (2007). Cardiovascular psychophysiology. Handbook of Psychophysiology, 3, 182–210. https://doi.org/10.1017/CBO9780511546396.008
doi: 10.1017/CBO9780511546396.008
Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., & Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience and Biobehavioral Reviews, 44, 58–75. https://doi.org/10.1016/j.neubiorev.2012.10.003
doi: 10.1016/j.neubiorev.2012.10.003
pubmed: 23116991
Castro, S. C., Strayer, D. L., Matzke, D., & Heathcote, A. (2019). Cognitive workload measurement and modeling under divided attention. Journal of Experimental Psychology: Human Perception and Performance, 45(6), 826.
pubmed: 30998070
Cohen, M. X. (2014). Fluctuations in oscillation frequency control spike timing and coordinate neural networks. Journal of Neuroscience, 34(27), 8988–8998.
doi: 10.1523/JNEUROSCI.0261-14.2014
pubmed: 24990919
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. https://doi.org/10.1007/BF02310555
doi: 10.1007/BF02310555
Delorme, A., & Makeig, S. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134, 9–21. https://doi.org/10.1016/j.jneumeth.2003.10.009
doi: 10.1016/j.jneumeth.2003.10.009
pubmed: 15102499
de Waard, D., & Lewis-Evans, B. (2014). Self-report scales alone cannot capture mental workload: A reply to De Winter, Controversy in human factors constructs and the explosive use of the NASA TLX: A measurement perspective. Cognition, Technology & Work, 16, 303–305.
doi: 10.1007/s10111-014-0277-z
Donoghue, T., Schaworonkow, N., & Voytek, B. (2022). Methodological considerations for studying neural oscillations. European Journal of Neuroscience, 55(11–12), 3502–3527.
doi: 10.1111/ejn.15361
pubmed: 34268825
Engström, J., Johansson, E., & Östlund, J. (2005). Effects of visual and cognitive load in real and simulated motorway driving. Transportation Research Part F: Traffic Psychology and Behaviour, 8(2), 97–120.
doi: 10.1016/j.trf.2005.04.012
Gratton, G., Coles, M. G., & Donchin, E. (1983). A new method for off-line removal of ocular artifact. Electroencephalography and Clinical Neurophysiology, 55, 468–484. https://doi.org/10.1016/0013-4694(83)90135-9
doi: 10.1016/0013-4694(83)90135-9
pubmed: 6187540
Heine, T., Lenis, G., Reichensperger, P., Beran, T., Doessel, O., & Deml, B. (2017). Electrocardiographic features for the measurement of drivers’ mental workload. Applied Ergonomics, 61, 31–43. https://doi.org/10.1016/j.apergo.2016.12.015
doi: 10.1016/j.apergo.2016.12.015
pubmed: 28237018
Hidalgo-Muñoz, A. R., Béquet, A. J., Astier-Juvenon, M., Pépin, G., Fort, A., Jallais, C., et al. (2018). Respiration and heart rate modulation due to competing cognitive tasks while driving. Frontiers in Human Neuroscience, 12, 525. https://doi.org/10.3389/fnhum.2018.00525
doi: 10.3389/fnhum.2018.00525
pubmed: 30687043
International Organization for Standardization (2016). ISO/TC 22/SC 39.
Jasper, H. H. (1958). The ten-twenty electrode system of the international federation. Electroencephalography and Clinical Neurophysiology, 10, 370–375.
Jensen, O., & Mazaheri, A. (2010). Shaping functional architecture by oscillatory alpha activity: Gating by inhibition. Frontiers in Human Neuroscience, 4, 186.
doi: 10.3389/fnhum.2010.00186
pubmed: 21119777
Kahneman, D. (1973). Attention and effort. Prentice-Hall.
Käthner, I., Wriessnegger, S. C., Müller-Putz, G. R., Kübler, A., & Halder, S. (2014). Effects of mental workload and fatigue on the P300, alpha and theta band power during operation of an ERP (P300) brain–computer interface. Biological Psychology, 102, 118–129. https://doi.org/10.1016/j.biopsycho.2014.07.014
doi: 10.1016/j.biopsycho.2014.07.014
pubmed: 25088378
Klimesch, W. (2012). Alpha-band oscillations, attention, and controlled access to stored information. Trends in Cognitive Sciences, 16(12), 606–617.
doi: 10.1016/j.tics.2012.10.007
pubmed: 23141428
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.
doi: 10.1016/j.jcm.2016.02.012
pubmed: 27330520
Lee, H. B., Kim, J. S., Kim, Y. S., Baek, H. J., Ryu, M. S., & Park, K. S. (2007). The relationship between HRV parameters and stressful driving situation in the real road. In 6th International Special Topic Conference on Information Technology Applications in Biomedicine, 2007 (Tokyo). https://doi.org/10.1109/ITAB.2007.4407380
Lenneman, J. K., & Backs, R. W. (2009). Cardiac autonomic control during simulated driving with a concurrent verbal working memory task. Human Factors, 51, 404–418. https://doi.org/10.1177/0018720809337716
doi: 10.1177/0018720809337716
pubmed: 19750801
Lohani, M., Cooper, J. M., Erickson, G. G., Simmons, T. G., McDonnell, A. S., Carriero, A. E., Crabtree, K. W., & Strayer, D. L. (2021). No difference in arousal or cognitive demands between manual and partially automated driving: A multi-method on-road study. Frontiers in Neuroscience, 15, 577418. https://doi.org/10.3389/fnins.2021.577418
doi: 10.3389/fnins.2021.577418
pubmed: 34177439
Lohani, M., Cooper, J. M., Erickson, G., Simmons, T., McDonnell, A., Crabtree, K. W., & Strayer, D. L. (2020). Driver arousal and workload under partial vehicle automation: A pilot study. Proceedings of the Human Factors Ergonomic Society., 64, 1955–1959.
doi: 10.1177/1071181320641471
Lohani, M., Payne, B. R., & Strayer, D. L. (2019). A review of psychophysiological measures to assess cognitive states in real-world driving. Frontiers in Human Neuroscience, 13, 57. https://doi.org/10.3389/fnhum.2019.00057
doi: 10.3389/fnhum.2019.00057
pubmed: 30941023
Lopez-Calderon, J., & Luck, S. J. (2014). ERPLAB: An open-source toolbox for the analysis of event-related potentials. Frontiers in Human Neuroscience, 8, 213. https://doi.org/10.3389/fnhum.2014.00213
doi: 10.3389/fnhum.2014.00213
pubmed: 24782741
Luck, S. J. (2014). An introduction to the event-related potential technique. MIT press.
Matthews, G., Reinerman-Jones, L. E., Barber, D. J., & Abich, J., IV. (2015). The psychometrics of mental workload: Multiple measures are sensitive but divergent. Human Factors, 57(1), 125–143.
doi: 10.1177/0018720814539505
pubmed: 25790574
McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, N.J.: L. Erlbaum Associates.
McDonnell, A. S., Simmons, T. G., Erickson, G. G., Lohani, M., Cooper, J. M., & Strayer, D. L. (2023). This is your brain on autopilot: Neural indices of driver workload and engagement during partial vehicle automation. Human Factors, 65(7), 1435–1450.
doi: 10.1177/00187208211039091
pubmed: 34414813
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30. https://doi.org/10.1037/1082-989X.1.1.30
doi: 10.1037/1082-989X.1.1.30
Mehler, B., Reimer, B., & Wang, Y. (2011). A comparison of heart rate and heart rate variability indices in distinguishing single-task driving and driving under secondary cognitive workload. In Proceedings of the 6th International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design: Driving Assessment, 2011 (Lake Tahoe, CA), 590–597. https://doi.org/10.17077/drivingassessment.1451
Mehler, B., Reimer, B., & Coughlin, J. F. (2012). Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task: An on-road study across three age groups. Human Factors, 54, 396–412. https://doi.org/10.1177/0018720812442086
doi: 10.1177/0018720812442086
pubmed: 22768642
National Highway Traffic Safety Administration (2013). Visual- Manual NHTSA Driver Distraction Guidelines for In-Vehicle Electronic Devices (Federal Register Vol.78, No. 81). National Highway Traffic Safety Administration.
National Transportation Safety Board. (2020). Collision between a sport utility vehicle operating with partial driving automation and a crash attenuator: Mountain View, California, March 23, 2018.
Navon, D., & Gopher, D. (1979). On the economy of the human-processing system. Psychological Review, 86, 214–255.
doi: 10.1037/0033-295X.86.3.214
Nunez, P. L., & Srinivasan, R. (2006). Electric Fields of the Brain: The Neurophysics of EEG. Oxford University Press.
doi: 10.1093/acprof:oso/9780195050387.001.0001
Reimer, B., Mehler, B., Coughlin, J. F., Godfrey, K. M., & Chuanzhong, T. (2009). An on-road assessment of the impact of cognitive workload on physiological arousal in young adult drivers. In Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications, (ACM), 115–118. https://doi.org/10.1145/1620509.1620531
Revelle, W. (2013). Using R and the psych package to find ω. Computer Software]. http://personality-project.org/r/psych/HowTo/omega.tutorial/omega.html #x1-150005.1.
Revelle, W., & Condon, D. M. (2019). Reliability from α to ω: A tutorial. Psychological Assessment, 31(12), 1395.
doi: 10.1037/pas0000754
pubmed: 31380696
Revelle, W., & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega and the glb: Comments on Sijtsma. Psychometrika, 74(1), 145–154. https://doi.org/10.1007/s11336-008-9102-z
doi: 10.1007/s11336-008-9102-z
RStudio Team. (2016). Rstudio: Integrated development environment for r [Computer software manual]. Boston, MA. Retrieved from http://www.rstudio.com/
Ruscio, D., Bos, A. J., & Ciceri, M. R. (2017). Distraction or cognitive overload? Using modulations of the autonomic nervous system to discriminate the possible negative effects of advanced assistance system. Accident Analysis and Prevention, 103, 105–111. https://doi.org/10.1016/j.aap.2017.03.023
doi: 10.1016/j.aap.2017.03.023
pubmed: 28399463
SAE (2016). Taxonomy and Definitions for Terms Related to Driving Automation Systems for on-Road Motor Vehicles (Surface Vehicle Recommended Practice: Superseding J3016 Jan 2014), SAE International. Available online at: https://www.sae.org/standards/content/j3016_201806/ .
Scerbo, M. (2007). Adaptive automation. Neuroergonomics The Brain at Work, 239252.
Schmidt, E. A., Schrauf, M., Simon, M., Fritzsche, M., Buchner, A., & Kincses, W. E. (2009). Drivers’ misjudgement of vigilance state during prolonged monotonous daytime driving. Accident Analysis & Prevention, 41, 1087–1093. https://doi.org/10.1016/j.aap.2009.06.007
doi: 10.1016/j.aap.2009.06.007
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420. https://doi.org/10.1037/0033-2909.86.2.420
doi: 10.1037/0033-2909.86.2.420
pubmed: 18839484
Shrout, P. E., & Lane, S. P. (2012). Psychometrics. Guilford Press.
Statista (2023). Number of autonomous vehicles globally in 2022, with a forecast through 2030. https://www.statista.com/statistics/1230664/projected-number-autonomous-cars-worldwide/
Strayer, D. L. (2015). Attention and driving. The Handbook of Attention, 1, 423–442.
Task Force of the European Society of Cardiology. (1996). Heart rate variability, standards of measurement, physiological interpretation, and clinical use. Circulation, 93, 1043–1065. https://doi.org/10.1161/01.CIR.93.5.1043
doi: 10.1161/01.CIR.93.5.1043
Taylor-Phillips, S., Jenkinson, D., Stinton, C., Kunar, M. A., Watson, D. G., Freeman, K., & Clarke, A. (2024). Fatigue and vigilance in medical experts detecting breast cancer. Proceedings of the National Academy of Sciences, 121(11), e2309576121.
doi: 10.1073/pnas.2309576121
Teo, T., & Fan, X. (2013). Coefficient alpha and beyond: Issues and alternatives for educational research. The Asia-Pacific Education Researcher, 22(2), 209–213. https://doi.org/10.1007/s40299-013-0075-z10.1007/s40299-013-0075-z
doi: 10.1007/s40299-013-0075-z10.1007/s40299-013-0075-z
Tozman, T., Magdas, E. S., MacDougall, H. G., & Vollmeyer, R. (2015). Understanding the psychophysiology of flow: A driving simulator experiment to investigate the relationship between flow and heart rate variability. Computers in Human Behavior, 52, 408–418. https://doi.org/10.1016/j.chb.2015.06.023
doi: 10.1016/j.chb.2015.06.023
Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. The Journal of Strength and Conditioning Research, 19(1), 231.
pubmed: 15705040
Wickens, C. D. (1980). The structure of attentional resources. In R. S. Nickerson (Ed.), Attention and performance VIII (pp. 239–257). Erlbaum.
Wickens, C. D. (1984). Processing resources in attention. In R. Parasuraman & R. Davies (Eds.), Varieties of attention (pp. 63–101). Academic Press.
Young, R. A., Hsieh, L., & Seaman, S. (2013). The tactile detection response task: preliminary validation for measuring the attentional effects of cognitive load. In Proceedings of the Seventh International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, New York, NY, 71–77. https://doi.org/10.17077/drivingassessment.1469
Zander, T. O., Andreessen, L. M., Berg, A., Bleuel, M., Pawlitzki, J., Zawallich, L., et al. (2017). Evaluation of a dry EEG system for application of passive brain-computer interfaces in autonomous driving. Frontiers in Human Neuroscience, 11, 78. https://doi.org/10.3389/fnhum.2017.00078
doi: 10.3389/fnhum.2017.00078
pubmed: 28293184
Zhao, C., Zhao, M., Liu, J., & Zheng, C. (2012). Electroencephalogram and electrocardiograph assessment of mental fatigue in a driving simulator. Accident Analysis and Prevention, 45, 83–90. https://doi.org/10.1016/j.aap.2011.11.019
doi: 10.1016/j.aap.2011.11.019
pubmed: 22269488
Zinbarg, R. E., Yovel, I., Revelle, W., & McDonald, R. P. (2006). Estimating generalizability to a latent variable common to all of a scale’s indicators: A comparison of estimators for ωh. Applied Psychological Measurement, 30(2), 121–144. https://doi.org/10.1177/0146621605278814
doi: 10.1177/0146621605278814