Machine learning based classification of presence utilizing psychophysiological signals in immersive virtual environments.
EDA
EEG
Machine learning
Presence
SHAP analysis
Virtual reality
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
17 Sep 2024
17 Sep 2024
Historique:
received:
11
03
2024
accepted:
06
09
2024
medline:
18
9
2024
pubmed:
18
9
2024
entrez:
17
9
2024
Statut:
epublish
Résumé
In Virtual Reality (VR), a higher level of presence positively influences the experience and engagement of a user. There are several parameters that are responsible for generating different levels of presence in VR, including but not limited to, graphical fidelity, multi-sensory stimuli, and embodiment. However, standard methods of measuring presence, including self-reported questionnaires, are biased. This research focuses on developing a robust model, via machine learning, to detect different levels of presence in VR using multimodal neurological and physiological signals, including electroencephalography and electrodermal activity. An experiment has been undertaken whereby participants (N = 22) were each exposed to three different levels of presence (high, medium, and low) in a random order in VR. Four parameters within each level, including graphics fidelity, audio cues, latency, and embodiment with haptic feedback, were systematically manipulated to differentiate the levels. A number of multi-class classifiers were evaluated within a three-class classification problem, using a One-vs-Rest approach, including Support Vector Machine, k-Nearest Neighbour, Extra Gradient Boosting, Random Forest, Logistic Regression, and Multiple Layer Perceptron. Results demonstrated that the Multiple Layer Perceptron model obtained the highest macro average accuracy of
Identifiants
pubmed: 39289475
doi: 10.1038/s41598-024-72376-1
pii: 10.1038/s41598-024-72376-1
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
21667Informations de copyright
© 2024. The Author(s).
Références
Lombard, M. & Ditton, T. At the heart of it all: The concept of presence. J. Comput. Commun. 3, JCMC321 (1997).
Lombard, M. & Jones, M. T. Defining presence. In Immersed in Media: Telepresence Theory, Measurement & Technology (eds. Lombard, M. et al.) 13–34 (Springer, 2015).
Slater, M. From presence to consciousness through virtual reality. Nature 1, 9–14 (2009).
Ijsselsteijn, W. & Riva, G. Being there: The experience of presence in mediated environments. Emerg. Commun. 5, 3 (2003).
Prothero, J. D., Parker, D. E. & Furness, T. A. I. Towards a robust, quantitative measure for presence. In Proceedings of the Conference on Experimental Analysis and Measurement of Situation Awareness (1995).
Kalawsky, R. S. et al. The validity of presence as a reliable human performance metric in immersive environments. In 3rd International Workshop on Presence, vol. 22 (2000).
Brooks, K. There is nothing virtual about immersion: Narrative immersion for VR and other interfaces. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=3a2bbc105c1464b217113a1cbbd9407232c09d98 [Accessed Nov 2023] (2003).
Diemer, J., Alpers, G. W., Peperkorn, H. M., Shiban, Y. & Mühlberger, A. The impact of perception and presence on emotional reactions: A review of research in virtual reality. Front. Psychol. 6, 111605 (2015).
doi: 10.3389/fpsyg.2015.00026
Cummings, J. J. & Bailenson, J. N. How immersive is enough? a meta-analysis of the effect of immersive technology on user presence. Media Psychol. 19, 272–309 (2016).
doi: 10.1080/15213269.2015.1015740
Stevens, J. A. & Kincaid, J. P. The relationship between presence and performance in virtual simulation training. J. Model. Simul. 10, 166–173 (2016).
Ma, Y., Zhang, L., Wu, M. & Roccetti, M. Research on the influence of virtual reality on the learning effect of technical skills of science and engineering college students: Meta-analysis based on 32 empirical studies. Int. J. Digit. Multimed. Broadcast. 2022, 6202370 (2022).
Sitzmann, T., Kraiger, K., Stewart, D. & Wisher, R. A meta-analysis of the effectiveness of immersive technology for learning. Hum. Resour. Dev. Rev. 9, 125–154 (2010).
Witmer, B. G. & Singer, M. J. Measuring presence in virtual environments: A presence questionnaire. Presence 7, 225–240 (1998).
doi: 10.1162/105474698565686
Jensen, L. & Konradsen, F. A review of the use of virtual reality head-mounted displays in education and training. Educ. Inf. Technol. 23, 1515–1529 (2018).
doi: 10.1007/s10639-017-9676-0
Lin, W., Wesson, M. & Hart, J. C. The effects of haptic feedback on task performance in a virtual environment. In Proceedings of the 8th annual ACM symposium on User interface and software technology, 1–10 (ACM, 2002).
LaViola, J. J. Jr. A discussion of cybersickness in virtual environments. ACM Sigchi Bull. 32, 47–56 (2000).
doi: 10.1145/333329.333344
Weech, S., Kenny, S. & Barnett-Cowan, M. Presence and cybersickness in virtual reality are negatively related: A review. J. Neurol. & Neuromed. 4, 39–44 (2019).
Saredakis, D. & Szpak, A. Development of a presence questionnaire for virtual reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–12 (ACM, 2019).
Morii, M., Sakagami, T., Masuda, S., Okubo, S. & Tamari, Y. How does response bias emerge in lengthy sequential preference judgments?. Behav. Res. Methods 52, 1709–1724 (2020).
Creswell, J. W. Research design: Qualitative, quantitative, and mixed methods approaches (Sage Publications, 2014).
Alsuradi, H., Park, W. & Eid, M. Eeg-based neurohaptics research: A literature review. IEEE Access 8, 49313–49328 (2020).
doi: 10.1109/ACCESS.2020.2979855
Schacter, D. L. The seven sins of memory: Insights from psychology and cognitive neuroscience. Am. Psychol. 54, 182–203 (1999).
pubmed: 10199218
doi: 10.1037/0003-066X.54.3.182
Slater, M. et al. A separate reality: An update on place illusion and plausibility in virtual reality. Front. Virtual Real. 3, 914392 (2022).
doi: 10.3389/frvir.2022.914392
Mishra, S., Kumar, A., Padmanabhan, P. & Gulyás, B. Neurophysiological correlates of cognition as revealed by virtual reality: Delving the brain with a synergistic approach. Brain Sci. 11, 51 (2021).
pubmed: 33466371
pmcid: 7824819
doi: 10.3390/brainsci11010051
Uhm, J.-P., Lee, H.-W. & Han, J.-W. Creating sense of presence in a virtual reality experience: Impact on neurophysiological arousal and attitude towards a winter sport. Sport Manag. Rev. 23, 588–600 (2020).
doi: 10.1016/j.smr.2019.10.003
Tregillus, K., Conkle, A. & Reiners, D. Design elements in immersive virtual reality: The impact of object presence on health-related outcomes. J. Med. Syst. 44, 156 (2020).
Sanchez-Vives, M. V. & Slater, M. From presence to consciousness through virtual reality. Nat. Rev. Neurosci. 6, 332–339 (2005).
pubmed: 15803164
doi: 10.1038/nrn1651
Slater, M., Khanna, P., Mortensen, J. & Yu, I. Visual realism enhances realistic response in an immersive virtual environment. IEEE Comput. Graph. Appl. 29, 76–84 (2009).
pubmed: 19642617
doi: 10.1109/MCG.2009.55
Khanna, P., Yu, I., Mortensen, J. & Slater, M. Presence in response to dynamic visual realism. In Proceedings of the ACM symposium on Virtual reality software and technology (ACM, 2006).
Hvass, J. S. et al. The effect of geometric realism on presence in a virtual reality game. In 2017 IEEE Virtual Reality (VR) (IEEE, 2017).
Palmisano, S., Kim, J. & Freeman, T. The effect of display lag on perceptual stability in head-mounted display-based virtual environments. Displays 31, 246–256 (2010).
Weber, S., Weibel, D. & Mast, F. W. How to get there when you are there already? Defining presence in virtual reality and the importance of perceived realism. Front. Psychol. 12, 628298 (2021).
pubmed: 34025504
pmcid: 8136250
doi: 10.3389/fpsyg.2021.628298
Kilteni, K., Groten, R. & Slater, M. The sense of embodiment in virtual reality. Presence Teleoper. Virtual Environ. 21, 373–387 (2012).
doi: 10.1162/PRES_a_00124
Lankoski, P. Embodiment in character-based videogames. In Proceedings of the 20th International Academic Mindtrek Conference, AcademicMindtrek ’16, 358–365 (Association for Computing Machinery, New York, NY, USA) (2016).
Slater, M. et al. Virtually being lenin enhances presence and engagement in a scene from the Russian revolution. Front. Robot. AI 5, 91 (2018).
pubmed: 33500970
pmcid: 7805740
doi: 10.3389/frobt.2018.00091
Fribourg, R., Argelaguet, F., Lecuyer, A. & Hoyet, L. Avatar and sense of embodiment: Studying the relative preference between appearance, control and point of view. IEEE Trans. Vis. Comput. Graph. 26, 2062–2072 (2020).
pubmed: 32070975
doi: 10.1109/TVCG.2020.2973077
Munyan, B. G., Neer, S. M. & Beidel, D. C. Jentsch, F, Olfactory stimuli increase presence in virtual environments. PLoS ONE 11, e0157568 (2016).
pubmed: 27310253
pmcid: 4910977
doi: 10.1371/journal.pone.0157568
Koijck, L. A., Toet, A. & Van Erp, J. B. Tactile roughness perception in the presence of olfactory and trigeminal stimulants. PeerJ 3, e955 (2015).
pubmed: 26020010
pmcid: 4435474
doi: 10.7717/peerj.955
Kern, A. C. & Ellermeier, W. Audio in vr: Effects of a soundscape and movement-triggered step sounds on presence. Front. Robot. AI 7, 20 (2020).
pubmed: 33501189
pmcid: 7805954
doi: 10.3389/frobt.2020.00020
Choi, W., Li, L., Satoh, S. & Hachimura, K. Multisensory integration in the virtual hand illusion with active movement. BioMed Res. Int. 2016, 1–9 (2016).
doi: 10.1155/2016/9195219
IJsselsteijn, W. A., de Ridder, H., Freeman, J. & Avons, S. E. Presence: concept, determinants and measurement. In Proceedings of the SPIE, vol. 3959, 520–529 (International Society for Optics and Photonics) (2000).
Gani, A., Pickering, O., Ellis, C., Sabri, O. & Pucher, P. Impact of haptic feedback on surgical training outcomes: A randomised controlled trial of haptic versus non-haptic immersive virtual reality training. Ann. Med. Surg. 83, 104734 (2022).
doi: 10.1016/j.amsu.2022.104734
Schuemie, M. J., Van der Straaten, P., Krijn, M., Van der Mast, C. & Research, V. Research on presence in virtual reality: A survey. CyberPsychol. Behav. 4, 183–201 (2001).
doi: 10.1089/109493101300117884
Kober, S. E., Kurzmann, J. & Neuper, C. Cortical correlate of spatial presence in 2d and 3d interactive virtual reality: An eeg study. Int. J. Psychophysiol. 83, 365–374 (2012).
pubmed: 22206906
doi: 10.1016/j.ijpsycho.2011.12.003
Grassini, S., Laumann, K. & Rasmussen Skogstad, M. The use of virtual reality alone does not promote training performance (but sense of presence does). Front. Psychol. 11, 553693 (2020).
doi: 10.3389/fpsyg.2020.01743
Dey, A., Phoon, J., Saha, S., Dobbins, C. & Billinghurst, M. A neurophysiological approach for measuring presence in immersive virtual environments. In 2020 IEEE International Symposium on Mixed and Augmented Reality (Ismar), 474–485 (IEEE, 2020).
Baumgartner, T., Valko, L., Esslen, M. & Jäncke, L. Neural correlate of spatial presence in an arousing and noninteractive virtual reality: An eeg and psychophysiology study. CyberPsychol. Behav. 9, 30–45 (2006).
pubmed: 16497116
doi: 10.1089/cpb.2006.9.30
Grassini, S., Laumann, K., Thorp, S. & de Martin Topranin, V. Using electrophysiological measures to evaluate the sense of presence in immersive virtual environments: An event-related potential study. Brain Behav. 11, e2269 (2021).
pubmed: 34173347
pmcid: 8413821
doi: 10.1002/brb3.2269
Clemente, M., Rodr’ıguez, A., Rey, B. & Alcaniz Raya, M. Measuring presence during the navigation in a virtual environment using eeg. Stud. Heal. Technol. Inform. 154, 62–66 (2010).
Usoh, M., Catena, E., Arman, S. & Slater, M. Using presence questionnaires in reality. Presence Teleoper. Virtual Environ. 9, 497–503 (2000).
doi: 10.1162/105474600566989
Patrick, C. J., Curtin, J. J. & Tellegen, A. Development and validation of a brief form of the multidimensional personality questionnaire. Psychol. Assess. 14, 150–163 (2002).
pubmed: 12056077
doi: 10.1037/1040-3590.14.2.150
Meehan, M., Insko, B., Whitton, M. & Brooks, F. P. Physiological measures of presence in stressful virtual environments. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, 645–652 (ACM, 2002).
Busscher, B., Vliegher, D., Ling, Y. & Brinkman, W.-P. Physiological measures and self-report to evaluate neutral virtual reality worlds. J. Cyber. Ther. Rehabil. 4, 15–25 (2011).
McCorry, L. K. Physiology of the autonomic nervous system. Am. J. Pharm. Educ. 71, 78 (2007).
pubmed: 17786266
pmcid: 1959222
doi: 10.5688/aj710478
Schubert, T., Friedmann, F. & Regenbrecht, H. The sense of presence in virtual environments: A three-component scale measuring spatial presence, involvement, and realness. Presence Teleoper. Virtual Environ. 10, 193–205 (2001).
Savalle, E. et al. Single trial classification of the level of presence in vr. In Journées Cortico 2023 (2023).
Magalie, O., Sameer, J. & Philippe, B. Toward an automatic prediction of the sense of presence in virtual reality environment. In Proceedings of the 6th International Conference on Human-Agent Interaction, 161–166 (2018).
Yildirim, C. A review of deep learning approaches to EEG-based classification of cybersickness in virtual reality. In 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), 351–357 (2020).
Kamińska, D., Smółka, K. & Zwoliński, G. Detection of mental stress through EEG signal in virtual reality environment. Electronics 10, 2840 (2021).
doi: 10.3390/electronics10222840
Pinilla, A., Voigt-Antons, J.-N., Garcia, J., Raffe, W. & Moller, S. Real-time affect detection in virtual reality: A technique based on a three-dimensional model of affect and EEG signals. Front. Virtual Real. 3, 964754 (2023).
doi: 10.3389/frvir.2022.964754
Yang, A. H. X., Kasabov, N. & Cakmak, Y. O. Machine learning methods for the study of cybersickness: A systematic review. Brain Inform. 9, 24 (2022).
pubmed: 36209445
pmcid: 9548085
doi: 10.1186/s40708-022-00172-6
https://www.unrealengine.com/en-US/unreal-engine-5 . [Accessed 19-08-2024].
https://www.unrealengine.com/marketplace/en-US/store . [Accessed 19-08-2024].
Breves, P. & Stein, J.-P. Cognitive load in immersive media settings: The role of spatial presence and cybersickness. Virtual Real. 27, 1077–1089 (2022).
doi: 10.1007/s10055-022-00697-5
Latoschik, M. E. et al. The effect of avatar realism in immersive social virtual realities. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, VRST ’17 (Association for Computing Machinery, New York, NY, USA,) (2017).
Li, Y. et al. The more you ask, the less you get: When additional questions hurt external validity. J. Mark. Res. 59, 963–982 (2022).
doi: 10.1177/00222437211073581
Castellanos, N. P. & Makarov, V. A. Recovering eeg brain signals: Artifact suppression with wavelet enhanced independent component analysis. J. Neurosci. Methods 158, 300–312 (2006).
pubmed: 16828877
doi: 10.1016/j.jneumeth.2006.05.033
Yuan, Z. & Oja, E. A FastICA algorithm for non-negative independent component analysis. In Independent Component Analysis and Blind Signal Separation, 1–8 (Springer) (2004).
Castellanos, N. P. & Makarov, V. A. Recovering eeg brain signals: Artifact suppression with wavelet enhanced independent component analysis. J. Neurosci. Methods 158, 300–312 (2006) (Epub 2006 Jul 7).
pubmed: 16828877
doi: 10.1016/j.jneumeth.2006.05.033
Morales, S. & Bowers, M. E. Time-frequency analysis methods and their application in developmental eeg data. Dev. Cognit. Neurosci. 54, 101067 (2022).
doi: 10.1016/j.dcn.2022.101067
Benedek, M. & Kaernbach, C. A continuous measure of phasic electrodermal activity. J. Neurosci. Methods 190, 80–91 (2010).
pubmed: 20451556
pmcid: 2892750
doi: 10.1016/j.jneumeth.2010.04.028
Greco, A., Valenza, G., Lanata, A., Scilingo, E. P. & Citi, L. cvxeda: A convex optimization approach to electrodermal activity processing. IEEE Trans. Biomed. Eng. 63, 797–804 (2015).
pubmed: 26336110
Jancke, L., Cheetham, M., Baumgartner, T. & Schnitzspahn, K. M. Virtual reality and the role of the prefrontal cortex in adults and children. Front. Neurosci. 12, 669 (2018).
Hosoda, C. et al. The structure of the superior and inferior parietal lobes predicts inter-individual suitability for virtual reality. Sci. Rep. 11, 23688 (2021).
pubmed: 34880322
pmcid: 8654954
doi: 10.1038/s41598-021-02957-x
Smith, M. Chapter 3—brain mapping. In Mechanisms and genetics of neurodevelopmental cognitive disorders (ed. Smith, M.) 49–76 (Academic Press, 2021).
doi: 10.1016/B978-0-12-821913-3.00004-4
Firat, R. B. Opening the “black box”: Functions of the frontal lobes and their implications for sociology. Front. Sociol. 4, 3 (2019).
Hwang, J., Park, S. & Chi, J. Improving multi-class motor imagery eeg classification using overlapping sliding window and deep learning model. Electronics 12, 1186 (2023).
doi: 10.3390/electronics12051186
Liu, Y., Sourina, O. & Nguyen, M. K. Real-time eeg-based emotion recognition and its applications. Transactions on Comput. Sci. XII: Special Issue on Cyberworlds 256–277 (2011).
Zhang, B. et al. Cross-subject seizure detection in eegs using deep transfer learning. Comput. Math. Methods Med. 2020, 7902072 (2020).
pubmed: 32454884
pmcid: 7231423
doi: 10.1155/2020/7902072
Raufi, B. & Longo, L. An evaluation of the EEG alpha-to-theta and theta-to-alpha band ratios as indexes of mental workload. Front. Neuroinform. 16, 861967 (2022).
pubmed: 35651718
pmcid: 9149374
doi: 10.3389/fninf.2022.861967
Eldenfria, A. & Al-Samarraie, H. Towards an online continuous adaptation mechanism (ocam) for enhanced engagement: An eeg study. Int. J. Hum.-Comput. Interact. 35, 1960–1974 (2019).
doi: 10.1080/10447318.2019.1595303
Putman, P., Verkuil, B., Arias-Garcia, E., Pantazi, I. & van Schie, C. Eeg theta/beta ratio as a potential biomarker for attentional control and resilience against deleterious effects of stress on attention. Cognit. Affect. Behav. Neurosci. 14, 782–791 (2014).
doi: 10.3758/s13415-013-0238-7
Duan, R.-N., Zhu, J.-Y. & Lu, B.-L. Differential entropy feature for EEG-based emotion classification. In 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER) (IEEE, 2013).
Dorosti, S. & Khosrowabadi, R. Fractal dimension of eeg signal senses complexity of fractal animations. bioRxiv 2021–02 (2021).
Holland, J. H. Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control, and artificial intelligence (University of Michigan Press, 1975).
Blagec, K., Dorffner, G., Moradi, M. & Samwald, M. A critical analysis of metrics used for measuring progress in artificial intelligence (2020).
Lundberg, S. & Lee, S.-I. Consistent individualized feature attribution for tree ensembles. In Advances in Neural Information Processing Systems, 4765–4774 (2018).
Li, X., Wang, X. & Duan, X. A limited memory BFGS method for solving large-scale symmetric nonlinear equations. Abstr. Appl. Anal. 2014, 1–9 (2014).
Lim, S., Yeo, M. & Yoon, G. Comparison between concentration and immersion based on EEG analysis. Sensors 19, 1669 (2019).
pubmed: 30965606
pmcid: 6479797
doi: 10.3390/s19071669
Wong, T.-T. Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern Recognit. 48, 2839–2846 (2015).
doi: 10.1016/j.patcog.2015.03.009
Ochs, M., Bousquet, J., Pergandi, J.-M. & Blache, P. Multimodal behavioral cues analysis of the sense of presence and social presence during a social interaction with a virtual patient. Front. Comput. Sci. 4, 746804 (2022).
doi: 10.3389/fcomp.2022.746804
Tan, E. et al. Theta activity and cognitive functioning: Integrating evidence from resting-state and task-related developmental electroencephalography (eeg) research. Dev. Cognit. Neurosci. 67, 101404 (2024).
doi: 10.1016/j.dcn.2024.101404
Valmaggia, L. R. et al. Virtual reality in the assessment and treatment of psychosis: A review of the literature. Schizophr. Bull. 33, 707–712 (2007).
Liu, Y. & Sourina, O. EEG-based subject-dependent emotion recognition algorithm using fractal dimension. In 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (IEEE, 2014).
Regenbrecht, H. & Schubert, T. Real and illusory interactions enhance presence in virtual environments. Presence Teleoper. Virtual Environ. 11, 425–434 (2002).
doi: 10.1162/105474602760204318
Uyanık, H., Ozcelik, S. T. A., Duranay, Z. B., Sengur, A. & Acharya, U. R. Use of differential entropy for automated emotion recognition in a virtual reality environment with eeg signals. Diagnostics 12, 2508 (2022).
pubmed: 36292197
pmcid: 9601226
doi: 10.3390/diagnostics12102508
Tuite, P. J. & Konczak, J. Cortical sensory dysfunction and the parietal lobe. In Encyclopedia of Movement Disorders, Three-Volume Set, V1–254 (Elsevier, 2010).
Kassuba, T., Menz, M. M., Röder, B. & Siebner, H. R. Multisensory interactions between auditory and haptic object recognition. Cereb. Cortex 23, 1097–1107 (2013).
pubmed: 22518017
doi: 10.1093/cercor/bhs076
Stančák, A. Jr., Riml, A. & Pfurtscheller, G. The effects of external load on movement-related changes of the sensorimotor eeg rhythms. Electroencephalogr. Clin. Neurophysiol. 102, 495–504 (1997).
pubmed: 9216482
doi: 10.1016/S0013-4694(96)96623-0
Jacobson, S. et al. Visual system and occipital lobe. In Neuroanatomy for the Neuroscientist (eds. Jacobson, S., Marcus, E.M. & Pugsley, S.). 445–476 (Springer, 2018).