A Graph-Based Nonlinear Dynamic Characterization of Motor Imagery Toward an Enhanced Hybrid BCI.
Graph-based feature extraction
Hybrid brain-computer interface (hBCI)
Motor imagery (MI)
Multimodal data fusion
Nonlinear dynamics
Recurrence quantification analysis (RQA)
Journal
Neuroinformatics
ISSN: 1559-0089
Titre abrégé: Neuroinformatics
Pays: United States
ID NLM: 101142069
Informations de publication
Date de publication:
10 2022
10 2022
Historique:
accepted:
05
07
2022
pubmed:
31
7
2022
medline:
26
10
2022
entrez:
30
7
2022
Statut:
ppublish
Résumé
Decoding neural responses from multimodal information sources, including electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), has the transformative potential to advance hybrid brain-computer interfaces (hBCIs). However, existing modest performance improvement of hBCIs might be attributed to the lack of computational frameworks that exploit complementary synergistic properties in multimodal features. This study proposes a multimodal data fusion framework to represent and decode synergistic multimodal motor imagery (MI) neural responses. We hypothesize that exploiting EEG nonlinear dynamics adds a new informative dimension to the commonly combined EEG-fNIRS features and will ultimately increase the synergy between EEG and fNIRS features toward an enhanced hBCI. The EEG nonlinear dynamics were quantified by extracting graph-based recurrence quantification analysis (RQA) features to complement the commonly used spectral features for an enhanced multimodal configuration when combined with fNIRS. The high-dimensional multimodal features were further given to a feature selection algorithm relying on the least absolute shrinkage and selection operator (LASSO) for fused feature selection. Linear support vector machine (SVM) was then used to evaluate the framework. The mean hybrid classification performance improved by up to 15% and 4% compared to the unimodal EEG and fNIRS, respectively. The proposed graph-based framework substantially increased the contribution of EEG features for hBCI classification from 28.16% up to 52.9% when introduced the nonlinear dynamics and improved the performance by approximately 2%. These findings suggest that graph-based nonlinear dynamics can increase the synergy between EEG and fNIRS features for an enhanced MI response representation that is not dominated by a single modality.
Identifiants
pubmed: 35907174
doi: 10.1007/s12021-022-09595-2
pii: 10.1007/s12021-022-09595-2
doi:
Types de publication
Journal Article
Research Support, N.I.H., Extramural
Research Support, Non-U.S. Gov't
Research Support, U.S. Gov't, Non-P.H.S.
Langues
eng
Sous-ensembles de citation
IM
Pagination
1169-1189Subventions
Organisme : NIGMS NIH HHS
ID : P20 GM103430
Pays : United States
Informations de copyright
© 2022. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.
Références
Acharya, U. R., Vinitha Sree, S., et al. (2011a). Application of Recurrence Quantification Analysis for the Automated Identification of Epileptic EEG Signals. International Journal of Neural Systems, 21(3), 199–211.
pubmed: 21656923
Acharya, U. R., Chua, E. -P., et al. (2011b). Automated Detection of Sleep Apnea from Electrocardiogram Signals Using Nonlinear Parameters. Physiological Measurement, 32(3), 287–303.
pubmed: 21285482
Ahn, S., & Jun, S. C. (2017). Multi-Modal Integration of EEG-FNIRS for Brain-Computer Interfaces – Current Limitations and Future Directions. Frontiers in Human Neuroscience, 11, 503.
pubmed: 29093673
pmcid: 5651279
Al-Shargie, F., et al. (2016). Mental Stress Assessment Using Simultaneous Measurement of EEG and FNIRS. Biomedical Optics Express, 8, 2583–2598.
Al-Shargie, F., Tang, T. B., & Kiguchi, M. (2017). Assessment of Mental Stress Effects on Prefrontal Cortical Activities Using Canonical Correlation Analysis: An FNIRS-EEG Study. Biomedical Optics Express, 8(5), 2583–2598.
pubmed: 28663892
pmcid: 5480499
Ayaz, H., et al. (2013). Continuous Monitoring of Brain Dynamics with Functional near Infrared Spectroscopy as a Tool for Neuroergonomic Research: Empirical Examples and a Technological Development. Frontiers in Human Neuroscience, 7, 871.
pubmed: 24385959
pmcid: 3866520
Baghdadi, G., Amiri, M., Falotico, E., & Laschi, C. (2021). Recurrence Quantification Analysis of EEG Signals for Tactile Roughness Discrimination. International Journal of Machine Learning and Cybernetics, 12(4), 1115–1136.
Bauer, C. M., et al. (2017). The Effect of Muscle Fatigue and Low Back Pain on Lumbar Movement Variability and Complexity. Journal of Electromyography and Kinesiology : Official Journal of the International Society of Electrophysiological Kinesiology, 33, 94–102.
Brunner, C., Delorme, A., & Makeig, S. (2013). Eeglab – an Open Source Matlab Toolbox for Electrophysiological Research. Biomedical Engineering.
Buccino, A. P., Keles, H. O., & Omurtag, A. (2016). Hybrid EEG-FNIRS Asynchronous Brain-Computer Interface for Multiple Motor Tasks. PLoS ONE, 11(1), 1–16.
Chiarelli, A. M., Croce, P., Merla, A., & Zappasodi, F. (2018). Deep Learning for Hybrid EEG-FNIRS Brain-Computer Interface: Application to Motor Imagery Classification. Journal of Neural Engineering, 15(3), 36028.
Cui, X., Bray, S., & Reiss, A. L. (2010). Functional near Infrared Spectroscopy (NIRS) Signal Improvement Based on Negative Correlation between Oxygenated and Deoxygenated Hemoglobin Dynamics. NeuroImage, 49, 3039–3046.
pubmed: 19945536
Deligani, R. J., Borgheai, S. B., McLinden, J., & Shahriari, Y. (2021). Multimodal Fusion of EEG-FNIRS: A Mutual Information-Based Hybrid Classification Framework. Biomedical Optics Express, 12(3), 1635–1650.
pubmed: 33796378
pmcid: 7984774
Donner, R. V., et al. (2010). Recurrence Networks-a Novel Paradigm for Nonlinear Time Series Analysis. New Journal of Physics, 12, 033025.
Donner, R. V., Small, M., Donges, J. F., Marwan, N., Zou, Y., Xiang, R., & Kurths, J. (2011). Recurrence-Based Time Series Analysis by Means of Complex Network Methods. International Journal of Bifurcation and Chaos, 21, 1019–1046.
Eckmann, J. P., Oliffson Kamphorst, O., & Ruelle, D. (1987). Recurrence Plots of Dynamical Systems. World Scientific Series on Nonlinear Science Series A, 16, 441–446.
Fazli, S., et al. (2012). Enhanced Performance by a Hybrid NIRS-EEG Brain Computer Interface. NeuroImage, 59(1), 519–529.
pubmed: 21840399
Feldhoff, J. H., et al. (2013). Geometric Signature of Complex Synchronisation Scenarios. EPL, 102, 30007.
Gao, J. B. (1999). Recurrence Time Statistics for Chaotic Systems and Their Applications. Physical Review Letters, 83(16), 3178.
Holper, L., Shalóm, D. E., Wolf, M., & Sigman, M. (2011). Understanding Inverse Oxygenation Responses during Motor Imagery: A Functional near-Infrared Spectroscopy Study. European Journal of Neuroscience, 33, 2318–2328.
pubmed: 21631608
Hong, K. S., Jawad Khan, M., & Hong, M. J. (2018). Feature Extraction and Classification Methods for Hybrid FNIRS-EEG Brain-Computer Interfaces. Frontiers in Human Neuroscience, 12, 246.
pubmed: 30002623
pmcid: 6032997
Hong, K. S., Raheel Bhutta, M., Liu, X., & Shin, Y. I. (2017). Classification of Somatosensory Cortex Activities Using FNIRS. Behavioural Brain Research, 333, 225–234.
pubmed: 28668280
Hosni, S. M., Borgheai, S. B., McLinden, J., & Shahriari, Y. (2020). An FNIRS-Based Motor Imagery BCI for ALS: A Subject-Specific Data-Driven Approach. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(12), 3063–3073.
pubmed: 33206606
Hosni, S. M., et al. (2019). An Exploration of Neural Dynamics of Motor Imagery for People with Amyotrophic Lateral Sclerosis. Journal of Neural Engineering, 17, 16005.
Hu, X. S., Hong, K. S., Ge, S. S., & Jeong, M. Y. (2010). Kalman Estimator- and General Linear Model-Based on-Line Brain Activation Mapping by near-Infrared Spectroscopy. BioMedical Engineering Online, 9, 1–15.
Ikegawa, S., et al. (2000). Nonlinear Time-Course of Lumbar Muscle Fatigue Using Recurrence Quantifications. Biological Cybernetics, 82, 373–382.
pubmed: 10836583
Ismail Hosni, S., et al. (2021). Graph-Based Recurrence Quantification Analysis of EEG Spectral Dynamics for Motor Imagery-Based BCIs. In 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (accepted).
Javorka, M., et al. (2009). The Effect of Orthostasis on Recurrence Quantification Analysis of Heart Rate and Blood Pressure Dynamics. Physiological Measurement, 30, 29.
pubmed: 19039163
Jiang, J., et al. (2020). Temporal Combination Pattern Optimization Based on Feature Selection Method for Motor Imagery BCIs. Frontiers in Human Neuroscience, 14, 231.
pubmed: 32714167
pmcid: 7344307
Kasahara, T., et al. (2012). The Correlation between Motor Impairments and Event-Related Desynchronization during Motor Imagery in ALS Patients. BMC Neuroscience, 13, 1–10.
Khan, M. J., Hong, M. J., & Hong, K. -S. (2014). Decoding of Four Movement Directions Using Hybrid NIRS-EEG Brain-Computer Interface. Frontiers in Human Neuroscience, 8, 244.
doi: 10.3389/fnhum.2014.00244
Kübler, A., et al. (2005). Patients with ALS Can Use Sensorimotor Rhythms to Operate a Brain-Computer Interface. Neurology, 64, 1775–1777.
pubmed: 15911809
Li, R., Potter, T., Huang, W., & Zhang, Y. (2017). Enhancing Performance of a Hybrid EEG-FNIRS System Using Channel Selection and Early Temporal Features. Frontiers in Human Neuroscience, 11, 462.
pubmed: 28966581
pmcid: 5605645
Lotte, F., et al. (2007). A Review of Classification Algorithms for EEG-Based Brain-Computer Interfaces. Journal of Neural Engineering, 4(2), R1-13.
pubmed: 17409472
Marwan, N. (2013). Cross Recurrence Plot Toolbox for MATLAB
Marwan, N., et al. (2002). Recurrence-Plot-Based Measures of Complexity and Their Application to Heart-Rate-Variability Data. Physical Review E - Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics, 66, 026702.
Marwan, N., Donges, J. F., Zou, Y., Donner, R. V., & Kurths, J. (2009). Complex Network Approach for Recurrence Analysis of Time Series. Physics Letters, Section A: General, Atomic and Solid State Physics, 373, 4246–4254.
Marwan, N., Carmen Romano, M., Thiel, M., & Kurths, J. (2007). Recurrence Plots for the Analysis of Complex Systems. Physics Reports, 438, 237–329.
Marwan, N., & Meinke, A. (2004). Extended Recurrence Plot Analysis and Its Application to ERP Data. International Journal of Bifurcation and Chaos in Applied Sciences and Engineering, 14, 761–771.
McFarland, D. J., McCane, L. M., David, S. V., & Wolpaw, J. R. (1997). Spatial Filter Selection for EEG-Based Communication. Electroencephalography and Clinical Neurophysiology, 103, 386–394.
pubmed: 9305287
McKenna, T. M., McMullen, T. A., & Shlesinger, M. F. (1994). The Brain as a Dynamic Physical System. Neuroscience, 60(3), 587–605.
pubmed: 7936189
Naseer, N., & Hong, K. S. (2013). Classification of Functional Near-Infrared Spectroscopy Signals Corresponding to the Right- and Left-Wrist Motor Imagery for Development of a Brain-Computer Interface. Neuroscience Letters, 553, 84–89.
pubmed: 23973334
Naseer, N., & Hong, K. -S. (2015). FNIRS-Based Brain-Computer Interfaces: A Review. Frontiers in Human Neuroscience, 9, 3.
pubmed: 25674060
pmcid: 4309034
Naseer, N., Noori, F. M., Qureshi, N. K., & Hong, K. S. (2016). Determining Optimal Feature-Combination for LDA Classification of Functional near-Infrared Spectroscopy Signals in Brain-Computer Interface Application. Frontiers in Human Neuroscience, 10, 237.
pubmed: 27252637
pmcid: 4879140
Ngamga, E. J., et al. (2016). Evaluation of Selected Recurrence Measures in Discriminating Pre-Ictal and Inter-Ictal Periods from Epileptic EEG Data. Physics Letters, Section a: General, Atomic and Solid State Physics, 380, 1419–1425.
Nguyen, T., et al. (2017). Utilization of a Combined EEG/NIRS System to Predict Driver Drowsiness. Scientific Reports, 7(1), 43933.
pubmed: 28266633
pmcid: 5339693
Pfurtscheller, G., & Lopes Da Silva, F. H. (1999). Event-Related EEG/MEG Synchronization and Desynchronization: Basic Principles. Clinical Neurophysiology, 110(11), 1842–1857.
pubmed: 10576479
Pitsik, E., et al. (2020). Motor Execution Reduces EEG Signals Complexity: Recurrence Quantification Analysis Study. Chaos, 30, 023111.
pubmed: 32113225
Qureshi, N. K., et al. (2017). Enhancing Classification Performance of Functional Near-Infrared Spectroscopy-Brain-Computer Interface Using Adaptive Estimation of General Linear Model Coefficients. Frontiers in Neurorobotics, 11, 33.
pubmed: 28769781
pmcid: 5512010
Saadati, M., Nelson, J., & Ayaz, H. (2020a). Convolutional Neural Network for Hybrid FNIRS-EEG Mental Workload Classification. In A. Hasan (Ed.), International Conference on Applied Human Factors and Ergonomics (pp. 221–232). Cham: Springer International Publishing.
Saadati, M., Nelson, J., & Ayaz, H. (2020b). Multimodal FNIRS-EEG Classification Using Deep Learning Algorithms for Brain-Computer Interfaces Purposes. In H. Ayaz (Ed.), Advances in Neuroergonomics and Cognitive Engineering (pp. 209–220). Springer International Publishing.
Santosa, H., Hong, M. J., & Hong, K. S. (2014). Lateralization of Music Processing with Noises in the Auditory Cortex: An FNIRS Study. Frontiers in Behavioral Neuroscience, 8, 418.
pubmed: 25538583
pmcid: 4260509
Sassaroli, A., & Fantini, S. (2004). Comment on the Modified Beer-Lambert Law for Scattering Media. Physics in Medicine and Biology, 49, N255.
pubmed: 15357206
Schalk, G., et al. (2004). BCI2000: A General-Purpose Brain-Computer Interface (BCI) System. IEEE Transactions on Biomedical Engineering, 51, 1034–1043.
pubmed: 15188875
Shin, J., et al. (2018). Simultaneous Acquisition of EEG and NIRS during Cognitive Tasks for an Open Access Dataset. Scientific Data, 5(1), 180003.
pubmed: 29437166
pmcid: 5810421
Takens, F. (1981). Detecting Strange Attractors in Turbulence. In D. Rand & L. S. Young (Eds.), Dynamical Systems and Turbulence, Warwick 1980. Lecture Notes in Mathematics 898. Berlin, Heidelberg: Springer.
Venugopalan, J., Tong, Li., Hassanzadeh, H. R., & Wang, M. D. (2021). Multimodal Deep Learning Models for Early Detection of Alzheimer’s Disease Stage. Scientific Reports, 11(1), 1–13.
von Lühmann, A., Ortega-Martinez, A., Boas, D. A., & Yücel, M. A. (2020). Using the General Linear Model to Improve Performance in FNIRS Single Trial Analysis and Classification: A Perspective. Frontiers in Human Neuroscience, 14, 30.
Webber, C. L., Jr., & Marwan, N. (2015). Recurrence Quantification Analysis – Theory and Best Practices. Understanding Complex Systems. Springer International Publishing, Cham Switzerland.
Wu, C. W., et al. (2019). Indication of Dynamic Neurovascular Coupling from Inconsistency between EEG and FMRI Indices across Sleep-Wake States. Sleep and Biological Rhythms, 17(4), 423–431.
Yin, X., et al. (2015). A Hybrid BCI Based on EEG and FNIRS Signals Improves the Performance of Decoding Motor Imagery of Both Force and Speed of Hand Clenching. Journal of Neural Engineering, 12(3), 36004.
Zbilut, J. P., Thomasson, N., & Webber, C. L. (2002). Recurrence Quantification Analysis as a Tool for Nonlinear Exploration of Nonstationary Cardiac Signals. Medical Engineering & Physics, 24(1), 53–60.