Imagined speech event detection from electrocorticography and its transfer between speech modes and subjects.
Journal
Communications biology
ISSN: 2399-3642
Titre abrégé: Commun Biol
Pays: England
ID NLM: 101719179
Informations de publication
Date de publication:
05 Jul 2024
05 Jul 2024
Historique:
received:
22
08
2023
accepted:
27
06
2024
medline:
6
7
2024
pubmed:
6
7
2024
entrez:
5
7
2024
Statut:
epublish
Résumé
Speech brain-computer interfaces aim to support communication-impaired patients by translating neural signals into speech. While impressive progress was achieved in decoding performed, perceived and attempted speech, imagined speech remains elusive, mainly due to the absence of behavioral output. Nevertheless, imagined speech is advantageous since it does not depend on any articulator movements that might become impaired or even lost throughout the stages of a neurodegenerative disease. In this study, we analyzed electrocortigraphy data recorded from 16 participants in response to 3 speech modes: performed, perceived (listening), and imagined speech. We used a linear model to detect speech events and examined the contributions of each frequency band, from delta to high gamma, given the speech mode and electrode location. For imagined speech detection, we observed a strong contribution of gamma bands in the motor cortex, whereas lower frequencies were more prominent in the temporal lobe, in particular of the left hemisphere. Based on the similarities in frequency patterns, we were able to transfer models between speech modes and participants with similar electrode locations.
Identifiants
pubmed: 38969758
doi: 10.1038/s42003-024-06518-6
pii: 10.1038/s42003-024-06518-6
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
818Subventions
Organisme : Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)
ID : 11K2324N
Organisme : Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)
ID : G0A4321N
Organisme : Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)
ID : G0C1522N
Organisme : Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)
ID : G0A4118N
Organisme : Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)
ID : G0A4321N
Organisme : Fonds Wetenschappelijk Onderzoek (Research Foundation Flanders)
ID : G0C1522N
Organisme : KU Leuven (Katholieke Universiteit Leuven)
ID : PDM/19/176
Organisme : KU Leuven (Katholieke Universiteit Leuven)
ID : C24/18/098
Organisme : Hercules Foundation
ID : AKUL 043
Organisme : EC | Horizon 2020 Framework Programme (EU Framework Programme for Research and Innovation H2020)
ID : 857375
Informations de copyright
© 2024. The Author(s).
Références
Nichelli, P. Consciousness and Aphasia. In The Neurology of Consciousness (Elsevier, 2016).
Schultz, T. et al. Biosignal-Based Spoken Communication: A Survey. IEEE/ACM Trans. Audio Speech Lang. Process 25, 2257–2271 (2017).
doi: 10.1109/TASLP.2017.2752365
Moses, D. A., Leonard, M. K., Makin, J. G. & Chang, E. F. Real-time decoding of question-and-answer speech dialogue using human cortical activity. Nat. Commun. 10, 3096 (2019).
doi: 10.1038/s41467-019-10994-4
pubmed: 31363096
pmcid: 6667454
Makin, J. G., Moses, D. A. & Chang, E. F. Machine translation of cortical activity to text with an encoder–decoder framework. Nat. Neurosci. 23, 575–582 (2020).
doi: 10.1038/s41593-020-0608-8
pubmed: 32231340
pmcid: 10560395
Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D. & Mesgarani, N. Towards reconstructing intelligible speech from the human auditory cortex. Sci. Rep. 9, 874 (2019).
doi: 10.1038/s41598-018-37359-z
pubmed: 30696881
pmcid: 6351601
Moses, D. A. et al. Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria. N. Engl. J. Med. 385, 217–227 (2021).
doi: 10.1056/NEJMoa2027540
pubmed: 34260835
pmcid: 8972947
Willett, F. R. et al. A high-performance speech neuroprosthesis. Nature 620, 1031–1036 (2023).
doi: 10.1038/s41586-023-06377-x
pubmed: 37612500
pmcid: 10468393
Metzger, S. L. et al. Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis. Nat. Commun. 13, 6510 (2022).
doi: 10.1038/s41467-022-33611-3
pubmed: 36347863
pmcid: 9643551
Metzger, S. L. et al. A high-performance neuroprosthesis for speech decoding and avatar control. Nature 620, 1037–1046 (2023).
doi: 10.1038/s41586-023-06443-4
pubmed: 37612505
pmcid: 10826467
Bocquelet, F., Hueber, T., Girin, L., Savariaux, C. & Yvert, B. Real-Time Control of an Articulatory-Based Speech Synthesizer for Brain Computer Interfaces. PLoS Comput. Biol. 12, e1005119 (2016).
doi: 10.1371/journal.pcbi.1005119
pubmed: 27880768
pmcid: 5120792
Anumanchipalli, G. K., Chartier, J. & Chang, E. F. Speech synthesis from neural decoding of spoken sentences. Nature 568, 493–498 (2019).
doi: 10.1038/s41586-019-1119-1
pubmed: 31019317
pmcid: 9714519
Ramsey, N. F. & Crone, N. E. Brain implants that enable speech pass performance milestones. Nature 620, 954–955 (2023).
doi: 10.1038/d41586-023-02546-0
pubmed: 37612488
Angrick, M. et al. Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity. Commun. Biol. 4, 1055 (2021).
doi: 10.1038/s42003-021-02578-0
pubmed: 34556793
pmcid: 8460739
Cooney, C., Folli, R. & Coyle, D. Neurolinguistics Research Advancing Development of a Direct-Speech Brain-Computer Interface. iScience 8, 103–125 (2018).
doi: 10.1016/j.isci.2018.09.016
pubmed: 30296666
pmcid: 6174918
Luo, S., Rabbani, Q. & Crone, N. E. Brain-Computer Interface: Applications to Speech Decoding and Synthesis to Augment Communication. Neurotherapeutics 19, 263–273 (2022).
doi: 10.1007/s13311-022-01190-2
pubmed: 35099768
pmcid: 9130409
Martin, S. et al. Decoding spectrotemporal features of overt and covert speech from the human cortex. Front. Neuroeng. 7, 14 (2014).
Martin, S. et al. Word pair classification during imagined speech using direct brain recordings. Sci. Rep. 6, 25803 (2016).
doi: 10.1038/srep25803
pubmed: 27165452
pmcid: 4863149
Proix, T. et al. Imagined speech can be decoded from low- and cross-frequency intracranial EEG features. Nat. Commun. 13, 48 (2022).
doi: 10.1038/s41467-021-27725-3
pubmed: 35013268
pmcid: 8748882
Kanas, V. G. et al. Real-time voice activity detection for ECoG-based speech brain machine interfaces. In 2014 19th International Conference on Digital Signal Processing 862–865, https://doi.org/10.1109/ICDSP.2014.6900790 (2014).
Milsap, G. et al. Keyword Spotting Using Human Electrocorticographic Recordings. Front Neurosci 13, 60 (2019).
Soroush, P. Z. et al. The nested hierarchy of overt, mouthed, and imagined speech activity evident in intracranial recordings. Neuroimage 269, 119913 (2023).
doi: 10.1016/j.neuroimage.2023.119913
pubmed: 36731812
Martin, S. et al. Decoding Inner Speech Using Electrocorticography: Progress and Challenges Toward a Speech Prosthesis. Front Neurosci 12, 422 (2018).
Leuthardt, E. C. et al. Temporal evolution of gamma activity in human cortex during an overt and covert word repetition task. Front. Hum. Neurosci. 6, 99 (2012).
Brumberg, J. S. et al. Spatio-Temporal Progression of Cortical Activity Related to Continuous Overt and Covert Speech Production in a Reading Task. PLoS One 11, e0166872 (2016).
doi: 10.1371/journal.pone.0166872
pubmed: 27875590
pmcid: 5119784
Pei, X. et al. Spatiotemporal dynamics of electrocorticographic high gamma activity during overt and covert word repetition. Neuroimage 54, 2960–2972 (2011).
doi: 10.1016/j.neuroimage.2010.10.029
pubmed: 21029784
Jayaram, V., Alamgir, M., Altun, Y., Scholkopf, B. & Grosse-Wentrup, M. Transfer Learning in Brain-Computer Interfaces. IEEE Comput Intell. Mag. 11, 20–31 (2016).
doi: 10.1109/MCI.2015.2501545
Dubey, A. & Ray, S. Cortical Electrocorticogram (ECoG) Is a Local Signal. J. Neurosci. 39, 4299–4311 (2019).
doi: 10.1523/JNEUROSCI.2917-18.2019
pubmed: 30914446
pmcid: 6538865
Haufe, S. et al. On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage 87, 96–110 (2014).
doi: 10.1016/j.neuroimage.2013.10.067
pubmed: 24239590
Pei, X., Barbour, D. L., Leuthardt, E. C. & Schalk, G. Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans. J. Neural Eng. 8, 046028 (2011).
doi: 10.1088/1741-2560/8/4/046028
pubmed: 21750369
pmcid: 3772685
Penfield W & Rasmussen, T. The Cerebral Cortex of Man; a Clinical Study of Localization of Function (Macmillan, 1950).
Wilson, G. H. et al. Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus. J. Neural Eng. 17, 066007 (2020).
doi: 10.1088/1741-2552/abbfef
pubmed: 33236720
pmcid: 8293867
Stavisky, S. D. et al. Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis. Elife 8, e46015 (2019).
doi: 10.7554/eLife.46015
pubmed: 31820736
pmcid: 6954053
Hertrich, I., Dietrich, S. & Ackermann, H. The role of the supplementary motor area for speech and language processing. Neurosci. Biobehav. Rev. 68, 602–610 (2016).
doi: 10.1016/j.neubiorev.2016.06.030
pubmed: 27343998
Lima, C. F., Krishnan, S. & Scott, S. K. Roles of Supplementary Motor Areas in Auditory Processing and Auditory Imagery. Trends Neurosci. 39, 527–542 (2016).
doi: 10.1016/j.tins.2016.06.003
pubmed: 27381836
pmcid: 5441995
Vansteensel, M. J. et al. Fully Implanted Brain–Computer Interface in a Locked-In Patient with ALS. N. Engl. J. Med. 375, 2060–2066 (2016).
doi: 10.1056/NEJMoa1608085
pubmed: 27959736
pmcid: 5326682
Gehrig, J. et al. Low-Frequency Oscillations Code Speech during Verbal Working Memory. J. Neurosci. 39, 6498–6512 (2019).
doi: 10.1523/JNEUROSCI.0018-19.2019
pubmed: 31196933
pmcid: 6697399
Giraud, A.-L. & Poeppel, D. Cortical oscillations and speech processing: emerging computational principles and operations. Nat. Neurosci. 15, 511–517 (2012).
doi: 10.1038/nn.3063
pubmed: 22426255
pmcid: 4461038
Knecht, S. Handedness and hemispheric language dominance in healthy humans. Brain 123, 2512–2518 (2000).
doi: 10.1093/brain/123.12.2512
pubmed: 11099452
Cheung, C., Hamilton, L. S., Johnson, K. & Chang, E. F. The auditory representation of speech sounds in human motor cortex. Elife 5, e12577 (2016).
doi: 10.7554/eLife.12577
pubmed: 26943778
pmcid: 4786411
Hincapié Casas, A. S. et al. Neural oscillations track natural but not artificial fast speech: Novel insights from speech-brain coupling using MEG. Neuroimage 244, 118577 (2021).
doi: 10.1016/j.neuroimage.2021.118577
pubmed: 34525395
van Wieringen, A. & Wouters, J. LIST and LINT: Sentences and numbers for quantifying speech understanding in severely impaired listeners for Flanders and the Netherlands. Int J. Audio. 47, 348–355 (2008).
doi: 10.1080/14992020801895144
Kleiner, M. et al. What’s new in psychtoolbox-3. Perception 36, 1–16 (2007).
Roussel, P. et al. Observation and assessment of acoustic contamination of electrophysiological brain signals during speech production and sound perception. J. Neural Eng. 17, 056028 (2020).
doi: 10.1088/1741-2552/abb25e
pubmed: 33055383
Fischl, B. FreeSurfer. Neuroimage 62, 774–781 (2012).
doi: 10.1016/j.neuroimage.2012.01.021
pubmed: 22248573
Tadel, F., Baillet, S., Mosher, J. C., Pantazis, D. & Leahy, R. M. Brainstorm: A User-Friendly Application for MEG/EEG Analysis. Comput. Intell. Neurosci. 2011, 1–13 (2011).
doi: 10.1155/2011/879716
Penny, W., Friston, K., Ashburner, J., Kiebel, S. & Nichols, T. Statistical Parametric Mapping: The Analysis of Functional Brain Images, https://doi.org/10.1016/B978-0-12-372560-8.X5000-1 (2007).
Duchateau, J. et al. Developing a reading tutor: Design and evaluation of dedicated speech recognition and synthesis modules. Speech Commun. 51, 985–994 (2009).
doi: 10.1016/j.specom.2009.04.010
Pedregosa, F. et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).