Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity.
Journal
Communications biology
ISSN: 2399-3642
Titre abrégé: Commun Biol
Pays: England
ID NLM: 101719179
Informations de publication
Date de publication:
23 09 2021
23 09 2021
Historique:
received:
05
05
2021
accepted:
11
08
2021
entrez:
24
9
2021
pubmed:
25
9
2021
medline:
15
12
2021
Statut:
epublish
Résumé
Speech neuroprosthetics aim to provide a natural communication channel to individuals who are unable to speak due to physical or neurological impairments. Real-time synthesis of acoustic speech directly from measured neural activity could enable natural conversations and notably improve quality of life, particularly for individuals who have severely limited means of communication. Recent advances in decoding approaches have led to high quality reconstructions of acoustic speech from invasively measured neural activity. However, most prior research utilizes data collected during open-loop experiments of articulated speech, which might not directly translate to imagined speech processes. Here, we present an approach that synthesizes audible speech in real-time for both imagined and whispered speech conditions. Using a participant implanted with stereotactic depth electrodes, we were able to reliably generate audible speech in real-time. The decoding models rely predominately on frontal activity suggesting that speech processes have similar representations when vocalized, whispered, or imagined. While reconstructed audio is not yet intelligible, our real-time synthesis approach represents an essential step towards investigating how patients will learn to operate a closed-loop speech neuroprosthesis based on imagined speech.
Identifiants
pubmed: 34556793
doi: 10.1038/s42003-021-02578-0
pii: 10.1038/s42003-021-02578-0
pmc: PMC8460739
doi:
Types de publication
Journal Article
Research Support, N.I.H., Extramural
Research Support, Non-U.S. Gov't
Research Support, U.S. Gov't, Non-P.H.S.
Langues
eng
Sous-ensembles de citation
IM
Pagination
1055Informations de copyright
© 2021. The Author(s).
Références
Nat Neurosci. 2020 Apr;23(4):575-582
pubmed: 32231340
J Neural Eng. 2018 Jun;15(3):036005
pubmed: 29378977
Front Neuroinform. 2017 Oct 31;11:62
pubmed: 29163118
J Neural Eng. 2020 Oct 15;17(5):056028
pubmed: 33055383
Neurology. 2001 Dec 11;57(11):2045-53
pubmed: 11739824
Front Neuroeng. 2014 May 27;7:14
pubmed: 24904404
PLoS One. 2013;8(1):e53398
pubmed: 23408924
Elife. 2019 Dec 10;8:
pubmed: 31820736
Nat Rev Neurosci. 2012 Jan 05;13(2):135-45
pubmed: 22218206
Sci Rep. 2016 May 11;6:25803
pubmed: 27165452
Nat Rev Neurol. 2019 Mar;15(3):148-160
pubmed: 30683913
PLoS One. 2016 Nov 22;11(11):e0166872
pubmed: 27875590
Hum Brain Mapp. 2007 Sep;28(9):868-79
pubmed: 17266104
Epilepsia. 2016 Mar;57(3):386-401
pubmed: 26899389
PLoS One. 2011;6(5):e18307
pubmed: 21625532
Mem Cognit. 2010 Dec;38(8):1147-60
pubmed: 21156877
PLoS One. 2009 Dec 09;4(12):e8218
pubmed: 20011034
Front Neurosci. 2019 Feb 19;13:60
pubmed: 30837823
Epilepsia. 2013 May;54(5):840-7
pubmed: 23551133
J Acoust Soc Am. 2002 May;111(5 Pt 1):2237-41
pubmed: 12051443
Front Neurosci. 2020 Feb 27;14:123
pubmed: 32174810
Brain Comput Interfaces (Abingdon). 2017;4(1-2):3-36
pubmed: 29152523
J Neural Eng. 2020 Feb 05;17(1):016049
pubmed: 32023225
Front Neurosci. 2019 Nov 22;13:1267
pubmed: 31824257
N Engl J Med. 2016 Nov 24;375(21):2060-2066
pubmed: 27959736
PLoS Biol. 2012 Jan;10(1):e1001251
pubmed: 22303281
Brain Lang. 2006 Mar;96(3):280-301
pubmed: 16040108
Nature. 2019 Apr;568(7753):493-498
pubmed: 31019317
Annu Int Conf IEEE Eng Med Biol Soc. 2016 Aug;2016:1540-1543
pubmed: 28268620
Nature. 2012 May 16;485(7398):372-5
pubmed: 22596161
Cereb Cortex. 2004 Jan;14(1):11-22
pubmed: 14654453
Neuroimage. 2014 Feb 15;87:96-110
pubmed: 24239590
J Neural Eng. 2019 Jun;16(3):036019
pubmed: 30831567
Sci Rep. 2019 Jan 29;9(1):874
pubmed: 30696881
Front Neurosci. 2015 Jun 12;9:217
pubmed: 26124702
Brain. 2008 Aug;131(Pt 8):2013-27
pubmed: 18669510
Neurosurgery. 2013 Mar;72(3):353-66; discussion 366
pubmed: 23168681
J Neurosci. 2008 Nov 5;28(45):11526-36
pubmed: 18987189
Clin Neurophysiol. 2019 Oct;130(10):1798-1803
pubmed: 31401488
Elife. 2017 Feb 21;6:
pubmed: 28220753
Nat Commun. 2019 Jul 30;10(1):3096
pubmed: 31363096
Nat Rev Neurosci. 2007 May;8(5):393-402
pubmed: 17431404
Lang Cogn Process. 2011 Jan 1;26(7):952-981
pubmed: 23667281
Front Hum Neurosci. 2012 May 03;6:99
pubmed: 22563311
Sci Rep. 2019 Jun 20;9(1):8881
pubmed: 31222030
Neuroimage. 2012 Aug 15;62(2):774-81
pubmed: 22248573