Temporal-spatial cross attention network for recognizing imagined characters.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
04 Jul 2024
04 Jul 2024
Historique:
received:
13
11
2023
accepted:
08
04
2024
medline:
5
7
2024
pubmed:
5
7
2024
entrez:
4
7
2024
Statut:
epublish
Résumé
Previous research has primarily employed deep learning models such as Convolutional Neural Networks (CNNs), and Recurrent Neural Networks (RNNs) for decoding imagined character signals. These approaches have treated the temporal and spatial features of the signals in a sequential, parallel, or single-feature manner. However, there has been limited research on the cross-relationships between temporal and spatial features, despite the inherent association between channels and sampling points in Brain-Computer Interface (BCI) signal acquisition, which holds significant information about brain activity. To address the limited research on the relationships between temporal and spatial features, we proposed a Temporal-Spatial Cross-Attention Network model, named TSCA-Net. The TSCA-Net is comprised of four modules: the Temporal Feature (TF), the Spatial Feature (SF), the Temporal-Spatial Cross (TSCross), and the Classifier. The TF combines LSTM and Transformer to extract temporal features from BCI signals, while the SF captures spatial features. The TSCross is introduced to learn the correlations between the temporal and spatial features. The Classifier predicts the label of BCI data based on its characteristics. We validated the TSCA-Net model using publicly available datasets of handwritten characters, which recorded the spiking activity from two micro-electrode arrays (MEAs). The results showed that our proposed TSCA-Net outperformed other comparison models (EEG-Net, EEG-TCNet, S3T, GRU, LSTM, R-Transformer, and ViT) in terms of accuracy, precision, recall, and F1 score, achieving 92.66
Identifiants
pubmed: 38965248
doi: 10.1038/s41598-024-59263-5
pii: 10.1038/s41598-024-59263-5
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
15432Subventions
Organisme : "Pioneer" and "Leading Goose" R&D Program of Zhejiang
ID : 2023C01143
Organisme : National Social Science Fund of China
ID : 19ZDA348
Informations de copyright
© 2024. The Author(s).
Références
Lotte, F. et al. A review of classification algorithms for EEG-based brain-computer interfaces: A 10 year update. J. Neural Eng. 4(2), R1 (2018).
doi: 10.1088/1741-2560/4/2/R01
Guillot, A., Moschberger, K. & Collet, C. Coupling movement with imagery as a new perspective for motor imagery practice. Behav. Brain Funct. 9(9), 8–8 (2013).
doi: 10.1186/1744-9081-9-8
pubmed: 23425312
pmcid: 3599464
Ullah, S. & Halim, Z. Imagined character recognition through EEG signals using deep convolutional neural network. Med. Biol. Eng. Comput. 59, 1167–1183 (2021).
doi: 10.1007/s11517-021-02368-0
pubmed: 33945075
Janapati, R., Desai, U., Kulkarni, S. A. & Tayal, S. Human-Machine Interface Technology Advancements and Applications (CRC Press, 2023).
doi: 10.1201/9781003326830
Pei, L. & Ouyang, G. Online recognition of handwritten characters from scalp-recorded brain activities during handwriting. J. Neural Eng. 18, 046070 (2021).
doi: 10.1088/1741-2552/ac01a0
Han, K. et al. A survey on vision transformer. IEEE Trans. Pattern Anal. Mach. Intell. 45, 87–110 (2020).
doi: 10.1109/TPAMI.2022.3152247
Lecun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436 (2015).
doi: 10.1038/nature14539
pubmed: 26017442
Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to sequence learning with neural networks. Adv. Neural Inf. Process. Syst. 27, 3104–3112 (2014).
Lotte, F., Congedo, M., Lécuyer, A. & Lamarche, F. A review of classification algorithms for EEG-based brain-computer interfaces. J. Neural Eng. 15, 031005 (2007).
doi: 10.1088/1741-2552/aab2f2
Ma, X., Qiu, S., Du, C., Xing, J. & He, H. Improving EEG-based motor imagery classification via spatial and temporal recurrent neural networks. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 1903–1906 (IEEE, 2018).
Alhagry, S., Fahmy, A. A. & El-Khoribi, R. A. Emotion recognition based on EEG using LSTM recurrent neural network. Int. J. Adv. Comput. Sci. Appl. https://doi.org/10.14569/IJACSA.2017.081046 (2017).
doi: 10.14569/IJACSA.2017.081046
Dai, Z. et al. Transformer-xl: Language modeling with longer-term dependency. ICLR 2019 (2018).
Beltagy, I. Peters, M. E. & Cohan, A. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020).
Willett, F. R., Avansino, D. T., Hochberg, L. R., Henderson, J. M. & Shenoy, K. V. High-performance brain-to-text communication via handwriting. Nature 593, 249–254 (2021).
doi: 10.1038/s41586-021-03506-2
pubmed: 33981047
pmcid: 8163299
Sun, P., Anumanchipalli, G. K. & Chang, E. F. Brain2char: A deep architecture for decoding text from brain recordings. J. Neural Eng. 17, 066015 (2020).
doi: 10.1088/1741-2552/abc742
Pascanu, R., Mikolov, T. & Bengio, Y. On the difficulty of training recurrent neural networks. JMLR.org (2012).
Gordon, S. M., Jaswa, M., Solon, A. J. & Lawhern, V. J. Real world bci: cross-domain learning and practical applications. In Proceedings of the 2017 ACM Workshop on an Application-oriented Approach to BCI out of the Laboratory, 25–28 ( 2017).
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D. & Houlsby, N. An image is worth [Formula: see text] words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
Dai, Z., Yang, Z., Yang, Y., Carbonell, J. & Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 (2019).
Wen, Q. et al. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125 (2022).
Zhou, D. et al. Refiner: Refining self-attention for vision transformers. arXiv preprint arXiv:2106.03714 ( 2021).
Song, Y., Jia, X., Yang, L. & Xie, L. Transformer-based spatial-temporal feature learning for EEG decoding. arXiv preprint arXiv:2106.11170 ( 2021).
Tibrewal, N., Leeuwis, N. & Alimardani, M. Classification of motor imagery EEG using deep learning increases performance in inefficient BCI users. PLoS One 17, e0268880 (2022).
doi: 10.1371/journal.pone.0268880
pubmed: 35867703
pmcid: 9307149
Chen, C. F., Fan, Q. & Panda, R. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021).
Han, K. et al. Transformer in transformer. Adv. Neural Inf. Process. Syst. 34, 15908–15919 (2021).
Vaswani, A. et al. Attention is all you need. arXiv (2017).
Devlin, J., Chang, M. W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
Shaw, P., Uszkoreit, J. & Vaswani, A. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 (2018).
Liu, Z. et al. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021).
He, K., Gkioxari, G., Dollar, P. & Girshick, R. Mask R-CNN. In International Conference on Computer Vision (2017).
Lawhern, V. J. et al. EEGNet: A compact convolutional network for EEG-based brain-computer interfaces. J. Neural Eng. 15, 0560131–05601317 (2018).
doi: 10.1088/1741-2552/aace8c
Ingolfsson, T. M. et al. EEG-TCNet: An accurate temporal convolutional network for embedded motor-imagery brain–machine interfaces. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC) 2958–2965 (IEEE, 2020).
Raviprakash, H. et al. Deep learning provides exceptional accuracy to ECoG-based functional language mapping for epilepsy surgery. Front. Neurosci. 14, 400 (2020).
doi: 10.3389/fnins.2020.00409
Wang, Z., Ma, Y., Liu, Z. & Tang, J. R-transformer: Recurrent neural network enhanced transformer. arXiv preprint arXiv:1907.05572 (2019).