A lightweight hybrid vision transformer network for radar-based human activity recognition.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
21 10 2023
21 10 2023
Historique:
received:
10
07
2023
accepted:
16
10
2023
medline:
1
11
2023
pubmed:
22
10
2023
entrez:
21
10
2023
Statut:
epublish
Résumé
Radar-based human activity recognition (HAR) offers a non-contact technique with privacy protection and lighting robustness for many advanced applications. Complex deep neural networks demonstrate significant performance advantages when classifying the radar micro-Doppler signals that have unique correspondences with human behavior. However, in embedded applications, the demand for lightweight and low latency poses challenges to the radar-based HAR network construction. In this paper, an efficient network based on a lightweight hybrid Vision Transformer (LH-ViT) is proposed to address the HAR accuracy and network lightweight simultaneously. This network combines the efficient convolution operations with the strength of the self-attention mechanism in ViT. Feature Pyramid architecture is applied for the multi-scale feature extraction for the micro-Doppler map. Feature enhancement is executed by the stacked Radar-ViT subsequently, in which the fold and unfold operations are added to lower the computational load of the attention mechanism. The convolution operator in the LH-ViT is replaced by the RES-SE block, an efficient structure that combines the residual learning framework with the Squeeze-and-Excitation network. Experiments based on two human activity datasets indicate our method's advantages in terms of expressiveness and computing efficiency over traditional methods.
Identifiants
pubmed: 37865672
doi: 10.1038/s41598-023-45149-5
pii: 10.1038/s41598-023-45149-5
pmc: PMC10590397
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
17996Informations de copyright
© 2023. Springer Nature Limited.
Références
Gurbuz, S. Z. & Amin, M. G. Radar-based human-motion recognition with deep learning: Promising applications for indoor monitoring. IEEE Signal Process. Mag. 36, 16–28. https://doi.org/10.1109/MSP.2018.2890128 (2019).
doi: 10.1109/MSP.2018.2890128
Kamal, S., Jalal, A. & Kim, D. Depth images-based human detection, tracking and activity recognition using spatiotemporal features and modified HMM. J. Electr. Eng. Technol. 11, 1857–1862. https://doi.org/10.5370/JEET.2016.11.6.1857 (2016).
doi: 10.5370/JEET.2016.11.6.1857
Jalal, A., Quaid, M. A. K., & Hasan, A. S. Wearable sensor-based human behavior understanding and recognition in daily life for smart environments. In 2018 International Conference on Frontiers of Information Technology (FIT), 105–110, https://doi.org/10.1109/FIT.2018.00026 (2018).
Waghumbare, A., Singh, U. & Singhal, N. DCNN based human activity recognition using micro-doppler signatures. In 2022 IEEE Bombay Section Signature Conference (IBSSC), 1–6, https://doi.org/10.1109/IBSSC56953.2022.10037310 (2022).
Wang, C. xi, Chen, Z. C., Chen, X., Tang, X. & Liang, F. T. Detection of MMW radar target based on Doppler characteristics and deep learning. In 2021 IEEE International Conference on Artificial Intelligence and Industrial Design (AIID, 266–271, https://doi.org/10.1109/AIID51893.2021.9456497 (2021).
Chen, V. C. Joint time-frequency analysis for radar signal and imaging. In 2007 IEEE International Geoscience and Remote Sensing Symposium, 5166–516, https://doi.org/10.1109/IGARSS.2007.4424025 (2007).
Kim, Y. & Ling, H. Human activity classification based on micro-Doppler signatures using a support vector machine. IEEE Trans. Geosci. Remote Sens. 47, 1328–1337. https://doi.org/10.1109/TGRS.2009.2012849 (2009).
doi: 10.1109/TGRS.2009.2012849
Fairchild, D. P. & Narayanan, R. M. Classification of human motions using empirical mode decomposition of human micro-Doppler signatures. IET Radar Sonar Navig. 8, 425–434. https://doi.org/10.1049/iet-rsn.2013.0165 (2014).
doi: 10.1049/iet-rsn.2013.0165
Karabacak, C. et al. Knowledge exploitation for human micro-Doppler classification. IEEE Geosci. Remote Sens. Lett. 12, 2125–2129. https://doi.org/10.1049/iet-rsn.2013.0165 (2015).
doi: 10.1049/iet-rsn.2013.0165
Kim, Y. & Moon, T. Human detection and activity classification based on micro-Doppler signatures using deep convolutional neural networks. IEEE Geosci. Remote Sens. Lett. 13, 8–12. https://doi.org/10.1049/iet-rsn.2013.0165 (2016).
doi: 10.1049/iet-rsn.2013.0165
Krizhevsky, A. & Sutskever, I. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. https://doi.org/10.1145/3065386 (2012).
doi: 10.1145/3065386
Zaremba, W., Sutskever, I. & Vinyals, O. Recurrent neural network regularization. Preprint at https://arXiv.org/arXiv:1409.2329 , https://doi.org/10.48550/arXiv.1409.2329 (2014).
Vaswani, A. et al. Attention is all you need. Adv. Neural Inf. Process. Syst. https://doi.org/10.48550/arXiv.1706.03762 (2017).
doi: 10.48550/arXiv.1706.03762
Zhu, J., Chen, H. & Ye, W. A hybrid CNN–LSTM network for the classification of human activities based on micro-doppler radar. IEEE Access 8, 24713–24720. https://doi.org/10.1109/ACCESS.2022.3150838 (2020).
doi: 10.1109/ACCESS.2022.3150838
Güneş, O. & Morgül, Ö. LSTM based classification of targets using FMCW radar signals. In 2021 29th Signal Processing and Communications Applications Conference (SIU), 1–4, https://doi.org/10.1109/SIU53274.2021.9477927 (2021).
Kwon, H. B. et al. Attention-based LSTM for non-contact sleep stage classification using IR-UWB radar. IEEE J. Biomed. Health Inform. 25, 3844–3853. https://doi.org/10.1109/JBHI.2021.3072644 (2021).
doi: 10.1109/JBHI.2021.3072644
pubmed: 33848253
Shrestha, A., Li, H., Le Kernec, J. & Fioranelli, F. Continuous human activity classification from FMCW radar with Bi-LSTM networks. IEEE Sens. J. 20, 13607–13619. https://doi.org/10.1109/JSEN.2020.3006386 (2020).
doi: 10.1109/JSEN.2020.3006386
Pan, M. et al. Radar HRRP target recognition model based on a stacked CNN–Bi-RNN with attention mechanism. IEEE Trans. Geosci. Remote Sens. 60, 1–14. https://doi.org/10.1109/TGRS.2021.3055061 (2022).
doi: 10.1109/TGRS.2021.3055061
Chakraborty, M., Kumawat, H. C., Dhavale, S. V. & Raj, A. A. B. DIAT-μ RadHAR (micro-doppler signature dataset) & μ RadNet (a lightweight DCNN)—For human suspicious activity recognition. IEEE Sens. J. 22, 6851–6858. https://doi.org/10.1109/JSEN.2022.3151943 (2022).
doi: 10.1109/JSEN.2022.3151943
Zhu, J., Lou, X. & Ye, W. Lightweight deep learning model in mobile-edge computing for radar-based human activity recognition. IEEE Internet Things J. 8, 12350–12359. https://doi.org/10.1109/JIOT.2021.3063504 (2021).
doi: 10.1109/JIOT.2021.3063504
Khalid, H.-U.-R., Gorji, A., Bourdoux, A., Pollin, S. & Sahli, H. Multi-view CNN-LSTM architecture for radar-based human activity recognition. IEEE Access 10, 24509–24519. https://doi.org/10.1109/ACCESS.2020.2971064 (2022).
doi: 10.1109/ACCESS.2020.2971064
Hua, W., Wang, X., Zhang, C. & Jin, X. Attention-based deep sequential network for polsar image classification. 2000 IEEE Int. Geosci. Remote Sens. Symp. https://doi.org/10.1109/IGARSS46834.2022.9883634 (2022).
doi: 10.1109/IGARSS46834.2022.9883634
He, Y., Li, X. & Jing, X. A mutiscale residual attention network for multitask learning of human activity using radar micro-doppler signatures. Remote Sens. 11, 2584. https://doi.org/10.3390/rs11212584 (2019).
doi: 10.3390/rs11212584
Touvron, H. et al. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139, 10347–10357, https://doi.org/10.48550/arXiv.2012.12877 (2021).
D’Ascoli, S. et al. ConViT: Improving vision transformers with soft convolutional inductive biases. In Proceedings of the 38th International Conference on Machine Learning, vol. 139, 2286–2296, https://doi.org/10.1088/1742-5468/ac9830 (2021).
Mehta, S. & Rastegari, M. MobileViT: Light-weight, general-purpose, and mobile-friendly vision transformer. arXiv https://doi.org/10.48550/arXiv.2110.02178 (2021).
doi: 10.48550/arXiv.2110.02178
Chen, Y., Dai, X., Chen, D. & Liu, M. Mobile-former: Bridging mobilenet and transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5270–5279, https://doi.org/10.48550/arXiv.2108.05895 (2022).
Ahmed, W., Naeem, U., Yousaf, M. H., & Velastin, S. A. Lightweight CNN and GRU network for real-time action recognition. In 2022 12th International Conference on Pattern Recognition Systems (ICPRS), 1–7, https://doi.org/10.1109/ICPRS54038.2022.9853854 (2022).
Huan, S., Wu, L., Zhang, M., Wang, Z. & Yang, C. Radar human activity recognition with an attention-based deep learning network. Sensors 23, 3185. https://doi.org/10.3390/s23063185 (2023).
doi: 10.3390/s23063185
pubmed: 36991896
pmcid: 10054704
Howard, A. G. et al. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv https://doi.org/10.48550/arXiv.1704.04861 (2017).
doi: 10.48550/arXiv.1704.04861
Hu, J., Shen, L., Albanie, S., Sun, G. & Wu, E. Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 42, 2011–2023. https://doi.org/10.1109/CVPR.2018.00745 (2020).
doi: 10.1109/CVPR.2018.00745
pubmed: 31034408
Jalal, A., Kim, Y.-H., Kim, Y.-J., Kamal, S. & Kim, D. Robust human activity recognition from depth video using spatiotemporal multi-fused features. Pattern Recognit. 61, 295–308. https://doi.org/10.1016/j.patcog.2016.08.003 (2017).
doi: 10.1016/j.patcog.2016.08.003
Fioranelli, D. F. et al. Radar sensing for healthcare. Electron. Lett. 55, 1022–1024. https://doi.org/10.1049/el.2019.2378 (2019).
doi: 10.1049/el.2019.2378
Dumoulin, V., & Visin, F. A guide to convolution arithmetic for deep learning. Preprint at https://arXiv.org/arXiv:1603.07285 , https://doi.org/10.48550/arXiv.1603.07285 (2016).
Yu, F., & Koltun, V. Multi-scale context aggregation by dilated convolutions. Preprint at https://arXiv.org/arXiv:1511.07122 , https://doi.org/10.48550/arXiv.1511.07122 (2015).
Ioannou, Y., Robertson, D., Cipolla, R. & Criminisi, A. Deep roots: Improving CNN efficiency with hierarchical filter groups. Proc. IEEE Conf. Comput. Vis. Pattern recogn. https://doi.org/10.1109/CVPR.2017.633 (2017).
doi: 10.1109/CVPR.2017.633
Hou, Q., Zhou, D. & Feng, J. Coordinate attention for efficient mobile network design. arXiv https://doi.org/10.48550/arXiv.2103.02907 (2021).
doi: 10.48550/arXiv.2103.02907
Agac, S. & Durmaz Incel, O. On the use of a convolutional block attention module in deep learning-based human activity recognition with motion sensors. Diagnostics https://doi.org/10.3390/diagnostics13111861 (2023).
doi: 10.3390/diagnostics13111861
pubmed: 37296713
pmcid: 10252329
Geng, F. et al. Light-efficient channel attention in convolutional neural networks for tic recognition in the children with tic disorders. Front. Comput. Neurosci. 16, 1047954. https://doi.org/10.3389/fncom.2022.1047954 (2022).
doi: 10.3389/fncom.2022.1047954
pubmed: 36405786
pmcid: 9673168
Piyathilaka, L., & Kodagoda, S. Gaussian mixture based HMM for human daily activity recognition using 3D skeleton features. In 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA), 567–572, https://doi.org/10.1109/ICIEA.2013.6566433 (2013).
Zhang, X., Zhou, X., Lin, M. & Sun, J. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. arXiv https://doi.org/10.48550/arXiv.1707.01083 (2017).
doi: 10.48550/arXiv.1707.01083
Tan, M. & Le, Q. V. EfficientNet: Rethinking model scaling for convolutional neural networks. arXiv https://doi.org/10.48550/arXiv.1905.11946 (2019).
doi: 10.48550/arXiv.1905.11946
Dey, R. & Salem, F. M. Gate-variants of gated recurrent unit (GRU) neural networks. In 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), 1597–1600, https://doi.org/10.1109/MWSCAS.2017.8053243 (2017).
Chen, C.-F. R., Fan, Q. & Panda, R. CrossViT: Cross-attention multi-scale vision transformer for image classification. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 357–366, https://doi.org/10.1109/iccv48922.2021.00041 (2021).
Wang, Z. et al. Attention-based vision transformer for human activity classification using mmwave radar. In Proc. of the 2022 4th International Conference on Video, Signal and Image Processing, 128–134, https://doi.org/10.1145/3577164.3577184 (2023).