Using computer vision of facial expressions to assess symptom domains and treatment response in antipsychotic-naïve patients with first-episode psychosis.
action units
drug‐naïve
facial expressions
psychopathology
schizophrenia
Journal
Acta psychiatrica Scandinavica
ISSN: 1600-0447
Titre abrégé: Acta Psychiatr Scand
Pays: United States
ID NLM: 0370364
Informations de publication
Date de publication:
12 Aug 2024
12 Aug 2024
Historique:
revised:
19
06
2024
received:
15
03
2024
accepted:
22
07
2024
medline:
13
8
2024
pubmed:
13
8
2024
entrez:
13
8
2024
Statut:
aheadofprint
Résumé
Facial expressions are a core aspect of non-verbal communication. Reduced emotional expressiveness of the face is a common negative symptom of schizophrenia, however, quantifying negative symptoms can be clinically challenging and involves a considerable element of rater subjectivity. We used computer vision to investigate if (i) automated assessment of facial expressions captures negative as well as positive and general symptom domains, and (ii) if automated assessments are associated with treatment response in initially antipsychotic-naïve patients with first-episode psychosis. We included 46 patients (mean age 25.4 (6.1); 65.2% males). Psychopathology was assessed at baseline and after 6 weeks of monotherapy with amisulpride using the Positive and Negative Syndrome Scale (PANSS). Baseline interview videos were recorded. Seventeen facial action units (AUs), that is, activation of muscles, from the Facial Action Coding System were extracted using OpenFace 2.0. A correlation matrix was calculated for each patient. Facial expressions were identified using spectral clustering at group-level. Associations between facial expressions and psychopathology were investigated using multiple linear regression. Three clusters of facial expressions were identified related to different locations of the face. Cluster 1 was associated with positive and general symptoms at baseline, Cluster 2 was associated with all symptom domains, showing the strongest association with the negative domain, and Cluster 3 was only associated with general symptoms. Cluster 1 was significantly associated with the clinically rated improvement in positive and general symptoms after treatment, and Cluster 2 was significantly associated with clinical improvement in all domains. Using automated computer vision of facial expressions during PANSS interviews did not only capture negative symptoms but also combinations of the three overall domains of psychopathology. Moreover, automated assessments of facial expressions at baseline were associated with initial antipsychotic treatment response. The findings underscore the clinical relevance of facial expressions and motivate further investigations of computer vision in clinical psychiatry.
Sections du résumé
BACKGROUND
BACKGROUND
Facial expressions are a core aspect of non-verbal communication. Reduced emotional expressiveness of the face is a common negative symptom of schizophrenia, however, quantifying negative symptoms can be clinically challenging and involves a considerable element of rater subjectivity. We used computer vision to investigate if (i) automated assessment of facial expressions captures negative as well as positive and general symptom domains, and (ii) if automated assessments are associated with treatment response in initially antipsychotic-naïve patients with first-episode psychosis.
METHOD
METHODS
We included 46 patients (mean age 25.4 (6.1); 65.2% males). Psychopathology was assessed at baseline and after 6 weeks of monotherapy with amisulpride using the Positive and Negative Syndrome Scale (PANSS). Baseline interview videos were recorded. Seventeen facial action units (AUs), that is, activation of muscles, from the Facial Action Coding System were extracted using OpenFace 2.0. A correlation matrix was calculated for each patient. Facial expressions were identified using spectral clustering at group-level. Associations between facial expressions and psychopathology were investigated using multiple linear regression.
RESULTS
RESULTS
Three clusters of facial expressions were identified related to different locations of the face. Cluster 1 was associated with positive and general symptoms at baseline, Cluster 2 was associated with all symptom domains, showing the strongest association with the negative domain, and Cluster 3 was only associated with general symptoms. Cluster 1 was significantly associated with the clinically rated improvement in positive and general symptoms after treatment, and Cluster 2 was significantly associated with clinical improvement in all domains.
CONCLUSION
CONCLUSIONS
Using automated computer vision of facial expressions during PANSS interviews did not only capture negative symptoms but also combinations of the three overall domains of psychopathology. Moreover, automated assessments of facial expressions at baseline were associated with initial antipsychotic treatment response. The findings underscore the clinical relevance of facial expressions and motivate further investigations of computer vision in clinical psychiatry.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Subventions
Organisme : Danish Agency for Science Technology and Innovation
Organisme : Gerhard Linds legat
Organisme : Lundbeck Foundation
Organisme : Region Hovedstadens Psykiatri
Organisme : Marie and Børge Kroghs Foundation
Informations de copyright
© 2024 The Author(s). Acta Psychiatrica Scandinavica published by John Wiley & Sons Ltd.
Références
Mandal MK, Habel U, Gur RC. Facial expression‐based indicators of schizophrenia: evidence from recent research. Schizophr Res. 2023;252(January):335‐344. doi:10.1016/j.schres.2023.01.016
Tandon R, Nasrallah HA, Keshavan MS. Schizophrenia, “just the facts” 4. Clinical features and conceptualization. Schizophrenia Res. 2009;110(1–3):1‐23. doi:10.1016/j.schres.2009.03.005
Gupta T, Haase CM, Strauss GP, Cohen AS, Mittal VA. Alterations in facial expressivity in youth at clinical high‐risk for psychosis. J Abnorm Psychol. 2019;128(4):341‐351. doi:10.1037/abn0000413
Galderisi S, Kaiser S, Bitter I, et al. EPA guidance on treatment of negative symptoms in schizophrenia. Eur Psychiatry. 2021;64(1):e21. doi:10.1192/j.eurpsy.2021.13
Norman RMG, Malla AK, Cortese L, Diaz F. A study of the interrelationship between and comparative interrater reliability of the SAPS, SANS and PANSS. Schizophrenia Res. 1996;19(1):73‐85. doi:10.1016/0920‐9964(95)00055‐0
Ekman P, Friesen WV. Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press; 1978.
Jiang Z, Luskus M, Seyedi S, et al. Utilizing computer vision for facial behavior analysis in schizophrenia studies: a systematic review. PLoS ONE. Vol 17. Public Library of Science; 2022:e0266828. doi:10.1371/journal.pone.0266828
Bishay M, Palasek P, Priebe S, Patras I. SchiNet: automatic estimation of symptoms of schizophrenia from facial behaviour analysis. IEEE Trans Affect Comput. 2021;12(4):949‐961. doi:10.1109/TAFFC.2019.2907628
Tron T, Peled A, Grinsphoon A, Weinshall D. Automated facial expressions analysis in schizophrenia: a continuous dynamic approach. Commun Comput Inform Sci. 2016a;604:72‐81. doi:10.1007/978‐3‐319‐32270‐4_8
Tron T, Peled A, Grinsphoon A, Weinshall D. Facial expressions and flat affect in schizophrenia, automatic analysis from depth camera data. 3rd IEEE EMBS International Conference on Biomedical and Health Informatics, BHI 2016, 220–223. 2016b. doi:10.1109/BHI.2016.7455874
Vijay S, Baltrušaitis T, Pennant L, Baker JT, Morency L‐P. Computational study of psychosis symptoms and facial expressions. Computing and Mental Health Workshop at CHI (Vol 2.). 2016.
Correll CU, Schooler NR. Negative symptoms in schizophrenia: a review and clinical guide for recognition, assessment, and treatment. Neuropsychiatr Dis Treat. 2020;16:519‐534. doi:10.2147/NDT.S225643
Sarkar S. Conceptualization and treatment of negative symptoms in schizophrenia. World J Psych. 2015;5(4):352‐361. doi:10.5498/wjp.v5.i4.352
Nielsen MØ, Rostrup E, Wulff S, et al. Alterations of the brain reward system in antipsychotic naïve schizophrenia patients. Biol Psychiatry. 2012;71(10):898‐905. doi:10.1016/j.biopsych.2012.02.007
Wechsler D. Manual for the Wechsler Adult Intelligence Scale—Third Edition (WAIS‐III). The Psychological Corporation; 1997. https://scholar.google.com/scholar_lookup?title=Manual+for+the+Wechsler+Adult+Intelligence+Scale+–+third+edition+(WAIS-III)&author=Wechsler+D.&publication+year=1997
Kay SR, Fiszbein A, Opler LA. The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophr Bull. 1987;13(2):261‐276. doi:10.1093/schbul/13.2.261
Baltrusaitis T, Zadeh A, Lim YC, Morency LP. OpenFace 2.0: facial behavior analysis toolkit. Proceedings—13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018, 59–66. 2018. doi:10.1109/FG.2018.00019
Baltrušaitis T, Mahmoud M, Robinson P. Cross‐dataset learning and person‐specific normalisation for automatic Action Unit detection. 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2015. 2015. doi:10.1109/FG.2015.7284869
Von Luxburg U. A tutorial on spectral clustering. Stat Comput. 2007;17(4):395‐416. doi:10.1007/s11222‐007‐9033‐z
Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit‐learn: machine learning in python. J Mach Learn Res. 2011;12(9):2825‐2830.
Damle A, Minden V, Ying L. Simple, direct and efficient multi‐way spectral clustering. Inform Inference. 2019;8(1):181‐203. doi:10.1093/imaiai/iay008
Zelnik‐Manor L, Perona P. Self‐tuning spectral clustering. Adv Neural Inform Process Syst. 2004;17:1601‐1608.
Leucht S, Kane JM, Kissling W, Hamann J, Etschel E, Engel RR. What does the PANSS mean? Schizophr Res. 2005;79(2–3):231‐238. doi:10.1016/j.schres.2005.04.008
Cheong JH, Jolly E, Xie T, Byrne S, Kenney M, Chang LJ. Py‐feat: python facial expression analysis toolbox. Affect Sci. 2023;4(4):781‐796. doi:10.1007/s42761‐023‐00191‐4
Kim M, Cho Y, Kim SY. Effects of diagnostic regions on facial emotion recognition: the moving window technique. Front Psychol. 2022;13(Sept):1‐12. doi:10.3389/fpsyg.2022.966623
Schurgin MW, Nelson J, Iida S, Ohira H, Chiao JY, Franconeri SL. Eye movements during emotion recognition in faces. J vis. 2014;14(13):1‐16. doi:10.1167/14.13.14
Wegrzyn M, Vogt M, Kireclioglu B, Schneider J, Kissler J. Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS One. 2017;12(5):1‐15. doi:10.1371/journal.pone.0177239
Friesen W, Ekman P. EMFACS‐7: Emotional Facial Action Coding System, Version 7. Unpublished Manual. 1984.
Jeganathan J, Campbell M, Hyett M, Parker G, Breakspear M. Quantifying dynamic facial expressions under naturalistic conditions. Elife. 2022;11:1‐20. doi:10.7554/eLife.79581
Daniel DG. Issues in selection of instruments to measure negative symptoms. Schizophr Res. 2013;150(2–3):343‐345. doi:10.1016/j.schres.2013.07.005
Marder SR, Kirkpatrick B. Defining and measuring negative symptoms of schizophrenia in clinical trials. Eur Neuropsychopharmacol. 2014;24(5):737‐743. doi:10.1016/j.euroneuro.2013.10.016
Kirkpatrick B, Strauss GP, Nguyen L, et al. The brief negative symptom scale: psychometric properties. Schizophr Bull. 2011;37(2):300‐305. doi:10.1093/schbul/sbq059
Forbes C, Blanchard JJ, Bennett M, Horan WP, Kring A, Gur R. Initial development and preliminary validation of a new negative symptom measure: the clinical assessment interview for negative symptoms (CAINS). Schizophr Res. 2010;124(1–3):36‐42. doi:10.1016/j.schres.2010.08.039