Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition
Joint Authors
Pan, Jiahui
Huang, Yongrui
Yang, Jianhao
Liao, Pengkai
Source
Computational Intelligence and Neuroscience
Issue
Vol. 2017, Issue 2017 (31 Dec. 2017), pp.1-8, 8 p.
Publisher
Hindawi Publishing Corporation
Publication Date
2017-09-19
Country of Publication
Egypt
No. of Pages
8
Main Subjects
Abstract EN
This paper proposes two multimodal fusion methods between brain and peripheral signals for emotion recognition.
The input signals are electroencephalogram and facial expression.
The stimuli are based on a subset of movie clips that correspond to four specific areas of valance-arousal emotional space (happiness, neutral, sadness, and fear).
For facial expression detection, four basic emotion states (happiness, neutral, sadness, and fear) are detected by a neural network classifier.
For EEG detection, four basic emotion states and three emotion intensity levels (strong, ordinary, and weak) are detected by two support vector machines (SVM) classifiers, respectively.
Emotion recognition is based on two decision-level fusion methods of both EEG and facial expression detections by using a sum rule or a production rule.
Twenty healthy subjects attended two experiments.
The results show that the accuracies of two multimodal fusion detections are 81.25% and 82.75%, respectively, which are both higher than that of facial expression (74.38%) or EEG detection (66.88%).
The combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources.
American Psychological Association (APA)
Huang, Yongrui& Yang, Jianhao& Liao, Pengkai& Pan, Jiahui. 2017. Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition. Computational Intelligence and Neuroscience،Vol. 2017, no. 2017, pp.1-8.
https://search.emarefa.net/detail/BIM-1139849
Modern Language Association (MLA)
Huang, Yongrui…[et al.]. Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition. Computational Intelligence and Neuroscience No. 2017 (2017), pp.1-8.
https://search.emarefa.net/detail/BIM-1139849
American Medical Association (AMA)
Huang, Yongrui& Yang, Jianhao& Liao, Pengkai& Pan, Jiahui. Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition. Computational Intelligence and Neuroscience. 2017. Vol. 2017, no. 2017, pp.1-8.
https://search.emarefa.net/detail/BIM-1139849
Data Type
Journal Articles
Language
English
Notes
Includes bibliographical references
Record ID
BIM-1139849