Eliminating other-race effect for multi-ethnic facial expression recognition

Mingliang Xue; Xiaodong Duan; Wanquan Liu

doi:10.3934/mfc.2019004

Article Contents

2019, Volume 2, Issue 1: 43-53. Doi: 10.3934/mfc.2019004

This issue Previous Article SEMANTIC-RTAB-MAP (SRM): A semantic SLAM system with CNNs on depth images Next Article Online optimization for residential PV-ESS energy system scheduling

Eliminating other-race effect for multi-ethnic facial expression recognition

1.
Dalian Key Lab of Digital Technology for National Culture, College of Computer Science and Engineering, Dalian Minzu University, Dalian 116600, Liaoning, China
2.
Department of Computing, Curtin University, Kent Street, Perth, WA 6102, Australia

^* Corresponding author: Xiaodong Duan
^* Corresponding author: Xiaodong Duan

Published: March 2019

This work is supported by National Natural Science Foundation of China (Grant No.61672132), Science and Technology Foundation of Liaoning Province of China (Grant No.20170520234)ìNational Natural Science Foundation of China (Grant No.61602321), and Natural Science Fund Project of Liaoning Province (No.20170540694).

Abstract / Introduction Full Text(HTML) Figure(4) / Table(1) Related Papers Cited by

Abstract

It has been noticed that the performance of multi-ethnic facial expression recognition is affected by other-race effect significantly. Though this phenomenon has been noticed by psychologists and computer vision researchers for decades, the mechanism of other-race effect is still unknown and few work has been done to compensate or remove this effect. This work proposes an ICA-based method to eliminate the other-race effect in automatic 3D facial expression recognition. Firstly, the depth features are extracted from 3D local facial patches, and independent component analysis is applied to project the features into a subspace in which the projected features are mutually independent. The ethnic-related features and expression-related features are supposed to be separated in ICA subspace. Hence, ethnic-sensitive features are then determined by an entropy-based feature selection method and discarded to depress their influence on facial expression recognition. The proposed method is evaluated on benchmark BU-3DFE database, and the experimental results reveal that the influence caused by other-race effect can be suppressed effectively with the proposed method.

Keywords:

Mathematics Subject Classification: Primary: 68T10; Secondary: 68U10.

Citation:

Full Text(HTML)

Figure 1. The average performance of facial expression recognition on East-Asian individuals when the ethnic-related features are removed gradually

Download: Full-size image PowerPoint slide

Figure 2. The confusion matrix of multi-ethnic facial expression recognition before(a) and after(b) ethnic-related feature elimination based on East-Asian individuals

Download: Full-size image PowerPoint slide

Figure 3. The average performance of facial expression recognition on White individuals when the ethnic-related features are removed gradually

Download: Full-size image PowerPoint slide

Figure 4. The confusion matrix of multi-ethnic facial expression recognition before(a) and after(b) ethnic-related feature elimination based on White individuals

Download: Full-size image PowerPoint slide

Table 1. The ethnicity distribution of BU-3DFE database

Ethnicity Sample Size Number of 3D Faces

White 51 1224

East-Asian 24 576

Black 9 216

Hispanic-Latino 8 192

Indian 6 144

Middle-East Asian 2 48

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	B. M. Craig, Z. Jing and O. V. Lipp, Facial race and sex cues have a comparable influence on emotion recognition in chinese and australian participants, Attention Perception and Psychophysics, 79 (2017), 2212-2223. doi: 10.3758/s13414-017-1364-z.
[2]	M. N. Dailey, G. W. Cottrell, C. Padgett and R. Adolphs, Empath: A neural network that categorizes facial expressions, Journal of Cognitive Neuroscience, 14 (2002), 1158-1173. doi: 10.1162/089892902760807177.
[3]	M. N. Dailey, C. Joyce, M. J. Lyons, M. Kamachi, H. Ishi, J. Gyoba and G. W. Cottrell, Evidence and a computational explanation of cultural differences in facial expression recognition, Emotion, 10 (2010), 874-893. doi: 10.1037/a0020019.
[4]	C. Darwin and I. J. Rachman, The Expression Of Emotions In Man And Animals, Julian Friedmann, 1979.
[5]	J. R. D'Errico, Understanding gridfit, Information available at: http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do.
[6]	P. Ekman and W. V. Friesen, Facial action coding system (facs): A technique for the measurement of facial actions, Rivista Di Psichiatria, 47 (1978), 126-138.
[7]	P. Ekman, E. R. Sorenson and W. V. Friesen, Pan-cultural elements in facial displays of emotion, Science, 164 (1969), 86-88. doi: 10.1126/science.164.3875.86.
[8]	G. A. Feingold, The influence of environment on identification of persons and things, Journal of the American Institute of Criminal Law and Criminology, 5 (1914), 39-51. doi: 10.2307/1133283.
[9]	S. Fu, H. He and Z.-G. Hou, Learning race from face: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 36 (2014), 2483-2509. doi: 10.1109/TPAMI.2014.2321570.
[10]	A. Hyvarinen, Fast and robust fixed-point algorithms for independent component analysis, IEEE Transactions on Neural Networks, 10 (1999), 626-634. doi: 10.1109/72.761722.
[11]	R. E. Jack, C. Blais, C. Scheepers, P. G. Schyns and R. Caldara, Cultural confusions show that facial expressions are not universal, Current Biology Cb, 19 (2009), 1543-1548. doi: 10.1016/j.cub.2009.07.051.
[12]	R. E. Jack, O. G. B. Garrod, H. Yu, R. Caldara and P. G. Schyns, Facial expressions of emotion are not culturally universal, Proceedings of the National Academy of Sciences of the United States of America, 109 (2012), 7241-7244. doi: 10.1073/pnas.1200155109.
[13]	H. Li, J. Sun and L. Chen, Location-sensitive sparse representation of deep normal patterns for expression-robust 3d face recognition, in Biometrics (IJCB), 2017 IEEE International Joint Conference on, IEEE, 2017, 234-242. doi: 10.1109/BTAS.2017.8272703.
[14]	H. Li, J. Sun, Z. Xu and L. Chen, Multimodal 2d+ 3d facial expression recognition with deep fusion convolutional neural network, IEEE Transactions on Multimedia, 19 (2017), 2816-2831. doi: 10.1109/TMM.2017.2713408.
[15]	R. S. Malpass and J. Kravitz, Recognition for faces of own and other race, Journal of Personality and Social Psychology, 13 (1969), 330-334. doi: 10.1037/h0028434.
[16]	V. Natu and A. J. O'Toole, Neural perspectives on the other-race effect, Visual Cognition, 21 (2013), 1081-1095. doi: 10.1080/13506285.2013.811455.
[17]	P. J. Phillips, F. Jiang, A. Narvekar, J. Ayyad and A. J. O'Toole, An other-race effect for face recognition algorithms, ACM Transactions on Applied Perception (TAP), (2010), 1-12. doi: 10.6028/NIST.IR.7666.
[18]	J. A. Russell, Cross-cultural similarities and differences in affective processing and expression, in Emotions and Affect in Human Factors and Human-Computer Interaction, Elsevier, 2017, 123-141. doi: 10.1016/B978-0-12-801851-4.00004-5.
[19]	J. M. Susskind, D. H. Lee, A. Cusi, R. Feiman, W. Grabski and A. K. Anderson, Expressing fear enhances sensory acquisition, Nature Neuroscience, 11 (2008), 843-850. doi: 10.1038/nn.2138.
[20]	M. Wang, W. Deng, J. Hu, J. Peng, X. Tao and Y. Huang, Racial faces in-the-wild: Reducing racial bias by deep unsupervised domain adaptation, arXiv preprint, arXiv: 1812.00194.
[21]	M. Xue, X. Duan, J. Zhou, C. Wang, Y. Wang, Z. Li and W. Liu, A computational other-race-effect analysis for 3d facial expression recognition, Chinese Conference on Biometric Recognition, 2016, 483-493. doi: 10.1007/978-3-319-46654-5_53.
[22]	M. Xue, A. Mian, W. Liu and L. Li, Fully automatic 3d facial expression recognition using local depth features, in IEEE Winter Conference on Applications of Computer Vision, IEEE, 2014, 1096-1103.
[23]	X. Yan, T. J. Andrews, R. Jenkins and A. W. Young, Cross-cultural differences and similarities underlying other-race effects for facial identity and expression, Quarterly Journal of Experimental Psychology, 69 (2016), 1247-1254. doi: 10.1080/17470218.2016.1146312.
[24]	L. Yin, X. Wei, Y. Sun, J. Wang and M. J. Rosato, A 3d facial expression database for facial behavior research, in 7th International Conference on Automatic Face and Gesture Recognition (FGR06), IEEE, 2006, 211-216.
[25]	Q. Zhen, D. Huang, Y. Wang and L. Chen, Muscular movement model based automatic 3d facial expression recognition, in International Conference on Multimedia Modeling, Springer, 2015, 522-533. doi: 10.1007/978-3-319-14445-0_45.
[26]	Q. Zhen, D. Huang, Y. Wang and L. Chen, Muscular movement model-based automatic 3d/4d facial expression recognition, IEEE Transactions on Multimedia, 18 (2016), 1438-1450. doi: 10.1109/TMM.2016.2557063.