text: Meaningful facial parts can convey key cues for both facial action unit detection and expression prediction. Textured 3D face scan can provide both detailed 3D geometric shape and 2D texture appearance cues of the face which are beneficial for Facial Expression Recognition (FER). However, accurate facial parts extraction as well as their fusion are challenging tasks. In this paper, a novel system for 3D FER is designed based on accurate facial parts extraction and deep feature fusion of facial parts. In particular, each textured 3D face scan is firstly represented as a 2D texture map and a depth map with one-to-one dense correspondence. Then, the facial parts of both texture map and depth map are extracted using a novel 4-stage process consists of facial landmark localization, facial rotation correction, facial resizing, facial parts bounding box extraction and post-processing procedures. Finally, deep fusion Convolutional Neural Networks (CNNs) features of all facial parts are learned from both texture maps and depth maps, respectively and nonlinear SVMs are used for expression prediction. Experiments are conducted on the BU-3DFE database, demonstrating the effectiveness of combing different facial parts, texture and depth cues and reporting the state-of-the-art results in comparison with all existing methods under the same setting.
  @cite_15: In this paper, we present a new idea to analyze facial expression by exploring some common and specific information among different expressions. Inspired by the observation that only a few facial parts are active in expression disclosure (e.g., around mouth, eye), we try to discover the common and specific patches which are important to discriminate all the expressions and only a particular expression, respectively. A two-stage multitask sparse learning (MTSL) framework is proposed to efficiently locate those discriminative patches. In the first stage MTSL, expression recognition tasks are combined to located common patches. Each of the tasks aims to find dominant patches for each expression. Secondly, two related tasks, facial expression recognition and face verification tasks, are coupled to learn specific facial patches for individual expression. The two-stage patch learning is performed on patches sampled by multiscale strategy. Extensive experiments validate the existence and significance of common and specific patches. Utilizing these learned patches, we achieve superior performances on expression recognition compared to the state-of-the-arts.
  @cite_32: This paper mainly studies facial expression recognition with the components by face parsing (FP). Considering the disadvantage that different parts of face contain different amount of information for facial expression and the weighted function are not the same for different faces, an idea is proposed to recognize facial expression using components which are active in expression disclosure. The face parsing detectors are trained via deep belief network and tuned by logistic regression. The detectors first detect face, and then detect nose, eyes and mouth hierarchically. A deep architecture pretrained with stacked autoencoder is applied to facial expression recognition with the concentrated features of detected components. The parsing components remove the redundant information in expression recognition, and images don't need to be aligned or any other artificial treatment. Experimental results on the Japanese Female Facial Expression database and extended Cohn-Kanade dataset outperform other methods and show the effectiveness and robustness of this algorithm.
summary: Having small patches as proposed by @cite_15 lacks the ability to automatically localize and determine what part is in effect for an expression. It can also have trouble distinguishing patches when applied on faces of different gender and ethnicity. Having a learned detector for the face and parts as proposed by @cite_32 cannot guarantee the robustness as much as face detection, alignment and localization techniques do, especially when there are peculiar samples provided. Both proposals used the JAFFE and CK+ database, which are not very diverse when it comes to their subjects ethnicities. In their experiments, they adopt a cross-validation approach, in which they do not ensure a subject independent protocol. This can result in a high performance which can be inconsistent when a new subject is ever tested.