The main focus of the proposed method presented in this chapter is to extract discriminative features from the informative regions of a face for efficient facial expression recognition. We pro- posed an informative region extraction (IRE) model that determines the importance of different facial sub-regions using projection analysis of expressive images. Procrustes-based approach is also proposed to estimate a common reference image (CRI). Subsequently, a weighted-PLBP (WPLBP) feature extraction approach is proposed, which extracts LBP features from the infor- mative regions and then features are weighted on the basis of importance of respective regions.
The region of importance is estimated by projecting an image onto the reference image. Our proposed modeling of informative regions is inspired by the theory proposed by Ekmanet al.[2], which says that any facial expression is resulted from the movements of a set of facial regions with respect to the their neutral state. Because of the movements of different facial regions, the texture patterns of underline regions differ from their original texture patterns of the corre- sponding regions of a neutral face. These deviations in terms of the projection errors indirectly give information of the informative regions of a face for a particular facial expression. In view of this, IRE model and WPLBP are proposed to estimate the informative regions of a face, and subsequently discriminative features are extracted for expression recognition. The importance of a region is judged on the basis of the analysis of projection errors. Higher projection er-
Table 3.7: Performance of the state-of-the-art methods
Existing works Features Selec- tion
Classifier Database Accuracy (%) Rahulamathavan et
al. [3]
Local fisher dis- criminant
LFDA MUG 95.24
- do - - do - JAFFE 94.37
Shan et al. [60] Local binary pattern
linear SVM CK+ 94.60
- do - - do - JAFFE 79.80
LBP Boost - do - CK+ 95.00±3.2
Oshidari et al. [69] Adaptive Ga- bor wavelet
SVM JAFFE 90.00
Dongcheng et al. [123]
Gabor wavelet phase
k-NN JAFFE 92.37
Gabor ampli- tude + phase
- do - - do - 93.48 Kotsia et al. [41] Gabor wavelet SVM JAFFE 88.10
- do - - do - CK 91.60
Zhong et al. [102] LBP MTSL CK+ 89.89
Liu et al. [103] LBP MTSL + SVM CK+ 97.70
ror signifies that the particular region undergoes significant deviation from their corresponding neutral state for showing the expression. As a result, that particular region conveys significant information of the considered facial expression. So, the features are only selected from the informative regions as they convey significant information of a facial expression. Hence, the feature vector derived only from the informative regions is more discriminative as compared to the feature vector obtained from the entire face image.
Performance of our proposed approach is evaluated on three well-known databases namely MUG, JAFFE, and CK+. Our proposed WPLBP approach consistently gives better perfor- mance as compared to other existing approaches. The proposed method is validated for different parameters, such as separability analysis and performance for different image resolutions. The Krippendorff’s alpha coefficients for different training-testing pairs show the reliability of the
proposed work. However, one of the shortcomings of the proposed method is that projection- based approach cannot be directly extended to non-frontal face images to determine the impor- tance of different facial regions. Moreover, the informative regions of a non-frontal face image is a subset of informative regions of a frontal face image, i.e., although some of the informative regions may be invisible in a non-frontal face images, still recognition can be made with the help of rest of the visible informative regions. Hence, the method proposed in this chapter can be extended to utilize informative regions of a face for multi-view or view-invariant face images.
4
An Informative Region-based Face Model for FER
The accuracy of facial expression recognition algorithms depends on the kind of face model used for recognition. Existing face models mainly extract geometrical features from some pre-defined facial points. Also, the face models available in the literature are not suitable for extracting features from all the informative regions of a face. The drawbacks of existing face models are highlighted in this chapter, and subsequently informative regions extracted in the previous chapter are deployed for generating an efficient face model. The advantage of our proposed face model is that both geometrical and texture features can be extracted from the facial points which are marked on the basis of informative regions. The efficacy of our proposed face model is validated by recognizing gestures only with the help of facial expressions.
4.1 Introduction
The motivation of introducing this chapter is to highlight structural difference of face models [14, 92, 96, 108], and to show the shortcomings of these models. A brief overview of different face models has been discussed in Section 2.3.2. Most of the existing face models extract geometrical facial features from a number of pre-defined facial points. In the literature, it has been established that texture features extracted from landmark points improve recognition accuracy [96, 97]. This is due to the fact that texture features add some of the additional informative regions/facial points which were not previously taken into consideration by the geometrical features. Skin textures around landmark points changes significant during facial expressions, and hence, all the regions showing texture pattern variations should be considered for the face model. However, existing face models are mainly intended to extract geometrical features, and so, existing face models are not optimal for texture-based FER. Also, there are different types of face models, and hence, it is difficult to select the best one for facial expression recognition. No standard analysis is available in the literature related to the selection of a suitable face model for FER.
In this view, few existing face models are analyzed, and the shortcomings of these models are highlighted in this chapter. Finally, we proposed a more efficient face model with the help of informative regions of a face. The performance of our proposed face model is validated by recognizing facial expressions from both images and videos.