Facial Expression Recognition (FER) is one of the relevant research fields in Computer Vision (CV) and Human-Computer Interaction (HCI). Using accuracy and total loss on BU-3DFE and CK+ datasets 127 4.3 The result of the comparative studies of multi-label models' performance.
Olusayo Ekundayo, Serestina Viriri, "Facial Expression Recognition
Olusayo Ekundayo, Serestina Viriri, "Facial Expression Recognition and Ordinal Intensity Estimation: A Multi-Label Learning Approach", Ad- and Ordinal Intensity Estimation: A Multi-Label Learning Approach", Advances in Visual Computing , LNCS Springer, vol. Viriri, "Deep Forest Approach For Facial Expression Recognition", Image and Video Technology, LNCS Springer, Vol.
Olusayo Ekundayo and Serestina Viriri, "Multilabel Convolutional Neural Network for Facial Expression Recognition and Ordinal Intensity Es-
- Facial Expression Recognition
- Facial Expression and Intensity Estimation
- Motivation
- FER Challenges
- Aims and Objectives
- Research Approach
- Thesis Contributions
- Datasets Sources
- Organisation of work
Facial expression intensity estimation is a noticeable difference between facial expression images with the same expression or the degree of facial expression difference measurements. This research aims to automate a simultaneous recognition of human emotional state with associated intensity from facial expression images.
The chapter covers the review and the suggestions of possible areas of FER applications and details of existing challenges in FER. It includes Categoris-
We claim that the data is relevant to the challenges we considered in this work and obtained copyright permission from the producer. We also claim that we have limited ourselves to the terms of use of the data.
Presents the implementation frameworks for FER, ranging from the deep learning approach for facial expression recognition, a multilabel convolutional
Contains the result presentation, analysis and discussion of each of the frameworks presented in chapter 3 including detail of the comparisons with the
In this Chapter, a conclusion with a general overview and contribu- tion of the thesis is presented with a highlight of possible future work
Facial Expression Recognition: A Review of Trends and Techniques
- Introduction
Facial Expression Recognition (FER) has received notable attention in computing, which is not limited to Computer Vision (CV) and Human-Computer Interaction (HCI).
Introduction
Deep Forest Application for Facial Expression RecognitionRecognition
- Introduction
- Background Model for Deep forest
- How Deep Forest Works
- Mathematical Illustration of the framework
Note that the output of each layer is the average of the probability distribution for cases in the leaf node of the trees for each forest. Section 3 contains a brief introduction to random forest and the description of the proposed deep forest framework for facial expression recognition. Summary of the investigation performed on the Deep Forest model with increasing number of classifiers.
Facial Expression recognition and ordinal In- tensity Estimation: A Multilabel Approach
- Introduction
- Method Discussion
- Model Evaluation
Convolution operation: with convolution operation, patches from expression images are extracted and transformed to generate a 3D feature map where depth is the number of lters encoding the unique part of the expression image. Equation (3.2) is the mathematical representation of the convolution operation where the image of the input expression is I(x,y), interpolated with the kernel H (f,f) and the output feature map G[p,q]of row p and emphq expressed accordingly. Furthermore, the dimension D(p,q) of the output feature map as given in (3.3) is calculated using the expression image size (m,n,c), s is the number of steps and lter H(f,f ), m is the image height, n is the width, c is the channel or depth of the image, and p is the padding number.
A Multilabel Learning Approach
1 Introduction
One of the information that accompanies emotion is the degree or intensity of the expressed expression. The order of arrangement of this work is as follows; Section 2 is a review of some of the existing work on intensity assessment of facial expression recognition. Section 4 contains the description of the multi-label approach and the adaptation of the CNN model to the multi-label problem.
2 Related Works
Section 5 presented the description of the experiment and provided information from both the databases and the multi-label statistics for evaluation. HMM, as a classifier detects the emotion, and a change point detector encodes the expression intensity from the weight vector. The aim of this work is to estimate facial expression intensity based on ordinal value and the emotion recognition simultaneously.
3 Convolution Neural Network
4 Multi-label Approach
Problem Description
Downsampling of the same pooling size is applied after the third and fifth layers. Equation (1) is a mathematical representation of the convolution operation, where the image of the input term f is convolved with the h kernel, and the output feature map of row x and column y is computed. We use the ReLu activation function on each convolutional layer of the network and the dense layer.
5 Experiment
- Database
- Data Pre-processing and Experiment Discussion
- Evaluation Metrics
- Result and Discussion
We used the sigmoid activation function in (3) for the final prediction at the fully connected layer and binary cross-entropy loss in (4) is used for calculating the model loss. This work considered two important and important preprocessing techniques to improve the performance of the system. The higher the value within the closed range between 0 and 1 the better the performance of the model.
6 Conclusion
We present a summary of the result obtained from four categories of the experiments in Table 1. In the evaluation, we observed overfitting when the epoch number is 25, and as shown in Table 1. The result of CK+ is relative to the BU-3DFE+Augmentation, which indicate the adaptability of the proposed method.
Introduction
Multilabel Convolution Neural Network (ML-CNN) used binary relevance to model the emotion with the respective intensity using ordinal metrics. Part of this paper was presented1 and published 2 and Improved method accepted for publication 3. 2020) Facial Expression Recognition and Ordinal Intensity Estimation: A Multi-label Learning Approach. eds) Advances in Visual Computing. 3Olusayo Ekundayo and Serestina Viriri Multi-label Convolutional Neural Network for Facial Expression Recognition and Ordinal Intensity Estimation", PeerJ computer science (under peer review).
ABSTRACT
INTRODUCTION
The classification of facial expressions into basic emotional states has been discussed several times in the literature in different ways, but the approach could not explain the intensity of the recognized emotion. To the best of our knowledge, none of the works on facial expression recognition and intensity estimation have considered a static FER dataset, and the environments investigated in the study are sequential and dynamic environments. This ensures that the prediction probability of any class is independent of the other classes.
RELATED WORKS
The summary of the existing method is shown in Table 1. 2020) proposed a multi-task learning system using a cascaded CNN, and the objectives tend to include student attention and emotion recognition and student intensity assessment in an intelligent classroom system . Some of the CNN optimization approaches focus on improving the discriminative power of the network through modifying the loss function to reduce intra-class variance and increase inter-class variance. The proposed multi-label deep learning model can learn hierarchical structure in FER datasets during network training and predict emotion and ordinal intensity in facial expression simultaneously.
MULTILABEL CONVOLUTION NEURAL NETWORK MODEL (ML-CNN) DESCRIPTION
We use accuracy and loss metrics (binary cross-entropy and islanding loss) to evaluate model performance on test data in each of the described experiments. When collecting data, other factors are also taken into account, which may threaten the recognition of FER. The model result on BU-3DFE shows that human variation factors have a limited effect on the model.
CONCLUSIONS
Serestina Viriri conceived and designed the experiments, performed the experiments, analyzed the data, performed the computational work, prepared figures and/or tables, authored or reviewed drafts of the paper, and approved the final draft. BU-3DFE: http://www.cs.binghamton.edu/~lijun/Research/3DFE/3DFE_Analysis.html - CK+: CK+ is available by approved request: https://www.jeffcohn.net/resources/.
Introduction
This section presents a framework that resolves the ambiguity of facial expression tags and the inconsistency of data annotations by learning correlations between tags using manifold learning, which models tag correlations with similarity distance and uses a convolutional graph network in a semi-supervised manner to recover emotion distributions from logical labels. Facial expression is described as a graph-related problem, where facial expression data points form the nodes of the graph, and similarity distances resulting from manifold learning form the edge information of the graph.
Facial expression recognition with manifold learning and graph convolutional network
Olufisayo Ekundayo 1* | Serestina Viriri 1*
Introduction
First, the results of the Deep Forest model and the performance comparison with the existing emotion classification models are presented. The hardware specifications for deep forest calculations include: (intel(R)Core(TM)i7-4770sCPU @3.10 GHz 3.10GHz and RAM: 8GB) Dell machine. CHPC high computing device was used for Multilabel Convolution neural network and Manifold graph convolution network was computed on (intel(R)Core(TM)i7-4770sCPU @3.10GHz 3.10GHz and RAM: 0.8GB) Dell machine.
Deep Forest Approach for Facial Expression Recog- nitionnition
Also, Figure 4.2 contains the confusion matrices of the model probability prediction accuracy on the CK+ and the BU-3DFE, respectively. The performance of Deep Forest is justified on the basis of facial expression classification by comparing its performance with the state-of-the-art DNN method (FERAtt) [36]. Similarly, FERAtt requires high computing devices such as GPU for its significant time of computation, unlike the Deep Forest which performed its layer by layer learning on the available computing device (intel(R)Core(TM)i7-4770sCPU @3.10 GHz 3.10 GHz and RAM: .8 GB) at a reasonable time of calculation.
Multilabel Convolutional Neural Network for Facial Expression RecognitionFacial Expression Recognition
The proposed Multilabel-CNN performance evaluation is based on the accuracy and loss function in Table 4.2 and Figure 4.3. The details of the comparison are presented in Table 4.4 The model output was also compared with some newer deep learning networks for FER classification that consider the same data sets; details of the comparison are available in Table 4.5. From Table 4.3 and Table 4.4, both the basic model (ML-CNN) and the optimized version (VGGML-CNN) outperform RAKELD, CC, MLkNN and MLARAM.
Facial Expression Recognition with Manifold Learning and Graph Convolutional NetworkLearning and Graph Convolutional Network
A comparative study with some existing models is presented in Table 4.8, and the model was compared with some basic methods on datasets using accuracy as a performance evaluation metric. Pro- Table 4.3: Result of comparative studies of the performance of multi-label models on BU-3DFE. Increasing the number of k parameters was considered to account for Table 4.4: Comparative performance studies of multi-label models on the extended BU-3DFE data set.
Conclusion
The learned information about nodes and neighbors in the model was visualized and this is presented as points on the graph. Since the output of the model has 16 vectors, 16-dimensional space is required for the image display. Visualization of output from manifold GCN on CK+ and BU-3DFE showed that the model learned the correlations between data annotations.
- Conclusion
- Future work
This thesis focused on facial expression recognition tasks by providing some frameworks for understanding emotion recognition techniques improvement. The outcome presented diverse facial expression recognition applications and discussion of single-label learning, multi-label learning, and label distribution learning trends. The combination of deep features and deep forest for a robust facial expression recognition system should be explored in future work.
IEEE, 2013. 1
FGR 2006: Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition, pages 211216. Evaluating the intricacies of emotional facial expression of mental patients using computer models. Investigating the eects of emotional valence and arousal on acquisition performance in conditional automated driving.
IEEE, 2019. 3
Optimized Deep Learning Model Based on Gravitational Search Algorithm with Different Feature Sets for Facial Expression Recognition. Facial expression recognition method with multi-label distribution learning for understanding non-verbal behavior in the classroom.
