5.2 Experimental results
5.2.3 Time Elapsed to Access AWS Rekognition
To determine the difference in AWS rekognition processing time with different types of images, this evaluation submitted all of the images to the AWS Rekognition service, including the originals, grayscale images, face-cropped images, and finally, face-cropped grayscale images. The same image type will be delivered repeatedly until the average time elapsed difference between two subsequent loops is as small as possible. The figure below illustrates the average time elapsed accessing AWS Rekognition using various types of images, as well as the accuracy associated with each image type.
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
72 Figure 5.2.3.1 Average Time Elapsed Against Different Types of images with
corresponding Accuracy.
In general, the improvement in AWS Rekognition processing time is not significant.
When original images are compared to face-cropped grayscale images, the difference is less than 0.05s. The visualization also demonstrates a mere reduction in the time required to obtain the result from AWS Rekognition, which is 0.031s. However, the processing time of the face detection algorithm used in this evaluation was greater than the time reduction associated with accessing the FER service. The primary issue here may be caused by the OpenCV Haar Cascade pre-trained face detection algorithm's low performance. Additionally, the effect of grayscale conversion is negligible. After converting the images to grayscale, the time reduced is less than 0.01s. However, the grayscale conversion took only 0.000257s and is required before performing OpenCV face detection on the images. To further reduce image processing time and the time required to obtain the AWS Rekognition result, a more efficient face detection model than OpenCV Haar Cascade face detection must be explored and implemented.
Original Grayscale Cropped Cropped Grayscale Accuracy 1 0.924528302 0.773584906 0.754716981 Average Time Elapsed 0.250520201 0.241044587 0.219466455 0.217090093
100.00%
92.45%
77.36% 75.47%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
0.2 0.21 0.22 0.23 0.24 0.25 0.26
Accuracy/%
Average Time Elapsed/s
Image Types
Average Time Elapsed Against Different Types of images with corresponding Accuracy
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
73 5.2.4 Accuracy Changes after Image Processing Implemented
First of all, to evaluate the accuracy changes when different image processing techniques are implemented on the input images, there are 53 images being chosen and labeled with the expressions. The images are then converted and cropped, generating another three sets of images and the expressions are expected to be the same as the original images. After obtaining the FER results from the AWS rekognition, the detected expression will be compared with the original labels, the true value, and compute the accuracy by dividing the number of correct detections by the total number of images which is 53. The computed accuracy for each type of image is shown in the table below.
Table 5.2.4.1 FER Accuracy when Sending Different Types of Images to AWS Rekognition
Image Types Original Grayscale Face-cropped Face-cropped grayscale Accuracy 53/53
= 100%
49/53
= 92.45%
41/53
= 77.36%
40/53
= 75.47%
The table shows that the accuracy is dropping along with the image processing technique implemented. Comparing the original and grayscale images, there are two confused expressions been misclassified as sad and angry, respectively, one sad expression misclassified as calm and lastly one calm expression as sad. The details are shown in table 5.2.4.2. At this point, it shows that the misclassified expressions are confused, sad, angry and calm. These expressions can be quite confusing when the expression is not strong and obvious. When people are a little confused, sad or angry, they can merely frown or even when people are calm and brainstorming, they frown.
Therefore, misclassifying these expressions is not a really big problem as humans also misunderstand them sometimes.
Table 5.2.4.2 Misclassified Grayscale images
Filenames Detected expressions True values
20.jpg SAD CALM
18.jpg ANGRY CONFUSED
3.jpg CALM CONFUSED
39.jpg CALM SAD
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
74 While comparing the face-cropped images with the original images, there are even more images that have been misclassified. The expressions that are not classified correctly are confused, happy and sad, as shown in table 5.2.4.3. The sad expression is the same case described in the section on grayscale images. Besides, there are seven confused expressions been misclassified into sad, disgusted, surprised and calm, including 18.jpg and 3.jpg, which are also misclassified when using the grayscale but into different expressions. The confused expression is misclassified into new expressions, disgusted and surprised. By looking at the images, there are common that the people were staring, mouth open or frowning, which might be confusing sometimes, even when identified by a human. On the other hand, the happy expressions are misclassified into surprised and angry. The case of a happy expression being classified as an angry expression is kind of extreme because the people in the image are shown as very exciting and make a funny face with mouth opened largely. While misclassifying happy expressions into surprised could be due to the similar positive vibes being expressed, sometimes people do feel happy when being surprised.
Table 5.2.4.3 Misclassified face-cropped images
Filenames Detected expressions True values
12.jpg SAD CONFUSED
15.jpg DISGUSTED CONFUSED
18.jpg SURPRISED CONFUSED
19.jpg SURPRISED CONFUSED
3.jpg DISGUSTED CONFUSED
51.jpg SAD CONFUSED
6.jpg CALM CONFUSED
29.jpg SURPRISED HAPPY
45.jpg SURPRISED HAPPY
47.jpg SURPRISED HAPPY
9.jpg ANGRY HAPPY
39.jpg CALM SAD
Lastly, 13 images are misclassified, and their true values are angry, confused, happy and sad. Firstly, the sad expression is still the same misclassified expression in the sections of grayscale images and face-cropped images. The combination of face cropping and grayscale conversion also caused new expressions to be recognized wrongly, which is the angry expression being classified as a surprised expression. By
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
75 looking into the images shown in Figures 5.2.4.1 and 5.2.4.2, the face-cropped image seemed less angry because some facial details were lost after the grayscale conversion.
Therefore, it has been misclassified as a surprised expression because of a big mouth.
While the misclassified confused images are the same in the previous section of the face-cropped images, some expressions are classified as calm expressions rather than disgusted and surprised expressions, which is probably due to the loss of face details after grayscale conversion. Lastly, there are four happy expressions that have been misclassified. Three of them are misclassified the same as in the previous section, but another expression which was classified as angry previously is now recognized as disgusted by the AWS rekognition. Using the first intuition, looking at the face-cropped grayscale 53.jpg and comparing it with the original image, there are really two different expressions. It might probably be due to the colour issue, which makes the expression less vivid and gives a different impression.
Table 5.2.4.4 Misclassified face-cropped grayscale images
Filenames Detected expressions True values
21.jpg SURPRISED ANGRY
12.jpg SAD CONFUSED
15.jpg DISGUSTED CONFUSED
18.jpg CALM CONFUSED
19.jpg CALM CONFUSED
3.jpg CALM CONFUSED
51.jpg SAD CONFUSED
6.jpg CALM CONFUSED
23.jpg SURPRISED HAPPY
29.jpg SURPRISED HAPPY
45.jpg SURPRISED HAPPY
53.jpg DISGUSTED HAPPY
39.jpg CALM SAD
Bachelor of Computer Science (Honours)
Faculty of Information and Communication Technology (Kampar Campus), UTAR
76 Figure 5.2.4.1 Original 21.jpg
Figure 5.2.4.1 face-cropped grayscale 21.jpg