Drowsiness and Yawn Detection System using Machine Learning

(1)

Vol. 71 No. 4 (2022) 190 http://philstat.org.ph

Drowsiness and Yawn Detection System using Machine Learning

Shubham Shukla, Vijay Kumar Sharma Computer Science and Engineering, Meerut Institute of Engineering and Technology,

Meerut, India

Article Info

Page Number: 190 - 201 Publication Issue:

Vol 71 No. 4 (2022)

Article History

Article Received: 25 March 2022 Revised: 30 April 2022

Accepted: 15 June 2022 Publication: 19 August 2022

Abstract

Face produce data that can be used to determine tiredness level. Many facial appearances derived from the face to decide the extent of fatigue. Yawning, head movements and eye blink are examples. In this paper we detect the driver’s tiredness condition without equipping their body to devices. However, developing a drowsiness detection system that is dependable and systematic is a difficult challenge that necessitates precise and robust algorithms. To identify driver tiredness, a number of procedures have been tested in the past. Because deep learning is becoming more popular, these algorithms must be re-evaluated to determine their capability to detect drowsiness. Therefore, this study examines machine learning approaches such as Hidden Markov Models (HMMs), Support Vector Machines (SVMs) and Convolutional Neural Networks (CNNs) in the context of drowsiness detection.

Keywords – Machine Learning, Drowsiness Detection System, facial expression, fatigue detection.

I. INTRODUCTION

Recently a few years, the demand of transportation is rises. In present scenario the automobile transportation is necessary for everyone. In 2019, total 100 million vehicle sold globally. In 2019, Union Territories (UTs) and States had logged that total of 4,49,002 traffic accidents occurred with 4,51,361 people were injured in road accident and 1,51,113 people dead. In 2019, there will be 4,49,002 accidents and 1,51,113 deaths, averaging 414 deaths and 1,230 accidents per day and about 17 deaths and 51 accidents each hour [1]. In 300Km Lucknow Agra Expressway, approximately 40% of

(2)

traffic accidents are happened because exhausted driver fall asleep at the wheel and this study was conducted by the Central Road Research Institute (CRRI) [2]. As a consequence, tired drivers are a prominent and often noticed risk factor in traffic accidents. In recent years, the drowsiness-driving- detection system has been a popular study issue. There are distinct indicators of drowsiness, such as:

Constant yawning.

Inability to maintain visual contact.

Leaning forward with the head.

This paper explains how to deploy a range of machine learning approaches to make accurate and reliable predictions for driver tiredness.

Figure 1 Major Causes of Road Accident Deaths during 2019

Convolutional neural networks and Keras were used to generate the model. A deep neural network of such that accurately identifies pictures is called a convolutional neural network. Three layers make up a CNN: an input layer, an output layer, and a multi-level hidden layer. Convolution is achieved on these layers applying filters that perform 2D matrix multiplication on both layer and filters.

The Convolutional Neural Network model is broken down into the following layers:

● In convolutional layer; kernel size and nodes are respectively 3 and 32.

● In convolutional layer; kernel size and nodes are respectively 3 and 64.

● Fully connected layers have 128 nodes.

The top layer is completely linked and has two nodes. All layers use Relu activation, except the output layer used by Softmax. This paper explains how to deploy a range of machine learning approaches to make accurate and reliable predictions for driver drowsiness.

(3)

II. DRIVER DROWSINESS DETECTION PROCESS

Behavioural approaches use mounted cameras in the automobile to analyse face traits such head movement, eye state, yawning and blinking rate to determine tiredness levels. To extract facial details from a video feed, most researchers use a standard procedure. After the gathering of these features, drowsiness is determined using machine learning approaches such as Support Vector Machines (SVM), Hidden Markov Models (HMM), and Convolutional Neural Networks (CNN). These strategies are used to train models that can predict drowsiness using attributes and tagged outputs. The most difficult aspect of this approach is locating a broad dataset that covers the predicted range among skin tones and races. Figure 2 depicts a similar paradigm for most approaches to detecting driver drowsiness.

The following facial characteristics are frequently obtained from a driver face:

1 Eye closure analysis: The condition of the driver eye is an essential factor that is widely used to detect weariness. Eye Aspect Ratio (EAR) and Percentage of eye closure (PERCLOS) [3] are two methods for determining the level of drowsiness. In 2016, Soukupova and Cech proposed EAR [4], which is the ratio of the eye's height and breadth. PERCLOS, on the other hand, is the frequency at which the eyes close with time. The key distinction is that EAR categorises eye proportions downwards, whereas PERCLOS determines that whether eye is closed or open.

EAR = 𝑝2 − 𝑝6 + | 𝑝3 − 𝑝5 | 2| 𝑝1 − 𝑝4 |

PERCLOS = 𝑁𝑜.𝑜𝑓 𝑓𝑟𝑎𝑚𝑒𝑠 𝑜𝑓 𝑐𝑙𝑜𝑠𝑒𝑑 𝑒𝑦𝑒

3 𝑚𝑖𝑛 𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙 𝑜𝑓 𝑎𝑙𝑙 𝑓𝑟𝑎𝑚𝑒 − 𝑏𝑙𝑖𝑛𝑘𝑖𝑛𝑔 𝑡𝑖𝑚𝑒

2 Eye blink rate: Blink rate quantification technique, blink rate is used to assess sleepiness. The average blink rate per minute is about 10. The blinking rate slows down when the driver is sleepy.

3 Yawning analyses: Exhaustion or boredom can induce yawning, and it can suggest that a driver is about to fall asleep behind the wheel. By keeping note of the shape of your mouth and the position of your lip corners can detect yawning tendencies in the driver by measuring the width of the mouth [5].

4 Facial expression analysis: To identify tiredness in a driver, this method uses a combination of multiple face features. This includes traits like forehead wrinkles and severe head postures [6] [7].

(4)

The procedures shown in fig. 2 offer a popular method for detecting drowsiness. These are the steps:

Video capturing: This is the process of converting a video image captured by a still camera or smartphone into a sequence of images. Video imaging is made in such a way that only the driver's face can be seen.

Face Detection: The face in the picture frames is often detected in the second stage. The most widely used algorithm for detecting the driver face from a picture is Viola and Jones [8]. When using CNNs, the entire image is usually put into a network with several filters and the features are automatically extracted. Face recognition and feature extraction are combined into a single phase by CNNs.

Feature Extraction: Face detection uses a variety of approaches to extract features, including Local Binary Patterns (LBP), Histogram of Oriented Gradients (HOG) and Landmark Localization.

Feature Analysis: The mined features can then be further refined, such as using PERCLOS or EAR for mouth or eye analysis algorithms to identify yawning.

Classification: Classifiers are applied in the classification step to make decisions about a driver's level of sleepiness. If weighted parameters determine fatigue, an audible alarm will signal that the driver needs to rest.

Figure 2 Driver drowsiness detection process

(5)

III. LITERATURE REVIEW

There has been a lot of discussion about classification methods and how they affect drowsiness detection systems. This section of the literature review systematically illustrates the advantages and disadvantages of each classification technique along with a comparative analysis of the error rates in each. Here is a detailed overview of the classification methods, their advantages and disadvantages, and a graphical comparison. In drowsiness recognition systems, different types of machine learning classifiers are used to learn data[25]. Choosing the right classifier can have a significant impact on system performance. HMM, CNN and SVM classifiers are commonly used in drowsiness systems due to their high accuracy when compared to other classifiers such as KNN, HOG, and others.

A. SUPPORT VECTOR MACHINE (SVM)

SVM is a learning approach that focuses on regression and classification. SVMs are first used to select the training data set from a predefined set of data. SVMs learn from categorised data into the classified form of data in drowsiness detection. Various measures are used to determine and quantify the driver's drowsiness. A fully automatic system for detecting driver drowsiness is presented [3]. SVMs are trained in states such as close, open, and raise the alarm, and the Haar feature technique is employed for eye identification and facial detection. This structure achieves 100% accuracy, but at a slower frame rate, which results in the facial expressions being lost. Table 1 shows a list of systems that use SVM classifiers.

B. HIDDEN MARKOV MODEL (HMM)

The Hidden Markov Model is a statistical model that predicts the hidden state from the visible state.

HMM is used to identify gene annotation, sequence mistakes, DNA modelling, facial expressions, computer virus classification, among other things. Table 2 lists the characteristics and methodologies employed by the HMM-based Downiness Detection. By calculating face intensity, we can detect wrinkle changes in facial expression. In both day and night time Infrared (IR) webcams are used to eliminate the changes. Older people have deeper wrinkles so that this system has a disadvantage. The Hidden Markov Model (HMM) approach is used to track the eye based on geometrical aspects and

(6)

color, but the system does not recognise the face because the driver is not looking ahead and the model is implemented for indoor use.

C. CONVOLUTIONAL NEURAL NETWORKS (CNN)

CNNs are similar to traditional neural networks in that they are built up of neurons that combine with learnable weights. CNN employs the spatial convolution layers deemed most appropriate for the image, resulting in strong correlations. CNN has proven to be effective in a variety of applications, including video analysis, image identification, and classification. The best results for object recognition were achieved in 2012, demonstrating the excellent results in Deep CNN, but CUN and Yoshua applied CNN first on computer vision. The suggested system for detecting driver drowsiness employs representation learning, with the Jones and Viola algorithm utilised to detect the face. The photo is first cropped at 48 * 48 pixels and passed to the outer layer of the network with 20 filters. The output goes to the SoftMax layer, but the system fails because it ignores the head posture. However, another author gave more precise results using a deep 3D neutral network, which is treated as a combination of two additional filters per page. If the driver changes his head the system works efficiently. Table 3 shows the method of assessing drowsiness according to the CNN classification.

Ref Measure Classifier Description Accura

cy [9] Eye state SVM It is based on the concept of

supervised learning.

98.4%

[10] Eye state Binary The eye condition was detected using a binary technique.

93.5%

[11] Eye closure Harr feature with SVM

For eye closure identification, the Haar algorithm is used with SVM.

99.74%

[12] Eye state SVM and HOG The HOG algorithm extracts features. To detect the present condition of the eye, SVM is deployed.

91.6%

(7)

[13] Yawing and Eye closure

Binary SVM with linear kernel

Binary SVM with linear kernel may be used to recognise eye closure and yawing.

94.5%

Table 1 Drowsiness detection methods based on SVM.

cy [14] Eye state HMM Used to detect the eye state

information.

99.7%

[15] Eye blink HMM Used HMM to detect eye blinking rate.

95.7%

[16] Eye blinks SVM and HMM Detects drowsiness using eye blinking rate.

90.99%

[17] Eye closure and other features

SVM and HMM Detect the closure of the eye and some other features.

97%

[18] Eye state HMM Used for the detection of the eye states.

95.9%

[19] Eye state and head position

HMM Drowsiness is detected using the condition of the eyes and the posture of the head.

Nil

Table 2 HMM-based methods for detecting drowsiness.

cy [20] Visual feature CNN with SoftMax

layer

Used voila and Jones algorithm with the CNN and SoftMax Layer of the visual features.

78%

[21] Eye state CNN Used voila and Jones algorithm with 98%

(8)

the CNN for the eye state.

[22] Eye state MTCNN and DDDN Used MTCNN and DDDN for the eye state detection.

91.6%

[23] Eye state CNN Used the Ada-boost and LBF and PERCLOS with CNN for the Eye state detection.

95.18%

Table 3 Drowsiness detection systems using CNN.

IV. COMPARATIVE STUDY OF CLASSIFICATION METHODS

Each classifier does not have to be appropriate in every case, because each classifier has its own set of advantages and disadvantages. The selection of a suitable classifier based on the system specification is critical for better performance and accuracy. Table 4 discusses the parameters for these classifiers as well as their merits and demerits. SVM, HMM, and CNN classification methods are compared in terms of test error rates. A slighter error rate leads to increased efficiency. Figure 3 depicts a comparative analysis. The comparative research in our study shows that HMM is more accurate than other two algorithm because of its lower error rate. SVM is the most widely used classification algorithm, owing to its ease of use.

Figure 3 Comparative graph of classifier.

0 1 2 3 4 5 6 7 8 9 10

SVM HMM CNN

Error Rate

(9)

Vol. 71 No. 4 (2022) 198 http://philstat.org.ph Techni

que

Parameters Merit Demerit

HMM Probabilities, hidden states, head position, eye closure, eye state and eye blinks.

Record the relationship between measurements that may indicate device power fluctuations.

Limited by the intrinsic structure.

CNN Visual features, Eye state, and Eye gaze.

Image recognition accuracy.

Needs lot of training data, high computational cost and slow training.

SVM Eye state, Eye closure, and Yawing.

Abundance of

implementation,

implicitly, good out of sample generalization, Guarantees optimal,

On noisy datasets, it's less effective, and it's not ideal for huge data sets.

Table 4 Merit and Demerit of classifiers.

V. CONCLUSION

There are many behavioural and machine learning strategies that can be used to detect driver drowsiness. This study provides an overview of driver drowsiness tactics using machine learning methods and provides a discussion of the various features and indicators used for classification. These systems' principal purpose is to sense a tiny shift in a driver's facial appearance that conveys drowsiness signals. There are several approaches (physiological, behavioural, vehicle-based) to measure sleep, and this study focuses on the behavioural method because it is non-invasive, works in a variety of lighting conditions, and does not require vehicle modifications.

The most effective supervised learning strategies have been discussed. The benefits and drawbacks of such methods are extensively analysed, as well as a comparison study. A classifier's accuracy varies depending on the situation, according to presents evidence. However, SVM is the most widely used classifier since it provides superior accuracy and speed in most scenarios, although it is not ideal for

(10)

large datasets. HMM has a low error rate, although it is slower to train and more expensive than the SVM classifier.

REFERENCES

1. https://morth.nic.in/sites/default/files/RA_Uploading.pdf

2. https://www.financialexpress.com/india-news/40-of-highway-accidents-occur-due-to-drivers- dozing-off/1659901/

3. Knipling, R., and P. Rau. "PERCLOS: A valid psychophysiological measure of alertness as assessed by psychomotor vigilance." Federal Highway Administration (FHWA) (1998).

4. Soukupova, Tereza, and Jan Cech. "Eye blink detection using facial landmarks." 21st computer vision winter workshop, Rimske Toplice, Slovenia. 2016.

5. Zainal, Mohd Shamian Bin, Ijaz Khan, and Hadi Abdullah. "Efficient drowsiness detection by facial features monitoring." Research Journal of Applied Sciences, Engineering and Technology 7.11 (2014): 2376-2380.

6. Nakamura, Taro, Akinobu Maejima, and Shigeo Morishima. "Detection of driver's drowsy facial expression." 2013 2nd IAPR Asian Conference on Pattern Recognition. IEEE, 2013.

7. Choi, In-Ho, Chan-Hee Jeong, and Yong-Guk Kim. "Tracking a driver’s face against extreme head poses and inference of drowsiness using a hidden Markov model." Applied Sciences 6.5 (2016): 137.

8. Viola, Paul, and Michael J. Jones. "Robust real-time face detection." International journal of computer vision 57.2 (2004): 137-154.

9. Sabet, Mehrdad, et al. "A new system for driver drowsiness and distraction detection." 20th Iranian Conference on Electrical Engineering (ICEE2012). IEEE, 2012.

10. Sabet, Mehrdad, et al. "A new system for driver drowsiness and distraction detection." 20th Iranian Conference on Electrical Engineering (ICEE2012). IEEE, 2012.

11. Punitha, A., M. Kalaiselvi Geetha, and A. Sivaprakash. "Driver fatigue monitoring system based on eye state analysis." 2014 International Conference on Circuits, Power and Computing Technologies [ICCPCT-2014]. IEEE, 2014.

(11)

12. AL-Anizy, Ghassan Jasim, Md Jan Nordin, and Mohammed M. Razooq. "Automatic driver drowsiness detection using haar algorithm and support vector machine techniques." Asian J.

Appl. Sci 8.2 (2015): 149-157.

13. Pauly, Leo, and Deepa Sankar. "Detection of drowsiness based on HOG features and SVM classifiers." 2015 IEEE International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN). IEEE, 2015.

14. Manu, B. N. "Facial features monitoring for real time drowsiness detection." 2016 12th International Conference on Innovations in Information Technology (IIT). IEEE, 2016.

15. Bagci, A. Murat, et al. "Eye tracking using Markov models." Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004.. Vol. 3. IEEE, 2004.

16. Pan, Gang, et al. "Eyeblink-based anti-spoofing in face recognition from a generic webcamera."

2007 IEEE 11th international conference on computer vision. IEEE, 2007.

17. Sun, Yijia, Stefanos Zafeiriou, and Maja Pantic. "A hybrid system for on-line blink detection."

Hawaii International Conference on System Sciences. 2013.

18. Tadesse, Eyosiyas, Weihua Sheng, and Meiqin Liu. "Driver drowsiness detection through HMM based dynamic modeling." 2014 IEEE International conference on robotics and automation (ICRA). IEEE, 2014.

19. Zhang, Bo, Wenjun Wang, and Bo Cheng. "Driver eye state classification based on cooccurrence matrix of oriented gradients." Advances in Mechanical Engineering 7.2 (2015): 707106.

20. Choi, In-Ho, Chan-Hee Jeong, and Yong-Guk Kim. "Tracking a driver’s face against extreme head poses and inference of drowsiness using a hidden Markov model." Applied Sciences 6.5 (2016): 137.

21. Dwivedi, Kartik, Kumar Biswaranjan, and Amit Sethi. "Drowsy driver detection using representation learning." 2014 IEEE international advance computing conference (IACC). IEEE, 2014.

22. George, Anjith, and Aurobinda Routray. "Real-time eye gaze direction classification using convolutional neural network." 2016 International Conference on Signal Processing and Communications (SPCOM). IEEE, 2016.

(12)

23. Reddy, Bhargava, et al. "Real-time driver drowsiness detection for embedded system using model compression of deep neural networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017.

24. Zhang, Fang, et al. "Driver fatigue detection based on eye state recognition." 2017 International Conference on Machine Vision and Information Technology (CMVIT). IEEE, 2017.

25. N. Lal, M. Singh, S. Pandey, and A. Solanki, "A Proposed Ranked Clustering Approach for Unstructured Data from Dataspace using VSM," 2020 20th International Conference on Computational Science and Its Applications (ICCSA), Cagliari, Italy, 2020, pp. 80-86, DOI:

10.1109/ICCSA50381.2020.00024.-