Vous êtes sur la page 1sur 9


A Robust Hybrid Design for Driver Fatigue Detection

Ijaz Khan, Hadi Abdullah and Mohd Shamian Bin Zainal
Abstract This paper presents a robust design for fatigue detection that analyse yawning and eyes status of driver to detect drivers level of vigilance. This system consists of seven stages starting from face detection. Combination of Viola Jones and skin color faces detection techniques are used for face detection in parallel that makes the design more accurate and fast. Both techniques are applied to detect face simultaneously. The technique that first detects face passes the image to next level, while image processed by second techniques is discarded. Snake counter is applied to track the detected face. Eyes and mouth are detected by using eye and mouth maps. To avoid complicated calculations to detect eyes and mouth in each image a snake counter is applied to track them continuously. The system uses skin color pixel information and a threshold to differentiate between eye and closed eyes. Using variation information of snake counter the system detects yawning state of driver. Yawning and closed eyes detection combined with two face detection techniques not only makes this system fast but also provides accuracy.
Index Terms fatigue detection, face detection, yawning analysis, eyes status analysis, snake counters .


RVING in drowsines have long been acknowledged to be one of the main hazard in safe driving. Drowsiness has a negative imapact on drivers abitiltes of driving and impairs drivers judgment and quick response time. The National Highway Traffic Safety Administration (NHTSA) estimates that 100,000 police-reported crashes are the direct results of driver fatigue each year[1].This results in an estimated 1550 deaths, 71,000 injuries, and $12.5 billion in monetary losses. Based on researches done by the Real Automvil Club de Espaa (RACE), driver drowsiness involves a high percentage (30%) of traffic accidents[2].According to national sleep foundation 2005 in America poll, 60% of adult drivers about 168 million people- say they have driven a vehicle while feeling drowsy in the past year and more than one third (37% or 103 million people) have actually fallen asleep at the wheel. Therefore an assistive system inside vehicle that monitors drivers lever of vigilance while he is drowsy, which can alert the driver after detection drowsiness is essential to prevent road side accidents. Researchers have been developing many techniques for drowsiness detection. Some of major techniques are shown in figure 1. These techniques can be classified into 3 categories. First is physiological approach in which drowsiness is de-

tected by recording brain activates in alpha and gamma

Fig. 1. Driver fatigue detection techniques.

Ijaz Khan is a researcher of Electrical engineering at University Tun Hussein Onn Malaysia (UTHM), Batu Pahat, Malaysia. Hadi Abdullah is a researcher of Electrical engineering at University Tun Hussein Onn Malaysia (UTHM), Batu Pahat, Malaysia. Shamian Bin Zainal is head of laboratory-faculty of electrical and electronices engineering at University Tun Hussein Onn Malaysia (UTHM), Batu Pahat, Malaysia.

band EEG (Electroencephalography) [3],[4],[5],blood pressure ,heart rate variability[6] and recording eyes movements EOG (Electrooculography) [7],[8],[9],[10],[11]. However sensitivity of physiological approach is still low compared with visual data. The second is driving behavior in which data observed from vehicle and road such as longitudinal and lateral speeds, accelerations, steering wheel angle are monitored[12],[13].Different equipments are used to acquire

Fig. 2. Fatigue detection phases

such data from vehicles and roads. However applications of exotic devices in this approach are limited to vehicle type and road conditions. The third approach is to detect drowsiness from facial expression such as yawning and eyes closure[14],[15],[16],[17].Numerous results demonstrates that facial expressions provides rich and reliable information about drowsiness[18]. In this paper we propose a hybrid drowsiness detection system which takes real time images of driver which are processed in four phases shown in figure 2. In first phase face is detected using hybrid method of Viola Jones and skin color detection technique (discussed in section 2.1 and 2.2). A snake counter is applied for continuous face tracking in phase 2(section 2.3). In 3rd phase mouth and eyes are detected and snake counter is applied to track eyes and mouth. To monitor yawning state and state of eyes (closed or open) snake counter and skin color pixel detection techniques are used (discussed in section 3). Once driver is observed in yawning state or with closed eyes for more than two seconds an alarm will on or a verbal statement will be given by the system advising driver to drive carefully. Using hybrid of Viola Jones and skin color detection for face detection not only increases accuracy but also decreases time required to detect face. Also a combination of yawning detection and eye state detection gives our system a promising solution for drowsiness detection.

Fig. 3. Face detection techniques.


Face detection techniques can be categorized into two major groups that are feature based approaches[19],[20] and image based approaches. Image based approaches use linear subspace method, neural networks[21],[22],[23] and statistical approaches for face detection. Feature based approaches can be subdivided into low level analysis, feature analysis and active shape model. Figure 3 demonstrate techniques and methods used to detect face in an image.

2.1 Viola Jones Method The basic principal in viola Jones[24] method is to scan a sub window that can detect a face in an input image. Viola Jones detector runs several times across an image with different size. This detector requires same number of calculations regardless of the size of the image which makes it faster than the standard image processing approach that on the contrary rescale the input image size and then

Fig. 4. Viola Jones integral image construction.

run a fixed size detector. Viola Jones detector is constructed of integral image and haar-like features shown in Figure 4 and 5. An image is first converted to integral image, which is done by summing up all the pixels above and to the left

To get the single resultant value of each feature, the sum of the white rectangles are subtracted from the sum of the black rectangles. Figure 6 shows application of haar-like feature on actual face image. For feature selection viola Jones used AdaBoost[25],[26] to build a simple classifier from computationally efficient features. Adaboost is a machine learning boosting algorithm that constructs a strong classifier from weak classifier. A weak classifier can be described mathematically as

1 pf ( x) > p h( x, f , p, ) = 1 otherwise


Fig. 5. Viola Jones Haar like features.

of the concerned pixel, and then set entire pixels equal to that sum. Sum of rectangle ABCD= D-(B+C)+A.

Where is polarity, is applied feature, is a sub window and ! is threshold. It is obvious that most of the region in an image consist of non-face. Viola Jones detector detects the non face area in an image and discards that area which results in detection of face area. For this purpose viola jones take advantage of cascading. When a sub window is applied to cascading stages, each stage decides whether the sub window is a face object or not. Sub windows which are not face are discarded and those which contain some percentage of having face objects are passed to next stage. Final stage is considered to have a high percentage of face objects. Higher number of stages in cascading provides more efficient result of detecting face in a sub window. Figure 7 shows cascading stages used in Viola Jones method.

Fig. 7. Cascading stages used in Viola Jones Method to discard non-


Fig. 6. Viola Jones Haar like features applied on actual face image.

The detector analyzes sub window with the help of features that consist of two or more than two rectangles.

2.2 Skin color detection technique The second method used for face detection is based on skin color face detection method. In face detection approaches detection of skin color is first step. Major advantages of this technique are its robustness, non sensitivity to position and shape invariance. However for this technique to work, it is vital to use an accurate color space model. Some existing color spaces are RGB, CMY, XYZ, UVW, LSLM, L*a*b*, L*u*v*, LHC, LHS, HSV, HSI, YUV, YIQ, YCbCr[27],[28] out of which Most commonly RGB, HIS , YCbCr are used. We are using RGB color space for skin color detection as it is native representation of color

images and it is widely used for processing and storing digital images. RGB color space consist of three basic colors R (red), G(green), B(blue) that can be combined to produce any resultant color. Although different people have different skin colors, studies have shown that actual difference lies between the intensities[29]. So if Brightness is removed from color representation, the difference between human skin colors can be reduced. In order to detect skin color following set of rules have been found to be more accurate than other models. (R>95) AND (G>40) AND (B>20) (first condition) AND (max {R, G, B} min{R, G, B} > 15) AND (|RG|> 15) AND (R>G) AND (R>B) AND (R>220) AND (G>210) AND (B>170) AND (R>B) AND (G>B) (2) Pixels of RGB image are detected as skin if first condition holds true. Rest of the conditions are used to ensure that RGB components must not be close together, that ensures grayness elimination. It also ensures that R and G component must not be together which must be true for fair complexion R is largest in R, G and B for pixel of skin color regions in RGB color space and B maybe be larger than G for pixels of skin color regions in the shadow. These rules are suitable for detection of skin color under regular illumination conditions as well as in shadow. The resultant image from this technique is a black and white image in which skin is converted to white and rest of the colors are converted to black. Face can be detected then by cutting the biggest white connected area with in black and white image.

(5) (6) Where D is known as the average distance between all the pairs from the snaxel set. External represent the external energy which moves the snake to the feature of interest in an image such as boundaries. An external energy equation is (7) Where ent operator. is a grey level image and is the gradi-

The drowsiness detection system is based on seven levels which can be classified as: Face detection Face tracking Eyes detection Mouth detection Eyes and mouth tracking Eye status analysis Yawning detection

A detailed flow chart of the system is given in figure 8.

2.3 Snake Counter for face tracking The snake model (active counter model) was introduced by Kass et al [30]. It was used to solve the problem of tracking humans face and other complex shapes and has been reported to successful track humans head and face boundaries [31]. Snakes are curves defined within an image domain that can move under the influence of internal and external forces. Internal force comes from within the curves itself while external forces are computed from the image data. The internal force imposes a piecewise smoothness constraint while the image forces push the snake towards the salient image features, for instance edges, lines or contours. A snake performed in the image could be represented by a set of snaxels, Vi=(Xi,Yi) for I=0,,N-1 where the X and Y coordination of the snaxel I are denoted as Xi and Yi respectively. The energy function could be written as
(3) E internal represents internal energy and can be computed as the sum of the snake contour continuity energy and the contour curvature energy. (4)

3.1 Face detection Face detection is accomplished by using two methods; Viola Jones and Skin color detection. To make the design robust these two methods are used simultaneously. Whichever detects face first sends it to next level and other image processed by any of these methods is discarded. This not only makes system faster but also gives high accuracy of both methods. 3.2 Face tacking Since this system is designed to take real time images of driver, it will be very slow and time consuming if the system applies face detection on every image it takes. Snake counter is applied to track face after detecting it. And once face tracking is lost by snake counter it is again detected in previous level. 3.3 Eyes detection To detect eyes in the face, we built an eye map that is based on the chrominance component [32]. The eye map is generated by observing that high and low values are found around the eyes. Eye map can be generated by equation (8).

1 C 2 2 EyeMapC = (Cb ) + (C r ) + ( b C ) r 3


Once the positions of eyes are highlighted by eye map, the image then can be converted to black and white image, in which eyes are denoted by white color pixels.

Fig. 8. System design flowchart showing application of different methods and techniques to detect drowsyness.

Since eyes are located in upper half of the face there for to reduce our calculations and to reject false alarm we only consider the upper part of the black and white image. Eyes then can be detected based on their geometrical shape and biggest connected component of white pixels.

3.4 Mouth detection In order to find mouth in the face, a mouth map can be generated based on the strongest differences in the color of face area and mouth region. Since mouth area contains strongest red component and weaker blue component [32] , hence the chrominance component will be less then in mouth area. Using equation (9 and 10) we can generate mouth map.

The image will be converted to black and white after the mouth area is highlighted by the mouth map. Since mouth is located in the lower half part of the face, hence the lower half part of the image will be taken into consideration only. Mouth can be detected as the biggest white globe in the lower part of the image.

MouthMap = (Cr )

Cr 2 (Cr ) C b


3.5 Eyes and mouth tracking Once mouth and eyes are detected in the face area, it will be cumbersome and time consuming to detect mouth and eyes each time in new image, therefore to reduce such calculations snake counters (active counters) are applied to mouth and eyes which will continuously track them in the image. The dynamic update feature of the snake contours makes it an appropriate choice for mouth and eyes tracking. Additionally the snake counter algorithm is fast enough to track eyes and mouth in real time environment. 3.6 Yawning detection Yawning while driving depicts that driver is getting lousy therefore detection of yawning is most important step in detection of drowsiness. During yawning the area of mouth increases vertically so when area of mouth increases the area of the applied snake counter will increase in sub sequent frames as well. The state of yawning is detected if the area of the snake counter increases vertically from its normal size which is already calculated when

1 2 C r ( x, y ) n ( x, y ) = 0.95 1 C r (x, y ) Cb (x, y ) n ( x , y )


3.7 Eye status detection When eyes are tracked using snake counter, skin color detection technique is applied to monitor state of eyes weither open or closed. When eyes are closed number of skin color pixel will increase compared to open eyes because colors of eyes are not skin (Sclera is mostly composed of white and some red part and Iris black, brown, grey etc no skin). Open and closed eyes detection using skin color technique can be seen in figure 10. To detect drowsiness from eyes the system must be able to differentiate between eye blinking and closed eyes due to sleepiness. Since human normal eye blinking takes less than a second the system threshold is set to two seconds. Whenever system detects closed eyes for more than two seconds it will alert the driver of being drowsy.

The simulation of drowsiness detection system is done in on a computer with specifications given in table 1. TABLE 1 Specification and simulator used.
System Parameters
Fig. 9. (a) Snake counter applied on mouth to track mouth while driver is moving his head. (b) Detection of yawning by observing area of snake counter which increases vertically

Values and versions 640X480 Pixels Matlab 7.1R2012a Intel(R) core(TM)i52450(2.50GHzX2)

Image size Simulation Software Processor

RAM Operating System

4GB Windows 7( 64bit)

Fig. 10. Closed and open eyes identification using Skin color detection. (a) Shows actual open and closed eyes, (b) shows processed skin color technique image where white part shows non skin pixels and black shows skin color pixels.

snake counter was applied to the mouth in normal state. Fiqure 9 demonstrates snake counter applied on mouth in normal and yawning state.

The system performance with six different cases their time of detection and system selection is given in detail in table 2 and figure 11. In first image shown in figure 11 (a), (b) and (c), (a) is face detection using Viola Jones which is accomplished by this method using system parameters given in table 1 in 0.946s. While the same image processed by skin color detection with same system parameters is done 1.132s shown in figure 11(b). Since Viola Jones technique detected image first, the system will take that image and pass it to next level where snake counter is applied. The difference of time is not much but drowsiness detection case of driver even this small duration can make increase system performance and help driver form any accident possibility.

Fig. 11. Viloa Jones and Skin Color Detection techniques applied on images and system selection. TABLE 2 Face detection using Viola Jones and Skin color detection, their detection time and system selection. Image Image (a),(b),(c) in Figure 11 Image (d),(e),(f) in Figure 11 Image (g),(h),(i) in Figure 11 Image (j),(k),(l) in Figure 11 Image (m),(n),(o) in Figure 11 Image (p),(q),(r) in Figure 11 Viola Jones Image (a) Detection time= 0.964s Image (d) Detection time= 1.019s Image (g) Detection time= 1.105s Image (j) Detection time= 0.9654s Image (m) Detection time= 0.990s Image (p) Detection time= 0.985s Skin Color Detection Image (b) Detection time= 1.132s Image (e) Detection time= 1.034s Image (h) Detection time= 1.059s Image (k) Detection time= 0.983s Image (n) Detection time= 1.119s Image (q) Detection time= 0.910s System Selection Image(c) Viola Jones selected Image(f) Viola Jones selected Image(i) Skin Color selected

Image(l) Viola Jones selected Image(o) Viola Jones selected Image(r) Skin Color selected
Fig. 12. Eyes and mouth detection of image selected by system.

The images that are selected by the system are then send to next levels in which eyes and mouth are detected (discussed in section 3.3 and 3.4). Figure 12 shows images with eyes and mouth detection. After these detections Snake counter is applied to track mouth and eyes continuously. The system then continuously monitors eyes and mouth. And eyes status and yawning condition are detected (discussed with examples in 3.6 and 3.7). The overall accuracy of system in average of all seven levels (face detection, face tracking, eyes detection, mouth detection, eyes and mouth tracking, yawning detection and eye status detection) using system parameters given in table is 96.7%. The highest accuracy of system design is shown in first level i.e. face detection.

Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE, 2010, pp. 14051408. [5] M. A. J. L. Cantero, and R. M. Salas, "Human Alpha Oscillations in Wakefulness, Drowsiness Period, and REM Sleep: 2002. [6] J. Vicente, et al., "Detection of driver's drowsiness by means of HRV analysis," in Computing in Cardiology, 2011, 2011, pp. 89-92. [7] S. Vitabile, et al., "Bright Pupil Detection in an Embedded, Real-Time Drowsiness Monitoring System," in Advanced Information Networking and Applications (AINA), 2010 24th IEEE International Conference on, 2010, pp. 661668. [8] H. Singh, et al., "Eye tracking based driver fatigue monitoring and warning system," in Power Electronics (IICPE), 2010 India International Conference on, 2011, pp. 1-6. [9 ] X. Xingliang, et al., "Objective Evaluation of Driver Fatigue by Using Spontaneous Pupillary Fluctuation," in Bioinformatics and Biomedical Engineering, (iCBBE) 2011 5th International Conference on, 2011, pp. 1-4. [10] N. Alioua, et al., "Eye state analysis using iris detection based on Circular Hough Transform," in Multimedia Computing and Systems (ICMCS), 2011 International Conference on, 2011, pp. 1-5. [11] I. G. Damousis and D. Tzovaras, "Fuzzy Fusion of Eyelid Activity Indicators for Hypovigilance-Related Accident Prediction," Intelligent Transportation Systems, IEEE Transactions on, vol. 9, pp. 491-500, 2008. [12] L. Yulan, et al., "Real-Time Detection of Driver Cognitive Distraction Using Support Vector Machines," Intelligent Transportation Systems, IEEE Transactions on, vol. 8, pp. 340-350, 2007. [13] Y. Takei, Furukawa, Y, "Estimate of drivers fatigue through steering motion," SMC 05, Vol. 2, pp. 1765-1770, 2005. [14] M. Omidyeganeh, et al., "Intelligent driver drowsiness detection through fusion of yawning and eye closure," in Virtual Environments Human-Computer Interfaces and Measurement Systems (VECIMS), 2011 IEEE Different Electroencephalographic Phenomena within the Alpha Band," Neurophysiol Clin, 32, pp. 5471,

In our study we have designed a drowsiness detection system with increased efficiency, accuracy and speed by using two different methods for face detection. To make design faster snake counters are applied on face eyes and mouth which decreases processing time of detecting face, eyes and mouth in each real time image. Yawning in image is detected by taking information by variation in size of snake counted that is applied on mouth. To check status of eyes a threshold value is applied with skin pixel detection. This threshold differentiates between eyes blinking and closed eyes due to sleepiness. This design can be easily implemented using FPGA and a simple VGA camera placed on the base of steering wheel. With further improvement in different levels the system can have more accuracy and become a more practical product.

The authors wish to thank Almighty Allah. The work was supported by University Tun Hussein Onn Malaysia.

[1] a. NHTSA. Drowsly driver detection and warning system for commercial vehicle drivers: Field proportional test design, and progress. National Highway Traffic Safety Administration,Washington, http://www.nhtsa.dot.gov/. [2] E. Rogado, et al., "Driver fatigue detection system," in Robotics and Biomimetics, 2008. ROBIO 2008. IEEE International Conference on, 2009, pp. 1105-1110. [3] L. Ming-ai, et al., "An EEG-based method for detecting drowsy driving state," in Fuzzy Systems and Knowledge Discovery (FSKD), 2010 Seventh International Conference on, 2010, pp. 2164-2167. [4] A. Garces Correa and E. Laciar Leber, "An automatic detector of drowsiness based on spectral analysis and wavelet decomposition of EEG records," in Engineering in [18] [17] [16] [Online]. Available: [15]

International Conference on, 2011, pp. 1-6. L. Weiwei, et al., "Driver fatigue detection through pupil detection and yawing analysis," in Bioinformatics and Biomedical Technology (ICBBT), 2010 International Conference on, 2010, pp. 404-407. S. Abtahi, et al., "Driver drowsiness monitoring based on yawning detection," in Instrumentation and Measurement Technology Conference (I2MTC), 2011 IEEE, 2011, pp. 14. B. Hariri, et al., "Demo: Vision based smart in-car camera system for driver yawning detection," in Distributed Smart Cameras (ICDSC), 2011 Fifth ACM/IEEE International Conference on, 2011, pp. 1-2. T. . M. Ingre, B. Peters, A. Anund, and G.Kecklund, "Subjective sleepiness, simulated driving performance and

blink duration: examining individual differences," Journal of sleep research, vol. 15, no. 1, pp.4754, 2006. [19] F. Bayoumi, et al., "Feature-based human face detection," in Radio Science Conference, 2004. NRSC 2004. Proceedings of the Twenty-First National, 2004, pp. C211-10. [20] C. Yao-Jiunn and L. Yen-Chun, "Simple Face-detection Algorithm Based on Minimum Facial Features," in Industrial Electronics Society, 2007. IECON 2007. 33rd Annual Conference of the IEEE, 2007, pp. 455-460. [21] [22] H. Rowley, "Neural network-based face detection,," PhD thesis, Carnegie Mellon University, Pittsburgh (1999). K. Curran, X. Li, et al. (2005). "Neural network face detection.," Imaging Science Journal, the 53(2): 105-115 [23] A. Mohamed, et al., "Face detection based neural networks using robust skin color segmentation," in Systems, Signals and Devices, 2008. IEEE SSD 2008. 5th International Multi-Conference on, 2008, pp. 1-5. [24] a. M. J. J. P. Viola, "Rapid Object Detection using a Boosted Cascade of Simple Features," Proc. IEEE CVPR, 2001. [25] P. a. M. J. J. Viola, "Robust Real-Time Face Detection," International Journal of Computer Vision 57(2): 137-154., (2004). [26] P. a. M. J. Viola, "Rapid Object Detection Using a Boosted Cascade of Simple Features,," in: Proc. IEEE Conf. Computer Vision and Pattern Recognition. [27] H. X. Xinyu Wang, Xi Chen and Heng Li, "Fast and Robust Face Detection with Skin Color Mixture Models and Asymmetric AdaBoost," Proc. of SPIE Vol. 7496 749618-1, 2009. [28] I. Hsieh, S., Fan, K., C., Lin, C.,, "A statistic approach to the detection of human faces in color nature scene," Pattern Recognition, 35(7), 1583-1596 (2002). [29] [30] V. Vezhnevets, Sazonov, V. and Andreeva, A. (2003). "A Survey on Pixel-Based Skin Color Detection Techniques.." e. a. M. Kass, ""Snakes: Active contour models,"" International Journal of Computer Vision, vol. 1, pp. 321331, 1988. [31] S. R. G. a. M. S. Nixon, "A dual active contour for head and boundary extraction," IEE Colloquium on Image Processing for Biometric Measurement, pp. 6/14 London, UK.,, Apr. 1994. [32] H. Rein-Lien, et al., "Face detection in color images," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 24, pp. 696-706, 2002. Ijaz Khan joined University Tun Hussein Onn Malaysia in 2012 as a student of M.S Electrical engineering. He did his Bachelor in Electrical engineering from COMSAT Abbottabad, Pakistan. His major research focus is FPGA designs of security systems, Wireless intruder tracking systems using FPGAs, economical solutions for large scale monitoring systems and face monitoring systems as road safety feature in automobiles. Hadi Abdullah received his bechlour degree in Electrical engineering from COMSAT Abbottabad, Pakistan in 2011. In 2012 he joined University Tun Hussein Onn Malaysia as a research student of M.S in Electrical engineering. The major fields of his research are wire-

less senor networks, Ultra Wide Band applications and FPGA design of wireless radar as a road safety feature in automobiles.

Mohd Shamian Bin Zainal received his Ph.D degree in Electrical engineering form Hokkaido University Japan in 2010. He did M.S also in Electrical engineering form University Tun Hussein Onn, Malaysia in 2003. His Bachelor is from University technology Malaysia 2001. He accepted position of Head of Laboratory - Faculty of Electrical Engineering and Electronics in University Tun Hussein Onn Malaysia. Also he is senior lecturer teaching different subjects of Electrical engineering at UTHM since 2004. He currently supervises M.S research students in different research projects, mostly related to FPGA, Real Time Embedded Systems and Communication. His current research is Smart Sensor For Inhaler, New Approach Of Hardware Modeling By Using FPGA For Cardiac Reentrant Arrhythmia Real-Time Analysis Tool, Radar designs for high precision localization of intruder in high security area, FPGA based face tracking and observation for drivers drowsiness detection and road monitoring. Journal: Mohd Shamian, Shingo Yoshizawa, Yoshikazu Miyanaga, ``Ultra-Low Power and Large Scale Design of Sub-threshold Digital Circuits for Wireless Communication Systems,'' RISP Journal of Signal Processing, Vol. 13, No.6, pp.487-496, Nov. 2009. Learning modules: Microprocessor and Microcontroller. He is a member of the IEEE and the IEEE Computer Society.