FACIAL FEATURE DETECTION

PROJECT REPORT

Amit Pillay &Ravi Mattani

Instructor: Dr. Roger Gaborski
Course: 405.757

Abstract
The aim of this project was to develop an algorithm that detects human
facial features like the mouth, nose and eyes in a full frontal face image. We
adapted a multi-step process in order to achieve the goal. To detect the face
region a skin-color segmentation method was used. Morphological
techniques were used to fill the holes that were created after the
segmentation process. From the skeletonization process, a skeleton of the
face was obtained from which face contour points were extracted. Facial
features can be located in the interior of the face contour. We used several
different facial-images to test our method. We achieved a detection rate of
over 90%.

Introduction
Face feature extraction is considered to be a key requirement in many
applications such as Biometrics, Facial recognition systems, video
surveillance, Human computer interface etc. Therefore reliable face
detection is required for the success of these applications.
The task of human facial feature extraction is not easy. Human face varies
from person to person. The race, gender, age and other physical
characteristics of an individual have to be considered thereby creating a
challenge in computer vision.
Facial feature detection aims to detect and extract specific features such as
eyes, mouth and nose.

Previous work & Research in the field of face recognition

There are many face extraction methods reported by researchers in the field
of face recognition. To combine image information and knowledge of the face
is the main trend.
In order to discriminate face candidates from the neck, ears and the incorrect
segmented parts, shape analysis of the color segmentation result is
necessary. One method (Saber and Tekalp, 1998; Lee et al., 1996) involves
first filling an ellipse to the segmented face for registration. Most of them do
not take into consideration the misalignment caused by ear and neck.
Ian-Gang Wang, Eric sung in their article have proposed a morphological
procedure to analyze the shape of segmented face region. This is an

unexplored approach in face detection research. We have incorporated some

of the methods proposed in this article. Several rules have been formulated
for the task of locating the contour of the face. These rules were primarily
based on the facial skeleton and knowledge of the face. The searching region
of the face features reduces. The facial features such as mouth, nostrils and
eyes can be more accurately located within the face contour.
Terrillon et al., 1998 mentions the problem of how other body parts such as
neck may lead to face localization error. A different approach was presented
in (Sobottka and Pitas, 1998) where the features are first detected and then
the contour was tracked using a deformable model. Brunelli and Poggio,1993
use dynamic programming to follow the outline in a gradient intensity map of
elliptical projection of face image.
Haalick and Shapiro, 1993 demonstrate how morphological operations can
simplify the image data while preserving their essential shape characteristics
and can eliminate irrelevances.

Face feature detection is a multi step process. We have incorporated several

methods and suggestions proposed in the articles mentioned in the above
section. We tried several methods for each step and the one that gave the
best results was selected.
In our project in order to detect the face region a skin-color segmentation
method was used. This method was developed by Yang and Waibel(1995)
This method is described in section 2.1.
To segment the face region we used morphological processes of dilation and
erosion. Section 2.2 describes them in detail.
The skeletonization process was used to extract face contour points. This
process is described in section 2.2.
The Skeleton thus obtained is used in the process of line segmentation which
includes line tracing and line merging. Facial features can be located in the

interior of the face contour. The process of Facial feature extraction is

described in section 2.3.
Flowchart depicting the entire process

Section 2.1: Skin Segmentation

This is one the important steps in face feature extraction process. The
efficiency of the color segmentation of a human face depends on the color
space that is selected. We used the finding by Yang and Waibel (1995,1996)
who noted that the skin colors of different people are very closely grouped in
the normalized r-g color plane. So we took some seed pixels from the face
area obtained the normalized r-g spread and classified the pixels that were
close enough to this spread as belonging to the face. The Figures below
demonstrate our results.

Fig1 : Skin color segmentation a)Original image b) Skin color segmentation

Section 2.2 Morphological Image Processing

(Process of filling interior holes of the face region and extraction of
the face skeleton for determining contour points)

Morphological image processing is a collection of techniques for digital

image processing based on mathematical morphology. Since these
techniques rely only on the relative ordering of pixel values, not on their
numerical values, they are especially suited to the processing of binary
images and grayscale images whose light transfer function is not known. [2]
Definitions of some morphological operations that we used in our project:
Erosion: Erosion is an operation that thins or shrinks the objects in the
binary image.Erosion are performed by IPT function imerode().
Dilation: Dilation is an operation that grows or thickens the objects in the
binary image. Dilation is performed by IPT function imdialate().
For successful facial feature extraction the accurate computation of the
contour points is very important. This helps in locating searching regions.
Both neck and ears have the same color as that of the face. Hence they are
connected to the face region. Therefore we need to separate them so as to
better locate the facial features.
In order to detect the skeleton, the face region was filled by applying
multiple morphological dilation operations. Then the same number of erosion
operations was applied. There exist holes that corresponds to the eyes, nose
and mouth etc. We fill these holes using the dilation operation. The erosion
operation is applied to the dilation result in order to restore the shape of the
face.
Then we obtained the skeleton of the face. We used the IPT function
bwmorph() in MATLAB to extract the skeleton. The ears and the neck can be
separated from the segmented face region by analyzing the skeleton.
The figures on the next page show the output we obtained after performing
operations.

Figure
a) The output after performing skin color segmentation

Figure
b) The output after filling the interior holes of the face region using dilation
and erosion operations.

Figure
c) Overlay skeleton with face region.

Contour Tracing

Some hn to tjvcadipcjascomks;lbvms;kcmadk;cmsa;he longest skeleton line

segment).
Rule 1.2 The left vertex will be selected as candidate for contour fitting if
most of the horizontal line segments are positioned at t
CSNVChe left of the symmetry axis.
JIRGJFRFJAECLC;MKSLRKMVSL/C NSFK.VNCD
Rule 1.3. The right nvkdAvjS;CMSL (left and right contour points,
respectively). They are each sorted from top to bottom, respectively.
X
DANC
Rule 1.6. If the difference between the horizontal coordinates of a point from
right sorted points set and any one of its previous and succeeding point is
large enough (a threshold is set), then the
Rule 2. The point set satisfying the above will be doubled using symmetry
axis, i.e., for a left fitting point there exists a right point that can be
calculated using the symmetrical axis and vice versa.
All the resulting skeleton lines are checked form Rules 1 to 2 in turn. The
points for fitting the contour of the human face can be collected while
excluding the points that lie on the ear and neck.

Section 2.3 Feature extraction within the ROI

After the contour has been found we obtained the Region of interest.
Features such as eyes, nose and mouth can be found within the ROI. We
locate the mouth first. Once the mouth is detected we can determine the
eyes and nose easily.
For finding the lips we tried several methods. We performed edge detection
using Sobel operator that finds the horizontal line within the ROI .Finding
vertical position of the line between the lips was done by using horizontal
integral projection ph of the binary image in the search region. ph is obtained
by summing up the pixels values in each row of the search. Because lip line

will have the longest horizontal line, its vertical position can be located
where ph has maximum value.
We also used some prior knowledge about the face in order to determine the
lips. We knew that lips are centered horizontally in the lower half of the face
and are isolated .We took advantage of this information in order to constrain
the vertical position of the search area.
Once we have located the lips finding the eyes and nose was done in similar
fashion, each time constraining the search area in vertical direction.

Figure: Contour points extracted from earlier process

Figure : The Region of Interest

Figure: Edges removed for detecting FF.

Figure: Final Results

Results & Analysis :

Number of images tested: 30
Success: 90%
The following Table shows the results of the first 13 images.
Image no

Name

Lips

Nose

Mouth

c37m.jpg

c4m.jpg

c1m.jpg

c2m.jpg

3
4

c5m.jpg

c6m.jpg

c7m.jpg

c8m.jpg

c9m.jpg

10

c10m.jpg

11

c11m.jpg

12

c12m.jpg

13

c13m.jpg

The Results and analysis of the experiments conducted are demonstrated in

the next page.

One of the experimental results with steps is shown in figures below.

The experimental face images were downloaded from www.faceresearch.org
site. A total of 30 images of different people were selected randomly. The
experimental results shown above illustrate that our method is quite good. A
correct recognition rate of 90% is obtained using out method. There were
some images which showed wrong detection of facial features. An analysis of
these images shows that our method failed to extract enough contour points
from these images which resulted from ill skeleton-extraction. Lack of
contour points resulted into restricted ROI in which needed features failed to
appear.
One way of improving the skeleton extraction could be to more generalize
the face region extraction process followed by sufficient dilation and erosion
process. Better the effect of skin-region extraction process, better will be the
resulting skeleton obtained and thus better will be the contour points
extracted.

Other issue that limits given experiment is the image size. Since the
experiment is performed using MATLAB IPL toolbox, running time of the
program is a major issue. Therefore large size images take longer time to
compute the result and some may not give desired results cause of image
complexity.

Figure: a) Original Face Image

Figure: c) After Dilation & Erosion

b) Skin Color Segmentation

d) Skeleton of the image

Figure: e) Contour points

Figure: g) After applying Edge Detection

to the ROI

f) ROI

h) Final image with features detected

References:
[1]Frontal-view face detection and facial feature extraction using color and
morphological operations by Jian-Gang Wang, Eric Sung *
[2] A Model-Based Gaze Tracking System by Rainer Stiefelhagen, Jie Yang,
Alex Waibel
[3] Digital Image Processing Using MATLAB by Gonzalez, Woods
&Eddins,Prentice
[4] Images taken from : www.faceresearch.org
[5] Prof. Gaborskis lecture slides.
[4] www.wikipedia.com