3D VIRTUAL ENVIRONMENT NAVIGATION USING

FACE TRACKING TECHNOLOGY

Jasper Bekkers, Sjef van Doorn, Bart Knuiman, Matthijs van der Meide, Mike van Mourik
NHTV, mADE, International Game Architecture and Design
Pakhuis, Triple O campus, Reduitlaan 33, Breda, The Netherlands
{bekkers, sjefvandoorn, bart.knuiman, mlvdmeide, omfgwee}@gmail.com

Abstract
Research into the viability of face-tracking technology for interaction with computer
generated 3D virtual environments was performed. An experimental software package
was created that allowed people to navigate a virtual environment using only a standard
webcam. Test subjects were asked to navigate a testing environment while the computer
recorded statistics, after which they completed a questionnaire inquiring about the
experience of the user during the experiment. We conclude that face-tracking technology
can be used as a viable alternative for computer interaction, specifically for navigating 3D
virtual environments. We list further research subjects in this area to improve user
experience or to improve the experimental software package we created.

Keywords
3D virtual environment, navigation, face tracking
Introduction
With high budgets and short developing times the software industry generally tries to get
the most value for their money; generally by optimizing for the largest possible target
audience. Generally this doesn’t include people with limited mobility (such as people
suffering from Paraplegia or other conditions) because they require specialized peripheral
devices.
We investigated the viability of one alternative method of computer interaction to allow
people to navigate a computer generated 3D virtual environment by using only head
movements.
We integrated an existing face-tracking software package, Seeing Machines’ faceAPI [1],
with an existing 3D computer game engine, Sauerbraten [2], to create an application that
allows people to navigate in a virtual environment using only input recorded from a
standard webcam.
We performed a survey among 16 test subjects, letting them move around in the 3D
virtual environment, while the computer recorded statistics of the experiment.
Afterwards, the test subjects filled out a questionnaire commenting on the process. In
this paper we will present the findings of this research.
Literary Survey
Research in the face-tracking field has several focuses, such as matching imagery to a
database of known faces [3] as being used by security enforcement agencies, and the
real-time tracking of a face from video stream input. For the purposes of this project, we
are only interested in the accurate and robust tracking of a face for use as an input to
real-time computer programs, such as games. Therefore we focus only on previous work
in the real-time face-tracking area.
Video-based face recognition algorithms take a video-sequence (still images characterized
by temporal continuity) as input, on which a probabilistic approach is used to match the
most likely face location and rotation [6]. There are two major types of real-time face
recognition algorithms “optical flow analyzing” and “feature tracking” algorithms the first
of these 2 is not a very handy approach to tracking the rotation of the face and this why
the second is becoming more popular in the field of real time face tracking [7].
The Algorithms in the feature tracking group generally focus on detecting either one [5]
or more [4] [7] face features. Algorithms tracking only a single face feature require that
the user stay in a fixed position in front of the camera in order to detect head rotation.
Algorithms tracking multiple face features can calculate the head rotation as well as head
location by cross-referencing several points on the face that are in fixed locations relative
to each other (the nose-eyes triangle).
Face-tracking is a powerful and intuitive cue in an interactive computer environment. The
head-pose estimation obtained using a face-tracking algorithm can be used for direct
pointing (using the nose) when hands and/or feet are unavailable [4]. Possible situations
where this might be useful include (combat) airplane control, where hands and feet are
already in use, as well as disabled people who have lost control over some of their limbs.
Research into the needs of disabled people for interacting with computers specifically is
being performed, and includes research into the possible benefits from face-tracking
systems [8].
In addition, there is a paradigm shift towards more natural ways of interacting with
computers in general, including face-tracking and body-motion-tracking systems.
Experiment
We wanted to know if using face-tracking technology is a viable alternative to the
commonplace input methods currently in use, when navigating a computer generated 3D
virtual environment. For the face-tracking technology to be a viable alternative, it has to
meet several criteria:
- It has to be a usable and reliable input method
- The input has to be sufficiently precise to navigate a 3D virtual environment
- Usage has to be easy and intuitive for the user
- Usage has to be sufficiently comfortable for the user

To test if these criteria are met, an experiment was set up. A software package was
created that integrated face-tracking technology [1] and a simple 3D virtual environment
created in and rendered by an existing 3D computer game engine [2]. The software
package was run on a single laptop with integrated webcam, placed on a table. A chair
was provided for the test subject to sit on during the experiment.

The 3D environment used in the experiment consisted of a non-linear path that the test
subject had to walk over. Test subjects were required to complete the path twice, while
the time taken to complete the path was recorded by the researchers.

Afterwards, the test subjects were asked to complete a small questionnaire in which the
test subjects were asked about the experience. The questionnaire addressed questions
regarding the test subject’s experience during the experiment, in the following categories:
- Test subject’s comfortability during the experiment
- Difficulty of using the interface
- Difficulty of completing the path
- Intuitiveness of the interface
- Immersiveness of the interface

The minimum requirements of the software package we created are listed in figure 1.

To run the experiment, we used a single Processor: Core2Duo at 1.6 GHz

laptop with integrated webcam in a well-lit
area. The laptop was positioned at the
user’s eye-height. Test subjects were seated
in a chair for comfort, but this is not
required.
Memory: 1GB of internal memory
Hard disk space: 200MB free space
Graphics: SM2.0 support, 128MB memory
Webcam: 640x480 resolution WDM compatible
Operating system: Windows XP
figure 1: minimal system requirements
Results
We performed the
experiment on 16 test
subjects (14 male + 2
female, average age 21),
and collected the results.

The time taken by the test

subjects for completing the
course is shows in figure 2.
69% of the test subjects
show an improved time the
second time the path is
completed when compared
to the first. The average
improvement of the figure 2: time taken to complete path
subjects is 6%.

Answers given in the questionnaire are graphed in figure 3 – 6

figure 3: interface usage difficulty figure 4: interface usage intuitiveness

figure 5: experiment difficulty figure 6: experiment comfortability

Conclusion
We have proven that face-tracking technology can be used to navigate in a computer
generated 3D virtual environment, using only a standard webcam. The software package
we developed does not require specialized computer peripherals, most modern computer
systems with a webcam attached can run the experiment.

Test subjects indicated that using face-tracking technology is an intuitive way of

interacting with the computer, but is due to being an unconventional input method,
requires getting used to. Our current software package lacks more advanced features that
would improve the quality of the experience even more.

Seeing that 69% of the test subjects improve their times when repeating the experiment,
indicates that there is a learning curve in using the face-tracking interface. This means
that with further exercise in using the interface, the technology may be used more
effectively.

It would appear that the usage of the face-tracking interface does not cause serious
discomfort in the test subjects. The majority of the test subjects indicated that they are
comfortable to very comfortable while using the interface.

Half of the test subjects stated they were immersed in the 3D virtual environment, this
would indicate that while using face-tracking interfaces, people are not losing their
experience of immersion in the virtual environment. This is good news for the software
industry, especially for the divisions specialized in entertainment software (such as
games), that partially rely on their ability to immerse users into the virtual environment.

We conclude that face-tracking technology is a viable alternative input method for

computers, and specifically, for navigating computer generated 3D virtual environments.
More research into this area might come up with a natural way of interacting with
computers for the general public, not only for people that cannot otherwise access virtual
environments.
Further Research
Our research should provide a valuable platform for further research into this field, we’ve
had some great results but we feel that they could be improved upon with additional
research.

The software package we developed is basic; it can be improved by adding dynamic

recalibration features that would allow the user of the interface more freedom in moving
around, as long as the user stays in the field of view of the camera. Additionally, an
algorithm to eliminate jitter from the face-tracking input might provide a smoother
experience for the user. Even a simple algorithm that discards extremes and allows for
integration of average input over time would improve the software package substantially
in out use-case, and most likely in most use-cases.

The range of input signals available from webcam imagery is extensive, and not all of it is
processed in our software package. There is room for more input into virtual
environments, for example, by using gesture recognition to indicate actions to perform
inside the virtual environment. Additionally, the blinking of a user’s eyes is detectable,
which might be used for interaction purposes as well.

Since some computer generated 3D virtual environments are already configured and
engineered to take analog user input in the form of joystick-peripherals, a software
package that converts input from webcam imagery to virtual joystick movements might
open up various virtual environments for users of face-tracking technology, without
requiring changes or modification of the 3D environment software. A generic human
input device (HID) driver for operating systems would allow for this to happen.

There are also possibilities to modify virtual environments to allow for an easier
interaction with face-tracking users, for example by prediction of user intentions by
analyzing common environment interactions by users. This would make the environment
more accessible and easier to use.

References
1. Seeing Machines’ faceAPI, http://www.seeingmachines.com/product/faceapi/
2. Cube 2: Sauerbraten, open source game engine, http://www.sauerbraten.org/
3. Human Face Recognition and the Face Image Set's Topology. Bichsel, M. and Pentland, A.P. 1994.
4. Fast Stereo-Based Head Tracking for Interactive Environments. Morency, L.P., et al. 2002.
5. Nouse ‘Use Your Nose as a Mouse’ – a New Technology. Gorodnichy, D.O., Malik, S. and Roth, G. 2004.
6. Handbook of Face Recognition. Chellapa, R. and Zhou, S.K. 2005.
7. Real-Time 3D Head Motion Tracking Using a Camera. Huang, C.F., Yang, T.J. and Ouhyoung, M. 2006.
8. Computers Helping People with Special Needs. ICCHP Proceedings. 2004.