Artif Life Robotics (2009) 14:397400
DOI 10.1007/s10015-009-0694-x
ORIGINAL ARTICLE
Djoko Purwanto Ronny Mardiyanto Kohei Arai
Electric wheelchair control with gaze direction and eye blinking
Received and accepted: May 18, 2009
Abstract We propose an electric wheelchair controlled by
gaze direction and eye blinking. A camera is set up in front
of a wheelchair user to capture image information. The
sequential captured image is interpreted to obtain the gaze
direction and eye blinking properties. The gaze direction is
expressed by the horizontal angle of the gaze, and this is
derived from the triangle formed by the centers of the eyes
and the nose. The gaze direction and eye blinking are used
to provide direction and timing commands, respectively.
The direction command relates to the direction of move-
ment of the electric wheelchair, and the timing command
relates to the time when the wheelchair should move. The
timing command with an eye blinking mechanism is designed
to generate ready, backward movement, and stop com-
mands for the electric wheelchair. Furthermore, to move at
a certain velocity, the electric wheelchair also receives a
velocity command as well as the direction and timing com-
mands. The disturbance observer-based control system is
used to control the direction and velocity. For safety pur-
poses, an emergency stop is generated when the electric
wheelchair user does not focus their gaze consistently in any
direction for a specified time. A number of simulations and
experiments were conducted with the electric wheelchair in
a laboratory environment.
Key words Electric wheelchair control Gaze estimation
Blinking measurement
D. Purwanto R. Mardiyanto
Department of Electrical Engineering, Institute of Technology
Sepuluh Nopember (ITS), Surabaya, Indonesia
K. Arai (
) firmed in terms of matching the accuracy of the electric
Department of Information Science, Saga University, 1 Honjo, Saga
840-8502, Japan
e-mail: arai@is.saga-u.ac.jp
This work was presented in part at the 14th International Symposium
on Artificial Life and Robotics, Oita, Japan, February 57, 2009
ISAROB 2009
1 Introduction
The ability to move freely is highly valued by all people.
However, it is sometimes difficult for a person with a physi-
cal disability. Nowadays, an electric wheelchair is commer-
cially available for disabled people. It generally requires
considerable skill to operate. Moreover, some disabled
people cannot drive an electric wheelchair manually, even
with a joystick, because they lack the physical ability to
control the movement. To enable a disabled person to drive
a wheelchair safely and easily so that they can enjoy a
higher quality of life, researchers have proposed several
electric wheelchair systems. The use of voice commands to
control an electric wheelchair is one research result. A small
number of command words and high-performance voice
recognition are employed in this system. An electric wheel-
chair control with electro-oculography (EOG) techniques
has also been proposed. In this case, the different com-
mands for the wheelchair are derived from the electro-ocu-
lography (EOG) potential signals of eye movements.
A system for electric wheelchair control using the eyes
was proposed in 2007. A commercially available web camera
on a head-mounted display (HMD) which the user wears is
used to capture moving pictures of the users face. A com-
puter mounted on the electric chair processes the captured
image data, detecting and tracking movements of the users
eyes, estimating the line-of-sight vector, and actuating the
electric wheelchair in the desired direction indicated by the
users eyes. One of the key essentials of the proposed system
is detecting and tracking the eye movements. This article
deals with the experimental results of the accuracy of the
estimation of the rotation angle of the eyeball, and with the
acquired image of the eyes and their surroundings. Through-
out the experiments, a success rate of almost 99% was con-
wheelchairs movement with the direction in which the user
intended to move.1
In this research, a method of electric wheelchair control
by gaze direction and eye blinking properties is proposed.
A camera is set up in front of the wheelchair user to capture
398
image information. The camera direction is focused onto
the face area of the wheelchair user. The camera is con-
nected to a computer with vision processing and electric
wheelchair motion control capabilities. With vision process-
ing, the sequential captured image is interpreted to obtain
the gaze direction and eye blinking properties. The gaze
direction is expressed as the horizontal angle of the users
gaze, and is derived from the triangle formed by the centers
of the eyes and the nose. The eye-blinking properties are
obtained from the timing of blinks. The gaze direction and
eye blinking are used to provide the direction and timing
commands, respectively, for the electric wheelchair. The
direction command relates to the direction of movement of
the electric wheelchair, and the timing command relates to
the time when the electric wheelchair should move. The
eye-blinking mechanism with a blink duration of at least
400 ms is designed to generate a timing command for when
to move. Furthermore, the electric wheelchair also receives
a velocity command to move at a certain velocity, as well
as the direction and timing commands. The disturbance
observer-based control system is used to control the direc-
tion and velocity. For safety purposes, an emergency stop
is generated when the electric wheelchair user does not
focus their gaze consistently in one direction for a specified
time.
2 Proposed system
2.1 System overview
The hardware of an electric wheelchair equipped with a
camera is shown in Fig. 1. The electronic system for vision
Fig. 1. Electric wheelchair hardware
processing and control purposes is placed in the bottom of
the wheelchair. The electric wheelchair uses a differential
steering mechanism. The wheelchair design considers the
following four important factors: safety, ease of operation,
low price, and convenience.
Four commands are use to control the electric wheel-
chair, i.e., the safety, timing, direction, and velocity com-
mands (Fig. 2). The safety command is for any situation in
which the wheelchair must stop immediately. The timing
command is obtained from the eye-blinking mechanism,
and is used to move or stop the chair. The direction
command is derived from the properties of the points of the
triangle made by both eyes and the nose of the user. The
velocity command is set independently and does not depend
on the vision system. Both the direction and velocity
commands refer to the wheelchairs motion in Cartesian
space.
2.2 Electric wheelchair model
Figure 3 illustrates the differential steering of the electric
wheelchair model. The desired wheelchair motion is deter-
mined by the command value of direction and linear
velocity v. The wheelchair motion depends on the electric
signal applied to the motor of each wheel. The angular
velocities l and r represent the direct input signals to
move the wheels of the electric wheelchair to the left or to
the right, respectively. The relation between direction and
velocity (, v) and the angular velocities of the wheels
Fig. 2. Commands for the electric wheelchair
Fig. 3. Electric wheelchair model
 399

Fig. 4. Electric wheelchair system

(l, r) is given by Eq. 1, where Rw is radius of a wheel,

is the gear ratio, and l is the distance between the wheels.
RW RW
2
v 2 l
 = R
R
(1)
W
l W r
l

2.3 Electric wheelchair system

The overall block diagram of the electric wheelchair system

is shown in Fig. 4. The system consists of a vision interpreta-
tion system and an electric wheelchair control system. All
the visual information processing, which includes gaze
direction estimation, eye blinking interpretation, and safety
detection, is carried out by a vision interpretation system.
The gaze direction estimation generates smooth-running
direction commands. The eye blinking interpretation and
safety detection systems generate command signals which Fig. 5. Relation between the triangular shape and the gaze
are sent directly to the controller. In the wheelchair con-
trol system, a disturbance observer-based control strategy
Fig. 6. Triangular shape from the
is applied to solve the two inputs/two outputs control
eyes and the nose
problem.

2.4 Vision interpretation system

2.4.1 Gaze direction estimation

The gaze direction is interpreted from the positions of both

eyes and the nose of the wheelchair user. As shown in Fig.
5, these three positions form a triangular shape that relates
to the direction of the gaze. With the parameters of the tri-
angle, as shown in Fig. 6, the estimation of the gaze direc- The direction command cmd is derived from the gaze
tion is formulated as Eq. 2. The function of f()is determined direction estimation with a smoothing function formula-
during the calibration procedure. tion to yield natural changes in commands and to suppress
a b measurement noise.
= f1
()
(2)
L cmd = f2 (3)
400

2.4.2 Eye-blinking interpretation Linear velocity response

0.4
An interpretation of eye blinking is useful to generate a
timing command which can be applied directly to the con-

(m/sec)
troller. Eye blinking with a duration of at least 400 ms is 0.2
used as the timing command. The algorithm of the timing command
command is as follows: actual
0
1. If <1 blink>then <disable controller> 0 5 10 15 20
2. If <2 blinks>then <enable controller> time (sec)
3. If <long blinking>then <set to move backward>
Rate of direction response
2.4.3 Safety detection 0.2
command

(rad/sec)
The safety detection control monitors the behavior of the actual
electric wheelchair user and the performance of the vision 0.1
system. Bad behavior is defined as when the user does
not focus during wheelchair control. Failure of the vision
system occurs when the system fails to detect the users face. 0
0 5 10 15 20
The safety command algorithm is defined as
time (sec)

If <bad user behavior>or <face detect failed>then <disable Actual response of wheelchair motors
controller> 30
(rad/sec)
2.5 Electric wheelchair control system 20

2.5.1 Electric wheelchair motors and controller 10 left

right
In both of the wheelchair motors, the disturbance observer 0
technique is applied. The disturbance observer is a tech- 0 5 10 15 20
nique to estimate the disturbance existing in a plant, and to time (sec)
make the motion controller be an acceleration controller
Fig. 7. Simulation results
also.3 With an acceleration controller, the wheelchair motors
can be modeled as an ideal integrator, and the block diagram
of the controller is realized by constant gain K.
v K 0 vcmd v
= (4)

0 K  cmd  4 Conclusion

2.5.2 Angular to linear velocity conversion and accelera- The feature points in the triangle made by the positions of
tion reference generator the eyes and the nose can be formulated to generate a direc-
tion command for the electric wheelchair. Combining the
The conversion from angular to linear velocity is formu- direction, velocity, timing, and safety commands yields an
lated in Eq. 1. Furthermore, the acceleration reference can effective method of controlling an electric wheelchair. With
be derived from Eq. 1 as follows: the disturbance observer technique applied as a control
1 strategy, the design criteria of the wheelchair control system
RW RW
l 2 2 
 can easily be satisfied.
 = R
RW
 (5)
r W
l
l References

3 Results 1. Arai K, Purwanto D (2007) Electric wheelchair control with the

To verify the effectiveness of the proposed method, several
simulations were carried out. Figure 7 shows that the actual
response follows the command in a specified time. Further-
more, the actual response of the motor shows that there
is a difference in the angular velocity of the left and right
motors when the command is change direction.
human eye. 7th International Conference on Optimization: Tech-
niques and Applications, December 2007, Kobe, Japan
2. Arai K, Uwataki H (2007) Computer input system based on viewing
vector estimation with iris center detection from the image of users
acquired with relatively cheap web camera allowing user move-
ments. J Inst Elec Eng Jpn C-127(7):11071114
3. Ohnishi K, Shibata M, Murakami T (1996) Motion control for
advance mechatronics. IEEE/ASME Trans Mechatronics 1(1):
5667