Vous êtes sur la page 1sur 2

The $3 Recognizer: Simple 3D Gesture Recognition on

Mobile Devices
Sven Kratz Michael Rohs
Deutsche Telekom Laboratories, TU Berlin Deutsche Telekom Laboratories, TU Berlin
Ernst-Reuter-Platz 7 Ernst-Reuter-Platz 7
10587 Berlin, Germany 10587 Berlin, Germany
sven.kratz@telekom.de michael.rohs@telekom.de

ABSTRACT
We present the $3 Gesture Recognizer, a simple but
robust gesture recognition system for input devices fea-
turing 3D acceleration sensors. The algorithm is de-
signed to be implemented quickly in prototyping en-
vironments, is intended to be device-independent and
does not require any special toolkits or frameworks,
but relies solely on simple trigonometric and geomet-
ric calculations. Our method requires significantly less
training data than other gesture recognizers and is thus
Figure 1. The reference gesture vocabulary containing
suited to be deployed and to deliver results rapidly. the gesture classes used for the preliminary evaluation.
(b) describes a clockwise circular motion, (c) a wrist
Author Keywords rolling motion, (e) a gesture resembling a tennis serve
and (j) a rapid forward-backwards motion.
Gesture recognition, recognition rates, classifier, user
interfaces, rapid prototyping, 3D gestures
prototyping for 2D touch-screen-based gesture recogni-
ACM Classification Keywords tion and therefore focuses on ease of implementation
H5.2 [Information interfaces and presentation]: User in- on novel hardware platforms. Our contribution is in
terfaces – Input devices and strategies. I5.2. [Pattern extending and modifying Wobbrock et al.’s algorithm
Recognition]: Design methodology – Classifier design to work with 3D acceleration data. Instead of captur-
and evaluation. I5.5 [Pattern Recognition]: Implemen- ing exact pixel positions on a touch screen, acceleration
tation – Interactive Systems data is of much lower quality because it is prone to noise,
and additionally, drift error accumulates as the path of
a gesture entry is integrated. We extend Wobbrock’s
General Terms
original algorithm with a scoring heuristic to lower the
Algorithms, Performance rate of false positives. The major contribution of this
work is the creation a simple gesture recognizer that is
INTRODUCTION AND RELATED WORK designed to recognize “true” 3D Gestures, i.e. gestures
An increasing number of mobile devices are equipped which are not limited to shapes that can be drawn in a
with 3D accelerometers, which calls for suitable meth- 2D plane. The advantage of true 3D gesture recognition
ods for 3D gesture recognition on these platforms. Ges- is that more natural movements, such a tennis serve or
ture input for mobile devices can be a way to overcome boxing punches can be input by the user. Like the “$1
the limitations of miniature input facilities and small Recognizer,” our approach is quick and cheap to imple-
displays, since the range of movement is not restricted ment, does not require library support, needs only mini-
by the size of the device. Our work is based on previous mal parameter adjustment and minimal training, about
work by Wobbrock et al. [4], who developed a simple 5 samples per gesture provide good recognition results.
“$1 Recognizer” using basic geometry and trigonome- It is therefore very valuable for user interface proto-
try. The “$1 Recognizer” is targeted at user interface typing and rapid application development. It can also
easily be integrated into mobile interfaces that take ad-
vantage of other modalities, like speech, or touch-based
interaction with RFID/NFC.

THE $3 GESTURE RECOGNIZER


Extending Wobbrock’s [4] work, we present a gesture
Copyright is held by the author/owner(s).
recognizer that can recognize gestures from 3D accel-
IUI’10, February 7-10, 2010, Hong Kong, China. eration data as input. A detailed description of the
ACM 978-1-60558-515-4/10/02.

419
algorithm we developed can be found in [2]. Our demo sorted by matching scores with the corresponding ges-
is implemented in Objective-C as an iPhone 3GS appli- ture class ID.
cation, and in Java running on the Android platform,
but the method is by no means limited to these devices. Scoring Heuristic
Wobbrock’s original algorithm did not feature a heuris-
tic to reduce the occurance of false positives, which is
Gesture Trace and Gesture Class Library a common problem for simple gesture recognition algo-
In contrast to [3], modification or pre-processing of the rithms operating on large gesture vocabularies [4]. The
raw acceleration data in any way (filtering, smoothing, matches obtained from gestures entered as 3D accelera-
etc.) is not required by our method. To determine the tion data are not as precise as strokes entered on a touch
current change in acceleration, we subtract the current screen. To compensate for the weaker matches, we have
acceleration value reported by the device’s acceleration developed our own scoring heuristic, which processes
the score table described in the previous section. After
sensor from the previous one. We thus obtain an accel- sorting the score table by maximum score, our heuris-
eration delta. By summation of the acceleration deltas, tic determines the recognized gesture with the following
we obtain a gesture trace T (which can be plotted in a rules:
3D space (Figure 1 (e),(j)) or projected into a 2D plane
(gestures (a)-(d), (f)-(i)) to obtain a graphical repre- • ε is defined as the threshold score.
sentation of the gesture [1]. The gesture class library L • Iff the highest-scoring candidate in the score table has a score
contains a predefined number of training gesture traces > 1.1ε, return this candidate’s gesture ID.
for each gesture class G. • Iff, within the top three candidates in the score table, two can-
didates exist of the same gesture class and have a score > 0.95ε,
respectively, return the gesture ID of these two candidates.
Gesture Recognition Problem • Else, return “Gesture not recognized!”.
The basic task of our algorithm is to find the best
matching gesture class from the gesture class library. To Using this heuristic, we achieved a considerable reduc-
find a matching gesture class, we compare the gesture tion of false recognitions compared to Wobbrock’s orig-
trace entered by the user to all known gesture traces inal strategy of selecting the gesture candidate with the
stored the gesture class library and thus generate a score highest matching score to determine the recognized ges-
table that lists the comparison score of the entered ges- ture.
ture with the known gestures. A heuristic is then ap-
plied to the score table to determine if a gesture has SUMMARY
been correctly recognized. We present a simple, easy-to-implement gesture recog-
nizer for input devices equipped with 3D acceleration
Resampling sensors. The idea behind our gesture recognition algo-
For optimal classification, the gesture trace needs to be rithm is to provide a quick and cheap way to imple-
resampled to have a number of points equal to the tem- ment gesture recognition for true 3D gestures (such as
plate gestures. Furthermore, our resampling method the reference gesture (e)), for devices equipped with 3D
ensures that the points are re-distributed to be at equal acceleration sensors. Our method does not require any
distances from each other. advanced software frameworks or toolkits. An exam-
ple application area for our gesture recognizer is user
Rotation to “Indicative Angle” and Rescaling interface prototyping.
To correct for rotational errors during gesture entry, the
resampled gesture trace is rotated once along the ges- REFERENCES
ture’s indicative angle. Like Wobbrock, we define the 1. Sanna Kallio, Juha Kela, Jani Mäntyjärvi, and
indicative angle as the angle between the gesture’s first Johan Plomp. Visualization of hand gestures for
point p0 and its centroid c = (x̄, ȳ, z̄). After rotation, pervasive computing environments. In Proc. AVI
the gesture trace is scaled to fit into a normalize cube, ’06, pages 480–483. ACM, 2006.
to compensate for scaling differences between gestures. 2. Sven Kratz and Michael Rohs. A $3 gesture
After these pre-processing steps, we have converted the recognizer – simple gesture recognition for devices
original user input to a gesture that is ready for match- equipped with 3d acceleration sensors. In Proc.
ing with candidates from the gesture class library. IUI’10, 2010.
3. Thomas Schloemer, Benjamin Poppinga, Niels
Search for Minimum Distance at Best Angle
Henze, and Susanne Boll. Gesture recognition with
We use the average MSE (Mean Square Error) as dis-
a wii controller. In Proc. TEI ’08, pages 11–14.
tance measure between the gesture entered by the user
ACM, 2008.
and candidate gestures in the gesture class library. To
compensate for rotational differences, we adapted a 4. Jacob O. Wobbrock, Andrew D. Wilson, and Yang
Golden Section Search (GSS). This
√ type of search uses Li. Gestures without libraries, toolkits or training:
the Golden Ratio (ϕ = 0.5(−1 + 5)) to iteratively es- a $1 recognizer for user interface prototypes. In
timate the optimal rotation around three axis of the Proc. UIST ’07, pages 159–168, New York, NY,
coordinate system. The end result of GSS is a table USA, 2007. ACM.

420