Design and Implementation of

Indonesian Sign Language to Speech Converter

Arry Akhmad Arman, Kudrat Soemintapoera, Evita T Sekar

Electrical Engineering Department,
Institut Teknologi Bandung
Email : aa@lss.ee.itb.ac.id

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 1

Outline

z Background
z Introduction
z System Description
z Testing
z Concluding Remark

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 2

1
Background

z Deaf people or mute people use sign

language to communicate each other
z Normal people who dont understand
sign language cannot receive
information from them
z There is a need to develop a system
that convert a sign language to a
speech.
z With this converter, normal people can
hear the message in the sign
language form.

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 3

Introduction

z Sign language is a language used to communicate

between deaf/mute people or deaf/mute people with
a normal people who understand that language.
z Sign language is a set of sign that formed by hand
and also several part of human face.
z Sign language is dedicated to a country.

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 4

2
Introduction : ANN

Learning Mode

Output sample
Input sample

Artificial
Neural
Network Production Mode

Input data
Artificial

Output
Neural
Network

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 5

Introduction : Text to Speech

Software
speech
synthesizer

text ucapan

z Text to Speech System (TTS) is system that convert

a text in specify language to a speech in the same
language.
z TTS is language dependent system.

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 6

3
Introduction : Text to Speech (2)

z TTS for certain language cannot be used for other

language. There are a lot of modification to be done
to use an existing TTS for other language.
z There are several type of TTS technologies.
z The most popular TTS technology is a diphone
concatination system. We have to build a diphone
database for specify language to use this type of
TTS. This type of TTS can generate the most natural
and intelligible voice.
z The first complete Indonesian Text to Speech System
have been developed in ITB using diphone
concatination technology.

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 7

Diphone Concatenation Principal

~k ko om mp pu ut te er r~

diphone Diphone database is

a complete set of diphones

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 8

4
Sign Language to Speech Converter

Hand Positions
(Indonesian Sign
Speech
Language)
(Indonesian Language)

Identification Text to
Sensor
and Mapping Speech
Module
Module Module

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 9

Sensor Module

z We use flexure sensors from Images Company that

integrated with a glove and other handmade holder.
z This sensor is variable resistance sensor. The value
of resistance will change according to the angle
position of the sensor.

Digital
Positions
Data

voltage Analog
Flexure
to Digital
Sensors
Converter

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 10

5
Sensor Unit

elbow

fingers wrist

Shoulder joint

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 11

Sensor Unit

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 12

6
Identification Modul
Positions Data
Text
(in digital format)

position
Artificial Neural code Code to Text
Network
Mapping
(Identification)

z In fact, the same input (i.e. finger) position for the sensor
cannot guarantee to desire the same exact value. User cannot
exactly form the same angle to represent every sign in sign
language. So, there is variated result from the sensors.
z To handle that condition, we choose Artificial Neural Network.

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 13

Text to Speech Module

z Text to Speech module implemented using existing
Indonesian TTS that already developed in previous
research activity in our laboratory.

Indonesian Phoneme to
Text to speech
text Speech
Phoneme
(MBROLA)

Indonesian Indonesian
Diphone Diphone
Text to Phoneme Database Database
Processed in Belgium
Published in Internet for non
commercial use
Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 14

7
Testing : Pre-Test

z Sensor system is one of the most important unit in this system.

The final result of the system is very determined by the sensor
accuracy.
z From the pre-test phase, we found that the result from sensor
module is not consistence.
z The inconsistency came from the sensor in the elbow and
shoulder joint position.
z The good result from a sensor at fingers and wrist position is
good.
z In fact, we use only one sensor for each position. Finger and
wrist movement or position need only one degree of freedom,
but the elbow and shoulder joint need than one more degree of
freedom.

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 15

Testing & Result : Final Test 1

z Testing procedure for finger and wrist position :

Choose a sample word that formed only by fingers and
wrist sensors (22 words)
Train ANN with sample words. Each word repeated 10 times.
Apply the network weights to the system.
Try every trained sign as an input to the system and let the
system identify the word. Repeat this test 10 time for each
sign.

z Result : 83.18% of trained signs (words) can be

identified by the system

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 16

8
Testing : Final Test 2

z Testing procedure for all sensor position :

Choose a sample word that formed by all sensors (72
words)
Train ANN with sample words. Each word repeated 10 times.
Apply the network weights to the system.
Try every trained sign as an input to the system and let the
system identify the word. Repeat this test 10 time for each
sign.

z Result : 49.58% of trained signs (words) can be

identified by the system

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 17

Concluding Remark

z For this application, the movement/position that only

consist of one degree of freedom, one sensor its
enough to identify the status.
z Sign language to speech converter with a sign
formed only by fingers and wrist is possible to
implement with good result.
z To detect a node status that consist of more that one
degree of freedom, one sensor it not possible to use.
So, to build a sign language to speech converter that
include the signs that formed by elbow and shoulder
joint, it is need to develop more sophisticated
sensory system.

Electrical Engineering Department of ITB, Indonesia Electrical Engineering Department of ITB, Indonesia Page 18