ST. FRANCIS INSTITUTE OF TECHNOLOGY (ENGINEERING COLLEGE) Mount Poinsur, S.V.

P Road, Borivali (W), Mumbai - 400103 2011-2012

Hand Gesture Recognition System for HCI

B. E. (Information Technology)
By

Bhavin Shah Tanya James Kaushik Vaghasiya Sinto Sunny

(54) (63) (70) (59)

Under the guidance of

Ms. Minal Lopes Lecturer

Department of Information Technology

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

Department of Information Technology Year 2010-2011

ST. FRANCIS INSTITUTE OF TECHNOLOGY (ENGINEERING COLLEGE)

CERTIFICATE OF APPROVAL For Project Synopsis

This to certify that, Bhavin Shah Tanya James Kaushik Vaghasiya Sinto Sunny (54) (63) (59) (70)

have satisfactorily carried out the Project work entitled Adaptive Learning System in partial fulfillment of Bachelor of Engineering in Information Technology as laid down by University of Mumbai during the academic year 2010-2011. Internal Guide: ____________________ (Ms. Minal Lopes)

Internal Examiner _________________________ ( )

External Examiner __________________________ ( )

Principal __________________________ (Prof. (Dr) A. K. Sen)

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

ACKNOWLEDGEMENT

It gives us immense pleasure to present the Synopsis of our B.E. Project Hand Gesture Recognition System for HCI. We wish to take this opportunity to thank few people, without whom our project would not have been possible. We would like to thank our respected Principal Dr. A. K. Sen for lending his support whenever we needed it. We are extremely grateful to our internal project guide Ms. Minal Lopes and for her valuable guidance, belief in our vision and unflagging support. We are also thankful to our Head of Department Mr. Pramod Shanbhag and our project coordinators Mrs. Vaishali Wadghare, Ms. Minal Lopes and Mrs. Prachi Raut for guiding us in the right direction. Lastly, we would like to thank our classmates and friends for their support and cooperation.

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

ABSTRACT

Even after more than two decades of input devices development, many people still find the interaction with computers an uncomfortable experience. Efforts should be made to adapt computers to our natural means of communication: speech and body language. Our aim is the proposal of a real time vision system for its application within visual interaction environments through hand gesture recognition, using general-purpose hardware and low cost sensors, like a simple personal computer and an USB web cam, so any user could make use of it in his office or home.

The basis of our approach is a fast segmentation process to obtain the moving hand from the whole image, which is able to deal with a large number of hand shapes against different backgrounds and lighting conditions, and a recognition process that identifies the hand posture from the temporal sequence of segmented hands.

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

TABLE OF CONTENTS Sr. No 1.0 Contents Introduction 1.1 Aims and Objectives 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 Proposed System Methodology Requirement gathering and Planning 9.1 Requirement elicitation 9.1.1 Use case diagram and description 9.2 Feasibility Study 9.2.1 Technical feasibility 9.2.2 Economic feasibility 9.3 Requirement analysis 9.3.1 Flow Chart 9.4 Timeline Charts 9.5 WBS Chart Analysis 10.1 ER Diagram Data Dictionary 10.2 Activity Diagram 10.3 Data Flow Diagram 11.0 12.0 13.0 UI Design(Snapshots) Appendix Bibliography And References Literature Surveyed Existing System Problem Statement Scope of the Project Page No 1 2 3 5 6 7 8 9 10

11

14

15 16 17

10.0

18 19 21

24 27 30 31

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

LIST OF FIGURES Fig. No 9.1(a) 9.1(b) 9.1(c) 9.3.1 9.4 9.5 10.1 10.2(a) 10.2(b) 10.2(c) 10.3(a) 10.3(b) 10.3(c) 10.3(d) 10.3(e) 10.3(f) Name of Figure Use Case of Admin Use Case of System Use Case of Student Flow Chart Time Line Chart WBS Chart ER Diagram Activity Diagram Administrator Activity Diagram Non-Registered User Activity Diagram Registered User DFD Level 0 DFD Level 1 DFD Level 2 DFD Level 2 DFD Level 3 DFD Level 3 Page No 11 12 13 15 16 17 18 21 22 23 24 24 25 25 26 26

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

1.0

INTRODUCTION

An important way of communicating among humans is through body language, thus an automatic posture recognition system can be used to improve human computer interaction (HCI). This kind of HCI would allow a human user to control remotely through hand postures a wide variety of devices.

The motivation of a Hand Gesture Recognition System is to represent

serious alternative to conventional input devices like keyboards, mice and applications based on computer vision .we are interested in controlling various computer applications by hand pose signs given by a human. Due to real-time operational requirements, we are interested in a computationally efficient algorithm.

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

1.1. AIM AND OBJECTIVES

Aim:
The Hand Gesture Recognition System aims at making use of a real time gesture recognition application for HCI.

Objectives:
To replace the traditional gesture recognition methods. To eliminate the need of markers, sensors, hand gloves etc. To allow the user to work in various light conditions and backgrounds. To work with applications in real time using a simple personal computer and a USB web-camera.

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

2.0 LITERATURE SURVEY

1) E Sanchez-Nielsen, La Laguna, M Hernandez-Tejera, Hand Gesture Recognition for Human-Machine Interaction, Journal of WSCG,Vol. 12, No.1-3, Date. February 2-6, 2003.

ABSTRACT Even after more than two decades of input devices development, many people still find the interaction with computers an uncomfortable experience. Efforts should be made to adapt computers to our natural means of communication: speech and body language. The PUI paradigm has emerged as a post-WIMP interface paradigm in order to cover these preferences. The aim of this paper is the proposal of a real time vision system for its application within visual interaction environments through hand gesture recognition, using general-purpose hardware and low cost sensors, like a simple personal computer and an USB web cam, so any user could make use of it in his office or home. The basis of our approach is a fast segmentation process to obtain the moving hand from the whole image, which is able to deal with a large number of hand shapes against different backgrounds and lighting conditions, and a recognition process that identifies the hand posture from the temporal sequence of segmented hands. The most important part of the recognition process is a robust shape comparison carried out through a Hausdorff distance approach, which operates on edge maps. The use of a visual memory allows the system to handle variations within a gesture and speed up the recognition process through the storage of different variables related to each gesture. This paper includes experimental evaluations of the recognition process of 26 hand postures and it discusses the results. Experiments show that the system can achieve a 90% recognition average rate and is suitable for real-time applications.

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

2) A Malima, E Ozgur, M Cetin, A Fast Algorithm for Vision-Based Hand Gesture Recognition for Robot Control, Signal Processing and Communications Applications,2006 IEEE 14th,pp. 1-4, Date. 17-19 April 2006.

ABSTRACT

We propose a fast algorithm for automatically recognizing a limited set of gestures from hand images for a robot control application. Hand gesture recognition is a challenging problem in its general form. We consider a fixed set of manual commands and a reasonably structured environment, and develop a simple, yet effective, procedure for gesture recognition. Our approach contains steps for segmenting the hand region, locating the fingers, and finally classifying the gesture. The algorithm is invariant to translation, rotation, and scale of the hand. We demonstrate the effectiveness of the technique on real imagery.

10

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

3.0

PROBLEM STATEMENT

The Hand Gesture Recognition System is used to create a user centric system. The user can interact with computers by just using various hand poses. These hand poses are captured using a web camera and recognized using an efficient pattern matching algorithm. These gestures can be used to control various computer applications to improve Human Computer Interaction.

11

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

4.0

SCOPE OF THE PROJECT

The proposed Hand Gesture Recognition System aims at making use of a real time gesture recognition application for HCI. This project tries to elude the problem of using markers, using marked gloves or requiring a simple background for gesture recognition. The basis of our approach is to develop an application which is able to deal with a large number of hand shapes against different backgrounds and lighting conditions.

Business Goals:

To improve Human Computer Interaction. To make high-quality marketable software.

Technical Goals:

To capture images and process them in real time. To build a user friendly application that is easy to understand and operate so that a person without much technical knowledge can operate on it.

To improve the precision and quality of service in gesture recognition technology.

12

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

5.0

CONSTRAINTS OF THE PROJECT

Business Constraints:

Project lacks self-learning capability.

Technical Constraints:

The web camera should be of minimum 4MP.

13

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

6.0

PROPOSED SYSTEM

The proposed Hand Gesture Recognition System aims at making use of a real time gesture recognition application for HCI. This project tries to elude the problem of using markers, using marked gloves or requiring a simple background for gesture recognition. The basis of our approach is to develop an application which is able to deal with a large number of hand shapes against different backgrounds and lighting conditions. The aim of this project is to develop a real time vision system for its application within visual interaction environments through hand gesture recognition, using general-purpose hardware and low cost sensors, like a simple personal computer and an USB web cam, so any user could make use of it in his office or home

14

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

7.0

METHODOLOGY

The Hand Gesture Recognition System for HCI uses an Agile software development methodology. Agile software development is a group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams. It promotes adaptive planning, evolutionary development and delivery, a time-boxed iterative approach, and encourages rapid and flexible response to change. It is a conceptual framework that promotes foreseen interactions throughout the development cycle. .

15

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

8.0
8.1

REQUIREMENTS GATHERING AND PLANNING

REQUIREMENTS ELICITATION 8.1.1 USE CASE DIAGRAM AND DESCRIPTION

User:

Start Application

Switch on Webcam User

Show Hand Gesture

Check if Action is Performed

Fig:9.1(a) Use Case of User

ACTOR

USECASE

DESCRIPTION Starts the software application. Switches on the web cam so images can be captured in runtime.

1. USER

Start Application Switch on Web cam

Show Hand Gesture

Shows hand gestures in front of the web cam

16

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

Check if action is performed

Checks if the required action is performed

System:

<<extend>> is Webcam off <<extend>> Check Status of Webcam Show Error Message is Webcam on <<include>>

System.

Capture Image <<include>> <<include>> Process Image Segment Image

Perform Pattern Matching Perform Required Action

ig:9.1(b) Use Case of System

ACTOR

USECASE

DESCRIPTION Checks if web cam is ON. Capture images on web cam in runtime.

1. SYSTEM

Check web cam status Capture images

Process images

Process images and segments the hand gesture from the image

Perform pattern matching

Matches the captured hand Gesture with stored patterns.

17

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

8.2 FEASIBILITY STUDY 8.2.1 TECHNICAL FEASIBILITY

HARDWARE REQUIREMENTS:
A personal computer with minimal requirements as

WEB CAMERA (4 MEGA PIXEL) ROCESSOR: PENTIUM 4 RAM: 1GB SOFTWARE REQUIREMENTS:
MATLAB JAVA Operating System: Windows 7/XP

18

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

8.3

REQUIREMENTS ANALYSIS 8.3.1 FLOW CHART

Flow Chart

19

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

8.4

TIMELINE CHARTS

Fig:9.4 Time Line Chart

20

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

8.5

W.B.S. CHART

Fig:9.5 WBS Chart

21

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

9.0
9.1

ANALYSIS

ACTIVITY DIAGRAM

User

System

Start Application

Check if Webcam is on

Show Hand Gesture

[if On]

[else] Switch Webcam On

Capture Image

Store Image

Process Image

Perform Pattern Matching

[else]

[if Pattern Recognise] Perform Action

Fig: 10.1 Activity Diagram

22

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

9.2 DATA FLOW DIAGRAM

DFD Level 0

23

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

DFD Level 1

24

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

DFD Level 2

25

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

10.0 DESIGN
10.1 UI DESIGN(LAYOUT)/SNAPSHOTS

26

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

27

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

28

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

11.0 APPENDIX

HCI PUI WIMP Paradigm

Human Computer Interaction Perceptual User Interface Windows, Icons, Menus and Pointing device An example or pattern, distinct concept

29

St. Francis Institute of Technology, Mumbai-400 103.

2011-2012

12.0 BIBLIOGRAPHY AND REFERENCES

IEEE References
1) E Sanchez-Nielsen, La Laguna, M Hernandez-Tejera, Hand Gesture Recognition for Human-Machine Interaction, Journal of WSCG,Vol. 12, No.1-3, Date. February 2-6, 2003.

2) A Malima, E Ozgur, M Cetin, A Fast Algorithm for Vision-Based Hand Gesture Recognition for Robot Control, Signal Processing and Communications Applications,2006 IEEE 14th,pp. 1-4, Date. 17-19 April 2006.

Websites
[1] www.google.com [2] www.wikipedia.com [3] www.mathworks.com

30