SixthSense

Technology
Prepared By:
3rd Year
Group 14
CSE(A) May 2017 CS 681
Rolls: 16, 33, 48, 131
Seminar
Prepared By:
Aritra Bhowmick 13000114016

Kaustav Roy Chowdhury 13000114033

Neelarghya Mandal 13000114048

Subhankar Mal 13000115131

3RD YEAR, 6TH SEMESTER

Dept. Of Comp. Sc. & Engineering,

Techno India, Saltlake

May, 2017

1
Acknowledgement
We are very thankful to be given this opportunity to present a seminar on
SixthSense Technology. We would firstly like to acknowledge the contribution and
guidance provided by our most dear and helpful faculties without whom this
seminar would not be possible.

We hereby, express my sincere gratitude to Prof. Poulami Dutta, HOD,

Department of Computer Science Engineering, for giving us an opportunity to
research and create a seminar on SixthSense Technology. We wish to thank our
faculties and especially our course coordinator Prof. Utpal Das for their guidance
and encouragement to proceed with this report. We wish to express a sense of
gratitude towards our respective families and friends for their encouragement and
support towards making this seminar a successful one.

2
Contents:
Sl No. Topic Page No.

1. Introduction 4

2. Brief History 5

3. Device Components 6

4. Working Principle 7

5. Applications 8

6. Advantages 9

7. Publications and Awards 10

8. Conclusion 11

9. References 12

3
Introduction:
'SixthSense' is a wearable gestural interface that augments the physical world
around us with digital information and lets us use natural hand gestures to interact
with that information.
We've evolved over millions of years to sense the world around us. When we
encounter something, someone or some place, we use our five natural senses to
perceive information about it; that information helps us make decisions and chose
the right actions to take. But arguably the most useful information that can help us
make the right decision is not naturally perceivable with our five senses, namely the
data, information and knowledge that mankind has accumulated about everything
and which is increasingly all available online. Although the miniaturization of
computing devices allows us to carry computers in our pockets, keeping us
continually connected to the digital world, there is no link between our digital
devices and our interactions with the physical world. Information is confined
traditionally on paper or digitally on a screen. SixthSense bridges this gap, bringing
intangible, digital information out into the tangible world, and allowing us to
interact with this information via natural hand gestures. ‘SixthSense’ frees
information from its confines by seamlessly integrating it with reality, and thus
making the entire world your computer.

4
Brief History:
SixthSense originated as a variety of wearable technologies including
headworn, neckworn, wristworn, etc. Including SixthSense technology (a camera
combined with a light source) developed in 1997 as a headworn device, and in 1998
as a neckworn object developed by Media Lab student Steve Mann. But the
SixthSense name for this work was not coined and published until 2001, when Mann
coined the term "SixthSense" to describe such devices.
In the 1990s and early 2000s, Mann used this project as a teaching example,
and taught several hundred students how to build the neckworn SixthSense system,
as part of the undergraduate teaching curriculum at University of Toronto. In the
1990s the early aremac did vector graphics rather than raster graphics, but a raster
graphics version based on a miniature wearable micromirror projector was
developed in 2001, which could project onto the wearer's hands, other objects, or the
floor or ground in front of the wearer, so that it could work with hand gestures or
foot gestures

Mann referred to this wearable computing technology as affording a

"Synthetic Synesthesia of the Sixth Sense", believing that wearable computing and
digital information could act in addition to the five traditional senses. Ten years
later, Pattie Maes, also with MIT Media Lab, used the term "Sixth Sense" in this same
context, in a TED talk.
Similarly, other inventors have used the term sixth-sense technology to
describe new capabilities that augment the traditional five human senses. For
example, in U.S. patent no. 9,374,397, timo platt et als, refer to their new
communications invention as creating a new social and personal sense, i.e., a
"metaphorical sixth sense", enabling users (while retaining their privacy and
anonymity) to sense and share the "stories" and other attributes and information of
those around them.

5
Device Components:
The components of the SixthSense Prototype include:

 Camera
 Projector
 Mirror
 Mobile Component
 Coloured Markers
The SixthSense prototype is comprised of a pocket projector, a mirror and a
camera. The hardware components are coupled in a pendant like mobile wearable
device. Both the projector and the camera are connected to the mobile computing
device in the user’s pocket. The projector projects visual information enabling
surfaces, walls and physical objects around us to be used as interfaces; while the
camera recognizes and tracks user's hand gestures and physical objects using
computer-vision based techniques. The software program processes the video
stream data captured by the camera and tracks the locations of the coloured markers
(visual tracking fiducials) at the tip of the user’s fingers using simple computer-
vision techniques. The movements and arrangements of these fiducials are
interpreted into gestures that act as interaction instructions for the projected
application interfaces. The maximum number of tracked fingers is only constrained
by the number of unique fiducials, thus SixthSense also supports multi-touch and
multi-user interaction.

6
Working Principle:
The working principle of the SixthSense device is fairly simple. The Camera
recognises the object or gesture in view and passes the information to a mobile
computing device. The device has no hardware restrictions; it can be a smart phone,
a tablet, iphone, laptop or even a smart watch; as long as the user can carry the
device. More over future research and development can lead to specific devices
which are made for the soul purpose of SixthSense. The device runs an open source
software that uses computer vision techniques to process the gesture or objects
streaming from the camera. The processer then transmits the output to a projector
which projects the output onto the desired surface via a mirror.

The prototype system runs on windows platform and majority of the code is
written in C++ and C#. The open source code is named WUW v0.1 beta.

WUW_v0.1 is the beta version of SixthSense code (WUW stands for Wear Ur World,
a former name of the project). This version runs on Windows':

 Microsoft DirectX
 Microsoft Visual Studio.
 Adobe Flash Player
 Microsoft Outlook

This technology is very much open to development, research, improvement and

production.

7
Applications:
The SixthSense prototype implements several applications that demonstrate
the usefulness, viability and flexibility of the system. The map application lets the
user navigate a map displayed on a nearby surface using hand gestures, similar to
gestures supported by Multi-Touch based systems, letting the user zoom in, zoom
out or pan using intuitive hand movements.

The drawing application lets the user draw on any surface by tracking the
fingertip movements of the user’s index finger. SixthSense also recognizes user’s
freehand gestures (postures). For example, the SixthSense system implements a
gestural camera that takes photos of the scene the user is looking at by detecting the
‘framing’ gesture.

The user can stop by any surface or wall and flick through the photos he/she
has taken. SixthSense also lets the user draw icons or symbols in the air using the
movement of the index finger and recognizes those symbols as interaction
instructions. For example, drawing a magnifying glass symbol takes the user to the
map application or drawing an ‘@’ symbol lets the user check his mail.

The SixthSense system also augments physical objects the user is interacting
with by projecting more information about these objects projected on them. For
example, a newspaper can show live video news or dynamic information can be
provided on a regular piece of paper. The gesture of drawing a circle on the user’s
wrist projects an analog watch.

8
Advantages:
Among the many advantages that the SisthSense Technology possesses
portability is one of the most fundamental. The whole concept of the SixthSense
revolves around portability and convenience to use. For example we don't walk into
a store and think of checking the QR code of an product by clicking a picture with
our smart phone and searching for a match, it's simply a lot of hustle. The SixthSense
can achieve this automatically.

The device is very cost effective. The current prototype system costs
approximate $350 to build but the instructions on how to make our own prototype
device can be found at http://www.pranavmistry.com/projects/sixthsense/diy/index.html
lowering the cost even more.

The software for the device (WUW v0.1 beta) is open source making it easy to
customize develop or improve with little effort that to without any copyright issues.

The ability to access data at real-time is another advantage which adds to the
convenience of use.

Since the coloured markers aren't restricted to a single user there can be a
single system installed for in a class room or even at different points on the globe
that allows multi-user interactions. Only restricted by the spectrum of colours that
the processor can recognise.

Finally the ability to connect the physical world of objects to the data world is
achieved in a seamless way making it the kind of technology that can take us many
decades into the future.

9
Publications and Awards:
The different publications regarding this Technology include:

 P. Mistry, P. Maes. SixthSense – A Wearable Gestural Interface. In the

Proceedings of SIGGRAPH Asia 2009, Sketch. Yokohama, Japan. 2009
 P. Mistry, P. Maes. SixthSense – A Wearable Gestural Interface. In the
Proceedings of SIGGRAPH Asia 2009, Emerging Technologies. Yokohama,
Japan. 2009
 P. Mistry. The thrilling potential of SixthSense technology. TEDIndia 2009.
Mysore, India 2009
 P. Mistry, P. Maes, L. Chang. WUW - Wear Ur World - A Wearable Gestural
Interface. In the CHI '09 extended abstracts on Human factors in computing
systems. Boston, USA. 2009
 P. Maes, P. Mistry. Unveiling the "Sixth Sense," game-changing wearable
tech. TED 2009. Long Beach, CA, USA 2009

The different awards regarding this Technology include:

 SixthSense wins 2009 INVENTION AWARD by Popular Science

 TR35 2009 Young Inventor Under 35 Award by Technology Review

10
Conclusion:
SixthSense recognizes the objects around us, displaying information
automatically and letting us to access it in any way we need. This leads to a seamless
experience and takes away the feeling of using a computer. The SixthSense
prototype implements several applications that demonstrate the usefulness, viability
and flexibility of the system without the manual use of the system itself. Thus
allowing us to interact with this information via natural hand gestures rather than
input devices.

The SixthSense Technology can potentially become the ultimate "transparent"

user interface for accessing information about everything around us at any time.

11
References:
 http://www.pranavmistry.com/projects/sixthsense/

 https://www.engineersgarage.com/articles/sixth-sense-technology

 https://en.wikipedia.org/wiki/SixthSense

 https://code.google.com/archive/p/sixthsense/

 http://wearcam.org/6ense.htm

12