Académique Documents
Professionnel Documents
Culture Documents
K.W.S.D. Kaluarachchi
October 2013
H. H. Rajamanthrie IT 10 0296 64
K.W.S.D. Kaluarachchi
Dissertation submitted in partial fulfillment of the requirements for the degree of Science
October 2013
H. H. Rajamanthrie IT 10 0296 64
Declaration
I declare that this is my own work and this dissertation does not incorporate without acknowledgment any material previously submitted for a Degree or Diploma in any Other University or Institute of higher learning and to the best of my knowledge and belief it does not contain any material previously published or written by another person except where the acknowledgment is made in the text.
Also, I hereby grant to Sri Lanka Institute of Information Technology the nonexclusive right to reproduce and distribute my dissertation, in whole or in part in print, electronic or other medium. I retain the right to use this content in whole or part in future works (such as articles or books)
Signature:
Date: 2013/10/23
The above candidate has carried out research for the B.Sc Dissertation under my supervision.
Date: 2013/10/23
Acknowledgement
We take this opportunity to express our deep sense of gratitude to those who contributed to either our collective or individual efforts. Our heartfelt thanks go out to the supervisor of our project, Dr. Rohana Priyantha Thilakumara and the co-supervisor, Mr. Darshika Koggalahewa for the kind patience, guidance and constant support rendered to us at all the stages of the project, right from the very be- ginning of the project. We would also like to thank the lecturer in-charge, Mr. Jayantha Amararachchi who provided the required lecture materials and the necessary guidance to do the project. The team extends sincere gratitude to all colleagues, who forwarded their enthusiastic ideas during the requirements gathering phase of the project. The teammates would also like to thank all the staff members of the SLIIT Malabe campus for their valuable suggestions and opinions that helped us to do the project. Finally, we thank all who lend their kind support; friends and families who continued to give their insights, patience, support and co-operation which motivated us in reaching greater heights.
ii
Abstract
Brain-Computer Interface (BCI) technology is a potentially powerful communication and control option in the interaction between Human and Computer systems. A Brain Computer Interface (BCI) is a direct communication pathway between the brain and an external device. This technology enables signals that are generated in the brain to control an external activity such as control of a keyboard of a computer or control of a wheelchair. Over the past few years, new games have been developed that are exclusively for use with an EEG headset by companies like Neurosky and Emotiv. In this project we have developed a set of programs that can be controlled by brain signals. In this report I explain about the Ludo Game which is controlled by Signals generated inside the Brain. Our research part is to find out about an effective and an efficient way to use to brain signals for this application. This Ludo Game is going to be a new experience and will define a new dimension in the Game industry. In another way, this can be an exercise for the Brain. ilitated in game developing. By proposing this frame work we try break down barrier between brain computer interface researchers and game development community. Our proposed framework will facilitate to professional game designers as well as independent gaming enthusiast to dive in to the depth of Brain Computer Interfacing gaming world without bothering about brain wave analyzing, feature extraction and classification. Furthermore they wouldn't want to brother about devices without considering devices they would be able to develop their creative gaming ideas. The objective of the proposed project is to bring expert knowledge of Neurophysicists toward the game developers and motivate them to develop more exciting and challenging games and expand the gaming industry.
iii
Table of Contents
DECLARATION ACKNOWLEDGEMENT ABSTRACT LIST OF FIGURES LIST OF ABBREVIATIONS 1 INTRODUCTION 1.1 1.2 1.3 2 BACKGROUND CONTEXT RESEARCH PROBLEM TO BE ADDRESSED RESEARCH QUESTIONS 1 II III V VI 1 1 2 3 4 4 5 5 7 7 9 10 11 11 12 18 24 25
CONTENT 2.1 ADDRESSING THE LITERATURE 2.2 METHODOLOGY 2.2.1 Overview 2.2.2 Overview of the System Design 2.2.3 User Characteristics 2.2.4 Product Functions 2.2.5 Tools and Technologies 2.2.6 Product Constraints 2.2.7 Assumptions and Dependencies 2.3 RESEARCH FINDINGS
3 4 5
iv
LIST OF FIGURES
Figure 1: System Diagram ........................................................................................................ 7 Figure 2: Student A Attention Level Values ........................................................................ 14 Figure 3: Student A Variance Between Attention Level Values .......................................... 14 Figure 4: Student B Attention Level Values ........................................................................ 15 Figure 5: Student B Variance Between Attention Level Values .......................................... 15 Figure 6: Student C - Attention Level Values while listening to a Song ................................. 17 Figure 7: Student D - Attention Level Values while listening to a Song ................................. 17 Figure 8: Attention Level Values ............................................................................................ 18 Figure 9: Running Average of Attention Level Values ........................................................... 19 Figure 10 - The simple keyboard ............................................................................................ 20 Figure 11 - Using the Simple Keyboard. ................................................................................. 21 Figure 12 - The Advance Keyboard. ....................................................................................... 22 Figure 13: Ludo Game User Interface .................................................................................... 23
List of Abbreviations
EEG BCI fMRI EEC SSVEP ALS API Electro - Electroencephalogram Brain Computer Interface functional Magnetic Resonance Imaging Encephalogram Steady State Visual Evoked Potential Amyotrophic Lateral Sclerosis Application Programming Interface
vi
1 INTRODUCTION
1.1 Background Context
The advent of computers was a significant breakthrough in this area of work, as it lead the way towards what we today know as Brain Computer Interface (BCI). The computational processing aspect of computers now opened up doors to the possibility of being able to analyze and distinguish the patterns of the electrical signals being produced by the human brain, and thus allowing to trigger a desired outcome for a given identified brain signal pattern. For example, when a brain signal pattern is detected by the computer then it could be associated to trigger a tangible action like activating a switch to turn on/off the lights. In simple terms this kind of a setup that detects electrical signals from the brain, and computes these signal patterns to initiate a substantial tangible action is known as a BCI.
BCI has always been a subject matter of interest in the fields of Rehabilitation and Assistive Technology. The potential of BCI is unbounded with regards to improving the lives people with disabilities. For example, BCI based systems could allow an individual with severely restricted or no range of movements (e.g. spinal cord injuries) to drive a power wheelchair, operate household appliances, etc. Or even help someone in a vegetative state to communicate by speaking out the words that the individual would like to say.
Primarily there are two categories of BCI systems invasive and noninvasive. Invasive systems interact with the brain directly via electrodes / sensors that are implanted into the brain or its surface. While noninvasive systems interact with the brain indirectly via electrodes / sensors placed on the surface of the head that detect brain signal emissions (e.g. Electro-Encephalography (EEG), functional Magnetic Resonance Imaging (fMRI), and Magnetic Sensor Systems).
Noninvasive BCI systems usually involve wearing a head cap (aka EEG cap) with multiple holes / slots to put on the electrodes at the relevant areas of the surface of the head to detect and record the electrical signals emitted by the brain. Electro gel is used on the electrodes to improve contact between the scalp and the electrode. Due to the requirement of a gel such electrodes are also known as wet-electrodes. Existing systems can have anywhere between a few to more than a 100 electrodes. The
practicalities of using wet electrodes, as an example drying up of gel, repeated cleaning of electrodes and head skin to setup EEG Cap, irritation of sensitive skin due to application of gel, etc. does not make them convenient for quick setup and daily use. Over the recent few years this has prompted the development of dry electrodes. Unlike wet electrodes, dry electrodes do not require the use of gel and can be setup direct into the EEG Cap. Although still in the infancy of its developmental cycle, dry electrodes are quickly catching up in terms of detecting high quality brain signals as their wet counterpart. Regular comparison studies are being carried out to evaluate the
performance of Wet vs. Dry Electrodes within the context of EEG based noninvasive BCI.
It is very difficult to use brainwaves to control BCI games, because each user will have a different distribution of brainwaves. For example, some users brainwaves will have a bigger amplitude than others. Therefore, we require to address this and find a suitable
method that transfers the brainwaves to the signals that are good enough to be used in a BCI game.
We checked and tested few methods to choose a best way to use the signals. We collected many signal samples using few SLIIT students. Then calculated them using our methods and chose the best method. Fourth, a systematic BCI serious game development process is needed for experts and game developers dealing with brain functions and brainwaves. This will allow diverse developers to quickly and easily produce games together.
2 CONTENT
2.1 Addressing the Literature
In what research has been done so far, the field of computer and information technology has developed a wealth of devices that capture brain signals. And there are applications that use them. These applications do not have a proper way to use the brain signals that gets captured by the device. In our research we test and suggest a proper method to use the captured signals. The current applications have faults in signal handling. The brain signals gets vary from person to person. The strengths of signals get dropped during occasions. In the applications we create, we use Running Average. The values that are given by the device are sent through a program function which calculates the Running Average of those values. And then Running Average values are sent to the game. Because of this we can avoid the big variances in the signals.
What is Running Average? In statistics, a moving average, also called rolling average, moving mean, rolling mean, sliding temporal average, or running average, is used to analyze a set of data points by creating a series of averages of different subsets of the full data set.
2.2 Methodology
2.2.1 Overview
Not only as an assistive technology, Brain Computer Interface technology can also be used to give entertainment in a new way taking the entertainment into a new level. Imagine a person playing a game using only his/hers thinking power.
In the past 10 years, the gaming industry has been a growing multi-billion-dollar business, this shows that the demand of videos games has been growing, and this rocketing demand also attracts a vast investment on new gaming interfaces, such as Dance Pad in PlayStation and Wii controllers in Wii, which furthered feedback to the snowball (i.e. the demand) positively. However, while new interfaces for console games (e.g. touch screens for NDS and remote motion sensors for Wii) has been developed, emergence of new gaming interfaces for PC games seem to slow down after the introduction of game pads, and we think a new gaming interface could perhaps give birth to a new genre of games in the big PC games market, where almost been captured by interfaces like Microsoft Kinect. Hence one may wonder if anything new would appear in the future gaming world, and what that would be recently, brain computer interfaces for consumer level have been released to the market, making BCI entertainment possible.
There are communities who are involved in developing Brain Computer Interface games. However, we see a discrepancy between the BCI games developed by the communities. Many of the BCI games developed by the BCI community aim at testing some psychological hypotheses or evaluating the performance of signal analysis and classification techniques. Thus, less attention is paid to game characteristics than to technical aspects. These games do not usually have any narrative or rich feedback or visuals. User (i.e. player) experience evaluations are almost never carried out. This leads to BCI games that are reliable but often not enjoyable. On the contrary, BCI
games from the games community are developed with respect to game design principles. However, the neurophysiology and signal analysis techniques they rely on are largely unknown. Because, these games mostly make use of the commercial BCI headsets which have their private technical details. This leads to BCI games that are potentially entertaining but unsatisfactory in terms of feeling of control.
In proposed method, we will try to transfer some knowledge from the games and the BCI communities into a shared preliminary framework to make them aware of each others research. From the games community, we will show some game playing motivations which can be satisfied by the features of BCI. From the BCI community, we will take the current interaction paradigms used in general and show the ways they can be used in games. This way, we hope to contribute to bridging the gap between the two communities and promoting the development of entertaining and reliable BCI games.
2.2.2
2.2.3
User Characteristics
The users of the system can be categorized under user groups as follows; 1. 2. 3. 4. BCI device makers. BCI game developers. BCI game players. Disabled persons.
The characteristics of those identified user groups of the Interactive Programming Assistance Tool are illustrated below.
BCI device makers. They are the people who develops and manufactures BCI devices. There are various brands in World Market nowadays. And from those, Neurosky and Emotiv owns a major market share.
BCI game developers. They develop the games which can be played using BCI devices. The most part of these of games get controlled directly using brain signals.
BCI game players. BCI game players get an experience which is much different from the experience that normal game players receive. They normally use their thinking power, concentration power to control games rather than hand. Disabled persons. Because this game is controlled by the signals that are generated inside the Brain, even a person who does not have any hands can play this game.
2.2.4
Product Functions
BCI enhanced Ludo Game This Ludo game is developed to be controlled by the brain waves. By wearing this device on your head and using your attention level, you can generate the value of the Die. There are two main functions in this Ludo Game. Calculating Running Average.
We tested few methods and finally chose to use running average to get a better signal value from the device. This function does the part of calculating running average of the signals values that gets captured by the device. Get a value to map with the game in order to control it.
We have developed a Ludo game to as a sample game. This function will map the signal values that get calculated by the Calculating Running Average function.
BCI enhanced Keyboard This application is a virtual keyboard which can be controlled by the eye blink instead of the hands. There is some functionality which provides users to press the appropriate keys using just the eye blink than using the hand.
BCI enhanced Calculator This application also is same as the above mentioned keyboard. It also provides the same functionality to the users as it allows users to control the options of the calculator using the eye blink. For calculating purposes user have to train his/her way of blinking in order to get the correct options.
BCI enhanced Snake and Ladders Game This gaming application is for entertaining the users providing opportunity to play the game using their eye blink than using the attention and the focus level. Depending on the eye blink user has to toss the dice and win the game. There fore it provides users a great opportunity to play the game in a new way apart from the traditional way.
2.2.5
We have used latest tools and technologies to do our research and to develop the Ludo game. The tools and technologies include the following: Microsoft Visual Studio 2012 .NET Framework 4.5 Neurosky MindWave Device
Microsoft Visual Studio 2012 is the IDE that we used to develop the Software components of our product. These Software components mainly include the Ludo game, the connection classes which connect the Neurosky device with the software components, the running average calculation classes.
Microsoft Excel was used to store signal values we have collected from some SLIIT students. Excel was also used to analyze the stored signal values by drawing charts and by calculating using few formulas.
10
2.2.6
Product Constraints
Our applications are specifically developed for Windows platform and tightly coupled with windows based computer architecture. Developers do not have sufficient hard-ware resources to build this application on different system architectures such as Linux platform.
When getting Brain Signal values using the device, the device takes like a second to calibrate. So , when you start collecting the signal values, the values come one second after you started it.
The BCI devices sensor which touches the forehead should have a good contact with the forehead. Otherwise it wont give good readings and will say the signal quality is poor.
In order for the device to work properly, the batteries that are inserted to the device should be good quality batteries with a good charge.
The driver for the BCI device should be installed in the Computer.
2.2.7
It is assumed that the computer which runs our applications is running on the Windows operating system and the Visual Studio framework is installed in the respective machine. It is essential that the Visual Studio is up and running by means of the accurate .NET framework.
The source codes that are to be input to the system should be developed in C# language.
11
Microsoft Windows based supported architecture should be available and implemented x86 x64
Microsoft Windows based operating system (Windows XP service pack 3 or higher version) should be available and functioning successfully.
All the machines should support the .NET framework 3.5 or higher.
While we are doing the research, through the tests we did, we found out that The device we are using have many limitations. The signals can get weakened at times. The variations of signal values are high. The way persons react to a one particular activity is different, because of that the signals their brains emit are different to each other.
The device we are using have many limitations. Our neurosky device comes with one sensor. And it can detect only meditation, attention, eye blink activities of the user. Because of that, the applications for this device can be only controlled by meditating, keeping the attention and by blinking eyes. Other movements of a user, like moving eyes, moving the head, moving a hand, can not be recognised by this device.
12
The signals can get weakened at times. The signal strength depends on few factors Whether the sensor is touching the forehead. Whether the battery of the neurosky device has a good charge level. The distance between the user who is wearing the device and the Computer which is running the application that is controlled by brain signals of the user.
The variations of signal values are high. The values of signals we gained by the tests were added to excel sheets. Then we generated charts using those values. When considering those charts, we noticed a big variation between the signal values. We calculated the variations of those signals. Those calculations also gave higher values. The following charts provide some examples for these higher variations. The data were collected from a SLIIT Student. First we asked him to play a Car Race Game, then we asked him to listen to a song. And while he was playing and listening to the song, we collected the data. And using those data we calculated Running Average of those values & variations of them.
13
14
15
The way persons react to a one particular activity is different, because of that the signals their brains emit are different to each other. The way different persons react to a particular activity is different from each other. Because of this the Brain Signals generated by different persons for a particular activity is different from each other. As an example some persons might like classic songs, while some dont. While some may have mixed emotions towards classic songs. The following two charts were created using the data collected which were collected using two SLIIT Students. The data were collected when they were listening to a song. Both of them listened to the same song. Though they listened to the same song, the way their brain reacted to that song was different. The two charts provide evidence for that.
16
17
18
We can notice the difference between the two charts. The first chart has a high variance. The second chart, which is drawn using the Running Average values, has a low variance. Because of that fact, using the Running Average is better. There can be occasions when there are sudden drops of signal strength. You can notice it when the signal value suddenly goes for 0 and then values start to come back again. During that occasions the Running Average can be a solution. Because of the running average, you can avoid sudden drops in signal values.
Why this device is not capable of controlling big games. You may have imagined of big games that get controlled totally by brain signals. Or you may have imagined when you think a letter, that letter gets automatically printed on the Computer. But since this device is giving out only the Attention level values, Meditation level values and Eyeblink values, it can not be done.
19
The Mind-Controlled Keyboards. We have developed two keyboards for disabled persons. These will be best suited for Mute Motor-disabled persons. A big challenge these type of disabled persons have is that they find it difficult to communicate with others. And this system we have developed will be helpful for them to communicate with others. They keyboard can be operated by using Eye-Blinks, which is convenient for the users.
20
The previous image shows our simple keyboard. It can be controlled by using EyeBlinks. In this keyboard, first row by row gets highlighted. When the user blinks, it goes one by one in the row that was highlighted when the user blinked. As an example : If the user want to tell someone to Switch on a Light, initially the user has to blink when the particular row gets highlighted. Then blink again when the Key which shows an emitted bulb gets highlighted.
21
Next we have developed a keyboard for the purposes of typing messages. This also can be operated just like the previous keyboard. Once the message is typed the user can make the Computer pronounce the message by clicking on SAY IT.
22
Ludo Game we have developed. We have developed a Ludo game also. The Ludo game is developed to provide entertainment for the user. The value of the Die of this game is generated according to the Attention level of the user. If the user is having a higher attention level, the Die will get higher values. If the attention level of the user is low, the Die will get lower values. The Attention level values of the user is given to the application by the BCI device that is worn by the user. Then Running Average of those values are automatically generated and those Running Average values decide the value that the Die should get.
23
4 CONCLUSION
For a long time, researchers have been working on a marriage of human and machine that sounds like something out of science fiction: a brain computer interface. The technology holds great promise for people who cant use their arms or hands normally because they have had spinal cord injuries or suffer from conditions such as amyotrophic lateral sclerosis (ALS) or cerebral palsy. BCI could help them control computers, wheelchairs, televisions, or other devices with brain activity.
To success with the projects like above mentioned, the capabilities of the used BCI devices should be in a high standard level with the fully functional options. Also there should be many sensors which are capable of capturing brain signal within vast areas of the brain.
Since we are using the NeuroSky mindwave head set which is contain only one sensor is not much capable of developing advance systems and applications. And the generated raw data also not much accurate and reliable as it can be used for a considerable application since it does not gives a stable value and can not find a certain pattern in order to use as an average value for controlling an application
Therefore the team decided to implement applications depending on the capabilities of the using BCI mindwave headset. So the implemented applications are Mind controlling Calculator and a Keyboard which is working based on the users eye blink. Also a Ludo Game controlling based on the users attention level and the focus level as well as a Snakes and Ladders game based on the eye blink. The sound generator is an application working based on the alpha, beta and gamma brain signals.
24
5 REFERENCES
1. "Brain-Computer Interfaces: Beyond Medical Applications," 2013. [Online]. Available: http://lifesciences.ieee.org/articles/114-brain-computer-interfacesbeyond-medical-applications. 2. E. Kader, B. P. B. P. Emilie Belley, J.-A. Filion, A. Nutter, M. Parent-Vachon, M. Saulnier, B. P. Stephanie Shedleur, B. P. Tsz Ting Wan, B. B. Elissa Sitcoff and P. O. Nicol Korner-Bitensky, "MOTOR IMAGERY - Information for Patients and Families," 2010. [Online]. Available: http://strokengine.ca/intervention/admin/patient/Motor%20ImageryFamily%20InformationDec2010.pdf. 3. "P300 (neuroscience)," 2012. [Online]. Available: http://en.wikipedia.org/wiki/P300_(neuroscience). [Accessed 2013]. 4. K. Nakayama and M. Mackeben, "Steady State Visual Evoked Potentials In the Alert Primate," 1982. [Online]. Available: http://visionlab.harvard.edu/members/ken/Ken%20papers%20for%20web%20pag e/027VisionRes82.pdf. 5. "Event-related Potential: An overview," 2011. [Online]. Available: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3016705/. [Accessed 2013]. 6. "Electroencephalogram (EEG)," WebMD, LLC., 2010. [Online]. Available: http://www.webmd.com/epilepsy/electroencephalogram-eeg-21508. 7. "What Is A Framework?," 2003. [Online]. Available: http://www.codeproject.com/Articles/5381/What-Is-A-Framework. 8. "Neurosky MindWave Mobile," Neurosky, 2012. [Online]. Available: http://neurosky.com/Products/MindWaveMobile.aspx. 9. Microsoft, "Microsoft Visual Studio," Microsoft, 2013. [Online]. Available: http://www.microsoft.com/visualstudio/eng/products/visual-studio-ultimate2012. 10. S. Du and M. Vuskovic, "Temporal vs. Spectral Approach to Feature Extraction," [Online]. Available: http://medusa.sdsu.edu/Robotics/Neuromuscular/Our_Publications/FE_Sijiang_pr ess.pdf.
25
Brain Computer Interface Application Framework 11. J. B. Ochoa, "EEG Signal Classification for Brain," 2002.
26