Hybrid Brain-Computer Interface

based on EEG and NIRS Modalities

Min-Ho Lee1, Siamac Fazli1, Jan Mehnert2 and Seong-Whan Lee1
1
Department of Brain and Cognitive Engineering, Korea University, Seoul, Korea
2
Department of Computer Science, Berlin Institute of Technology, Berlin, Germany

Abstract—Non-invasive brain-computer interfaces (BCIs) allow

users to control external devices by their intentions. Currently,
most BCI systems are synchronous, which means, they rely on
cues or tasks to which a subject has to react. It would be more
useful for users if they could control a device at their own will
(i.e., asynchronous BCIs). However, previous asynchronous BCI
systems that rely on non-invasive electroencephalogram (EEG)
measurements, are not accurate and stable enough for real world
applications. Previously, hybrid BCI systems, relying on
simultaneous EEG and near-infrared spectroscopy (NIRS)
measurements, have been shown to increase the classification
performance of synchronous motor imagery (MI) tasks. In this
study, we present a first report on an asynchronous multi-modal
hybrid BCI, based on simultaneous EEG and near-infrared
spectroscopy (NIRS) measurements and propose novel subject-
dependent classification strategies for combining both
measurements.
Keywords-Hybrid Brain-Computer Interface, Asynchronous
BCI, Combined EEG-NIRS, Subject-dependent Classification.
I. INTRODUCTION
Brain-computer interface (BCI) systems have been
developed to allow users to control computers or external
devices using their intentions, which are decoded from their
brain activity [1]. However, current asynchronous BCIs lack
stability in single trial classification and their performance
show a high variability across subjects. Therefore, enhanced
accuracy and universality of BCI decoding are priorities of
current BCI research. One solution of stabilizing the outcome
of BCIs lies in using two or more modalities of neuroimaging
data, i.e., measurement devices forming a hybrid BCI [2]. In
this paper we validate the capability of a hybrid NIRS-EEG
BCI for an asynchronous MI paradigm. Furthermore, we
discuss the optimal way of combining the signals from both
measurement modalities on a single subject basis.
II. METHODS
A. Participants and Eperimental Paradigm
14 healthy, right-handed volunteers (aged 20 to 30)
participated in the experiment. The subjects were seated in a
comfortable chair with armrests and were instructed to relax
their arms. The experiment consisted of 2 blocks of motor
imagery (50 trials per block per condition) with visual feedback.
For all blocks the first 2s of each trial began with a black
fixation cross, that appeared at the center of the screen. Then,
as a visual cue an arrow appeared pointing to the left or right
during the motor imagery.
Figure 1. Setup of NIRS sources and detectors and EEG electrodes.
B. Data Acquisition
During the whole experiment EEG and NIRS were
recorded simultaneously. The NIRS-System (NIRScout 8-16,
NIRx Medizintechnik GmbH, Germany) was equipped with 24
optical fibers (8 sources with wave-length of 760nm and
850nm, 16 detectors convolving to 24 measurement channels)
with sampling rate of 6.25Hz. EEG signals were recorded with
a multichannel EEG amplifier (BrainAmp by Brain Products,
Munich, Germany) using 37 Ag/AgCl electrodes at a sampling
rate of 100Hz. EEG electrodes and the optical probes (sources
and detectors) were located according to the international 10-20
system and covered frontal, motor and parietal areas of the
head.
C. Data Analysis
1) Pre-processing: To reduce the volume conduction
effects in the EEG signals we performed a simple Laplacian
spatial filter and subject-dependent band-pass filters. For the
NIRS signals, raw data (attenuation changes) were transformed
to hemoglobin concentration (oxygenated (HbO) and
deoxygenated (HbR) hemoglobin) changes with a modified
Beer-Lambert law (differential path length factors: 5.98 (for
higher wavelength: 830 nm) and 7.15 (for lower wavelength:
760 nm)). NIRS data was then low-pass filtered at 0.2Hz using
a one-directional digital filter, namely a 3rd order Butterworth-
filter. A baseline correction was performed by subtracting the
average signal from −2s to 0s before stimulus onset from each
trial.
2) Feature Extraction and Classification: For the EEG
data, Common Spatial Patterns (CSP) were used as spatial
filters. Log-variances of the temporally and spatially filtered
data were calculated as features. For the NIRS signal, the
features were calculated as the mean amplitudes of the
hemodynamic level of HbO in the interval between from 4s to
6s with respect to stimulus onset.
 TABLE I. THE EXPERIMENTAL RESULTS OF ASYNCHRONOUS BCI

EEG EEG+NIRS
Subject
Acc. Sen. Spec. AUC Acc. Sen. Spec. AUC
VPeaa 53.0 20.7 63.9 0.56 58.5 74.9 54.6 0.76
VPeab 77.8 72.6 80.2 0.79 77.1 72.7 79.0 0.78
VPeac 31.0 84.4 17.3 0.56 34.2 86.5 21.0 0.59
VPead 71.3 60.1 75.3 0.76 71.5 64.0 74.3 0.77
VPeae 61.7 56.6 63.4 0.69 56.6 64.8 53.9 0.68
VPeaf 36.6 83.5 25.0 0.60 38.1 80.3 27.5 0.61
VPeag 66.7 59.4 69.5 0.71 67.5 70.9 66.3 0.75
VPeah 66.2 55.4 70.9 0.67 67.4 58.1 71.3 0.69
VPeai 36.3 46.9 33.1 0.50 48.5 45.1 49.6 0.58
VPeaj 63.5 38.3 75.5 0.59 63.0 65.8 61.8 0.66
VPeak 27.0 80.8 13.1 0.52 61.4 87.7 55.2 0.80
VPeal 57.4 40.0 64.5 0.59 57.4 44.5 62.4 0.61
VPeam 45.9 55.9 42.9 0.60 57.0 63.7 55.1 0.70
VPean 75.3 64.4 80.2 0.76 76.1 74.1 77.0 0.80
Mean 55.0 58.5 55.3 0.64 59.6 68.1* 57.8 0.70*

For separation of the three classes ('left hand imagery', 'right
hand imagery' and 'idle') we considered multi-classification
strategy namely One Versus the Rest (OVR) [3]. Three types of
OVR-based linear discriminant analysis (LDA) classifiers were
trained for EEG: left vs others, right vs others, idle vs rest (all
OVR). Furthermore, one OVR classifier was estimated from
NIRS (HbO) data: idle vs rest. To emulate classification in
real-time, the classifiers were fed with data from a moving
window (width=1s, step size=0.5s). This procedure emulates a
pseudo-online paradigm [4].
3) Subject-dependent Classifier Combination: To estimate
the three mental states (right, left and idle state), we propose
subjects-dependent strategy based on EEG and NIRS
classifiers. At first, the current mental state was estimated by
OVR-based EEG classifiers, then the output of classifiers were
adjusted (or weighted) by the output of the NIRS-based
classifier. In order to find the optimal weighting parameter for
each subject, a grid search was performed on the validation
data set using Area under the Receiver Operating Curve
(AUC) as a loss function. For the performance evaluation of
an asynchronous BCIs, the accuracy, sensitivity, specificity,
and AUC scores were calculated for stand-alone EEG and the
proposed hybrid EEG-NIRS system.
III. RESULTS
Our first goal is to show that the proposed subject-
dependent classifier strategy can improve the classification
accuracy of asynchronous BCI systems. Table 1 shows the
classification results for stand-alone EEG and the proposed
multi-modal hybrid EEG-NIRS system. The hybrid asyn-
chronous BCI outperformed the EEG stand-alone system. The
results showed that all the evaluation parameters (accuracy,
sensitivity, specificity, and AUC score) enhance the
performance on average across all subjects. Paired t-tests
yielded significant performance increases for the sensitivity
and AUC scores (p = 0.03 and p = 0.01, respectively). However,
accuracy and specificity did not reach significant levels
(p = 0.10 and p = 0.52, respectively).
IV. DISCUSSION AND CONCLUSION
To stabilize the performance of asynchronous BCIs, we
introduced a multi-modal hybrid BCI. Our results outline that
for some subjects a single modality is not capable to estimate
mental states reliably enough for the robust control of
asynchronous BCIs. By employing a hybrid-BCI approach, the
overall performance was significantly enhanced and some
subjects gained remarkably improved accuracies. Furthermore,
subjects (and patients) who are not able to operate a BCI (so-
called 'BCI illiteracy' [5]) will find a viable alternative in the
presented hybrid asynchronous BCI approach. However, the
analyzing routines presented here, have been solely validated in
an offline fashion using pseudo-online technique. Our future
work will extend our research to a real-time asynchronous
hybrid BCI with visual feedback or for controlling external
devices.
ACKNOWLEDGMENT
This work was supported by National Research Foundation
of Korea (NRF) funded by the Ministry of Science, ICT and
Future Planning (No. 2012-005741).
REFERENCES
[1] J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller and, T.
M. Vaughan, “Brain–Computer Interfaces for Communication and
Control,” Clinical neurophysiology, vol. 113, 2002, pp.767–791.
[2] S. Fazli, J. Mehnert, J. Steinbrink, G. Curio, A. Villringer, K. R. Müller,
and B. Blankertz, “Enhanced Performance by a Hybrid NIRS-EEG
Brain Computer Interface,” Neuroimage, vol. 59, 2012, pp. 519-529.
[3] G. Dornhege, B. Blankertz, G. Curio, and K.-R. Müller, “Boosting Bit
Rates in Noninvasive EEG Single-trial Classfications by Feature
Combination and Multiclass Paradigms,” Biomedical Engineering, vol.
51, 2004, pp. 993–1002.
[4] B. Blankertz, G. Curio, and K.-R. Müller, “Classifying Single Trial
EEG: Towards Brain Computer Interfacing,” Advances in neural
information processing systems, vol. 1, 2002, pp. 157–164.
[5] C. Guger, G. Edlinger, W. Harkam, I. Niedermayer, and G. Pfurtscheller,
“How many people are able to operate an EEG-based brain-computer
interface (BCI)?,” Neural Systems and Rehabilitation Engineering, vol.
11, 2003, pp. 145–147.