Cognitive Infocommunications

PterBaranyi AdamCsapo
GyulaSallai
Cognitive
Infocommunications
(CogInfoCom)
Cognitive Infocommunications (CogInfoCom)
Pter Baranyi Adam Csapo Gyula Sallai
Cognitive
Infocommunications
(CogInfoCom)
123
Adam Csapo
Institute for Computer Science
and Control of the Hungarian
Academy of Sciences
Budapest, Hungary
Pter Baranyi
Szchenyi Istvn University
Gyor, Hungary
Institute for Computer Science
and Control of the Hungarian
Academy of Sciences
Budapest, Hungary
Szchenyi Istvn University

Gyor, Hungary
Gyula Sallai
Budapest University of Technology
and Economics
Hungary
Future Internet Research
Coordination Centre
University of Debrecen
Hungary
ISBN 978-3-319-19607-7
DOI 10.1007/978-3-319-19608-4
ISBN 978-3-319-19608-4 (eBook)
Library of Congress Control Number: 2015953263

Mathematics Subject Classification (2000): 0101, 0401, 11Axx, 2601
Springer Cham Heidelberg New York Dordrecht London
Springer International Publishing Switzerland 2015
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, express or implied, with respect to the material contained herein or for any
errors or omissions that may have been made.
Printed on acid-free paper
Springer International Publishing AG Switzerland is part of Springer Science+Business Media (www.
springer.com)
Preface
In the past decades, the role and value of information has significantly increased in
nearly all aspects of daily life. Information besides matter and energy has become
a primary conceptual component of human thinking, as evidenced by the growing
appearance of the word information in a widening range of contexts. Alongside
the factories, power plants and relevant industries which provide, store, process
and transport matter and energy, a system of infocommunication services providing
analogous functionalities with respect to information now pervades our everyday
experience. As a result, access to infocommunications has become a basic need, in
much the same way as the ability to access electric power through outlets in walls is
taken for granted. In fact, most services pertaining to matter, energy and information
are co-dependent at various levels, with each service relying on others for its own
sustained efficiency.
At the same time, it is important to realize that while services relevant to matter
and energy primarily stem from and serve physical needs, infocommunications
is motivated by and serves high-level mental activities, which are supervenient
on human knowledge and intelligence. Given that human mental capabilities are
generally more flexible to adaptation than material-energetic capabilities governed
purely by physical-biological reality, new patterns, modes and structures are
constantly appearing in the ways we interact with information. Coupled with recent
advances in technology, this inventiveness in means of interaction is culminating in a
process of human entanglement with information and communication technologies
(ICT). This process has been identified and described by many authors, and can
be observed from a number of different perspectives relevant at different scales
of time, including the key measures used to characterize the performance of new
technologies, the multiple (physical) levels at which interaction can occur, and
the increasing relevance of what we refer to as human-oriented fields in the
development of new technologies.
With respect to the ways in which the performance of new technology is characterized, it can be observedat least in the case of personal informatics devices
and from the perspective of end usersthat measures of hardware performance
are losing importance in comparison with high-level (soft) functional capabilities.
v
vi
Preface
While 20 years ago the specifications of personal computers would have included
details on power consumption, processor clock frequency, cache size and other
low-level hardware details, later the capacity of random access memory, as well
as networking and graphical capabilities gained relative importance. Today as end
users of personal informatics devices, we rarely take note of such specifications,
despite their still being accessible. Instead, we take for granted that a new device
will include capabilities for the most common communication protocols, and will
include a camera, for instance, that produces images of decent quality for most
practical purposes. Even our focus on battery life is much more oriented towards
capability than towards raw measures of hardware performance: if the battery
life of a device can last for a full working day, most practical requirements for
functionality will be satisfied. As this tendency continues, the specifications of
personal informatics devices most important to us will soon beand already are
to some extentthose that focus on high-level capabilities built into and around the
operating systems running on them, such as the richness of associated application
stores, the various capabilities of built-in personal assistants such as Siri, Google
Now and Cortana (including capabilities for generating and understanding speech,
text, gestures and other emotional behaviours), or capabilities for handling multiple
or multi-user profiles for family use. In the end, our personal informatics devices
can be expected to evolve into tools that merely provide an intelligent window
into a ubiquitously and collaboratively accessible augmented world of personalized
information servicesa fact that yields significant motivations, and important
support for key notions behind research areas such as speechability, ethology-based
CogInfoCom and mathability, as discussed extensively in the book. The capabilities
of future devices will be judged based on the extent to which they are motivated by
and fulfill the goals of such branches of CogInfoCom, and once this is the case, it can
be said that these are the research areas which supply the most relevant parameters
requiring specification.
With respect to the multiple levels at which interaction can occur (i.e. in terms of
the second perspective behind entanglement listed earlier), it can be observed that
the phenomenon of entanglement is gaining strength at levels of communication
ranging from the low level of cell-electronics connections (as in, e.g., braincomputer interfaces and some areas of bionics) through connections expressed
at the representational, structural or command level (involving, e.g., multimodal
interaction), all the way to high-level collective relationships among populations of
humans and ICT (involving e.g. high-level services based on data aggregation and
analysis). These three levels and their impacts are discussed extensively in the book,
in relation to research areas such as CogInfoCom channels and socio-cognitive ICT.
The point is also made that as the three levels of interaction gain strength, they can
be expected to influence each other in as yet unforeseen ways.
Finally, with respect to the increasing relevance of human-oriented fields in
new technologies (i.e. in terms of the third perspective behind entanglement listed
earlier), it can be observed that as our interaction with information evolves, both
the theory and practice of infocommunications are increasingly relying on results
from fields of scientific inquiry motivated by the goal of better understanding how
Preface
vii
humans think, feel and act. In particular, all fields dealing with high-level aspects
of human cognition, including even the humanities, can ultimately be expected to
make important contributions. It is true that this is a slower process, characterized
by a larger time constant than the processes of change behind functional capabilities
and levels of interaction; nevertheless, it is important to note that the effects of such
human-oriented fields can already be observed in the plethora of new research
directions emerging within ICT. For example, the Future Internet concept that is
actively supported by the EU has several branches that are strongly relevant to this
phenomenon, such as the Internet of Things (IoT) and 3D Internet. In IoT, the goal
is to deploy as many sensors and actuators as possible into the surrounding physical
network, enabling the measurement and actuation of previously inaccessible parts of
physical-biological-mental reality. Irrespective of how the data collected in this way
is represented, it will serve as a kind of informational, or virtual image of everything
in the world that is important to us. Through the concept of 3D Internet, researchers
are seeking ways to attribute 3D geometry to this growing collection of data
including both data that is naturally and directly amenable to 3D representation
and abstract data sets requiring virtual visualizationin order to support our
capabilities for interpretation and manipulation. The increasingly popular domain of
Big Data also represents an initiative towards processing IoT data, albeit primarily in
more automated, less human-interpretable ways. A common effect of these research
areas is that a new infrastructure is emerging, which can be seen as a collection
of artificial sensory organs that extend across the globe and are capable of
extracting different kinds of information in a way that integrates humans into the
picture. It can be expected that this human integration will eventually bring about
new forms of entanglement, which extend through periods of time that are much
longer than the relatively episodic interactions imagined today. New generations
are already growing up in a kind of co-evolution with ICT, as evidenced by
the growing number of psychological effects that have been documented with
respect to changing cognitive-social-technological relationships. In the book, the
term cognitive entity is introduced to characterize such entangled relationships
in terms of cognitive capabilities. Further, the term generation CE (generation
of cognitive entities) is introduced by analogy with the generations X, Y and Z,
to reflect the fact that members of todays generation are growing up in this new
world starting from the first phases of their personal and social development, and
that through their entangled co-evolution with ICT, they can be seen to embody
an entirely new set of cognitive capabilities. The general conclusion is that if
researchers would like to record, explain and understand the diverse phenomena
surrounding human-ICT entanglement and newly emerging cognitive entities, as
well as build new technologies based on those phenomena, then having recourse to
results from relevant human-oriented research fields will be essential, in much the
same way as the development of personal informatics devices today involves joint
efforts from information and electrical engineers as well as experts of psychology
and ergonomics.
The goal of the emerging field of cognitive infocommunications (CogInfoCom) is to support the investigation of human-ICT entanglement related
viii
Preface
phenomena as well as how such phenomena lead to new cognitive entities and
the appearance of the generation CE. The field places special emphasis on the
qualification and quantification of cognitive capabilities, and aims to adopt a multiand interdisciplinary approach in doing so. The distinction between multi- and interdisciplinarity is an important one: while the formulation of the scope and goals of
the field encourages the involvement of a large variety of pertaining disciplines, it
also aims towards an integration of the perspectives and methodologies which they
represent. Thus, besides aiming to create a common language, it is also the goal
of CogInfoCom to establish a common way of thinking about relevant problems,
and a common approach towards designing new technologies. This is not without
challenges. The engineering sciencesand by extension, infocommunications
generally emphasize the synthesis of novel systems so as to provide functionalities
that are guaranteed to be stable and reliable in at least some respect. Crucially, even
the modeling of component structures and phenomenaserving as a prerequisite to
engineering designproceeds with these ultimate goals in mind. Considerations
in engineering often do not extend to the requirement of holistic completeness;
instead, specific interpretations are focused on that are relevant to specific design
goals. In contrast, the primary focus of most branches of the cognitive sciences
is analytical in nature, with the goal of recording, understanding and explaining
cognitive phenomena in a way that is phenomenologically complete. However, such
analyses do not necessarily lead to functional reproducibility. As a result, it is often
the case that a set of experiments and models separately deemed precise from the
perspective of an engineering field are seen as inadequate from the perspective of
a cognitive science-related field, and vice versa. If a personal computer from our
century were to have suddenly appeared functionally complete 100 years ago, surely
its operation would have been described from a different set of perspectives than
those descriptions that are accepted as blueprints for their production today. The
former set of descriptions would also likely have been inadequate for the design of
a fully functional computer. From the perspective of CogInfoCom, the key point is
that while investigations in the cognitive sciences focus on a biological structure that
has been highly differentiated through the complex processes of evolution, and that
as a result cannot easily be copied or reproduced using todays engineering
concepts, it is nevertheless possible to obtain new knowledge by attempting to
extend human cognitive capabilities through the synthesized design of technologies
in infocommunications. This latter goal in turn necessitates a broadening in scope
of the human-oriented methodologies used in the cognitive sciences to address
(artificial) cognitive phenomena relevant to artificial systems.
It can often be observed that well-established research fields become more
specialized, or perhaps show increasing signs of segmentation, while specific
segments meet with previously distinct research fields to form completely new
synergies. This process can be likened to the roots of a tree, which meet at various
points to create the trunk, which in turn gives distinction to different branches, which
then meet with the foliages of neighbouring trees. In this process, some researchers
choose to maintain focus and achieve completeness in their own specialized
research, while others instead are in constant search of new possibilities for synergy
Preface
ix
with other fields. Although seemingly there may be a contradiction between the
two approaches, in fact both of them together are necessary for the emergence
of valuable new research directions. So it is the case with CogInfoCom: one can
find in it the influence of well-established research fields (e.g. augmented cognition,
human-computer interaction, virtual reality and others), while at the same time new
directions and specializations are also appearing due to the unique viewpoint of the
field, as discussed in several chapters of this book. It is important to emphasize
that these new specializations born under CogInfoCom are often difficult, if not
impossible, to categorize into already existing fields. Interesting and novel changes
in viewpoint are often possible by considering various phenomena from tighter, or
broader perspectives. All of these tendencies can be observed in parallel at the IEEE
CogInfoCom conferences series. In some cases, sessions with well-specified topics
have included presentations of researchers coming from widely different areas, such
as information, medical or cognitive sciences, which has led to valuable exchanges
with respect to a focused topic. In other cases, sessions named after completely
new research directions appeared, and have included presentations which aimed
to establish novel perspectives. This heterogeneity can also be observed in the
fields represented by researchers who have co-authored papers at conferences and
special issues on CogInfoCom; for example, joint papers were published together
by researchers with backgrounds in computer science, systems and control theory,
mathematics, ethology and cognitive psychology.
As in the case of all newly established conference series, the CogInfoCom conferences have been attended by both supporters and sceptics of the motivations and
ideas behind the field. At the 3rd IEEE International Conference on CogInfoCom,
it happened that a professor dealing with human-computer interaction voiced his
criticism, saying that There is nothing new in CogInfoCom: : : it is just another
name for what I am working on. Later at the same event, another scientist from
the field of computational linguistics made almost the same remark, expressing
his view that CogInfoCom was just another label given to the already existing
field he was dealing with. As the same criticism was later raised with respect to
cognitive informatics and neurocomputing as well, it is worth considering what
these criticisms would entail, if accepted as true:
CogInfoCom D HCI
CogInfoCom D computational linguistics
CogInfoCom D cognitive informatics
CogInfoCom D neurocomputing
Were these equalities to be accepted as true, clearly they would also suggest
relationships of equivalence among their right-hand sidessomething that has never
been claimed by any serious researcher acquainted with the scope and goals of these
fields:
Preface
HCI D computational linguistics

computational linguistics D cognitive informatics
cognitive informatics D neurocomputing
HCI D cognitive informatics
:::
No researcher has ever suggested that cognitive informatics is just another name for
HCI, or that computational linguistics is actually the same as any of these fields.
These facts go well to demonstrate the multi-disciplinary nature of CogInfoCom:
the criticisms of novelty were raised by researchers who were all able to discover
the motivations for their own work behind it, and yet, they all came to the same
conference representing different fields, never before seen to have much in common!
However, in addition to having this strongly multi-disciplinary background, the
CogInfoCom conferences have also led to the formulation of a wide range of new
research perspectives and initiatives, which, as outlined in this book, cannot be
trivially broken down into the contributions of previously existing individual fields.
Thus, the emergence of CogInfoCom has shown that the time is ripe for all of the
cognitive + social + technological phenomena discussed above to be investigated
from a unified perspective. For these reasons, we respectfully ask readers to look
further than the ways in which their own research perspectives are represented in the
book, and to support the development of CogInfoCom by actively looking for ways
to approach this new and common perspective. In parallel to the process through
which humans and ICT are merging together and becoming increasingly entangled,
we encourage readers to help develop both our analytical and synthesis-oriented
understanding of newly emerging cognitive entities.
Writing a book that does justice to the influence provided by the large number
of fields behind CogInfoCom and also adequately represents the new perspectives
it has brought about is a significant challenge. No attempt at meeting this challenge
would have been possible without the invaluable suggestions and contributions
of many people who have influenced our thinking about CogInfoCom over the
years. Although providing a complete list of mentors, colleagues and students
who have made such contributions would be impossible to compile, we would
like to express our gratitude to Nick Campbell, Tom Gedeon, Hideki Hashimoto,
Kristiina Jokinen, Toshikazu Kato, Tetsuo Kotoku, Joo-Hoo Lee, Gabor Magyar,
Helen Meng, Adam Miklosi, Geza Nemeth, Mihoko Niitsuma and Csaba Pleh
for their active involvement in initiating a common conceptual framework for
further discussions at the very beginning of the CogInfoCom conference series.
We would also like to thank Hassan Charaf, Valeria Csepe, Anna Esposito, Peter
Foldesi, Hamido Fujita, Attila Gilanyi, Andras Hajdu, Karoly Hercegfi, Thomas
Hermann, Ferenc Honbolygo, Laszlo Hunyadi, Lajos Izso, Joni Jamsa, Zsolt Janko,
Anita Komlodi, Maria Koutsombogera, Laszlo Kovacs, Szilveszter Kovacs, Mika
Luimula, Jean-Claude Martin, Frederic Noel, Harris Papageorgiou, Sakari Pieska,
Claudiu Pozna, Radu-Emil Precup, Laura Slaughter, Bjorn Solvang, Wei Deng
Preface
xi
Solvang, Peter Varlaki, Gabor Vattay, Klara Vicsi, Alessandro Vinciarelli, Carl
Vogel and Yeung Yam, who have significantly contributed to the success of the
conference series by helping to organize sessions and tracks that were often
pioneering in their subject matter, and also by supporting through their work the
appearance of several special issues on CogInfoCom in international scientific
journals. We thank the entire CogInfoCom community for helping to create a
friendly and scientifically stimulating atmosphere, as well as for bringing their
valuable perspectives into the fruitful discussions held during and oftenthrough email and Skypebetween the CogInfoCom conferences. Last but not least, special
thanks is due to Anna Szemereki for her Herculean efforts in the organization of
every last detail of the CogInfoCom conferences year after year.
Budapest, Hungary
April 2015
Pter Baranyi
Adam Csapo
Gyula Sallai
Contents
Part I
Foundations of CogInfoCom
Emergence of CogInfoCom in Science and Technology . . . . . . . . . . . . . . . .

1.1 What Is CogInfoCom? . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
1.2 Scientific Priming Effects .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
1.2.1 Convergence from a General Perspective . . . . . . . . . . . . . . . . .
1.2.2 Convergence from an Infocommunications Perspective ..
1.3 Technological Priming Effects. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
3
4
5
5
7
8
Definitions, Concepts and Assumptions . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

2.1 Defining CogInfoCom . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
2.2 Concepts Emerging from CogInfoCom . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
2.2.1 Mode of Communication .. . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
2.2.2 Type of Communication .. . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
2.3 Implicit and Explicit Assumptions . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
2.3.1 Levels of Merging and Entanglement .. . . . . . . . . . . . . . . . . . . .
2.3.2 Levels of Cognitive Capability . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
2.3.3 An Emergent Concept of Information .. . . . . . . . . . . . . . . . . . . .
2.3.4 Transitions from Operation to Functionality . . . . . . . . . . . . . .
13
13
14
14
15
16
16
18
21
21
ICT-Based Foundations of CogInfoCom .. . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

3.1 The Digital Convergence.. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
3.2 From Telecommunications to Cognitive Infocommunications .. . . .
3.2.1 Digitization of Separate Communication Sectors .. . . . . . . .
3.2.2 Unification of Telecommunications:
Electronic Communications .. . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
3.2.3 Expansion of Telecommunications: Infocommunications
3.2.4 Expanding Content Space: Cognitive
Infocommunications .. . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
3.3 ICT, TIM, DES and CogInfoCom . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
23
23
25
26
27
28
31
32
xiii
xiv
Contents
Part II
Research Directions Under CogInfoCom
Synergies Between CogInfoCom and Other Fields .. . . . . . . . . . . . . . . . . . . .

4.1 Affective Computing .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
4.2 Augmented Cognition .. . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
4.3 Body Area Networks .. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
4.4 Brain-Computer Interfaces.. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
4.5 Cognitive Informatics . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
4.6 Cognitive Networks for Future Internet .. . . . . . . .. . . . . . . . . . . . . . . . . . . .
4.7 Human-Computer Interaction and Multimodal Interaction . . . . . . . .
4.8 Sensory Substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
4.9 Social Signal Processing .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
4.10 Virtual and Augmented Avatars . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
4.11 Virtual and Augmented Reality . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
39
40
41
43
44
45
45
47
50
51
52
54
CogInfoCom-Driven Research Areas.. . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

5.1 CogInfoCom Channels .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
5.2 Speechability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
5.3 Socio-Cognitive ICT . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
5.4 Common Features and Techniques . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
5.4.1 Icon-Based Approaches and Applications.. . . . . . . . . . . . . . . .
5.4.2 Context-Based Approaches and Applications .. . . . . . . . . . . .
5.4.3 Pattern-Based Approaches and Applications .. . . . . . . . . . . . .
5.5 A Summary of Common Concepts .. . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
57
57
59
60
62
63
66
69
71
Novel Research Initiatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

6.1 The Virtual Collaboration Arena . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
6.1.1 Affordances for Multi-user Collaboration .. . . . . . . . . . . . . . . .
6.1.2 Synergies Supported by the VirCA Platform .. . . . . . . . . . . . .
6.2 Augmented Virtual Reality Mediated Neuroscience Research.. . . .
6.3 Ergonomics of Augmented Virtual Collaboration . . . . . . . . . . . . . . . . . .
6.3.1 Adaptation of Rasmussen and Reasons
Framework of Human Errors to VirCA. . . . . . . . . . . . . . . . . . . .
6.3.2 Types of Rule-Based Human Errors Identified .. . . . . . . . . . .
6.4 Ethologically Informed CogInfocom (EtoCom) . . . . . . . . . . . . . . . . . . . .
6.4.1 CogInfoCom Tanglefaces and Engines .. . . . . . . . . . . . . . . . . . .
6.4.2 Examples: The EtoMotor and Its Applications.. . . . . . . . . . .
6.5 CogInfoCom-Aided Industrial Engineering.. . . .. . . . . . . . . . . . . . . . . . . .
6.5.1 Design Issues Relevant to Cognitive Networks . . . . . . . . . . .
6.5.2 Industrial Robotics and Production Management .. . . . . . . .
6.5.3 Industrial Capabilities of VirCA . . . . . . .. . . . . . . . . . . . . . . . . . . .
6.6 Mathability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
6.6.1 Granular and Analytical Representations.. . . . . . . . . . . . . . . . .
6.6.2 Example Application: Human-Aided
Resolution of Linear Functional Equations . . . . . . . . . . . . . . .
73
74
75
76
77
78
79
79
80
81
82
91
92
92
93
95
97
98
Contents
Part III
7
xv
CogInfoCom Channels
Structure of Sensory Signals: Icons and Messages. .. . . . . . . . . . . . . . . . . . . .

7.1 Motivations Behind CogInfoCom Channels . . . .. . . . . . . . . . . . . . . . . . . .
7.2 The Need for a Unified View of Communication:
Overlaps, Ambiguities, and Contradictions .. . . .. . . . . . . . . . . . . . . . . . . .
7.3 Unified Structure of Sensory Signals: Streams,
Icons and Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
7.3.1 CogInfoCom Streams . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
7.3.2 CogInfoCom Icons . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
7.3.3 CogInfoCom Messages. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
7.4 Icons and Messages: A Summary of Key Points . . . . . . . . . . . . . . . . . . .
CogInfoCom Channels: Perceptual Unification of
Representation and Meaning . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
8.1 CogInfoCom Channels and Relevant Concepts .. . . . . . . . . . . . . . . . . . . .
8.1.1 CogInfoCom Message Generated Perceptual Concepts . .
8.1.2 Definition of CogInfoCom Channels . .. . . . . . . . . . . . . . . . . . . .
8.1.3 A Novel Interpretation of the Concept of Modality .. . . . . .
8.2 Formal Representation of CogInfoCom Channel Parameters .. . . . .
8.2.1 Nomenclature and Notations . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
8.2.2 Mathematical Background . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
8.2.3 Generative Model of CogInfoCom Channels . . . . . . . . . . . . .
8.3 Design Challenges Behind CogInfoCom Channels:
A Summary of Key Points . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
Mapping Meaning onto CogInfoCom Channels . . . .. . . . . . . . . . . . . . . . . . . .
9.1 Concept Algebra Based Background .. . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
9.1.1 Key Definitions in the OAR Model .. . .. . . . . . . . . . . . . . . . . . . .
9.1.2 Message Generated Perceptual Concepts
in Terms of OAR . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
9.1.3 Tools for Analogy: Interactive Concepts . . . . . . . . . . . . . . . . . .
9.2 Concept Mapping Techniques as Inference in OAR . . . . . . . . . . . . . . . .
9.2.1 Direct Mapping . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
9.2.2 Analogy-Based Mapping .. . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
9.2.3 A Note on the Joint Use of Mapping Techniques . . . . . . . . .
9.3 Future Perspectives for Automated Mapping . . .. . . . . . . . . . . . . . . . . . . .
10 Tunability of CogInfoCom Channels . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

10.1 Preliminary Discussions . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
10.2 Mathematical Background .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
10.2.1 Tensor Product Form of Multivariate Functions . . . . . . . . . .
10.2.2 HOSVD-Based Canonical Form of
Discretized Multivariate Functions . . . .. . . . . . . . . . . . . . . . . . . .
10.3 Generic Tuning Model for CogInfoCom Channels .. . . . . . . . . . . . . . . .
10.4 The Spiral Discovery Method (SDM) .. . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
105
105
107
109
110
111
114
116
119
120
120
121
122
123
123
125
126
127
129
129
130
131
134
134
135
136
138
139
141
141
144
144
145
146
147
xvi
Contents
10.5 Implementation of SDM . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 151

10.6 Usability of SDM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 152
11 Temporal Aspects of CogInfoCom Channel Design . . . . . . . . . . . . . . . . . . . .
11.1 Challenges Behind Temporal Aspects . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
11.2 Aspects of Low-Level Biological Communication . . . . . . . . . . . . . . . . .
11.2.1 CogInfoCom Cues and Signals . . . . . . . .. . . . . . . . . . . . . . . . . . . .
11.2.2 Cues, Signals and Messages in Past Research . . . . . . . . . . . .
11.3 Aspects of High-Level Human Communication .. . . . . . . . . . . . . . . . . . .
11.3.1 Volition .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
11.3.2 Directness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
11.3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
11.4 Putting It All Together: Progressive CogInfoCom Channels .. . . . . .
Part IV
157
157
159
160
162
162
164
165
166
166
Future Perspectives of CogInfoCom
12 Cognitive Capabilities in the Future Internet . . . . . . .. . . . . . . . . . . . . . . . . . . .

12.1 Challenges of Todays Internet . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
12.2 Future Internet Visions .. . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
12.3 Recent Future Internet Capabilities. . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
12.4 Clustering Future Internet Research Themes . . .. . . . . . . . . . . . . . . . . . . .
12.4.1 Cluster 1: Internet Basic Research (Internet Science) . . . .
12.4.2 Cluster 2: Future Internet Modeling, Analysis and Design
12.4.3 Cluster 3: Future Internet Network Architectures.. . . . . . . .
12.4.4 Cluster 4: Data and Content Technologies . . . . . . . . . . . . . . . .
12.4.5 Cluster 5: 3D Internet and Cognitive Infocommunications
12.4.6 Cluster 6: Internet of Things (IoT).. . . .. . . . . . . . . . . . . . . . . . . .
12.4.7 Cluster 7: Cyber-Physical Systems and Applications .. . . .
12.4.8 Cluster 8: Future Internet Based Community
Applications.. . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .
12.4.9 Cluster 9: Experimentation, Standardization, Regulation
173
173
175
178
179
181
181
181
182
183
183
184
184
184
13 Towards a Cognitive Internet Ecosystem . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 187

13.1 Reshaping Digital Ecosystems.. . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 187
13.2 A Long-Term Vision: Cognitive Internet Ecosystem .. . . . . . . . . . . . . . 190
References .. .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 193
Glossary . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 211
Index . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 213
About the Authors
Pter Baranyi is a professor at the Multidisciplinary Doctoral School of the

Szchenyi Istvn University, and head of the 3D Internet based Control and
Communications Laboratory at the Institute for Computer Science and Control of
the Hungarian Academy of Sciences.
Adam Csapo is a postdoctoral researcher at the Institute for Computer Science and
Control of the Hungarian Academy of Sciences, and an assistant professor at the
Department of Informatics of the Szchenyi Istvn University.
Gyula Sallai is a professor at professor at the Department of Telecommunications
and Media Informatics of the Budapest University of Technology and Economics,
and Scientific Director of the Future Internet Research Coordination Centre of the
University of Debrecen.
xvii
About the Book
This book describes the theoretical foundations of cognitive infocommunications

(CogInfoCom), and provides a survey on state-of-the-art solutions and applications
within the field. The book covers aspects of cognitive infocommunications in
research fields such as affective computing, BCI, Future Internet, HCI, HRI,
sensory substitution, and virtual/augmented interactions, and also introduces newly
proposed paradigms and initiatives under the field, including CogInfoCom channels,
speechability and socio-cognitive ICT. The book focuses on describing the merging
between humans and information and communications technology (ICT) at the level
of cognitive capabilities with an approach towards understanding the perspectives
of generation CE (after generations X, Y and Z), and developing future cognitive
ICT.
xix
Part I
Foundations of CogInfoCom
This part presents the scope and goal of cognitive infocommunications

(CogInfoCom), and discusses the primary motivations behind the field from various
perspectives.
In Chap. 1, a research historical overview is given on the phases and synergies
leading to the emergence of cognitive infocommunications, both from a scientific
and technological point of view. Based on the chapter, it can be concluded that the
emergence of the field is a natural development in view of the broader evolution
of relevant research domains, including infocommunications and the cognitive
sciences.
In Chap. 2, a deeper discussion on the scope and goals of CogInfoCom is
provided. This is followed by an overview of novel concepts which have emerged
based on the definition, as well as the various implicit and explicit assumptions
that serve as a basis for CogInfoCom research, including the assumptions of level
of cognitive capability and entanglement between humans and information and
communications technology (ICT).
Finally, in Chap. 3, the foundations of CogInfoCom are discussed from the
perspective of the digital convergence process leading to the diffusion of ICT.
Although the discussion is motivated by the fact that CogInfoCom is strongly
influenced by engineering fields, focus is also placed on changes in associated value
chains and social-technological phases (in particular, the cognitive phase). Based on
the chapter, the position of CogInfoCom within the Digital Ecosystem is clarified.
Chapter 1
Emergence of CogInfoCom in Science

and Technology
In this chapter, an overview is given of the scientific and technological triggers

leading to the emergence of cognitive infocommunications (CogInfoCom).
Section 1.1 of the chapter begins with a brief introductory description of the
motivations behind the field. This is followed by a discussion of how the field
has emerged, first from the point of view of the scientific influences (Sect. 1.2),
and later from the point of view of technologies (Sect. 1.3) contributing to it. The
discussions provided by the chapter reveal that the evolution of all scientific fields
is characterized by a process of maturation and ultimate convergence into newer
fields of inquiry. In many cases such processes are supported by both conceptual
and technological advances which bring to light possibilities for new theoretical
approaches as well as previously unimagined application areas. It is demonstrated
that the emergence of CogInfoCom can be attributed to a similar process.
Some explanation is in order here regarding the term priming effect, which
is used in Sects. 1.2 and 1.3 to describe the nature of the influence of scientific
research fields and technology on CogInfoCom. The term was taken from the
study of cognitive biases, where a concept manifesting itself in any of a variety
of forms (i.e. as a subconscious impression, a conscious idea, or alternatively as
a behavior or emotion experienced) is capable of selectively priming other related
concepts, effectively rendering them easier to recall and/or experience (Kahneman
2011; Gilovich et al. 2002). The loose analogy with this technical notion is that the
scientific currents and technological advances that define our contemporary world
inevitably influence the way in which new problems, challenges and research goals
are defined. It is with respect to these influences that the chapter aims to position the
field of CogInfoCom.

P. Baranyi et al., Cognitive Infocommunications (CogInfoCom),
DOI 10.1007/978-3-319-19608-4_1
1 Emergence of CogInfoCom in Science and Technology
1.1 What Is CogInfoCom?

Cognitive infocommunications (CogInfoCom) is an interdisciplinary research field
that has emerged as a synergy between infocommunications and the cognitive
sciences.1 One of the key observations behind CogInfoCom is that humans and
ICT are becoming entangled at various levels through a convergence process
between these disciplines, as a result of which new forms of cognitive capability
are appearing. Crucially, these capabilities are neither purely natural (i.e., human),
nor purely artificial; therefore, it is suggested that they should be treated in a
way that unifies both engineering and human-oriented perspectives. Thus, while
CogInfoCom aims to provide qualitative and quantitative analyses of emergent
communicative phenomena through the analogy of natural cognitive capabilities, it
also focuses on the development of approaches and methodologies for the synthesis
of new human-ICT capabilities based on engineering principles.
A more specific description of the scope and goals of CogInfoCom is provided
in Chap. 2. A deeper discussion of assumptions underlying the field and specific
terminologies to which it has given rise is also provided in that chapter. At this
point, it is nevertheless worth highlighting that some of the most important features
of the field are as follows:
Focus on cognitive capability: rather than merely focusing on ways in which
humans, devices and ICT infrastructures interact, the field acknowledges the
notion of cognitive capabilityan abstraction which allows for the introduction of temporal and contextual considerations into the analysis and design of
relevant phenomena;
. . . from a generic perspective: rather than restricting the notion of cognitive
capability to humans alone, the field adopts the point of view that such capabilities are emergent properties of any continued interaction and communication that
potentially involves both natural and artificial components and systems;
. . . at various (particularly at large) temporal scales: rather than merely
focusing on individual goal-oriented interactions at clearly specified points in
time, the field adopts the point of view that the entanglement between humans
and ICT is resulting in the necessity to consider their co-evolution at larger scales
of time;
. . . in the context of emergent functionality: rather than focusing exclusively on
function-oriented interactions, the field also considers cases where functionalities
developed at one time to fulfill a specific goal acquire new roles and usesa
phenomenon that is caused by constant change and growth in complexity of the
elements (and relationships thereof) defining human-ICT networks.
The term cognitive sciences is used here as a comprehensive term for those fields that deal
with the capabilities of biological systems, includingamong othersthe fields of psychology,
neuroscience, cognitive modeling, cognitive ergonomics and human factors, linguistics, biology,
anthropology, some branches of artificial intelligence, etc.
1.2 Scientific Priming Effects

In this section, it is argued that the appearance of CogInfoCom is a natural
development from a research historical point of view, and that this is well reflected in
the terminologies emerging from the field. In order to show this, the past evolution of
informatics, media, communications and the cognitive sciences is briefly described.
Chapter 3 adds further perspectives to the discussion provided here by considering
the value chains and social-technological phases relevant to the convergence of these
fields.
1.2.1 Convergence from a General Perspective

It is a common phenomenon for newly established fields to go through a process
of maturation and ultimate convergence. The evolution of informatics, media and
communications is no different: although these fields initially had different goals
and applied different methodologies, their maturation and growing pervasiveness
has led to the emergence of new synergies. Thus the fields of infocommunications,
media informatics and media communications appeared in the latter half of the
twentieth century (Sallai 2012b; Fransman 2002; Chaffee and Metzger 2001). The
subsequent evolution of these disciplines, in turn, has resulted in still newer patterns
of convergence. As modern network services aim to provide an increasingly holistic
user experience, the infocommunications sector now encompasses all information
processing and content management functions [. . . ] of information technology and
electronic media (Sallai 2012a, 2007; Preissl and Muller 2006). Thus, the modern
ICT/TIM sector was born.
Parallel to these developments, with the enormous growth in scope and technological relevance of the cognitive sciences, the new fields of cognitive media
(Nannicelli and Taberham 2014; Hokanson and Hooper 2000; Recker et al. 1995;
Kozma 1991), cognitive informatics (Wang 2002; Wang and Kinsner 2006; Vernon
et al. 2007) and cognitive communication(s) (Roschelle 1996; Hewes 1995); and
also2 (WUN CogCom 2008; Mitola and Maguire 1999) are gradually emerging. By
today, these fields have either fully made their way, or are steadily on their way into
standard university curricula and will eventually become a natural part of collective
awareness.
For example, a quick search reveals that several research groups and companies
around the world have cognitive media (sometimes together with the suffix
technologies) in their name. While originally the field was strongly motivated by
new prospects of virtually augmented, interactive education, today it is driven by a
more general focus on how various forms of media can be analyzed in terms of their
2
Cognitive communication and cognitive communications have different meaning in different
research communities, as described later.
effects on human cognition, and how they can be applied to sharing information
in ways that appeals to human cognitive capabilities. As a result, any research
focusing on interactive media, or interaction technologies in general will have strong
relevance to the field. However, another important factor behind cognitive media is
the growing prevalence of artificial sensory capabilities implemented in media: in
a way analogous to the human nervous system, the Internet as an interconnection
of globally distributed devices and nodes, together with the media applications
based on it can be seen as an artificial nervous system and an artificial set of
sensory modalities. Data available on social network platforms such as Facebook
and Twitter are increasingly used to predict and understand physical, mental and
social processes taking place in the world. According to this view, cognitive media
targets not only the cognitive effects that media has on humans, but also the
cognitive capabilities of media itself. As this viewpoint gains increasing viability, it
can be expected that at a certain point the term cognitive will no longer be reserved
exclusively for the description of human-oriented phenomena: although this was the
original meaning of the term (from a time when modern media, not to mention ICT
was still a dream of the future), today the ICT infrastructure surrounding us has
growing capabilities for sensing, representing and interpreting a growing scope of
phenomena.
Similarly, the terms cognitive informatics and cognitive communication/cognitive communications are omnipresent in delineations of research
focus. A definition of cognitive informatics can be found on the official web page of
a number of research organizations (e.g., the Pacific Northwest Laboratory funded
by the U.S. Department of Energy). Today there are several research institutes
dedicated to cognitive informatics, such as the Institute for Cognitive Informatics
(ICON) in Sheffield, UK, which has an annual budget of over one million GBP.
An IEEE International Conference on Cognitive Informatics has been held every
year since 2002, and several symposia with leading researchers invited as keynote
lecturers have been held in the past few years. Cognitive communication(s), when
used without the trailing s, can refer to the study of ways in which humans
anticipate context as a contributor to the choice of communication style, and
perceive the consequences of communication choices (in short, it is the study of
human mental models of communication). On the other hand, with the trailing
s, cognitive infocommunications is used to refer to devices and networks which
can assign resources and functionalities in dynamic and intelligent ways (as in
e.g., cognitive radio or cognitive networks). It is possible that in the future, these
two directions will reach common formulations for certain problemssee e.g.
Mitola and Maguire (1999), which specifically mentions user needs and application
scenarios as possible contributors to the adaptivity of cognitive radio.
In a way analogous to the evolution of media informatics, media communications
and infocommunications, examples of results which can be categorized as belonging
to cognitive media informatics, cognitive media communications and cognitive
infocommunications are gradually emerging, even ifas of yetthese fields are not
always clearly defined. The relations between the fields discussed here are shown in
Fig. 1.1 (further significant details on this figure are provided later in Chap. 3).
Fig. 1.1 Relationships between fields relevant to the convergence process behind CogInfoCom.
The figure highlights the fact that the emergence of CogInfoCom is a result of the way in which
the fields of informatics, media and communications have evolved. From a historical perspective,
CogInfoCom can also be regarded as an extension of cognitive communications (an in-depth
discussion on this aspect is provided in Chap. 3 of this book)
1.2.2 Convergence from an Infocommunications Perspective

As a result of the convergence process described above, infocommunications
today adopts a broader focus than before. An up-to-date and comprehensive
outline of the past evolution of infocommunications, as well as prospects of
its future development, is provided in Sallai (2012a). The convergence leading
to the infocommunications of today was thoroughly analyzed in the mid-1990s,
and was soon recognized by both by the Commission of European Communities
(European Commission 1997) and by the International Telecommunication Union
(ITU 1999).
The consensus is that this convergence took place at three levels, manifesting
itself in the unification of technologies, the integration of markets and the harmonization of regulation (Sallai 2012a). Thus, the same devices that are used to
communicate with others can also be used to access and process digital media
content (hence, both the technology and the market is available for unification).
Further, it is due to the harmonization of regulation that the cost of using modern
infocommunications devices is transparent with respect to the kind of information
that they are used to process and transmit. The convergence process and its future
prospects can be summarized in the following four steps (for further details, see
Chap. 3):
1. Traditional separation, internal digitization. The technology, market and
regulation behind various content types (e.g., voice, text, audio-visual) are
managed separately.
2. Unified telecommunications. A partial unification was possible from a technological point of view with the rapid development of digital technology. Hence,
it became possible to handle different kinds of content with the same devices.
On the other hand, the electronic media production industry had yet to become
involved in the convergence process.
3. Infocommunications. The electronic media content producing industry, as well
as the appropriate institutions for regulation joined the convergence process
to produce the technological and social-economic revolution that is todays
infocommunications industry.
4. Cognitive infocommunications. A natural fourth step is the integration of
cognitive sciences into the convergence process behind infocommunications. On
the one hand, this involves an expanding content space, in which new information
types are gathered, inferred and reasoned upon in novel ways. On the other hand,
it is essential that this content be used in effective ways to extend the cognitive
capabilities of both users and ICT, allowing for the formation of new cognitive
entities.3
As a result of the synergy between the cognitive sciences and infocommunications, the content space that is accessed and manipulated by both users and ICT is
expected to grow in size and richness of interpretation. If the potential behind this
expanding content space is to be harnessed, it can be expected that the respective
unique capabilities of humans and ICT will be applied through new, long-term
interaction patterns, leading to the emergence of new cognitive entities. A significant
challenge in CogInfoCom is how to power these cognitive entities with the kinds
of information and functionality that are suitable to their roles and requirements.
1.3 Technological Priming Effects

An important tool using which emerging technologies are often characterized is the
technological hype cycle, a graphical representation developed by ICT research and
advisory firm Gartner Inc. (Fenn and Raskino 2008). The main premise behind the
hype cycle is that new technologies most often evolve through a process that can be
broken down into the following five phases (Fig. 1.2):
1. Technology trigger. A potential breakthrough becomes apparent, and significant
public interest is raised through early proof-of-concept success stories.
2. Inflated expectations. The number of success stories associated with the technology increases, but is also accompanied by failures; although some enterprises
join efforts to develop and improve the technology, many are deterred by growing
perceptions of risk.
3. Trough of disillusionment. Interest wanes as the technology fails to deliver on
its (both real and perceived) promises. Most investments are tied to the condition
that existing implementations are improved to the minimum level of satisfaction
among early adopters.
3
For a fully detailed explanation of what is meant by the term cognitive entity, the reader is
referred to Sect. 2.3.2.
Fig. 1.2 Five phases of Gartners hype cycle in terms of technological expectations with respect
to time
4. Slope of enlightenment. Potential benefits of the technology become increasingly widely understood as an increasing number of successful use cases become
apparent. Second- and third-generation implementations of the product appear,
new pilots are increasingly funded, while more conservative actors remain
cautious.
5. Plateau of productivity. Mainstream adoption begins to take off. The broad
market applicability of the technology becomes clear, and its adoption yields
clear advantages.
One of the messages of the hype cycle model is that almost all new technologies
are initially surrounded by a kind of hype that is, in a way true to the definition of
the word, founded more on a combination of illusions, expectations and a desire
to provide something new than on a true realization of benefits, drawbacks and
associated tradeoffs. This is what leads to the trough of disillusionment, which
in turn often (or in the case of successful technologies, always) brings about a
reassessment with respect to potential manufacturing approaches and use cases.
The end result of this reassessment combined with a few instances of successful
applications is a better informed proliferation of the technology.
Besides making possible these general conclusions, however, the model also
allows for the tracing of technological advances through time, both in nominal and
relative terms. On the one hand the location of an individual technology on the
hype cycle curve allows for assessments to be made with respect to the timeliness
and the expected success of that technology in the near future. On the other hand,
10
technologies can also be assessed relative to one another, or alternatively in groups

if considered together based on some criterion. The latter type of group assessment
is especially informative if it is performed through a period of 510 years instead of
at a specific point in time: such a perspective allows for the maturation process of
technologies behind specific fields or industries to be evaluated.
In this section, we consider a group of those technologies and R&D initiatives
which can be interpreted as ready-to-use components of cognitive entities; or
alternatively as either catalyzing or at the very least influencing CogInfoCom
research (these technologies and R&D areas are further discussed in Chaps. 4 and 5).
Figure 1.3 provides two hype curves with several examples as of 2010 and 2014.
Based on a broad comparison of these two snapshots of technological progress,
several points can be made in terms of relevance to CogInfoCom:
Trends in 2010 were characterized by a comparatively higher proportion of
technologies directed at what may be referred to as elementary cognitive capabilitiesincluding speech recognition, gesture recognition, and location awareness. In contrast, the period between 2010 and 2014 has seen the appearance and
comparative proliferation of more refined, higher level cognitive capabilities
including emotional capabilities represented by affective computing, capabilities
for human augmentation, speech-to-speech translation and health monitoring.
Capabilities belonging to this latter category are often directed at capturing the
global state of an entire cognitive entity rather than merely providing details
on lower-level aspects of interaction.
As the former set of elementary capabilities have matured over the past
years, they have evolved into building blocks for the latter, higher level
capabilities. For example, earlier technologies directed at recognizing discrete
gestures, speech utterances or facial information have become a crucial part
of applications directed at e.g. input control through fine joint movements and
emotion recognition; while smart network technologies such as mesh sensor
networks have begun to form the basis of health monitoring infrastructures.
Many of those technologies which focus on higher-level cognitive capabilities
still havent reached the slope of enlightenment and are often prognosticated
to require 810 more years before maturationin much the same way as they
were 4 years ago (in case they were already present on the hype curve at
I
Fig. 1.3 Evolution of expectations between 2010 (top) and 2014 (bottom) in terms of CogInfoCom. Technologies less relevant to the field, such as cryptocurrencies, machine-to-machine
communication services and others were left out for improved clarity. Care was also taken to ensure
that technologies shown on the curve for 2014 were also included on the curve for 2010 if also
present in the originally published figure. Darker shades represent technologies expected to have
relatively longer maturation times. For example, the maturation process of human augmentation
technologies has been, and still is expected to take over 10 years, while the maturation of speech
recognition can be expected to take less than 2 years as of 2014. The original hype cycle curves can
be found at e.g. http://www.smartinsights.com/managing-digital-marketing/marketing-innovation/
technology-for-innovation-in-marketing/
Fig. 1.3 (continued)
11
12
that time). This suggests that the maturation processes of such capabilities are
governed by larger time constants and are expected to remain in the main focus
of CogInfoCom-related research areas for years to come.
In summary, the evolution of hype curves from the past 45 year period strongly
suggests that the primary technological components underlying CogInfoCom have
reached the phase of maturation, while the integration of these components as
building blocks into more complex capabilities has begun both in subtle ways as
well as at a very explicit conceptual level. This integration process can be expected
to gain further traction in the coming years as the technologies based on which it is
grounded remain at the stable end of maturation, and as new technological triggers
continue to appear based on those technologies (as demonstrated by the hype curve
of 2014, many new concepts are already appearing). Such salient processes of
technological integration are a strong motivating factor behind CogInfoCom.
Chapter 2
Definitions, Concepts and Assumptions
In this chapter, a description of the scope and goals of CogInfoCom is provided. This
is followed by an overview of novel conceptssuch as those of mode and type of
communication, as well as the more general notion of cognitive capabilitywhich
have emerged through the field. Further, a set of assumptions, primarily founded on
the existence and consequences of the merging process between humans and ICT,
are described in terms of their relevance to CogInfoCom research.
2.1 Defining CogInfoCom

Humans and the infocommunications network (ICT in a broader sense) surrounding
them are merging together and becoming entangled at various levels, ranging from
low-level connectivity at the cellular and electrotechnical level, all the way to the
highest level of sensing collective behaviors such as mass movements, mass habits
etc. As a result, humans (more generally, living beings) and infocommunications
will soon coexist as an entangled web, resulting in an augmentation of both
natural and artificial cognitive capabilities.1 This process is occurring today, and
is expected to gain further impact in the near future. By analogy, it also necessitates
a merging process between the scientific fields related to natural cognitive systems
and the scientific and technological fields related to infocommunications. The above
motivations have contributed significantly to the definition of CogInfoCom. A draft
proposal of the definition was provided in Baranyi and Csapo (2010), and was later
refined by Professors Nick Campbell, Tom Gedeon, Hideki Hashimoto, Toshikazu
Kato, Tetsuo Kotoku, Kristiina Jokinen, Joo-Hoo Lee, Gbor Magyar, Helen Meng,
As we will see, in many cases this separation between natural and artificial is no longer
meaningful.

DOI 10.1007/978-3-319-19608-4_2
13
14
2 Definitions, Concepts and Assumptions
Gza Nmeth, Mihoko Niitsuma and Gyula Sallai at the 1st International Workshop
on CogInfoCom, held in Tokyo, Japan in 2010. The finalized definition is as follows:
Definition 2.1. Cognitive infocommunications (CogInfoCom) investigates the
link between the research areas of infocommunications and the cognitive sciences,
as well as the various engineering applications which have emerged as a synergic
combination of these sciences. The primary goal of CogInfoCom is to provide a
systematic view of how cognitive processes can co-evolve with infocommunications
devices so that the capabilities of the human brain may not only be extended through
these devices, irrespective of geographical distance, but may also interact with the
capabilities of any artificially cognitive system. This merging and extension of
cognitive capabilities is targeted towards engineering applications in which artificial
and/or natural cognitive systems are enabled to work together more effectively.
2.2 Concepts Emerging from CogInfoCom

As will be discussed later in this chapter in further detail, the implicit and explicit
assumptions underlying CogInfoCom together form a unique viewpoint. As a
result, new notions and concepts capable of leading to new research directions are
continuously emerging. In this section, two early concepts central to multi-sensory
communication between various levels of cognitive capability are introduced: the
mode of communication, and the type of communication. In the future, these
concepts may be extended to provide a more detailed qualification (and in the
long run: quantification) of cognitive capabilities independent of the exchange of
communicational messages.
2.2.1 Mode of Communication

The mode of communication refers to the way in which the relationship between
actors at the two endpoints of can be characterized:
Intra-cognitive communication: information transfer occurs between two cognitive entities with equivalent cognitive capabilities (e.g., between two humans,
or between two humans in the same social-technological environmentas
determined by what is relevant to the application).
Inter-cognitive communication: information transfer occurs between two cognitive entities with different cognitive capabilities (e.g., between a human and an
artificially cognitive system, or between two humans in different social or technological environmentsas determined by what is relevant to the application).
2.2 Concepts Emerging from CogInfoCom
15
In accordance with the introductory remarks to this section, future developments

are expected to lead to a deeper, perhaps quantified understanding of what is meant
by level of cognitive capability. Further discussions on this notion can be found in
Sect. 2.3.2.
2.2.2 Type of Communication

The type of communication refers to the way meaning is conveyed between the two
communication entities:
Sensor-sharing communication: cognitive entities on both ends use the same
sensory modality to receive information.
Sensor-bridging communication: sensory information is not only transmitted,
but also transformed to a different, more appropriate sensory modality of the
receiving cognitive entity.
Representation-sharing communication: the same information representation
is used on both ends of communication.
Representation-bridging communication: sensory information is filtered
and/or adapted so that a different information representation is used on the
two ends of communication.
A sensor-sharing application brings novelty to traditional infocommunications in
the sense that it can convey any kind of normally perceptible signal (i.e., a signal that
could be perceived if there were no distance to communicate across) to the other end
of the communication line. The key determinant of sensor-sharing communication is
that the same sensory modality is used to perceive the information on the receiving
end of communication as would be if there were no distance between the sending
and receiving ends.
Sensor bridging can reflect both the way in which information is conveyed (i.e.,
by changing sensory modality) as well as the novelty of the information type that
is conveyed. Whenever the transmitted information type is imperceptible to the
receiving entity due to a lack of appropriate sensory modality, communication will
necessarily occur through sensor bridging.
A CogInfoCom application can be regarded as an instance of representation
sharing even if it bridges between different sensory modalities. By considering the
general characteristics of a representation (e.g., its character-based, icon-based etc.
nature) rather than the specific details of its physical or biological manifestation, it
becomes possible to describe representations of different modalities in unified ways.
As mentioned in the introductory remarks to this section, aspects encompassed
by the concept of type of communication are relevant only when explicit communication occurs in a way that is directed at information sharing. As this is only a small
part of what it means to communicate, further extensions can be expected to emerge
in future work (see also Sect. 2.3.3).
16
2.3 Implicit and Explicit Assumptions

The definition of CogInfoCom as well as the discussions on the scope and goals
of the field lay emphasis on the merging process between humans and ICT, as
well as on different levels of cognitive capabilities and long-term co-evolution of
biological and artificial systems (Baranyi and Csapo 2010, 2012). In this section,
a brief discussion is provided on these aspects to further highlight the focus of the
field. The section is concluded by observations on the new information concept that
is implicitly assumed by the CogInfoCom, as well as the transition from operation
to emergent functionality that implicitly motivates research within the field.
2.3.1 Levels of Merging and Entanglement

As mentioned earlier in Sect. 1, one of the key observations behind CogInfoCom
is that there is a merging process between humans and ICT that is resulting in
increasingly complex forms of human-ICT entanglement, and is at the same time
creating the necessity for an analogous convergence between technology and the
human-oriented cognitive sciences. The phenomena of merging and entanglement
in the context of ICT are clear not only from everyday experience, but have also been
remarked and analyzed to various degrees and in various contexts by many authors,
as in e.g. Romportl et al. (2015), Pang (2013), Gripenberg (2011), and Dahlbom
(1996).
From the point of view of interaction modes, the merging process between
humans and ICT can be observed at three different levels:
1. The first level of entanglement corresponds to low-level, direct relationships,
including those that rely on invasive and non-invasive forms of interface (as in
e.g. brain-computer interfaces). Entanglement at this low level allows for direct
sensing and control, however, it is also relatively cumbersome in that it requires
sensors to be implanted or worn and is also difficult to operate at conceptually
higher levels of command.
2. A different form of entanglement is possible at the level of personal informatics
devices, in which communication and interaction occur through (humanbut
crucially not only human) sensory modalities. The question of what kind of
communication language to use (i.e. in terms of message encoding) depending
on the semantics of the information, as well asamong othersthe modality
to be used, the application environment, and the users individual cognitive
capabilities are strongly relevant to this level of entanglement. It is important
to note that the challenge consists not only in providing effective and ergonomic
interface design, but also in accommodating the transfer of an expanding set
of semantic conceptsrelevant at large temporal scales, for instance in coexistive smart home and other augmented virtual reality applicationsthrough
the limited possibilities afforded by human sensory modalities.
17
3. Finally, a third level of entanglement can be seen to occur at the collective level of
multi-user interactions. Applications in this layer can have relevance to collective
behaviors in two ways: by making use of collective behaviors in order to support
individual users interaction with a system; or alternatively, by supporting the
prediction or a posteriori analysis of collective events based on an analysis of past
behaviors (both individual and collective). Such applications often rely on the
mining and analysis of vast amounts of heterogeneous data sourcesincluding
e.g. activity on social communication platforms.
From a general perspective, Hodder defines entanglement as the sum of four
types of relationships in which humans depend on things (HT), things depend on
other things (TT), things depend on humans (TH) and humans depend on humans
(HH) (Hodder 2014, 2012). All four of these co-dependence relationships can be
equally observed in the particular case of human-ICT entanglement. However, it is
important to note that all of them can, and should be interpreted at multiple temporal
scales, ranging from episodic interactions that are point-like in time to decadeslong periods of co-existence. An important quality of human-ICT entanglement
is that co-existence with ICT is becoming an inseparable feature of the everyday
experience of children growing up today. This is a new phenomenon that is opening
new avenues of research which extend far beyond human-computer interactions;
such research must take into consideration the not only the capability to achieve
certain functions in comfortable and effective ways, but also the psychological and
mental effects of long-term use, and how ICT can evolve together with humans in
order to become a natural, ecologically valid part of the everyday human experience.
For example, the psychological effects of ICT can be grasped by considering the
effects of ubiquitous e-mail access and connectednessas suggested by popular
notions such as e-mail apnea,2 phantom vibrations or internet indispensability
(Stone 2011; Drouin et al. 2012; Platzer and Petrovic 2011); as well as the general
feeling one experiences when a device or network malfunctions (the feeling that
part of oneself is malfunctioning rather than an external technology) as described
in Pang (2013). It is also worth considering how the Internet is physically closer
to users than before: while a decade ago, shutting down the computer for the night
meant that access to the Internet was finished for the day, today nothing is easier
than checking our e-mail or the weather report one more time on any of the number
of mobile devices surrounding us.
Such long-term co-existence in turn raises awareness on the usefulness of
applications with long-term goals. Apart from enabling users to perform a specific
task in a specific domain, an equally viable goal might be to support, through
ICT, the development of a capability to perform an increasingly complex set of
tasks (this is the case, for example, in speechability and mathability, as will be
described in Sects. 5.2 and 6.6). It should be noted that long-term interactions are
also capable of generating functionalities that are not planned for a priori (see
also Sect. 2.3.3). For example, as users continuously interact with, and increasingly
2
A temporary absence or suspension of breathing, or shallow breathing, while doing email
(Stone 2011).
18
become entangled with systems of ICT components, new use case requirements,
and new possibilities for previously unknown functionalities are discovered. The
accommodation of such functionalities, in turn, can create new patterns of usage
many of which may then have a recurrent effect on usage patterns, and potentially
even on the social organization of the community of users involved in the merging
process. This mutual influence between usage patterns and use-case requirements
is creating an open-ended evolution of functionality. From a different perspective,
long-term co-evolution between humans and ICT is also enabling the offline
collection of vast amounts of data which can later be instrumental in developing new
applications (for example, if those applications rely on machine learning techniques
requiring large amounts of data). As a result, the creation of new functionalities is
facilitated in more than one way through the implicit knowledge that is generated
from past interactions. Based on the above, we introduce the term tangleface to
be used instead of the word interface when characterizing ubiquitous human-ICT
relationships. Additionally, we use the term tangleaction instead of interaction
when such relationships persist through extended periods of time. These notions
will be used often and elaborated from a variety of perspectives in later parts of this
book.
2.3.2 Levels of Cognitive Capability

From a CogInfoCom perspective, any kind of hardware or software component that
actively collects/stores/understands/communicates data can be seen as a component
with a set of cognitive capabilities. Whenever users become entangled with a system
of such capabilities, the border between the natural and artificial gradually becomes
vague. In other words, it is often the case that there is no longer any objective border
between purely human and purely artificial cognitive capabilities. For example, in a
scenario where a user controls an industrial robot with one hand using knowledge
obtained from a smartphone held in her other hand, the question immediately arises:
should this interaction be characterized from the perspective of communication
between three different entities, or is there benefit in viewing the user and the
smartphone as one entity that is communicating with the robot? The answer to this
question is important, if only for the fact that both the robot and the supporting
smartphone application might be designed differently if it is known in advance
that they will used together in this specific scenario, or if the cognitive effects
that the smartphone application will have on the usersuch as limited dexterity
and attention, increased capabilities for information access, etc.are known in
advance. To consider two other examples, the boundary between artificial and
human capabilities would be equally blurred in a scenario where a users lower arm
is augmented through a robot arm that is capable of downloading new skills from
an ICT network; or in a scenario where a pair of augmented glasses, or an augmented
helmet is used to provide an industrial operator with real-time information feeds
complementing the task at hand (such technologies are already present in industry,
and are on the verge of commercial breakthrough).
19
The bottom line is not that one would be philosophically inclined to specify a
boundary between entities, but that it is also necessary to specify such boundaries
from the functional perspective of engineering design. On the one hand, in a domain
where difficult problems of synthesis can be effectively tackled only by breaking
them down into smaller components and gluing those components together through
some form of communication once they are complete, the functional boundaries
at which this is done can make or break the tractability and sustainability of an
implementation. On the other hand, once it is accepted that the boundaries between
artificial and natural are not as clean as they were a few decades ago, unprecedented
possibilities emerge for the development of new functionalitieseven cognitive
capabilities. Such capabilities can be seen as implemented in the dependencies
between components in much the same way as lower-level functionalities are
created as a result of several different components working appropriately in
mutually specified contexts. This hierarchical dependence among capabilities can
be seen as leading to a hierarchical organization of cognitive capabilities.
The embodied perspective of cognition that is currently favored in the cognitive
sciences adopts the view that human cognitive capabilities and human intelligence
are emergent properties which cannot be separated from the physical, biological,
neurophysiological and higher-level bases of our mental existence (Deacon 2013).
It is also clear that the social and technological context of our interactions with other
humans and ICT further influences the kinds of mental and physical work that we
are able to perform (Hollan et al. 2000; Deacon 2013). Further, analogous emergent
properties can be identified in the functionalities of ICT devices and networks.
Although the view that computers are merely symbol processing systems has been
implicitly accepted for decades by thinkers and technologists of all backgrounds, an
emergentist view of computing is now gaining acceptance. Though fundamentally
different from living systems it can be argued that all computational systems
(apart from purely theoretical constructs such as the Turing Machine) have some
form of embodiment, and that furthermore, computation in general has physical
underpinnings and physical ramifications (Heder 2014). As highlighted by several
authors, even lexical knowledge such as knowing the derivative of the sine function
or knowing the capital of a country is strongly embodied in the sense that without
direct experience in working and manipulating functions, or without being able to
travel and experience through our bodies what really constitutes a city, our notions
would be entirely different (Picard 2003a; Heder 2014).
The extension of such notions to human-ICT entanglement and emergent
cognitive capabilities can be seen as a natural development. It can be argued that
not only are new functionalities and cognitive capabilities formed through humanICT entanglement, but that they can also be seen as higher-order in the sense that
they are dependent on lower-level foundations. However, this point of view also
makes it possible for such higher-order capabilities to be combined into newer
ones that are located at still higher levels of hierarchy. This process is illustrated
in Fig. 2.1. From a practical point of view, the figure shows that all cognitive
capabilities, however trivial in a human-ICT context, can be analyzed in terms of
other capabilities on which they depend. This observation can serve as an important
20
human sensory
modalities
artificial sensory
modalities
newer use cases

more data accumulated
newly available tanglefaces
emergent cognitive entities
time
Fig. 2.1 New, higher-level artificial capabilities and emergent cognitive entities are created
through time as new use cases are generated through a broadening of artificial sensory modalities
and the increasing possibility for users to co-evolve with them through extended periods of time
starting point in the design of CogInfoCom systems, especially when combined

with the goal of providing functionality rather than the ability to utilize operational
procedures (see also Sect. 2.3.4). In this book, the term cognitive entity will be used
when describing any synergic combination of humans, devices, infrastructure and
environment that is identifiable from the perspective of some (high-level) cognitive
capability. Whenever a cognitive capability is considered at a lower, less complex
leveland pertains directly to a human or an artificial device, the capability is said
to be embodied by a cognitive being, or a cognitive thing, respectively.3 Based
on this terminology, cognitive entities are formed when information on various
relationships between cognitive beings, cognitive things and their environment are
stored, interpreted and acted upon in identifiable ways. As a result, a new perspective
automatically emerges based on which cognitive entities can be analyzed and
understood in terms of holistic cognitive capabilities without being broken up into
constituent components. At the same time, the emergence of a new generation of
users can also be prognosticated, which we refer to as the generation of cognitive
entities, or generation CE by analogy with the well-known terminology of the
generations X, Y and Z. It can be argued that members of generation CE, growing
up starting from around the year 2010, are unique in the sense that the maturation of
their personality and social life unfolds in an environment that is inseparable from
cognitive ICT. The extended cognitive capabilities that ICT provides are merged into
The analogy with the Internet of Things is clear, when IoT is regarded as a cyberization of the
physical world together with humans, as will be detailed Chaps. 3 and 4.
21
the daily experience of this generation, and become irrevocably intertwined with its
expectations and thinking processes, both at a conscious and subconscious level.
Expressed more directly, one can say that ICT cannot be taken away from this
generation without bringing about significant cognitive effects and psychological
discomfort.
2.3.3 An Emergent Concept of Information

It is important to clarify that the notions of merging, entanglement and levels of
cognitive capability also suggest a unique concept of information that is implicitly
present in CogInfoCom. Importantly, rather than being treated as a commodity that
can be transmitted from one end of a communication line to another, information is
seen as an artifacta functionally relevant physical by-productthat emerges from
embodied patterns of interaction and communication.
In this sense, the design of a CogInfoCom system or application involves not
only the identification of various data and information types that are to be sent to
various components at certain points in time, but also a broader consideration of
how nuanced differences in behavior can lead to functionally relevant by-products,
how these by-products cause the communication between cognitive entities to
evolve through time, and how this evolution can be directed towards further,
novel functionality in flexibly re-usable ways. Thus, a common pasti.e. one that
is accumulated in goal-independent ways through a progressive human-ICT coevolutioncan be expected to eventually yield increasingly rich, functional models
of how humans are capable of communicating and operating in ICT settings.
While it may be the case that in an information system, a solution to any of
these problems will eventually require suitable data structures for representation and
manipulation at a more atomic level, the CogInfoCom perspective nevertheless
suggests that the starting point of design should be the observation and detection of
emergent possibilities for novel interpretation leading to novel functionality.
2.3.4 Transitions from Operation to Functionality

As described earlier in Sect. 2.3.1, new human-ICT capabilities are formed based
on new kinds of co-dependence relationships among and between humans and ICT.
It can also be observed that the details behind these capabilities (i.e., how they
are implemented) become gradually less important through timeat least from the
22
point of view of the user.4;5 For instance, a driver using Google Maps or Waze for
direction information does not need to know whether and how many other drivers
are queried for traffic congestion information before the system recommends a
specific route. Similarly, to a user interacting with several social networks at the
same time, the route taken by a link before it is shared with the user is unimportant;
only its original source is important besides that fact that it eventually reaches
the user. Many of the technological services we use on a daily basis we do so
unconsciously, while taking for granted that they will be available through the
flexible coordination of available devices and software components.
In this way, the emergence of CogInfoCom is implicitly based upon, and
also further supports a transition from operation to high-level functionality. This
transition is characterized by a decreasing need for explicit specification of use cases
and operational details, and an increasing prevalence of high-level and adaptive,
transparent functionalities that are seen and treated as tools rather than procedures
of operation.
An important characteristic of emergent phenomena is that they cannot be analyzed in a

reductionist manner, by separating them into the parts from which they are constituted (Deacon
2013).
Of course, from the perspective of an application designer, implementation details will always be
important.
Chapter 3
ICT-Based Foundations of CogInfoCom
This chapter discusses the foundations of CogInfoCom from the historical perspective of the diffusion of information and communications technology (ICT)a
phenomenon resulting from the evolution of digital electronics and the convergence of telecommunications and information technology. A holistic overview of
this digital convergence process is provided, with special focus on changes in
technological background, changes in value chains, social-technological phases (in
particular, the cognitive phase) as well as newly emerging applications. Through an
understanding of this convergence process, the position of CogInfoCom within the
Digital Ecosystem is clarified.
3.1 The Digital Convergence

During the last two decades, considerable progress was made in breaking down
barriers among certain scientific disciplines. The integration of voice (telephony),
data and video communications based on digital communication technologies has
led to the emergence of electronic communications. The convergence of communication, information and media technologies through a common digital technology
has resulted in the birth of the first real convergent sector, referred to as the sector
of infocommunications (infocom), ICT or TIM (telecommunications, information
and media technologies). The uniformization of those digital technologies upon
which these sectors are based, in turn, has not only enabled an increase in effectiveness of economies of scale and in efficiency and complexity of products/services,
but has also provided additional opportunities for the combination of functions.
Today, this convergence process is extended still further, through the synergic
combination of infocommunications and the cognitive sciences, leading to the

DOI 10.1007/978-3-319-19608-4_3
23
24
3 ICT-Based Foundations of CogInfoCom
emergence of CogInfoCom.1 In general, convergence is more than the sum of the

component sciences; it brings surplus advantages, generates synergic impacts, and
gives rise to new approaches, platforms, products, services and applications. These
convergence phenomena are relevant to the realization of the information society,
because they are not restricted to the level of technology, but rather influence
increasingly wider fields and are thus becoming a social phenomenon. As a case
in point, when it comes to the deployment of a set of synergies, convergence
necessitates a reconsideration of earlier cooperation forms between companies,
as well as a reconsideration of market structures; further, the harmonization of
regulations governing converging areas also becomes an important issue.
The convergence process is triggered by the development of digital electronics
on a huge scale, and hence it is generally referred to as digital convergence.
Specific costs of the fundamental functions in digital integrated circuits, such as the
transmission, storage and processing of information have been continuously halved
every 18 : : : 24 months for 40 years. The evolution of digital technology has had a
pervasive impact on telecommunications, information and media technologies and
resulted in a convergent sector, which is a relevant part of the future information
(networked knowledge) society. An integrated information (TIM) value chain, as
the intelligent, digital infrastructure of information society has been shaped. This
convergence process on the infrastructure level was supported by the rapid diffusion
of the Internet concept. Though the Internet itself has existed since 1969, it was with
the introduction of the World Wide Web in 1991 that the Internet became an easily
accessible network. Since then, the Internet has grown into a huge global system
of interconnected computer networks that use the standard Internet protocol suite
(TCP/IPv4). Internet technology radically reshaped telecommunications, integrated
information processing and content management, and they are transformed into
relevant strategic components of a Digital/Internet Ecosystem and an open Digital/Internet Age. For integrating customers and smart objects into the ecosystem
new approaches as Cognitive Infocommunications and extensions as Internet of
Things were generated and the reconsideration of Internet has been required leading
to Future Internet concepts.
The first papers dedicated to the comprehensive analysis of the convergence of
voice, data and video communications were published in the mid-1990s (OECD
1992; Freeman 1994; Saito 1994; Telpolicy 1994; Melody 1997). The Green
paper on the convergence of the telecommunications, information and media
sectors was issued by the European Commission (EC 1997). The International
Telecommunication Union identified the convergence as a main trend in 1999 (ITU
1999). The key issue in these and latter documents is the investigation of the
impact of the technological trend on the market structure and the regulation of the
telephone-based telecommunication, datacommunication and broadcasting sectors;
the main challenge is how to capitalize on the upgraded electronic communication
The next step of the convergence is also outlined by the combination of infocommunications,
cognitive science, quantum mechanics, nanotechnology and biotechnology
3.2 From Telecommunications to Cognitive Infocommunications
25
networks and services, and to find the best regulatory regime to deploy this process
(Henten et al. 2003; Melody 1997). Electronic communications provide the bearing
digital infrastructure for the digitalized content services and applications, whereby
the convergence process has been naturally expanded to all information processing
and content management functions; and the integration of telecommunications with
information technology and electronic media has been realized, the infocommunication sector was born (European Commission 1997, 2010; WEF 2007; Sallai 2007;
Liu 2013). Content involved voice, data and visual information, integration of media
streams and explosion of multimedia are to be highlighted. However content space
cannot be limited to these media contents. All other senses, including touching,
smelling, as well as emotions, sensory information, three-dimensional gestures, and
any other cognitive information can be also processed, transmitted and displayed.
Recently the content space is expanding by cognitive and sensory contents, billions
of smart devices are to be interconnected (CogInfoCom 2010, 2013; Nishinaga
2010; Smith 2012; Vermesan and Friess 2013; European Commission 2013a).
Cognitive infocommunications by definition aims to manage the expanded content
space, and to provide opportunities for creating enhanced, richer applications
(Baranyi and Csapo 2010, 2012). At the same time the identification capacity
of the todays Internet is running out, Internet architectures are reconsidered for
better managing scalability, mobility and quality requirements, application creation,
for integrating cognitive capabilities and security issues, for handling billions of
connected devices (Internet of Things) and big volume of data generated (Big Data)
as well as for exploiting the opportunities derived from the intensive technological
development. This convergence process transforms business and bank spheres,
administration, production, agriculture, transport, health, education and knowledge
systems, etc., our everyday life. The future networked knowledge society is going
to be established on Internet base, but the limitations of the todays Internet must be
eliminated. Recognizing the challenges of the todays Internet and the opportunities
for a more advanced Internet, European Commission supported intensively the
research activity on the Future Internet (FI) and initiated the organisation of the
Future Internet Assemblies (FIAs). Since 2008 eleven FIAs were held and four
books were issued on the research results (Tselentis et al. 2010; Domingue et al.
2011; Alvarez et al. 2012; Galis and Gavras 2013).
3.2 From Telecommunications to Cognitive

Infocommunications
Digital technology has radically reshaped telecommunications and led to the
integration with more and more functional areas and the birth of cognitive infocommunications. We identify four main overlapping phases of the evolution of
the digital convergence: (1) the digitization of the separate communication sectors,
(2) the digital convergence of the different communication forms, (3) the digital
26
Fig. 3.1 Phases of digital convergence and penetration of Internet
convergence of communications, information technology and content management,

and (4) the expansion of the managed content space toward cognitive contents and
beyond (Fig. 3.1).
The convergence of technologies provides technical opportunities; the real
exploitation of the digital convergence depends on the business benefits, on
the added value and the cost-effectiveness of the integrated solutions. Therefore
the history and the steps of the convergence process are well characterised by the
evolution of the value chains. The information value chains represent the consequent
value-generating functions from information sources to the usage at the destination.
To emphasize the evolutionary phases, we use a simple value chain model only, and
we find that the information value chain in each phase is radically different; the
separated vertical value chains are merged and transformed into a single value chain
with more and more horizontal layers (Henten et al. 2003; Sallai 2007, 2013c).
3.2.1 Digitization of Separate Communication Sectors

Traditionally, the various contents, such as voice, data and text, and audio-visual
(AV) programs, have been associated with separated networks, services and user
terminals, and their markets, value chains and regulation have been separately
managed. Voice has been managed by telephony (voice communications), data
and text by data communications, audio-visual programs by radio and television
broadcasting and distribution (AV communications) (Fig. 3.2). These separated
27
Fig. 3.2 Phase1: separated value chains
sectors had their own specific technology; the penetration of digital technology
has been started by implementing the sector-specific communication functions. The
sector-specific technologies are more and more characterized by the intensive use of
the digital technology and the whole separate network, services and terminals have
been reintegrated on digital base (separated internal digitization). The digitization of
telephony started with the use of digital transmission, followed by the introduction
of digital control and switching, which led to an integrated digital telephone network
(OECD 1992; Freeman 1994; Telpolicy 1994; Melody 1997; ITU 1999). Modern
mobile telephony is already fully based on digital technology. Fixed and mobile
telephony convergence (FMC) can be considered a specific case in the frame of the
voices value chain. Digital solutions have also penetrated into AV communications.
3.2.2 Unification of Telecommunications: Electronic

Communications
Any kind of digitized information content can be transmitted in the same way
through various digital networks and therefore the integrated realization of these
networks is reasonable (Saito 1994; European Commission 1997; ITU 1999). The
merger of the separated value chains has begun. Computer networks became
capable of voice communications, using the PC as a voice terminal (VoIP).
Downloading media streams to PC-s, they can operate as audiovisual terminals. The
SMS type data transmission has been developed for mobile phones. A broadband
network as an integrated network is equally able to transmit voice, data, text,
audio-visual programs etc. Interconnecting disparate networks and shaping an
28
Fig. 3.3 Phase2: value chain of electronic communications with horizontal convergences
integrated network, the Internet protocol suite (TCP/IPv4) was proved to be most
efficient connectivity technology. Additionally, the combination of voice, data and
audio-visual services offered new IP-based multimedia service opportunities. At
the users, various integrative terminals appeared. The value chains of the voice,
data and AV communications have been merged; horizontal convergence and some
integration of the services, networks and terminals can be identified, and instead of
the separated vertical value chains a single value chain with three horizontal layers
can be shaped (Fig. 3.3) (Henten et al. 2003; Sallai 2007). A unified telecommunication sector has emerged, which is formally called electronic communications.
In deploying these horizontal convergences, uniform regulation was introduced for
electronic communications in the European Union (Ryan et al. 2003).
3.2.3 Expansion of Telecommunications: Infocommunications

Telecommunications combined with some information processing and content
handling functions on digital technology base are called infocommunications,
or in short form, infocom(s) or infocomm(s). The term first emerged in the
beginning of eighties at scientific conferences and then was gradually adopted in
the 1990s by the players of telecommunications sector, including manufacturers,
service providers, regulatory authorities and international organizations to clearly
express their participation in the convergence process. It was regularly used by the
International Telecommunication Union (ITU 1999). Electronic communications
provide the bearing digital infrastructure for the digitized content services and
applications, whereby the digital convergence process has been naturally expanded
29
to all information and media technology functions. The same digital message
form is used in the computer sector for passing messages within and between
the computers, together with the growing operational use of computers within
telecommunications, resulting in a synthesis between the telecommunications sector
and the computer-based information sector (info-telecom/info-com convergence).
The IP-based connectivity technology provided a platform on which higher functions can be built. The IP-based solutions in both sectors generated an integrated
structure for processing, storing, accessing and distributing information (ITU 1999;
Valtonen 2001). The electronic media and content production and management
have also been involved into the convergence process (media convergence) by
using the opportunities delivered by IP-based telecommunications and information
technology (ICT), which is demonstrated by the integration of media streams
(multimedia) and the explosion of social media, and the immense spread of the
Internet provided by telecommunications and Internet service providers (ISPs)
(Domingue et al. 2011). Electronic content services and applications based on web
technologies and delivered by electronic communication networks and services
have emerged, e.g. e-business, e-commerce, e-government, e-health, e-learning,
on-line administrations, voting, population census or games, etc. Sophisticated
Internet applications have also been conceived and developed for managing a smart
home or office, an intelligent transportation or energy system, or intelligent digital
cities (smart cities). In general they can be called e-content or infocommunication
applications, the terms information societys services, content services and eservices are also used (European Commission 1997, 2010; Henten et al. 2003; Sallai
2007; Liu 2013). The third step of digital convergence process is the key step of the
process; it is specially called infocommunications convergence. The value chain of
infocommunications contains three additional layers (Fig. 3.4) (Sallai 2012a,b):
the layer of content space, symbolising the jointly-managed information sources
and the customers payable demands;
the layer of e-content or infocommunication applications, including from simple
content services to the wide variety of secure and multi-content Internet services;
the layer of the common IT infrastructure for applications, the middleware
layer, including common message handling, content management functions
(e.g. directory assistance, editing, indexing), browsers, portals, search engines,
security systems etc.
The layers of IT infrastructure, e-communication services and networks together
can be considered the infocommunication infrastructure. Similar layer models of
infocommunications embracing the functions provided by the Internet technology
have been shown and discussed to allocate the players of infocommunications to
layers (Fransman 2002; Krafft 2003, 2010).
Today, the term infocommunications as expanded telecommunications in
the above mentioned meaning has generally been used by telecommunications
30
Content space
Websites, AV programs, telephone calls, text messages
E-content / infocom applications
Information society service: e-commerce, e-government, ...
E-content IT infrastructure
Content management and security systems.
Fig. 3.4 Phase3: value chain of infocommunications
manufacturers, service providers and regulatory authorities, in scientific papers and

university curriculums, and in the name of scientific and professional conferences
and journals (e.g. IEEE Infocom, Infocommunications Journal). It is important
to emphasis, that in the Phase 3 the telecommunications are expanded to the
infocommunications through an infocommunication convergence process and an
infocommunication sector is arisen, but at the same time the three convergent
sectors are really merged (TIM convergence) and a unified TIM sector is born
(a comparison of the terms of infocom and others is explained in Sect. 3.3).
In the past decade, researchers have begun to address the fact that the complex,
heterogeneous and dynamic nature of modern infocommunications networks raises
the need for a set of capabilities for self-adaptivity, in order to ensure that networklevel objectives are fulfilled and maintained. Examples of such objectives can
include e.g. controllable quality of service, differentiated access control, resilience
to congestion-related bottlenecks, etc. Research focused in this direction has led to
the emergence of the cognitive networka network concept in which individual
components are capable of observing network conditions, and of planning, deciding
and acting based on those conditions in order to achieve certain end-to-end goals
of a data flow (Thomas et al. 2006; Fortuna and Mohorcic 2009). The cognitive
network concept is rapidly gaining acceptance, to the extent that in many cases no
explicit qualification is necessary: capabilities for self-awareness and self-adaptivity
are becoming a defining feature of infocommunications, and the prefix cognitive
can be dropped.
31
3.2.4 Expanding Content Space: Cognitive

Infocommunications
TIM convergence transforms the business, government and civil spheres, the
healthcare, transport, education, knowledge and other systems, and it has societal
implications. The TIM convergence pervades our everyday life, the convergent
TIM technology is a relevant pillar of the future networked knowledge society.
Nevertheless, traditionally the TIM convergence is related to the traditional types of
content, such as telephone calls, text messages, AV programs, as well as web sites,
the sensory information managed has been limited to sight and hearing. However
the content space can be expanded to all senses, including touching, smelling or
any other modality, in general the human emotions and feeling, and gestures in
3D space. Technological innovation is increasingly supported by research results
on user attitudes, and human-oriented solutions can be developed by taking them
into account. The multitude of multiple sensors can be used to detect the state of
people, devices, narrower or wider environment, or describe the dynamics of motion
or measured parameters, and there is real demand to connect them to the Internet,
leading to the concept of Internet of Things. Analysing the collected huge data sets
valuable cognition and knowledge can be derived (Big Data analytics).
Cognitive infocommunications (CogInfoCom) expands the content space with
cognitive and sensory contents, and merges cognitive sciences and infocommunications. The goal of the convergent cognitive applications is to extend the
capabilities of the human brain through infocommunication devices, irrespective
of geographical distance, including any combination of artificial and biological
cognitive systems. The sensory information obtained or experienced can not only
be transferred to the destination, but also transformed to an appropriate sensory
modality in a way that the user can process it effectively (CogInfoCom 2010, 2011;
Baranyi and Csapo 2010, 2012). A simple example is the reverse radar, which
transforms visual sensory information to hearing to help the reversing driver to
sense obstacles with an accelerating beeping sound. Thereby in the value chain of
CogInfoCom, the content layer is expanded and the applications layer involves the
bridging of the sensory information to an appropriate, manageable one, if necessary
(Fig. 3.5) (Sallai 2012a).
Three Dimensional (3D) Internet provides widespread opportunities for CogInfoCom by radically expanding the environment of communications (Alpcan et al.
2007; Daras and Alvarez 2009). 3D is natural for people, and 3D communications
can embrace our cognitive systems, including not only hearing and vision, but
gestures, touch, smell etc. Users intensively use smart phones, laptops and tablet
screens; thus some gestures detected by simple or sophisticated touches within
the display area are involved and the navigation practice is significantly changing.
Games are already excellent application area to test different cognitive input solutions. Virtual walking and navigation can be done in a fine-arts museum or a shop,
e.g. a drug-store, and goods, such as drugs in box can be seen and virtually smelled
and moved. The 3D visualization techniques may promote multi-dimensional
32
Fig. 3.5 Phase 4: expanding the content space: cognitive infocommunications
evaluation and comparison of different goods or shops. Efficient warnings can be

conceived for critical situations by combining visual and voice signals and vibration
feedback, which are already widespread in vehicles and gaming (Prekopcsak et al.
2008; Papadimitratos et al. 2009; Csapo and Baranyi 2010; Smith 2012; Vermesan
and Friess 2013). Virtual collaborative platform can be build, where 3D content
can be manipulated, users at different remote locations collaboratively interact
with real-time processes in a 3D context, while the participating hardware and
software devices can be spatially and logically distributed and connected via IP
network (Galambos et al. 2012a). In general, by enhancing the intelligence of things,
in particularly their artificial cognitive components, sensory capabilities, several
proper CogInfoCom systems are created (Castrucci et al. 2011; Minutolo et al. 2012;
CogInfoCom 2012, 2013).
The concept of CogInfoCom will further enhance the business and societal
implications of the digital convergence process and prepare the further expansion
of the content space.
3.3 ICT, TIM, DES and CogInfoCom

Different terms have been formed for the same entity from different aspects;
the same term is often used in many different contexts. Some of the related
terminologies are clarified in this section, and the relative position of the different
terms is shown by an additive colour mixing scheme to build up a consistent
terminology.
33
Infocommunications (Infocom) is considered in the Sect. 3.2 the natural expansion of telecommunications with information processing and content handling
functions including all types of electronic communications (fixed and mobile
telephony, datacommunications, videocommunications, broadcasting, etc.) on a
digital technology base, mainly through Internet technology.2
Information and Communications Technology, usually abbreviated as ICT,
has been in use from the second half of the 1990s (Stevenson 1997) and is used as an
extended synonym for information technology (IT) to emphasize the role of unified
(tele)communications, the integration of telecommunications with computers, as
along with the necessary software, middleware, storage and audio-visual systems
that enable users to create, access, store, transmit and manipulate information. In
other words, ICT consists of IT as well as telecommunications, broadcast media and
all types of audio and video processing and transmission (EITO 2014; ITU 2009).
At present, the term ICT is generally used and usually refers to the integration of
information and telecommunication technology sectors involving their convergence
with the media technology sector based on common digital technology. The term
Information Society Technologies (IST) was generally used to the integration of
telecommunications, IT and media sectors in EU research, technological development and demonstration framework programmes (FP5 and FP6) between 1998
and 2007 (Arend 2002). However, since 2007 in FP7 and Horizon 2020 the term
ICT has been used. The EU Horizon 2020 programme (20142020) preferably
supports the ICT research and innovation, in particularly the development of
(European Commission 2013a):
next generation computing, advanced computing systems and technologies;
infrastructures, technologies and services for the Future Internet;
content technologies and information management, including ICT for digital
content and creativity;
advanced interfaces, robotics and smart spaces;
nano-electronics and photonics technologies and embedded systems.
The member countries of the Organisation for Economic Cooperation and
Development (OECD) agreed in 1998 to define the ICT sector as a combination
of manufacturing and services industries that capture, transmit and display data
and information electronically. In 2011, the OECD set a number of standards for
measuring and comparing the information society across countries (OECD 2011),
including definitions of ICT as well as Media and Content products (goods and
services). Accordingly, ICT products must primarily be intended to fulfill or enable
the function of information processing and communication by electronic means,
including transmission and display. Content published in mass communication
The terms info-com(s), info-communications (with a hyphen) are used to express the integration
of the IT and (tele)communication sectors (Krafft 2010; Valtonen 2001).
34
media such as printed, audio-visual and online contents and related services are not
considered as ICT products, but are referred to as the Media and Content sector.3
Over the past several years, the abbreviations TIM, as the Telecom IT
Media sector or TIME, as the Telecom IT/Internet Media and Electronics/Entertainment/Edutainment sector, are frequently used to express the full
integration of these sectors and to enhance the significance of content respectively.
The integrated sector was aptly called IST (Information Society Technology) (Arend
2002), and Digital Technology sector seems to be also appropriate. Generally we
use the term TIM, where M denotes Media and Content sector in wide sense,
including cognitive and sensory contents, too.
The alternative terms Digital Ecosystem (DES), Internet Ecosystem, Digital/Internet Age and Digital World have also emerged to embrace all those sectors
that are already or on the verge of being based on digital technologies (WEF 2007;
IDATE 2009). However these terms generally have a broader meaning, the terms
involve the participants, the customers too. The Digital Ecosystem is defined by the
World Economic Forum as the space formed by the convergence of the Telecommunications, IT and Media and Content sectors, and consists of users, companies,
government and civil society, in addition to the infrastructure that enables digital
interactions (WEF 2007). The evolution of digital technology reached and integrates
the customers by their terminals, devices, sensors, comprises cognitive and sensory
content and the applications may involve the active collaboration and context of
customers (human-centric community applications). Smart phones are part of the
personal life; one device with more and more sensory capability is for private and
business. If it is so, the term Digital Ecosystem organically incorporates cognitive
infocommunications, Internet of Things, Big Data, 3D Internet and community
applications.
On the usage of the term ICT, some additional remarks are in order here: (1) Originally,
only information and communications technology (with communications in the plural) was
considered correct since ICT refers to communications (in the sense of a technology of sending
and receiving information), not communication (the act of sending or receiving information by
speaking, writing, phoning, emailing, etc.). Nevertheless, recently, the single form information
and communication technology is becoming increasingly common, and it is also used by the
International Telecommunication Union (ITU 2009). In order to express these dual forms, we use
communication(s) (Giles 2009); (2) The International Telecommunication Union (ITU) and the
European Telecommunications Standards Institute (ETSI)according to their own definition
deal with not only telecommunications, but also ICT issues; however, their relevant activity, their
recommendations and standards resp. are focused on network-centric issues (ETSI 2011; ITU
2009). Some characteristic citations are from their web-sites: ITU is committed to connecting all
the worlds people. : : : We allocate global radio spectrum and satellite orbits, develop the technical
standards that ensure networks and technologies seamlessly interconnect, and strive to improve
access to ICTs to underserved communities worldwide. ETSI produces globally-applicable standards for ICT, including fixed, mobile, radio, converged, broadcast and internet technologies. : : :
ETSI is the recognized regional standards body dealing with telecommunications, broadcasting and
other electronic communications networks and services; (3) The term infocommunications is also
used in politics in a wider sense as a shorter form of information and communications technology
or simply to interpret the abbreviation ICT.
35
Fig. 3.6 Digital convergence

prism: positioning
Infocommunications
Fig. 3.7 Digital convergence

prism: positioning cognitive
infocommunications
The relationship and position of the terms is presented by a digital convergence

prism (Fig. 3.6) (Sallai 2012b, 2013c), which shows the three components (T, I, M)
and their pairs and the triple combination (convergent TIM triplet) according to
the rule of additive colour mixing. Assuming that telecommunications (Telecom) is
blue, informatics (IT) is green and Media and Content is red, then teleinformatics
or telematics or info-coms is cyan, telemedia or networked media or media
communications is magenta, media informatics is yellow, and the convergent TIM
is white. In such a way, the integrated TIM sector corresponds to the prism as a
whole, the ICT sector to the whole minus the red area (Media and Content), and the
infocommunications sector relates to Telecom and neighbouring three areas (blue,
36
cyan, magenta and white). That means that, for example, media informatics is a part
of ICT but not part of infocommunications.
Figure 3.7 shows the transformed prism representing the convergence with
cognitive sciences, and the relationship of the cognitive infocommunications, cognitive communications, cognitive informatics and cognitive media. In this historical
perspective cognitive infocommunications can be considered as an expansion of
cognitive communications.
Part II
Research Directions Under CogInfoCom
This part provides an overview of those research areas that are either in a
synergic relationship with CogInfoCom, or have emerged as a result of the unique
perspectives and research efforts represented by the field. A wide range of research
areas are treated here, and it is important to bear in mind that all of them are multifaceted and constitute individual research domains in their own right. At the same
time, all of them incorporate at least some aspect that makes them relevant to the
use or support of cognitive capabilities in infocommunications. For this reason, a
discussion on how they may contribute to CogInfoCom is well justified.
In Chap. 4, several key points of synergy are discussed from the perspective
of existing research fields relevant to the merging process between humans and
ICT. In particular, relationships with the fields of affective computing, augmented
cognition, body area networks, brain-computer interfaces, cognitive informatics,
cognitive networks and Future Internet, HCI and multimodal interaction, sensory
substitution, social signal processing, virtual and augmented avatars and virtual
and augmented reality are considered. It is important to emphasize that while
all of these fields have their own motivations and unique set of methodologies,
primarily those aspects are highlighted which represent opportunities for synergy
with CogInfoCom.
In Chap. 5, an overview is provided of three key research areas which have
emerged through the influence of CogInfoCom. In particular, the areas of CogInfoCom channels, speechability and socio-cognitive ICT are considered. Research
efforts and results relevant to these areas are presented from a perspective that
focuses on a set of generic techniques referred to as iconic, context-based and
pattern-based. The purpose of this discussion style is to highlight possibilities for
the unified treatment of challenges relevant to the entanglement between humans
and ICT. It should be noted that a detailed treatment of CogInfoCom channels is
provided later in Part III of the book.
Finally, in Chap. 6, several new initiatives are described which have been
proposed at the various scientific fora on CogInfoCom to guide future research in the
field. Specifically, areas relevant to augmented virtual reality mediated neuroscience
research, ergonomics of augmented virtual collaboration, ethologically informed
38
II Research Directions Under CogInfoCom
CogInfoCom (EtoCom), CogInfoCom-aided industrial engineering and augmented

mathematical capabilities (mathability) are addressed. Although the initiatives
discussed here represent future directions in much the same way as the fields
detailed in the previous chapter, they are nevertheless treated separately for the
reason that they are still relatively young, and continued research is expected to
lead to significant developments in their scope and goals. Regardless, the fact that
they were proposed and exist in their current form reflects well the interdisciplinary
outlook that is promoted by the CogInfoCom conference series.
Chapter 4
Synergies Between CogInfoCom

and Other Fields
In this chapter, several key points of synergy are discussed from the perspective
of existing research fields relevant to the merging process between humans and
ICT. It is important to emphasize that while all of these fields have their own
motivations and unique set of methodologies, they also incorporate some aspect,
or some future potential that makes them relevant to the use and support of
cognitive capabilities in infocommunications. In this chapter, we aim to focus
primarily on such aspects. However, partly due to the fact that the information
concept underlying CogInfoComas discussed earlier in Sect. 2.3.3focuses on
functionally relevant by-products of interaction rather than exclusively on the
transfer of explicit knowledge, it will not always be possible to draw a clear line
between what is relevant and what is not. Nevertheless, those aspects that are already
clearly relevant to CogInfoCom are presented in some detail. Modes of usage which
focus on long-term co-evolution rather than episodic interactions are of particular
interest.
A structural visualization of the relationships treated in this chapter is provided
in Fig. 4.1. As shown in the figure, all of these fields, including CogInfoCom
itself, are simultaneously characterized by both overlaps and unique distinguishing
features. For example, an important notion that sets apart the goals of CogInfoCom
from any of the research fields discussed in this chapter is that it aims to reach
an understanding of how extended periods of co-evolution can trigger novel
functionalities. This aspect of long-term co-evolution is rarely acknowledged, not to
mention planned for, in other fields. The concepts of tangleface and tangleaction
as analogies to the words interface and interactionwere introduced earlier
in Sect. 2.3.1 precisely for this reason: to clearly distinguish interfaces designed
for episodic/stateless interactions from tanglefaces designed for, or evolved through
extended periods of stateful entanglement. Nevertheless, it goes without saying that
a host of challenges, both in terms of analysis and design, can be better addressed if
past results from synergically related fields are considered.

DOI 10.1007/978-3-319-19608-4_4
39
40
4 Synergies Between CogInfoCom and Other Fields
Fig. 4.1 Relationships between fields with synergic contribution to CogInfoCom discussed in this
chapter. While the figure shows that CogInfoCom has common motivations with these fields,
and may also benefit from finding suitable adaptations of their respective methodologies, it also
highlights the fact that the borders between the synergically relevant fields themselves have begun
to disappear. For example, as discussed in the chapter, sensory substitution is increasingly regarded
as a form of sensorimotor extension, which in turn increases its relevance to augmented cognition;
at the same time, the motivations of augmented cognition are strongly salient when laid aside and
considered together with those of human-computer interaction; similarly, body area networks have
strong relevance to the concepts of Future Internet and Internet of Things, and so on. Today, it is
not uncommon for papers focusing primarily on one of these fields to be equally well suited to,
and most often accepted to conferences dealing with another: this fact alone supports the message
conveyed by the figure
4.1 Affective Computing

Affective computing is a research field proposed by R. Picard at MIT in the 1990s
that focuses on computing that relates to, arises from, or influences emotions
(Picard 1995, 1997, 2003a). While computation in general is often regarded as a
subject area that should ideally be devoid of emotion, mounting evidence from
various human-oriented sciences has brought about the realization that all high-level
cognitionincluding perception, reasoning and decision makingis intimately
linked with emotion. This view is supported not only through anatomical findings,
but also through investigations of how reasoning and decision making is affected in
subjects with physical lesions and/or emotional disorders.
4.2 Augmented Cognition
41
As a result, research on affective computing has focused both on introducing

aspects of emotionally influenced reasoning into computing, as well as on the
perception, understanding and emulation of human emotions. With respect to this
latter goal of emulation, Picard formulated four key components of emotion to be
taken into consideration (Picard 2003b):
emotional appearance: behavioral manifestations give the appearance that a
system has emotions;
multiple levels of emotion generation: depending on e.g. the specific roles of
emotions, or the availability of computational resources, different mechanisms
can be used to generate emotions;
emotional experience: a system is capable of attributing semantic labels to its own
emotions (and of experiencing subjective feeling/intuition about them, although
given limitations in our current understanding of the famous hard problem of
consciousness, the fulfillment of these latter criteria cannot be prognosticized);
mind-body interactions: signaling and regulatory mechanisms are provided by
emotions which create links between cognitive and other bodily activities.
Based on the above, the field of affective computing is multi-faceted and is
under continuous development. From a CogInfoCom perspective, certain aspects
of the field are more relevant than others. Specifically, results of the field can
become highly relevant when they are applied to the modulation of information
in infocommunications settings with the purpose of strengthening human-ICT
co-evolution from an emotional perspective. At the higher-level scale of collective social interactions, understanding, reasoning about and influencing, through
infocommunications, the emotions of a group of people in a city or a region
would be a possible CogInfoCom-oriented extension to the field. For an example
on new research directions motivated by such perspectives, we refer readers to
the discussion on socio-cognitive ICT in Sect. 5.3a newly established research
direction that goes well to show the relevance of CogInfoCom in social ICT
settings, and that cannot be trivially decomposed into its constituents motivations
and methodologies.
4.2 Augmented Cognition

Augmented cognition (AugCog) is a research field that was proposed as part
of a DARPA program by D. Schmorrow and his colleagues in the early 2000s
(St. John et al. 2004; Schmorrow et al. 2006; Stanney et al. 2009). AugCog aims to
transform human-computer interactions by making information systems adapt
to the changing capabilities and limitations of the user (St. John et al. 2004). The
first international conference on the field was held in 2005.
42
AugCog primarily addresses cognitive aspects such as attention, memory, cognitive biases and learning capabilities using cognitive state gauges based on
psychophysiological and neurophysiological measures derived from sources such as
EEG, pupil dilation, mouse pressure, heart rate and many others (St. John et al. 2004;
Stanney et al. 2009). By creating a closed loop system between the user and the
device, measurements on cognitive state can be directly incorporated into both shortterm control and long-term adaptation strategies, allowing for the compensation of
cognitive limitations (Fuchs et al. 2007; Hale et al. 2008).
A research area of growing importance within augmented cognition is that of
cognitive biases (Kahneman 2011; Gilovich et al. 2002). Some authors suggest
that the use of a corrective interface between incoming data and the user can
help eliminate cognitive biasesthat is, systematic errors in diverse aspects of
human thinking and decision makingwhile at the same time supporting new
ways of thinking about problems (Greitzer and Griffith 2006; Griffith and Greitzer
2007; Staal et al. 2008). As pointed out recently by Professor Peter Foldesi and
his colleagues, this can be achieved in several ways, e.g. by modifying input data,
modifying underlying processing and optimization algorithms, or transforming the
way in which output data are presented (Foldesi and Botzheim 2012; Danyadi et al.
2012). Such possibilities are well reflected in the area of human augmentation
detailed in the Gartner hype curves reproduced in Fig. 1.3.
Based on the above, AugCog can be seen as a research area that provides ways to
tighten the coupling between users and systems by improving sensing capabilities
of cognitive states and manipulating operation and feedback information in appropriate ways. The field shows strong parallels with human-computer interactionas
highlighted in its definition quoted abovebut it can also be seen as providing a
complementary perspective in the sense that its name speaks about the augmentation
of (human) cognition as opposed to the augmentation of artificial capabilities (i.e. to
render devices more suited to seamless interaction). When ideas and approaches
from AugCog are applied to the modulation of functionality and information flow in
infocommunication systems, the links between the field and CogInfoCom are clear.
Especially interesting applications are possible when results in AugCog are applied
to the sensing of cognitive states in scenarios with multiple participants and multiple
devices, as suggested, for example, in (Skinner et al. 2013). Such applications are
eventually expected to lead to new kinds of augmented sensors that are capable
of understanding distributed phenomena based on multimodal activities in ICT
networks (possible examples include the filtering of relevant data from vast records
of social network activity on e.g. Facebook or Twitter; or from sensor measurements
in spatial memory systems (Niitsuma and Hashimoto 2009; Niitsuma et al. 2007) or
other distributed physical environments).1
These aspects are strongly relevant to socio-cognitive ICT, a field that was inspired by CogInfoCom as described in Sect. 5.3.
4.3 Body Area Networks
43
4.3 Body Area Networks

Body area networks (BANs), or body sensor networks (BSNs) are specialized
applications in which a network of sensors either attached to or implanted in
the human body communicate physiological information to a central location for
further processing and analysis. The predecessors of BANs were referred to as
Personal Area Networks (PANs), which were first developed in the second half
of the 1990s as a network of electronic devices on or near the human body,
exchanging information via small currents passed through the body (Zimmerman
1996, 1999). The term body sensor network was coined in the twenty-first century
as the concept of PAN evolved towards wearable and implantable sensors used
for health monitoring related purposes. Today, BANs/BSNs are seen as involving
all kinds of monitoring of physical, physiological and biochemical parameters
without activity restriction and behavior modification for applications supporting
healthcare, sports and general wellbeing (Yang 2014).
The transition from wireless to body area networks brings with itself a variety of
challenges, including issues of security (both with respect to data theft/manipulation
and interference between neighboring BSNs), biocompatibility, power source
miniaturization, and context awareness (Hanson et al. 2009; Sayrafian-Pour et al.
2010; Yang and Sayrafian-Pour 2012; Yang 2014). Much work has been carried out
in the past two decades to address such challenges, as discussed in recent surveys
of the field (Yuce 2010; Chen et al. 2011; Latre et al. 2011; Ullah et al. 2012).
Although technologies relevant to BANs still focus primarily on healthcare
applications, their long-term use can be expected to reach a broader scope of
domains. In particular, from a CogInfoCom perspective, the long-term collection
of physiological data coupled with machine learning techniques can lead to a kind
of cyberization of the human body that in turn may be useful for the design of
increasingly contextually aware and physiologically-augmented infocommunication technologies. Such possibilities for cyberization once again highlight the fact
that the term cognitive is increasingly applicable to artificial besides biological
systems as the ICT network is increasingly characterized by contextually aware
components capable of sensing, representing and interpreting sensory patterns from
their environment. In the long run, it is conceivable that research on BANs may
inspire the creation of body cyberizations belonging to multiple users as a new
kind of multi-BAN architecture. Such applications would further contribute to
the emergence of collective cognitive entities, like those targeted by the field of
socio-cognitive ICT (for details on the emergence of that field, readers are referred
to Sect. 5.3).
44
4.4 Brain-Computer Interfaces

Brain Computer Interfaces (BCI), sometimes also referred to as Brain Machine
Interfaces (BMI), is a research field that deals with the creation of direct communication pathways between the brain and external devices. Research on BCI began in
the early 1970s at UCLA, and was funded by the U.S. government through DARPA
(Vidal 1973).
There are several points of similarity between BCI and body area networks in
that both focus on technologies that bypass explicit forms of human interaction
(i.e. specification of intent through physical movement) and instead use more direct
forms of communication. On the other hand, several key differences between the two
approaches can also be highlighted. Perhaps the most important difference is that
while the measurements obtained through BANs cannot be voluntarily controlled,
quite often the use of brain-computer interfaces relies on voluntary interactions from
the user, in terms of deciding what to think about, or where to look on a particular
display. This property has led to a broader set of terminologies involving distinctions
between active, reactive and passive forms of BCI (Zander and Kothe 2011), and
has also directed focus at a differentand wideningset of application areas.
While initially both invasive and non-invasive forms of BCI were used primarily for
rehabilitation (as described in e.g. Dobelle 2000; Hochberg and Donoghue 2006),
today BCI is seen as a new kind of modality that can be used for more general
applications, including recreation and gaming, education, robot (tele-)operation, as
well as planning and control phases (besides steering phases) of navigation (Nijholt
and Tan 2008; Tan and Nijholt 2010; Gurkok and Nijholt 2012).
From the perspective of CogInfoCom, BCI allows for synergies to be created
between natural and artificial cognitive systems at a direct, low level. Simply
expressed, BCI connects high-level brain activity to infocommunication networks,
allowing for increasingly rich information to flow from brainsand eventually,
heterogeneous cognitive entitiesinto the ICT network. The online use of BCI
can in itself provide important enhancements to a wide range of infocommunication
technologies. On the other hand, as it is applied to long-term measurements
allowing for the formulation of a more precise mapping between brain signals
and cognitive state, BCI will also make possible the development of applications
with increasing levels of contextual awareness. In particular, it is well known that
the success of machine learning approacheswhich serve as an important glue
behind CogInfoCom applicationsdepend to a large extent on the amount of data
available. In this sense, measurements through BCIeven when they are offline
in that they are not directed towards a specific application when collectedcan play
an important role in CogInfoCom by providing the large amounts of data necessary
for the creation of knowledge that is necessary for new cognitive capabilities. After
reaching a critical mass, the amount of data collected through BCI can be seen as
a new kind of knowledge instead of a set of individual measurements.
4.6 Cognitive Networks for Future Internet
45
4.5 Cognitive Informatics

Cognitive informatics (CI) emerged as a field of research in the early twentyfirst century (the first IEEE international conference in the field was organized by
Y. Wang in 2002) (Wang 2002; Wang and Kinsner 2006). Several definitions have
been proposed to specify the goals of CI. The definition on the webpage of Pacific
Northwest National Laboratory (which is a research institute funded by the U.S.
Dept. of Energy) states that CI is the multidisciplinary study of cognition and information sciences, which investigates human information processing mechanisms and
processes and their engineering applications in computing.2
One of the main purposes of CI is to investigate the internal information storing
and processing mechanisms in natural intelligent systems such as the human brain.
Much like CogInfoCom, CI also aims to create numerically tractable models which
are well grounded from an information theoretical point of view, and are applicable
to engineering systems. However, while the results of CI largely converge towards
and support the creation of artificially cognitive systems, the goal of CogInfoCom
is to enable these systems to communicate and co-evolve with each other and their
users efficiently. Hence, CogInfoCom builds on a large part of results in CI, as
the latter can provide support both in enhancing the communication space among
cognitive entities, as well as between various meta-levels of co-evolution directed
towards the creation of new cognitive entities.
4.6 Cognitive Networks for Future Internet

As discussed earlier in Sects. 3.2.3 and 3.2.4, modern ICT networks are characterized by an increasingly general capability for self-reflective analysis and adaptive
reconfiguration in order to fulfill network-level objectives. Hence, a growing
awareness of the term cognitive network can be observed in research and industryoriented sectors alike (Thomas et al. 2006; Fortuna and Mohorcic 2009). It is
important to note, however that the list of self-reflective capabilities available to
modern networks will not end here: through the entanglement between humans and
ICT, future networks are expected to be capable of understanding and acting upon
human behavior and social situationsa new dimension that will soon become an
integral part of the end-to-end goals relevant to optimal data flow. As a result, the
scope of the term cognitive network can be constantly extended based on newly
emerging human-oriented synergies, as new data and information types become a
part of the cognitive content space that is handled by ICT. Eventually, it can be
expected that cognitive networks will be aptly described as artificially cognitive
neural networks that incorporate findings relevant to sensing, representation and
http://www.pnl.gov/coginformatics.
46
interpretation from the cognitive sciences, and merge those findings with modern
network technologies. This aspect of networked technologies described through
the cognitive sciences further strengthens the notion that not only human-oriented,
but also technology-oriented research fields (whether focusing on individual or on
networked technologies) can lay a strong and direct claim to the connotations of the
term cognitive (such a development can be in fact be expected in a wide range of
fields relevant to informatics in the near future).
One research area that seems particularly relevant to the emergence of cognitive
networks is Future Internet, which deals with the theoretical and practical aspects
of creating cognitive networks through which users can interact not only with other
people and computers, but also with everyday objects and abstract computational
capabilities (Tselentis et al. 2010). Two major directions in Future Internet research
are the Internet of Things and 3D Internet.
The Internet of Things (IoT) focuses on the integration and virtualization of the
physical-biological world (hence both physical objects and humans) together with
their capabilities into a comprehensive network through billions of interconnected
sensor technologies.3;4 In a sense, IoT creates a strong physical analogy to the
human nervous system: just as the latter provides humans with a sensor network,
IoT implements a global, physical sensor network [examples of this analogy abound
in the literature, through less direct terms such as the central nervous system
as in the case of the Central Nervous System of the Earth project carried out
by HPas well as in direct architectural considerations (Ning and Wang 2011)].
Inasmuch as Internet of Things enhances the capabilities of humans for effective
communication, it is expected to provide key insights into the field of CogInfoCom
as well. The possibility of treating the human sensory system and the global
sensory system implemented by IoT in a unified framework is also strongly relevant
to CogInfoCom. Although today the engineering and cognitive science based
perspectives through which the two areas are addressed are still markedly distinct,
it is conceivable that similar terminologies and methodologies for investigation will
be developed in the not too distant future. An overview of architectural designs and
applications in IoT can be found in Uckelmann et al. (2011), while (Perera et al.
2014) provides an in-depth survey from an application-oriented point of view.
The concept of 3D Internet (3DI), which is a more recent development, focuses
on the growing expectation of users for high-quality 3D imagery and immersive
online experience (Alpcan et al. 2007; Kapahnke et al. 2010). 3D Internet is seen
as a natural part of the Future Internet concept, because with the appearance of
virtualized interfaces to everyday objects, users will still expect to be able to handle
them in the same ways (importantly, in relation to the same functionalities) as
they do in the physical world. However, once this is established, it also becomes
This means that rather than seeing IoT as a network of objects, it can regarded as humans and
objects that matter to them.
Through the human aspect, other domains such as affective computing and body area networks
when used for infocommunication purposesalso become particularly relevant.
4.7 Human-Computer Interaction and Multimodal Interaction
47
clear that 3DI can also be useful for handling 3D representations of content that
is not in its natural form amenable to direct physical representation. Further, once
modifiable representations of both (virtualized) physical objects and abstract data
are created, the ability to navigate (i.e. move along trajectories and remember
those trajectories) between objects and data sets in a way that seems natural also
becomes important. Taken together, these perspectives clearly show that 3D Internet
is about much more than visualization and interaction in 3D: it is a mapping between
the digital world and (physical) 3D metaphors based on highly evolved human
capabilities for representation, manipulation and navigation in 3D. Any technology
that achieves this is an important contribution to the field. As a case in point, spatial
memory systemsproposed by Professor Niitsuma (Niitsuma and Hashimoto 2009;
Niitsuma et al. 2007)which map physical locations and gestures to digital content
and automated functionality are strongly relevant to 3D Internet.
Both the Internet of Things and the 3D Internet are expected to pervade our
everyday lives in the near future. A consequence of both of these research directions
is that users are expected to be able to communicate with both physical and virtual
things (i.e. everyday objects and objectified abstractions) through the Internet,
and also to collaborate with them in ways that depend on both the (artificially
cognitive) capabilities of the objects and on the context (i.e., users will need access
to different components of the network depending on situational context). These
criteria introduce a host of challenges. First, there is the question of augmented
collaboration (i.e., the objects, as viewed from the Internet, can be a combination
of physical and virtual componentssee also Sect. 4.11 for more details on the
relationship between virtual reality and CogInfoCom). Second, there is the question
of scalability (i.e., due to the rapid expansion of content space as an increasing
number of objects try to communicate with the user in the Internet of Things). In
these regards, CogInfoCom has common interests with Future Internet in terms of
selecting the information types that are relevant to the given context, and the modes
of communication that are necessitated by those information types. For further
perspectives on the relevance of Future Internet research to CogInfoCom, readers
are referred to Chap. 12.
4.7 Human-Computer Interaction and Multimodal

Interaction
Human-computer interactions (HCI) is a highly influential multidisciplinary field
that focuses on the psychology of interactions between users and computer systems
and aims to formulate design principles that are guaranteed, at least in some context,
to lead to improved usability (Card et al. 1983; Preece et al. 1994). Although the
results produced by the field will be foundational as long as humans use computers,
the proliferation of information systems providing users with less and less direct,
or simply different possibilities for interaction has led to the emergence of fields
48
such as human-machine interaction, human-robot interaction, human-information

interaction, and even human-Internet interaction and human ubiquitous computing
interaction.
As remarked by Streitz and Nixon, we have to ask whether we are still interested
in interacting with computers, rather than interacting with information and
collaborating with people due to the fact that computers are disappearing from our
lives in two senses of the word: in a physical and a mental sense (Streitz and Nixon
2005). Physical disappearance refers to changing modes of interaction, while mental
disappearance reflects the fact that even if computers are increasingly important in
the background, users are also gradually becoming less aware of their existence as
their interfaces blend into the everyday physical surroundings. However, this train
of thought can be continued further: it may be added that not only the computer,
but in many cases the user as an individual entity is also disappearing. On the one
hand, the word user suggests that we are using something, whereas the longterm co-evolution targeted by CogInfoCom research well extends into the territory
of functionally agnostic interaction patterns. On the other hand, there is often
value in considering cognitive entities at larger spatio-temporal scales than those
characterizing single-user interactions: such entities are by definition heterogeneous
and often lack the kinds of clearly delineated interfacesboth internally and
externallywhich originally gave rise to words such as user and interaction.
Regardless, in cases where individual users are being targeted by new infocommunication technologies, results in HCI are strongly relevant. Given that
infocommunications is directed primarily towards the sharing of knowledge through
ICT, the following dimensions of interactionwhich have been extensively studied
in the pastare particularly important to consider:
Negative effects of reduced resolutionThere are convincing studies which
show that it is better to use different modalities than the ones that are normally
used for a given task when the resolution of data flow is reduced through the
normal modality. For example, it was shown that providing force feedback that
is reduced in degrees of freedom can result in the destabilization of teleoperation
systems (Verner and Okamura 2006). In applications such as remote knot-tying
in telesurgery, it was shown that the forces applied by the telesurgeon were
closer to the normal, manual case when auditory and graphical displays were
used instead of direct, but reduced-precision force feedback (Kitagawa et al.
2005) (a specific solution to the problem of vibrotaction-based force feedback
was recently proposed by Galambos and Baranyi 2011a). All of this suggests
that sensor-bridging CogInfoCom can have important functional benefits: the
use of different modalities than the ones normally used in settings when no
infocommunication is needed can be advantageous.
Intersensory integrationThere is extensive proof in the literature that different sensory channels are not independent of each other. While contradicting
information from various senses can cause confusion, simulation sickness or
other discomfort, illusions in which stimulation in one sensory channel leads
to the illusion of stimulation in another can be very powerful in virtual and/or
4.7 Human-Computer Interaction and Multimodal Interaction
49
remote teloperation (Biocca et al. 2001). The ability of human cognition to

integrate experience from various sensory channels is referred to as intermodal
(or intersensory) integration. Whereas the previous point on the effects of
reduced resolution highlighted the value of using different sensory modalities;
intersensory integration supports the use of multiple sensory modalities. This can
be essential whenever the use of a combination of senses adds further intelligence
to CogInfoCom applications (on the technological side, the implication that
2 C 2 can be greater than 4 when it comes to combining information from
separate sources has long been made, as evidenced by areas such as multi-sensor
fusion and big data).
Cross-effects between sensory modalitiesResearchers have long ago discovered that the impression that different sensory modalities are independent
of each other is more illusory than real (Stein et al. 1995). Thus, when
designing feedback strategies in teleoperation systems, care must be taken to
ensure that the operator is not overloaded with sensory information. The question
as to whether multi-sensory feedback is productive or not has much to do
with the degree of redundancy in the information that is presented (Massimino
1992; Rochlis 2002). However, Biocca et al. (2001, 2002) also suggest that
it is possible for one sensory modality to yield realistic sensations normally
perceived through another modality, while another sensory modality gives no
contribution to realistic sensations, but rather serves to increase the users sense
of telepresence. Differences between feedback contributing to telepresence and
semantic interpretability should ideally be considered when designing future
infocommunication technologies.
Sensory dominanceAnother key point of interest when designing multimodal
interfaces is how the various sensory modalities relate to one another in terms
of importance to human cognition. This is referred to as the question of sensory
dominance. There have been a number of studies which show that vision dominates haptic touch and audition (Sheridan 1994; Welch and Warren 1986; Pavani
et al. 2000; Hecht and Reiner 2009), but it was also shown that relationships
of dominance can become more complex if more than two modalities are under
stimulation at the same time (Hecht and Reiner 2009). This suggests that it is
important to consider the effects of new technologies from a unified conceptual
view of perception.
To summarize, HCI is strongly relevant to CogInfoCom when it comes to the
analysis and design of functionally motivated user-device interactions. At the same
time, questions of indirect interactions (essentially tangleactions) will increasingly
arise as the entanglement process between humans and ICT develops. Charting
out suitable methodologies for the analysis and design of capabilities for such
tangleactions is one of the key goals of CogInfoCom.
50
4.8 Sensory Substitution

Sensory substitution is a research direction that provides room for synergy with
sensor-bridging CogInfoCom. The basic idea behind sensory substitution, and its
utility was first described by Bach-y-Rita and his colleagues, who, in one of their
more recent works, define sensory substitution as the use of one human sense to
receive information normally received by another sense (Bach-y Rita et al. 2003).
There are at least two aspects in which sensory substitution leaves room for future
research. The first aspect, highlighted by Auvray and Myin (2009) is the question
of whether sensory substitution can truly be regarded as substitution. Some have
argued that despite sensory substitution, the new stimuli should still be regarded
as generating percepts from the original, substituted modality (this is known as
the deference thesis, e.g. as in Hurley and Noe 2003). Others have argued the
opposite, maintaining that the new, substituting modality dominates the substituted
modality, and that the stimuli should be regarded as generating percepts in the
substituting modality (this is known as the dominance thesis, e.g. as in Prinz 2006).
The key novelty of Auvray and Myins investigations is that they demonstrate
using concepts from the psychophysics and psychology of sensory modalities
(e.g., sensory organs, qualitative experience, behavioral experience, dedication and
sensorimotor equivalence)that the modality used after sensory substitution is in
fact a completely new one, which is different from both the substituting and the
substituted modalities. This has led to the term sensorimotor extension, which
suggests that through the creation of new sensory modalities, the sensorimotor
capabilities of humans can be extended.
The second aspect is the realization that it may be useful to broaden the scope of
sensory substitution, at least in engineering systems, based on the source and role of
the information that is presented through extended modalities. This was highlighted
in Bach-y Rita et al. (2003) as follows:
However, in the context of mediated reality systems, which may incorporate multiple
modalities of both sensing and display, the use of one sense [. . . ] to display information
normally acquired via another human sense [. . . ] or alternatively via a non-natural sense
such as sonar ranging, could be considered to be a form of sensory augmentation (i.e.,
addition of information to an existing sensory channel). [. . . ] We therefore suggest that, at
least in multimodality systems, new nomenclature may be needed to independently specify
(a) the source of the information (type of environmental sensor, or virtual model); (b) the
type of human information display (visual, auditory, tactual, etc.); and finally (c) the role
of the information (substitutive or augmentative), all of which may play a role in reality
mediation.
In this statement, Bach-y-Rita, Tyler and Kaczmarek clearly demonstrate that

although sensory substitution is sufficient in describing many applications, it could
be valuable to broaden the scope of sensory substitution to describe forms of
communication between humans and machines which had previously not been
considered, even if the source or destination of the communication cannot be
described using the traditional senses of the human nervous system. Such a position
seems viable, primarily because the types of information that must be communicated
4.9 Social Signal Processing
51
between artificial systems and the user may not always be directly perceptible by the
available sensory and cognitive subsystems (hence the need for sensor-bridging). In
other cases, the transformation of information from one sensory modality to another
(sensor-bridging) and from one representation to another (representation-bridging)
may provide more effective interaction between the user and the system.
It is clear that the terminology used in CogInfoCom is in many respects a
reflection on the suggestion of Bach-y-Rita and his colleagues. The distinction
between intra-cognitive and inter-cognitive forms of cognitive infocommunications
reflects their first criterion regarding the new terminology (i.e., regarding the
source of the information). The distinction between sensor-sharing and sensorbridging reflects the proposition to distinguish between different types of human
information display. The concepts of representation-sharing and representationbridging are somewhat related to the third point, namely to the question of whether
the transferred information is substituted or augmented.
4.9 Social Signal Processing

The research area of social signal processing (SSP) focuses on the automated
detection of human attitudes in social interactions through the sensing and analysis
of behavioral cues that are appropriate to the social context (Pentland 2007;
Vinciarelli et al. 2008, 2009). Several taxonomies have been proposed for what
constitutes a relevant behavioral cue. For face-to-face interactions with access
to visual and auditory feedback, examples of dimensions from which typically
useful cues can be extracted include physical appearance, gesture and posture,
facial expressions, vocal behavior and spatially relevant behaviors such as distance between interlocutors (Vinciarelli et al. 2009). Although the relative weight
attributed to different behavioral cues in forming a comprehensive understanding of
cognitive state has been a subject of many debates, it is now agreed upon that nonverbal cues alone can often provide more relevant information than a knowledge of
only verbal messages (Ambady and Rosenthal 1992).
In a comprehensive overview of the field, Vinciarelli and his colleagues distinguish four phases of social signal processing (Vinciarelli et al. 2009):
1. Data capture: a natural prerequisite for social signal processing is that social
interactions have to be recorded through some medium, however, this has to
be done in a way that respects privacy and that is passive in the sense that the
recording process itself does not influence the recorded behaviors.
2. Detection of people: depending on the kinds of media used in the previous step,
behaviors originating from different participants in the social interaction have to
be separated and attributed correctly to individual participants.
3. Extraction of behavioral cues: the behavioral cues of interest have to be identified
and extracted from a potentially noisy stream of behaviors.
52
4. Context-sensitive classification: the behavioral cues extracted in the previous step

have to be interpreted in terms of cognitive state based on an awareness of the
context in which the entire interaction is taking place.
It should be noted that, understandably, SSP focuses on ways in which human
intentions and attitudes are seen from an external point of view rather than how
they are understood or intended from within. Although it has been remarked in
a variety of contexts that social signals are particularly honest (e.g. Ekman 2003;
Pentland 2008), this does not mean that someone who is less well trained can use
or interpret them to their full potential. Thus, an important research direction within
CogInfoCom would be to support human capabilities for displaying and discerning
social signals more accurately through extended periods of co-evolution.
Another important direction in which the study of social signals can be extended
is into the domain of virtualized social interactions, including applications using
avatars or relying on augmented social signals (for example, in the area of citizen
sensing, as described in Sheth 2009; Zhang et al. 2011).
4.10 Virtual and Augmented Avatars

The term avatar is derived from the Sanskrit avatarawhich refers to an
incarnation of an immortal beingand was first used in a technological context in
a role-playing game designed by R. Garriott called Ultima IV: Quest of the Avatar
(Garriott 1985). Since then, the term has been defined in many ways and applied in
increasingly diverse contexts outside of the gaming industry.
Strictly from a gaming perspective, an avatar can be seen as the users
representative in the virtual universe (Waggoner 2009). However, this view can be
further qualified by taking into consideration the various modes of control through
which users may influence avatars, and the psychological effects the use of an avatar
can have on users perception of identity. As highlighted by Wilson, avatars can lead
to a part of ones self being extended or projected onto the screen, allowing it
to enact a dynamic of agency by proxy (Wilson 2003). As described in the same
paper, avatars can be multifarious and polymorphous, resulting in the ability to
create the perception that the self is evolving through time. Further, this evolution
can be actively influenced by the user through the choice of how avatars are created,
socialized, and applied in the generation of shared meaning through communicative
exchange (Wilson 2003; Waggoner 2009).
Qualifications such as these have played an important role in shaping the general
understanding of avatars. Gradually through the past decades avatars have come to
be viewed as semi-autonomous embodied communicational agents linked to a user
(Benford et al. 1997; Vilhjalmsson and Cassell 1998; Cassell et al. 1999; Taylor
2002). Increasingly, avatars are supposed to provide access points in the creation
of identity and social life, and to live digitally (Taylor 2002). However, this is not
4.10 Virtual and Augmented Avatars
53
without significant challenges, as highlighted by many authors. Some challenges

include (Benford et al. 1997; Cassell et al. 1999):
Identity: users controlling different avatars should ideally be distinguishable and
in most cases recognizable at more than one levelincluding e.g. the identity of
the user, or the role of the user in the systembased on the appearance of their
avatar.
Presence: the fact that a user is present in the system, and the context of
that presenceincluding what the user is doing, how busy or available for
communication the user is, how the users cognitive/emotional state can be
characterized, and so onshould ideally be ascertainable from the representation
of the avatar.
Relationships with objects and information: the past and present interactions of
users with both the physical and virtual world should ideally be ascertainable to
a degree that is necessitated by the application based on the representation of the
avatar.
Metacommunicational fidelity: communication through avatars should give users
the sense that they themselves are communicating in much the same way as in
real life, such that they can use their metacommunicational (e.g. prosodical, gestural and other body movement related) capabilities, both in terms of generation
and recognition to the fullest extent possible.
These challenges are strongly relevant to the well-known mind-body problem,
which results from the fact that while the goal is to create a conceptually strong
link between users and their avatars (i.e. in users minds), the two bodies will
necessarily continue to inhabit physically separate worlds. Especially when multiuser conversation occurs in a virtual reality, the fact that users are required to control
a physically distinct replica of their own bodies leads to important challenges.
Such difficulties can be mitigated by using shared control schemes in which partial
behavioral control of semi-autonomous avatars is sufficient for detailed, highfidelity interactions. Further, the undesirable effects of the mind-body problem may
also be alleviated through the use of non-antropomorphic avatars, i.e. personified
functional representations that are abstract in their mapping to users (such abstract
mappings can be equally successful as human psychology is known to attribute
personalities to abstract representations with ease).
Both the challenges associated with episodic interactions (such as creating
functionally adequate mappings between human and non-antropomorphic joints),
and the psychological effects as well as potential benefits of maintaining a virtual
identity through extended periods of time bring important perspectives to CogInfoCom. Conversely, through the integration of results in a range of fields detailed
earlier, as well as through results in various CogInfoCom-driven fields detailed later
(i.e. CogInfoCom, speechability and socio-cognitive ICT as detailed in Chap. 5),
new inspiration may be brought into the design practice of virtual avatars.
54
4.11 Virtual and Augmented Reality

In everyday human-machine interactions, both human and machine are located
in the same physical space and thus communication can proceed through the
users natural cognitive capabilities. In contrast, when 3D virtualization comes
into play, physical contact is lost, and users are forced to interact with virtual
representations of the same machine (Riva and Davide 2001; Smith et al. 2009).
Thus, the problem of human-machine interaction is transformed into a problem of
human-virtual machine interaction. Once human avatars come into play, the human
counterpart of human-virtual machine interactions also become virtualized. The
key observation to be made here is that virtualization can progress through several
stepstransitioning through human, machine, virtual human and virtual machine
interactions. This process of virtualization can be further complicated when multiple
users and multiple machines are applied towards operating a single process.
Based on the above, CogInfoCom and virtual reality research can have fruitful
synergies in the following aspects of virtual human-virtual machine interaction:
The natural communication capabilities of humans become limited due to the
restricted interfaces provided by the virtual representations of machines (for
instance, while the senses of vision and audition still receive considerable amount
of information, the tactile and olfactory senses are almost completely restricted
in virtual environments, i.e. it is usually not possible to touch or smell the virtual
representation of a machine). For this reason, it becomes necessary to develop
a virtual/augmented cognitive system which can extend the natural one so as to
allow humans to effectively communicate with the virtual representations. In this
extended scenario, the virtual representation of a remote system can be viewed as
an infocommunication system through which the remote system communicates
with the user.
After dealing with virtual reality/augmented reality (VR/AR) for a whiledue
to limitations in virtualized user representationsusers can have the impression
that they are interacting merely as a ghost. Thus, everything can be seen and
heard, but no body is available, and nothing can be touched or manipulated as in
the real, physical world. The problem of situation awareness (also commonly
referred to as telepresence)which is a measure of the degree to which the
user feels present in the remote or virtual environment (Sheridan 1992; Endsley
and Garland 2000)addresses this question directly. In an early work on the
subject, Sheridan outlines three key components of telepresence: the extent of
sensory information, the control of relation of sensors to the environment, and
the ability to modify the physical environment (Sheridan 1994). Encumberment
is a term used often in the literature to describe the extent to which the user is
burdened with having to wear various kinds of sensors in order to interact with
a system (Ellis 1991; Adams and Hannaford 1999). It is natural to try to reduce
encumberment in virtual environments, however, doing this conflicts the goal of
increased situation awareness.
4.11 Virtual and Augmented Reality
55
CogInfoCom has common interests with virtual and augmented reality research
along these two aspects. One of the primary goals of CogInfoCom in this respect
is to be able to augment the sensory capabilities of both the human user and the
artificially cognitive system so that they can communicate with each other at a
higher level, thus alleviating the difficulties of human-virtual machine interaction
and the conflicting goals of situation awareness and unencumberment.
Chapter 5
CogInfoCom-Driven Research Areas
In this chapter, an overview is provided of the research areas of CogInfoCom

channels, speechability and socio-cognitive ICTall three of which have recently
emerged under CogInfoCom (further recent initiatives are presented in the following
chapter). CogInfoCom channels focus on how multi-sensory messages between
cognitive entities can be structured in such a way that semantic meaning can be
effectively interpreted; while speechability and socio-cognitive ICT address various
aspects of linguistic and social tangleactions in geographically and temporally
distributed cognitive networks. The second half of the chapter briefly presents
research efforts and results that are representative of these areas and have appeared
at CogInfoCom conferences and special issues.1 A unified approach is proposed
towards describing existing applications in order to facilitate the identification
and future application of universal principles behind them. This discussion style
highlights the benefits of approaching problems related to the entanglement between
humans and ICT from the perspective of hierarchical cognitive capabilities.
5.1 CogInfoCom Channels

Interaction and communication between cognitive entities can occur at numerous
levels depending on the semantics and pragmatics of the situation. Finding the
interaction mode, or language that is most suitable for a given application is an
important challenge, especially if the information type to be conveyed is such that
the cognitive entity has no past experience in interpreting it through any form of
The decision to present results from CogInfoCom fora serves the goal of tractability, and in no
way intends to suggest that important and relevant results have not been published elsewhere.

DOI 10.1007/978-3-319-19608-4_5
57
58
5 CogInfoCom-Driven Research Areas
sensory pattern. As the expansion of cognitive content space described earlier in

Sect. 3.2.4 progresses, this is often precisely the case. As a result, the question of
how sensory representations can be created anew to characterize novel concepts is
expected to gain increasing relevance.
The framework of CogInfoCom channels addresses the above challenges by
combining both structural and semantic elements to define sets of sensory messages
with graded references to high-level concepts2 (Csapo and Baranyi 2012b). From a
syntactical point of view, icon-like and message-like design elements in interfaces
developed for the auditory, haptic and other modalities [e.g., auditory icons and
earcons (Blattner et al. 1989), haptic icons and hapticons (Maclean and Enriquez
2003; Enriquez and MacLean 2003), olfactory icons and smicons (Kaye 2004)] are
generalized into a layered framework in which messages are built up of lower-level
icons (Csapo and Baranyi 2012d). While icons support direct meaning, messages
are generally multi-dimensionalboth from a perceptual and from a conceptual
point of viewand are often characterized by the sequential occurrence of iconic
elements. The CogInfoCom channel framework includes a concept algebra-based
toolset for the mapping of semantic meaning to messages, as well as a parametric
cognitive artifact, referred to as the spiral discovery method (SDM), which allows
users to fine-tune the parametric mapping between generating parameters (used
to generate messages) and semantic gradations (Csapo and Baranyi 2012c). More
recently, the framework has been extended to include concepts adapted from biology, with the goal of modeling the evolution of communication as it occurs in natural
settings. If communication is seen as an adaptive process that evolves through
time to convey meaning about emergent concepts, the biological phenomenon of
ritualization, by which implicit cues evolve into purposeful signals can provide
important ideas for further development (Scott-Phillips et al. 2012). Accordingly,
signal ritualization has been extended in the CogInfoCom channel framework with
channel differentiation, through which individual messages can evolve into sets of
graded messages with variations that depend on the contextual background (Csapo
and Baranyi 2013).
The framework of CogInfoCom channels plays an important role in CogInfoCom
research as it highlights several important concepts that are relevant not only to
describing ways in which humans are capable ofand prefer tocommunicate,
but also to synthesizing new communication patterns. Several application examples
relevant to the research area are described in Sect. 5.4 of this chapter. Further, an indepth overview of recent theoretical advances in relation to CogInfoCom channels
is provided in Part III of the book.
In this case, high-level refers to the conceptual level that can be readily interpreted by a cognitive
entity, as opposed to lower-level concepts, which constitute higher-level ones but are not usually
interpreted directly.
5.2 Speechability
59
5.2 Speechability
There is strong empirical support for the view that humans evaluate interactions with
ICT and human-human social interactions based on analogous criteria (Nass and
Yen 2010). As a result, the various aspects of how humans communicate in everyday
social interactions cannot be neglected when augmenting the social capabilities of
cognitive networks.
The modality of speech is a central component of social interactions in everyday
life. For several decades before the emergence of CogInfoCom, researchers have
aspired not only to transmit speech between geographically distant locations, but
also to enable artificially cognitive systems to understand and communicate through
verbal utterances in the most natural (i.e., human) way possible. The reductionist
approach applied to this latter problem has quickly resulted in the appearance
of increasingly specialized sub-branches focusing on a wide range of verbal
phenomena. This fragmentation of a research paradigm that was once fueled by
a single goal is not surprising, given that speech itself cannot be fully characterized
by any single dimension. Rather, it involves co-dependent interactions among such
aspects as word usage, prosody, facial expressions, hand gestures, body postures
and movements, as well as higher-level constraints on the dynamics of dialogue
imposed by social norms and cultural specificities. A further source of heterogeneity
is brought about by the fact that the modality of speech is used for more than a single
purpose: its goal is not merely to support the sharing of information, but also to help
create trust and more generally strengthen those kinds of social/behavioral patterns
that are conducive to the maintenance of society. Removing any of these aspects
from human speech, as is clear from extensive research on virtual communication
agents, renders interaction unnatural and cumbersome.
Speechability aims to reverse the tendency towards fragmentation described
above by attempting to link cognitive linguistics with verbal and non-verbal social
communicative signals through human-ICT tangleactions (Campbell 2012; Benus
et al. 2014b). The latter qualification refers to the emergent effects of long-term coevolution in terms of a broadening range of natural and artificial modalities applied
together towards long-term goals, and consequently the increasing availability
of raw data that is collected for e.g. machine learning or other post-processing
purposes. Such long-term entanglement will lead to capabilities whose scope
extends far beyond the generation and understanding of speech, so as to encompass
application areas such as speech rehabilitation, tutoring for the learning of foreign
languages, or communicational training.
Approaches applied in speechability research are also rooted in the observation
that speech is an embodied phenomenon, i.e. it is interpreted through reference
to physical interaction in specific social contexts. Thus, a unified approach is
suggested that acknowledges the fact that humans and machines have different
embodimentsalbeit ones that are converging as novel cognitive entities. If this
point of view is adopted, then even without long-term tangleactions, speech-related
capabilities and phenomena can be mapped onto artificial ICT capabilities in ways
60
Fig. 5.1 Adaptation of central notions in Fig. 2.1 to the domain of speechability. New, higher-level
artificial capabilities and emergent cognitive entities are created through time as new use cases are
generated through a broadening of artificial sensory modalities and the increasing possibility for
users to co-evolve with them through extended periods of time
that suit underlying differences in embodiment rather than denying them. In this
way, human capabilities are supported rather than copied, and the criterion for
technology to be human-like becomes relatively less important. This explains the
term speechability, as distinct from speech ability (i.e., the ability to speak as
humans do), which reflects the complex, embodied nature of speech phenomena in
cognitive entities. Crucially, while speechability includes speech ability (including
artificial capabilities for speech generation and recognition), it also encompasses a
broader range of tanglefaced applications.
Based on these observations, the central notions of Fig. 2.1 can also be applied
to speechability, as illustrated in Fig. 5.1. The CogInfoCom conference series has
hosted several tracks and sessions on speechability, initiated by Professors Benus,
Campbell and Esposito from Columbia University, Trinity College Dublin, and the
University of Naples II, respectively. Several application examples relevant to the
research area are described in Sect. 5.4 of this chapter.
5.3 Socio-Cognitive ICT

Computer networks of all kinds have emergent cognitive properties due to the
constraints imposed on their operation through various user interactions. Todays
Internet is no exception to this rule. However, as the content space that is handled
through the Internet is augmented with new cognitive data and information types, a
growing set of functions and value chains are conceivable.
5.3 Socio-Cognitive ICT
61
Many recently developed applications can be seen as directed towards augmenting the social capabilities of cognitive network based on the analysis, manipulation
and management of information flow. For example, content and metadata-based
analysis of user activity is used to gain a better understanding of spatially distributed,
crowd-generated phenomenaincluding social-political tendencies, the spread of
ideas and epidemics, etc. Similarly, high-level structural and organizational management of cognitive networks is applied to the optimization of information flow
in critical situations, such as in workflow management and disaster recovery. The
effective treatment of such critical situations necessitates a hierarchical allocation
of both human and technological resources that is crucially enforced by technology,
in much the same way as face-to-face human interaction patterns are governed and
to some extent enforced by social conventions.
Applications such as these can be categorized as operating at the highest level of
entanglement between humans and ICT, or among heterogeneous cognitive entities
in more complex cases (see also Sect. 2.3.1). In this case, collective behaviors
are used to support either individual user interactions, or the prediction/analysis
of collective behaviors and events. Due to the clear relevance of both social
and cognitive capabilities to such applications, the term socio-cognitive ICT
proposed by Professor Hassan Charaf and his research group at the Budapest
University of Technology and Economicshas been increasingly used to describe
them (Szegletes et al. 2014; Fekete and Csorba 2014; Kovesdan et al. 2014).
Interestingly, this choice of description is not without parallels from the past,
even from a technological perspective. Hemingway and Gough described design
challenges relevant to ICT as a conflict between different goals and underlying
values in the fields of software engineering, human-computer interaction and
information systemsthe latter of which is generally concerned with the longerterm impacts of information and communication systems on organizations and
society (Hemingway and Gough 1998). In this interpretation, the term sociocognitive is used to describe all aspects encompassing the ways in which social
phenomena affect, and the ways in which they are affected by ICT infrastructure.
Sharples et al. describe socio-cognitive engineering as a general methodology that
aims to analyze the complex interactions between people and computer-based
technology and then transform this analysis into usable, useful and elegant sociotechnical systems involving technology and social context. In a way somewhat
similar to earlier works, the paper also views this approach as an integration of
fields such as software, task, knowledge and organizational engineering (Sharples
et al. 2002). In general, the important difference between such earlier uses of the
term socio-cognitive and its use in socio-cognitive ICT is that socio-cognitive
ICT focuses on all aspects of network management and networked experience as
opposed to the design-centric perspective adopted in earlier works.
62
5.4 Common Features and Techniques

When the particularities of specific approaches are abstracted away, it becomes clear
that applications and characteristic solutions in the areas of CogInfoCom channels,
speechability and socio-cognitive ICT have important features in common. Thus, by
adopting a higher-level perspective as suggested by the entity and modality-agnostic
notion of cognitive capability, the following three broad classes of techniques can
be distinguished:
icon-based (iconic) techniques: efficiently, reliably and immediately recognizable features are derived from communication signals and used to identify or
synthesize events and attribute values that are relevant to cognitive content;
context-based (contextual) techniques: features derived from communication
signals are used in combination with contextual information to detect or synthesize events and attribute values that are relevant to cognitive content;
pattern-based techniques: behavioral/emotional patterns that are abstracted
away from (human) social communication are mapped onto patterns in the
analysis and synthesis of cognitive content.
It should be noted that in the description of icon-based techniques, immediacy
refers not to a specific time span, but rather to the possibility of recognizing relevant
features in a signal without having recourse to contextual information. Of course,
the question of what constitutes context and what does not is one that leads to a
figure-ground problem that is not amenable to a formal definition, and the answer to
the question largely depends on the level of cognitive capability of the interpreting
entity, as well as on the meaning that is being conveyed. For example, a stop sign
on the side of a road is an icon with respect to human visual capabilities and with
respect to the meaning that one should stop; however, if the sign were much larger
than the human field of vision, one would have to re-construct its meaning based on
a serial discovery of the relative configuration of individual parts.
As the example shows, icons, in a sense, are a direct representation, while
contextual representations (also referred to in this book as message-like) can be
better described as indirect. In some cases, the two can be distinguished between
by scalability in space or time. For example, if a feature can be detected from a
series of acoustic signals irrespective of the length of time within the signal that is
considered (i.e. irrespective of whether the signal is sampled for 500 ms or 2 s), then
the feature derived is iconic. Similarly, if a the derivation of a visual feature is largely
independent of the spatial extension that is taken into consideration, the feature
can be regarded as iconic. Conversely, in the case of message-like features, the
spatio-temporal relationships between constituting elements are often not resistant
to scaling.
An interesting property of icons and messages is that through time, icons can
evolve into messages and vice versa; i.e. new meaning can be constructed based on
contextual relationships among sets of icons (if such relationships can be learned),
63
and message-like representations can also be simplified into iconic ones (if the
structure of events in the world is such that the meaning of the message can still
be unambiguously reconstructed).
Finally, pattern-based techniques often employ both iconic and contextual
approaches, but are also crucially different from the two in that they also take into
consideration characteristic patterns in cognition and behavior that, while often are
extrinsic to the technology that is developed, nevertheless have important effects on
the success of the application.
In this section, example applications presented or published at various CogInfoCom fora are surveyed in terms of the three classes of icon-based, context-based
and pattern-based approaches, as well as in terms of application area. This will help
elucidate some additional properties of these techniques, and how they can be used
together to develop powerful applications.
5.4.1 Icon-Based Approaches and Applications

As described above, icon-based analysis focuses on direct relationships between
the profile of a communication signal and semantic meaning. Many research
contributions relevant to CogInfoCom channels, speechability and socio-cognitive
ICT clearly reflect the utility of such approaches. Often, however, they is also
complemented with higher-level, contextual and pattern-based approaches. In the
following, a brief overview is provided of the utility of icons in various sub-domains.
5.4.1.1 Icons in Physiological Signals

A variety of physiological signals can serve as a direct and effective source of information relevant to human cognitive states, including, among others, information
on mental effort, level of engagement, and emotional arousal/valence. For example,
increased activity in low beta waves (i.e. 12.516 Hz) of EEG signals, low-frequency
(i.e. 0.040.15 Hz) peaks in the heart rate variability/heart period variability power
spectrum, as well as pupil dilation have been shown to correlate well with increased
mental load (ChuDuc et al. 2013; Hercegfi 2011a,b; Kimura et al. 2009). Relative
tensions in different facial muscles as measured using electromyography, changes
in skin conductance, amplitude sizes in event-related potentials in EEG, and relative
activeness of specific brain regions as measured by fMRI, in turn can be highly
indicative of emotional state (Hercegfi 2011a,b; Lee et al. 2009).
In a recent application, Szegletes and her colleagues applied such physiological
signals to create a biofeedback-driven framework for games with dynamically
adjustable difficulty levels (Szegletes et al. 2014). By supporting the automatic
regulation of difficulty level, users engagement can be maintained. Approaches
such as this one can be useful not only for entertainment purposes, but also in the
context of serious gaming applications and any other solution in which the sensing
of human cognitive states is important.
64
5.4.1.2 Icons in Verbal Signals

Important sources of information can be found in the different dimensions encompassed by verbal aspects of speechability, including both low-level features within
the phonological, spectral, prosodical and articulatory dimensions, and higher-level
features of dialogue. Depending on how the derived features are usedwhether as
individual detectors of specific concepts, or as a stream of inputs to a larger-scale
decision mechanismthey can be regarded as iconic or contextual. Commonly
used lower-level features include mean figures of energy and intensity, variations
in energy and intensity (i.e. jitter and shimmer), first and second formants in
frequency spectrum, and articulation rate. Higher-level, dialogue-related features
include the occurrence of various conversation fillers, discourse markers, laughter,
interruptions, silences, etc.
The use of low-level icons alone in reaching conclusions on high-level, semantically interpretable phenomena is relatively rare due to the fact that often such
icons often correspond to small sections of human speech and have ambiguous
meaning; thus, they are best aggregated with other elements prior to further analysis.
Generally speaking, this is done by extracting groups of icons over time and making
aggregated decisions using machine learningas discussed further in Sect. 5.4.2.
Nevertheless, attempts can sometimes be made to find correlations between iconic
parameters and semantic meaning. In a recent set of publications, Kiss and his
colleagues used low-level iconic parameters to investigate the possibility of detecting, based on speech, various environmentally induced physiological and cognitive
states such as depression (specifically, seasonal affective depression) and hypoxia.
Differences in the investigated parameters were shown to be significant in the case of
both targets (Kiss and Vicsi 2014; Kiss et al. 2014). Emotional informationat least
to the extent of presence of emotionally charged utterancescan also be extracted,
sometimes based on icons alone. For example, Raptis applied principal component
analysis on a large number of zero-order and first-order iconic features to find
ways to identify regions of expressive interest (RoEIs) in speech corpora (zeroorder icons in this case included word-level acoustic features relevant to intensity,
energy and rate, while first-order icons reflected changes in such qualities between
neighboring words) (Raptis 2013). As the regions of expressive interest were local
phenomena, the results of the paper suggest that they can be identified to at least
some degree using word-level icons.
At a higher, dialogue level of hierarchy, Bonin et al. have reported an interesting
set of experiments that have confirmed that the occurrence of topic shifts can be
predictable based on the timing between laughter, interruptions and silences (Bonin
et al. 2014a,b). Conversational fillers and discourse markers have also been shown
to be relevant to iconic meaning extraction. Both linguistic elements are highly
iconic in the sense that their value is taken from a relatively limited, discrete set of
alternatives and the time at which they occur are relatively well determined by their
function. Not surprisingly, several efforts to link them to salient events and attributes
have been successful. For example, Benus linked the conversational fillers uh and
mm to the disambiguation of communicative functions (Benus 2013). Similarly,
65
with respect to discourse markers, Abuczki demonstrated that when more than a
single lexical meaning is possible, duration with speech and the accompanying
presence of certain manual gestures can serve to disambiguate between them
(Abuczki 2014).
5.4.1.3 Icons in Written Text

The analysis of written text also makes possible the definition or generation of icons
indicative of semantic properties. Approaches used in this context generally operate
at the level of words (sometimes pairs of words), and are often complemented
with databases containing information about e.g. semantic relationships, sentiment
attributes, social/geographical structures and tendencies, etc.
Examples of such approaches include a recent analysis that studied the correlation between semantic labels and specific word occurrences in Tumblr URLs as
indicated by sentiment lexicons across five languages (Vogel 2013). Although most
semantic labels studied in that work were not amenable to such correlation, when
it came to content that was deemed likely to be a source of concern for users who
would choose implement filters based on offensiveness, a direct correlation was
observed with words reflecting negative sentiment. In another study, it was shown
that the use of strongly emotional expressions, as determined from an emotion
lexicon, could be correlated with the relative popularity of content in social media
(Trung et al. 2013).
While both of these examples integrate more than one source of information
(i.e., semantic databases besides word occurrences), they are iconic approaches in
because they make decisions based on primary features of word usage. It is also the
case that the semantic databases themselves are static in the sense that their content
does not change as a function of content; hence, the result of any specific database
lookup can be seen as a direct mapping from word to label that is independent of
the text under analysis.
5.4.1.4 Icons in Multimodal Feedback

Iconic approaches can become increasingly powerful when events occurring in
more than a single modality are taken into consideration, as no single modality
is in general sufficient for the detailed characterization of high-level cognitive
phenomena. In such cases, depending on the goals with respect to the application
area, it is usually possible to distinguish between a primary modality and a set of
auxiliary modalities.
For example, in the application area of emotion recognition in speech, both
behavioral and communicative features derived from e.g. facial expressions, head
movements, as well as bodily and hand gestures can complement acoustic features
derived from speech to provide a stronger support for semantic analysis. In a set
of recent studies, Navarretta demonstrated that the relatively co-located appearance
66
of such multimodal iconic features in audio-visual streams of dyadic conversations

shows significant correlations (Navarretta 2014a,b,c). The fact that some of these
dimensions were fairly coarse-grained but could nevertheless contribute to improvements in classification shows that a fusion of modalities can serve as a viable basis
for iconic approaches to speech processing.
In the research area of CogInfoCom channels, iconic feedback has been applied
to force feedback through vibrotactile stimuli (Galambos 2012), audio-based tactile
feedback (Csapo and Baranyi 2012a, 2011), audio-based navigation feedback
(Watabe and Niitsuma 2013) and audio feedback enhanced virtual oversketching (Csapo et al. 2013). While most of these approaches combine iconic and
higher-level, contextual and pattern-based forms of mapping, icons are crucial
in simplifying the conceptual mapping between signal and meaning whenever
possible. For example, when mapping several dimensions to sensory signals at the
same timesuch as in the case of mapping tactile softness, roughness, stickiness
and temperature to auditory signalsthe use of iconic representations for auditory
softness and roughness significantly reduces complexity and improves interpretability (Csapo and Baranyi 2012a, 2011).
Multi-modal icons are also indispensable in the various domains of sociocognitive ICT, including social signal processing. In some cases, they are combined
from temporally distributed moments, possibly in non-linear ways, leading to
solutions that are primarily context-based. In other cases, multi-modal data related
to a single event is extracted or displayed in order to understand or control cognitive
processes. In a recent example, Leung et al. developed an audio-visual application
that uses a multimodal iconic display support the learning of pronounciation for
language students on mobile devices (Leung et al. 2013). Tanaka et al. also use a set
of multimodal icons in their NOCOA application for the development of cognitive
skills in autistic patients (Tanaka et al. 2012).
5.4.2 Context-Based Approaches and Applications

Context-based approaches focus on aggregate relationships between the profile of a
communication signal and semantic meaning. Solutions in this category generally
combine icons from various points in time and from various spatio-temporal
resolutions in order to select or synthesize a specific semantic interpretation. It has
generally become clear that this is a natural option in various domains, as more often
than not the formulation of one-to-one relationships between icons and semantic
labels is difficult, if not impossible.
Often, context-based approaches are complemented with machine learning
techniques in one of two ways: icons are either concatenated and then processed,
or separate decisions are made based on individual icons at the lower, iconic level,
and the decisions are then combined into a final decision. It is important to note
that at the same time, the use of machine learning does not in itself mean that the
overall approach is context based. For instance, if machine learning is used to find
67
very simple correlations between structure and meaning, which can later form the
basis of the definition of iconic relationships, then the resulting approach is icon
based. If instead an algorithm is applied to uncovering a set of highly intertwined
relationships between temporally distributed instances icons, potentially at different
levels of granularity, then the resulting approach is context based. As the level of
complexity at which cognitive phenomena are analyzed increases, the latter case
can be expected to become more common; nevertheless, this cannot be regarded as
a general rule.
5.4.2.1 Context in Verbal Signals
One aspect of human dialogue which exemplifies well the importance of context is
the notion of entrainment, which can be explained as the propensity of conversational partners to behave like one another, or as a natural inclination of people
who interact with each other to develop similar, matching, or synchronized forms of
behavior (Benus et al. 2014a,b, 2012). Entrainment is a high-dimensional concept
that can be analyzed from the perspective of temporal changes in any of a number
of lower-level features, including filled pauses, conversational fillers, patterns of
turn-taking, linguistic styleand can even body postures and gestures. In a detailed
investigation, Benus et al. have shown that in many cases, the result of supreme
court case hearingsboth in terms of polarity of decision and in terms of level
of agreement between judgescan be predicted based on features of entrainment
between lawyers and judges (Benus et al. 2014a). While several of the measures
used are local in terms of the length of speech they characterize, many of the same
measurements, aimed at detecting changes in time contribute to the final decision.
Past research suggests that time is essential: humans react highly positively to
communication patterns which change to match their own communication style
through time (and even more positively than if they match to begin with) (Nass
and Yen 2010).
In the area of social signal processing, a recent work by Cheng et al. provides
further examples in which machine learning based aggregations of low-level icons
of voice quality are used to support inference on the popularity of videolecture
content (as demonstrated in the paper, the approach can be combined with data
from other modalities for increased accuracy) (Cheng et al. 2014).
Contextual approaches can be used to effect in other complex dimensions as
well, such as in emotional analysis. In a recent contribution, Galanis et al. applied
a support vector machine based learning approach on a set of over 1500 acoustic
icons defined at various levels of temporal granularity. These features were learned
in conjunction with a small number of contextual parameters such as the gender
and role of the speaker (Galanis et al. 2013). Contextuality, however, in the sense of
the word specified earlier, pervades the entire decision process applied in the study:
a number of low-level representations are combined using machine learning into a
single decision. In another example, Origlia et al. introduce context into a syllablebased emotion tracking algorithm by adding features from neighboring syllables,
and demonstrate that this approach improves system performance (Origlia et al.
2014).
68
Generally speaking, the Bayesian framework of learning can also be very supportive of contextual solutions when the goal is to introduce contextual information
into the decision making process by influencing maximum a posteriori estimates.
For example, Dropuljic and his colleagues applied Gaussian mixture model based
a priori knowledge to emotional classification, both with respect to the distribution
of the training set, and with respect to individual valence-arousal categories within
the training set (Dropuljic et al. 2013). In both cases, the a priori knowledge used to
calibrate categorization results was based on an aggregate characterization of a large
number of independent measurements based on structural units (i.e., the training set
of the estimator) that were defined in the same way as the texts under analysis (in
any other case, they could not have been used to effect within the training set).
5.4.2.2 Context in Written Text

While in the case of spoken dialogue, iconic elements can often reveal important
background information, relying on word usage alone, without any acoustic or
gestural support in the case of written text can lead to highly ambiguous interpretations. As highlighted in Puszta et al. (2013), the analysis of text data increasingly
proceeds through sampled network traffic (this is natural as local storage loses
relevance in distributed cognitive networks), hence documents of often incomplete
and are not available for batched processing. These characteristics further reduce
the applicability of purely iconic approaches.
In the analysis of activity on social networks, information on users (including
e.g. number of followers, number of other users influenced, number of other
users providing influence) is often used in conjunction with semantic-level and
geographical analysis. Plovics et al. successfully applied such features to the goal
of predicting the success of individual Twitter messages in terms of the number of
expected re-tweets (Palovics et al. 2013). Other sources of contextual information in
this application area include regional correlations of language usage as well as intralanguage word usage frequencies, as demonstrated by Kondor et al. (2013); or the
relevance of time of day and location in the filtering of false positives in keyword
frequency based approaches (Matuszka et al. 2013). Patterns of diffusion can be
taken into consideration in terms of the rate at which the number of reshares of a
Twitter message reaches peak, and the rate at which the number of reshares grows
until it peaks (Trung et al. 2013). Finally, it has been shown that the analysis of
crowd opinions is increasingly becoming possible through the use of unigram and
bigram-defined features together with e.g. sentiment and acronym databases and
combined through machine learning (Hangya and Farkas 2013).
69
5.4.2.3 Context in Multimodal Feedback

In cases where purely iconic approaches are insufficient for the filtering out or
conversely, for the generation of relevant features, context-based solutions are often
useful.
In application areas relevant to socio-cognitive ICT, it is often the case that
a number of modalities can contribute to improving performanceeven if their
role seems less than well understood, and even counter-intuitive. Recent examples
are provided in the aforementioned contribution on the prediction of lecture
ratings (Cheng et al. 2014). Also related to the recognition of emotions as hidden
cognitive states, Berthelon et al. have convincingly argued the need to take into
consideration normative aspects individual personality (i.e. contextual aspects with
respect to the signal taken at face value) when detecting emotions (Berthelon
and Sander 2013a,b). However, when modality is interpreted from the viewpoint
of artificial capabilitiese.g. in terms of the modes of interaction provided to
users by devices, and in terms of the network protocols used for communication,
further new perspectives emerge.3 In a recent example, Fekete and Csorba have
developed a Markov model based approach for contextually informed energy saving
by dynamically allocating the most appropriate transfer media (i.e. WiFi, Bluetooth,
2G EDGE, 3G, and LTE) for specific mobile applications (Fekete and Csorba 2014).
As will be described later in Chap. 9, contextual aspects of interaction are also
essential in the design of CogInfoCom channels as well, especially when no direct
representation is available about a concept in the modality that is used by the channel
(either because it is normally perceived in a different sensory modality, or because
it is not usually encountered in any sensory modality). In such cases, the spatial
and/or temporal context in which the concept is normally encountered, or presented
to users can provide essential support for successful conceptual mappingsee also
Csapo and Baranyi (2012b,d).
5.4.3 Pattern-Based Approaches and Applications

In a broad review of the field of social signal processing, Vinciarelli et al. highlight
three issues or factors which they consider as important to take into consideration
for the future development of the field. These issues are (1) the tightening of
collaboration between social sciences and engineers, (2) implementing multi-cue,
multi-modal approaches, and (3) the use of real-world data rather than artificial
laboratory conditions (Vinciarelli et al. 2009). It can be argued, in fact, that all three
of these factors have strong relevance not just to social signal processing, but to
many other research directions under CogInfoCom. While the second issuerelated
For a possible unification of the many ways in which the concept of modality is interpreted, see
Sect. 8.1.3.
70
to multi-modal approachesis mostly relevant to the context-based approaches

described earlier, the first and third issues have strong pattern-based motivations.
Pattern-based techniques are different from both icon-based and context-based
approaches in that they focus on high-level temporal regularities as blueprints
for analysis and synthesis without directly considering parts of individual communication signals. This is the level of the various social sciences, in which
aspects contributing to the long-term usability and effectiveness of interaction and
communication are essential. Often, the question of how a broad group of users
can be convinced or motivated to participate and contribute to the success of an
application is equally important (i.e. as in crowdsourced or participatory sensing
applications).
The value of ICT-enhanced community support has been demonstrated through a
number of applications at the CogInfoCom fora, including a solution in which blind
users were given on-line navigation help based on the experience of other members
of the blind community (Balata et al. 2014), a protocol for mobile participatory
sensing in smart city service environments (Szabo et al. 2013), and well as a
gamified social platform to motivate physical exercise (Mulas et al. 2013). In the
application area of disaster management, Blum et al. have argued that existing
approaches are effectively blind in that they do not take into consideration
the different levels of capability in different contexts and at different levels of
the management chain (Blum et al. 2014). Thus, the authors have proposed a
structurally managed hierarchical social-technological architecture using which
real-time crisis management can be better coordinated through on-demand voice and
video streaming (Blum et al. 2014). In the training of professionals in maintaining
security and order, Spitters et al. have demonstrated the value of practicing the
interpretation of interpersonal stance relationships true virtually mediated study
environments (Spitters et al. 2013). Such applications show great promise not
only towards the training of professionals, but also towards arriving at a clearer
understanding of how humans communicate in realistic situations through extended
periods of time.
It can be expected that in an way that is analogous to the hierarchical link
between iconic and contextual approaches, increasingly rigorous and methodical
approaches will be developed for the effective transitioning from contextual to
pattern-based approaches. Machine learning will in a similar way be expected to be
at the center of this evolution. At the same time, increasingly elaborate (i.e. hybrid
in the sense of semi-automation) ways of capturing human knowledge (also referred
to as knowledge management) are expected to provide significant contributions
(Torok and Kosa 2015; Liao 2003; Bhatt 2001). Examples of such development
are already gaining momentum. Further ideas relevant to the evolution into patterns
of CogInfoCom channels are described in Chap. 11.
5.5 A Summary of Common Concepts
71
5.5 A Summary of Common Concepts

In this chapter, a brief overview was provided on the three recent research
directions of CogInfoCom channels, speechability and socio-cognitive ICT within
CogInfoCom. Through a discussion on application examples from the point of view
of use of icons, context and patterns of interaction, significant commonalities were
uncovered in the approach adopted within these research domains.
The general outline of these commonalities is as follows. When tackling new
problems relevant to human-ICT entanglement, it is often the case that an analysis of
the structural makeup of communication brings to light iconic elements that are, in
themselves, relevant to some form of semantic meaning that is of interest. However,
for the reason that the question of what constitutes an icon and what constitutes
context is a function of the cognitive capabilities of the (human) interpreter and
on the semantic meaning with respect to which the icon is to be considered, it is
suggested that methodologies for finding artificial icons could be just as valuable
as using human icons. Following the identification of icons that work at least in
some measurable proportion of cases, sets of icons can be combinedoften through
machine learninginto higher-level, contextually informed structures of meaning.
This step can be seen as a kind of sensitivity analysis aimed at deciding which icons
contribute most to useful aggregated conclusions. Although the techniques applied
here can originate from a broad spectrum of approaches, and are not at this point
amenable to broad generalization, several interesting ideas have been highlighted,
such as the explicit observation of changes in icons (as in the case of entrainment),
or the use of a priori, contextual information as a conditioning factor in Bayesian
models. Finally, at the level of social patterns, the challenge of finding aggregate
tendencies among contextual factors was highlighted as a key challenge of the
future.
Although this account leaves holes in a number of places which are to be filled in
through continued research, the chapter demonstrated that the various CogInfoComdriven research areas have much in common, and it is suggested that perhaps a
fruitful direction in the coming years would be to formulate a generic framework
for the extraction and use of icons, messages and patterns.
Chapter 6
Novel Research Initiatives
In this chapter, several new research initiatives are described which have been
proposed at various scientific fora on CogInfoCom since the first international
workshop on the field in 2010. These initiatives are discussed in a separate chapter
because they are still relatively young, and continued research can still be expected
to lead to significant developments in their scope and goals. Nevertheless, it is
important to mention them for the reason that all of them reflect the freshness of
perspective and interdisciplinary outlook that is promoted by CogInfoCom.
The chapter is structured as follows. In Sect. 6.1, a recently developed virtual
collaboration platform called VirCA is introduced that in many ways reflects the
conceptual underpinnings of CogInfoCom approaches. As a result of its strong
relevance to the field, the platform has been used in many studies, both as a
development tool and as a tool for investigation, and is extensively referred back
to in later sections of the chapter. Section 6.2 focuses on new perspectives created
by the VirCA platform and the CogInfoCom approach in general in the augmented
virtual reality mediated study of issues in neuroscience. Section 6.3 gives an
overview of how the VirCA platform has been used in ergonomics-related studies of
augmented virtual collaboration. Section 6.4 introduces the emerging research area
of ethologically-based CogInfoCom (EtoCom), in which design goals targeting the
communicational behaviors of devices are supported by concepts from ethology.
Section 6.5 provides insight into to the directions in industrial engineering that can
be supported by CogInfoCom research as evidenced by contributions appearing
at the CogInfoCom conference series. Finally, Sect. 6.6 provides an overview of
the new research initiative of mathability, which focuses on the modeling of
collaborative human-ICT capabilities towards mathematics.

DOI 10.1007/978-3-319-19608-4_6
73
74
6 Novel Research Initiatives
6.1 The Virtual Collaboration Arena

Before turning to the research initiatives that are the main focus of the chapter, this
section introduces the Virtual Collaboration Arena (VirCA) platform. VirCA will be
a recurrent theme in later parts of the chapter on account of the fact that it has been
used extensively to develop, conduct and evaluate a large variety of experiments.
VirCA (Virtual Collaboration Arena) is an extensible 3D augmented/virtual
collaboration platform developed at MTA SZTAKI (Institute for Computer Science
and Control, Hungarian Academy of Sciences)1 (Galambos et al. 2015, 2012b;
Galambos and Baranyi 2011b). The platform makes it possible to rapidly design,
develop and deploy distributed collaborative scenarios by combining 3D visualization capabilities with network communications and flexible affordances for
interactive interface development.
The basic building blocks of a VirCA-based application are a set of distributed,
loosely coupled components referred to as cyber devices (CDs). Each CD represents
either an object in the environment, or a capability with which one or more of those
objects can be extended. CDs can be used to manipulate scenes (i.e., add, delete,
modify and move objects), to acquire state information, or to generate/manage
notifications on various events (e.g. user actions, collisions, etc.) that can be
handled through callback functions. By handling objects and capabilities together
and allowing for their interconnection in networked scenarios, VirCA allows
developers to:
Actively share and collaboratively manipulate 3D content;
Collaborate in augmented ways by synchronizing the 3D content and processes
in VirCA with the real world;
Make use of flexible knowledge plug-and-play by integrating into VirCA-based
applications already existing hardware and software components (including those
supporting and implementing e.g. robot systems, sensors, speech technology,
machine vision, or semantic reasoning capabilities) developed at research facilities with different competence profiles;
Proactively participate in the development of future technologies by virtualizing technologies that are either inhibitively expensive, or are not as yet
commercially available. In this way, it is possible to investigate whether the
incorporation of a given technology into a real-world system would yield the
anticipated benefits without necessarily committing to its long-term use.
The networked extensibility of VirCA is provided through the Robotic Technology Component standard, often referred to as RT-Middleware or RTM for
short (RTC 2008), through its open source implementation OpenRTM-aist (AIST
RTM 2010; Ando et al. 2005). Although RTM was originally conceived for the
purpose of modular robot control, it appropriately serves the goals of VirCA
http://www.sztaki.mta.hu.
6.1 The Virtual Collaboration Arena
75
Fig. 6.1 Structure of the VirCA framework. The VirCA backend is composed of two server
applications: the RTM nameserver and the VirCA system editor. Each component (VirCA clients,
CDs and RCs) appears as a set of CORBA objects in the naming service. The System Editor is
responsible for handling data flow (RTM Data Port) and Remote Procedure Call (RTM Service
Port) connections according to the application scenario that is defined by users through a webbased front-end
development through underlying CORBA-based data-flow and Remote Procedure

Call mechanisms (Galambos and Baranyi 2011b). The plug-in interfaces for VirCA
are implemented as mutually provided and consumed RTM service ports on the core
component. The component structure of the VirCA platform is presented in Fig. 6.1.
6.1.1 Affordances for Multi-user Collaboration

A key distinguishing feature of VirCA as compared to other simulation platforms,
such as Gazebo (2012), is its support of multi-user scenarios. Since version 0.2,
VirCA has been capable of connecting multiple endpointssituated in locations
that are potentially far from each otherin order to encourage collaborative sharing
and manipulation of active 3D augmented/virtual content. In practical terms, this
means that multiple VirCA instances can be connected in a single-master multislave topology, while CDs can be attached to any of the VirCA nodes. In this setup,
participants share the same VR scene, in which their position and gaze direction are
represented by a symbolic avatar (Fig. 6.2).
76
Fig. 6.2 Two scenarios which demonstrate some of the interaction capabilities afforded by the
VirCA platform. The top subfigure shows a virtual representation of a PR2 household robot (the
original robot was developed for research purposes by Willow Garage). The bottom subfigure
shows a use-case in which multiple usersrepresented through virtual avatar headsare capable
of collaboratively manipulating and functionally controlling the virtual scenario
6.1.2 Synergies Supported by the VirCA Platform

VirCA can be seen as a platform that integrates motivations from several disciplines,
and as a result supports the development of new synergies. Specifically, it builds
on de facto standards to create a freely extensible collaborative environment
6.2 Augmented Virtual Reality Mediated Neuroscience Research
77
that merges together physical and virtual realities, thus providing an important
testbed environment for new concepts in cognitive networks and, by association,
CogInfoCom research.
The emphasis that VirCA lays on the flexible creation and merging together
of augmented/virtual capabilities is also very much in line with the point of view
adopted by CogInfoCom, and given the opportunity for users to increasingly apply
full-body interaction approaches in collaborative ways, it supports the kind of
experimental data collection that can lead to important new insights in a variety of
research areas. By significantly reducing the time and cost requirements associated
with the deployment of multi-disciplinary testbed environments, VirCA encourages
researchers to focus more on their research questions and less on the technical
details of implementation. This will be demonstrated through several examples in
the following sections.
6.2 Augmented Virtual Reality Mediated

Neuroscience Research
The CogInfoCom approach, together with tools like the VirCA system, can provide
rich opportunities for research in neuroscience from at least two directions:
CogInfoCom mediated neuroscientific analysis: the use of infocommunication
technologies that are tightly integrated with human capabilities to gain a better
understanding of the neuroscience-related aspects of both short-term interactions
and long-term human-ICT co-evolution;
Neuroscience inspired tools for synthesis: insight obtained through investigations characterized by the previous point can be applied to the development of
highly optimized applications in infocommunications, by allowing for the finetuning of functionalities to optimally suit the capabilities of individual users.
An important barrier in neuroscience research is posed by the lack of tools
to present and measure brain responses in ecologically valid situations (De Vos
and Debener 2014; Persa et al. 2014). At the same time, it has been noted that
neuroscience is traditionally multi-disciplinary, often involving the collaboration
of physicians, psychologists, engineers, data scientists and biologists (Persa et al.
2014). For this reason, recent collaborations surrounding the VirCA platform
initiated by Professor Valeria Csepe and her research grouphave been targeted at
the creation of a virtual laboratory framework for neurocognitive experiments that
can provide VR support to a network of researchers interested in creating, sharing
and performing experiments in collaborative ways (Persa et al. 2014; Torok et al.
2014; Koles et al. 2014). VirCA is well suited to these efforts given its support
for the deployment of reproducible 3D content in immersive environments, as well
as the remote collaboration among researchers with diverse backgrounds. Initial
requirements formulated with respect to the platform include:
78
Parametric tunability of experimental stimuli;

Possibility to create realistic (immersive) stimuli;
Capabilities for precise timing and logging.
The developed framework provides an XML schema for describing experiments
in terms of scheduled eventsi.e., groups of stimuli with respective starting
conditions. An Experiment Descriptor generally contains:
A definition of the 3D virtual scene in which the experiment is to be displayed;

A specification of stimuli used in the experiment;
An organization of stimuli into groups in order to specify events;
A schedule of events via appropriate starting conditions.
Such Experiment Descriptors are interpreted by an Experiment Controller, which

acts as a Cyber Device within VirCA, allowing it to have access to virtual
representations, external devices as well as user events defined with respect to those
representations and devices.
Preliminary studies have shown that the developed platform can be used in
conjunction with a variety of analysis tools, including both wired and wireless EEG
configurations, heart rate variability and mobile eye-tracking (Persa et al. 2014;
Torok et al. 2014; Keszei et al. 2014; Koles et al. 2014). Such results highlight a
new vision of the possibility to perform laboratory experiments based on a serviceoriented concept: a group of collaborators provide access to (potentially expensive)
laboratory measurement equipment, while a separate group of collaborators monitor
the experimental sessions through, and a third group of collaborators receive and
process the measurement data. All of these tasks are managed through the VirCA
system, allowing for significant improvements in the time and cost requirements
associated with the design, deployment and carrying out of laboratory experiments.
6.3 Ergonomics of Augmented Virtual Collaboration

As described earlier in various sections of Chap. 4, the increasing use of augmentative reality mediation is leading to new challenges with respect to multi-sensory
usability. These new challenges are in turn generating interesting opportunities for
experimentation in the fields of ergonomics and human factors, as highlighted by
several recent works from Professor Karoly Hercegfi, Anita Komlodi and their
research groups. This section describes a set of recent example studies which were
based on the VirCA platform (Keszei et al. 2014; Koles et al. 2014; Logo et al.
2014). Discussions suggest that the adaptation of approaches used in ergonomics
poses new requirements and challenges, but at the same time leads to important
synergies with CogInfoCom.
6.3 Ergonomics of Augmented Virtual Collaboration
79
6.3.1 Adaptation of Rasmussen and Reasons Framework

of Human Errors to VirCA
It has recently been suggested that the interpretation of the three levels of the
Rasmussen/Reason framework of errors (namely, those of skill-based, rule-based
and knowledge-based levels (Rasmussen 1982; Reason 1990)) can be extended to
the domain of augmented virtual collaboration, and thus applied specifically to
interaction and collaboration within the VirCA system (Keszei et al. 2014; Koles
et al. 2014; Logo et al. 2014). The three levels of error types can be summarized as
follows:
skill-based errors are characterized by slips in sensory-motor actions that lack
conscious control;
rule-based errors are characterized by lapses in the application of stored
procedures induced by experience or learned through training;
knowledge-based errors are characterized by mistakes in formulating plans of
action due to a misinterpretation of the situation.
It has been remarked in the above mentioned studies that rule-based errors are
of particular interest in augmented virtual collaboration systems such as VirCA.
This claim can be supported through more than a single consideration. In general,
it is unlikely that skill-based and knowledge-based errors inherent to VirCA could
present themselves, given that the platform handles both input and output modalities
in a device-agnostic way (hence, most skill-based errors would be attributable to
a sub-optimal selection of peripheral devices); and most knowledge-based errors
would be more relevant to the task at hand and the instructions given to users with
respect to that task rather than the system itself. For this reason, relative emphasis
was placed on the analysis of rule-based errors; i.e., errors that would most likely
emerge as a combination of the task and the platform used to accomplish it.
6.3.2 Types of Rule-Based Human Errors Identified

Using the evaluation criteria associated with rule-based errors as a starting point in
the context of a 3D environment based collaborative scheduling task, the following
classes of usability errors were identified:
Acts defying the laws of physics such as reaching through walls or placing
objects in midair;
Scanning and usage of off-topic objects such as searching for collaborators
avatars, or trying to manipulate objects that have no relation to the task at hand.
A common rule-based error uncovered in Logo et al. (2014) was when users
intended to grab an object and instead reached through it because they were
expecting some form of feedback that the object was in a state where it could be
80
selected. The key point here is not that unrealistic interactions are a problem, but
rather that it is important to address users expectations encapsulated in implicit
rules.
At the same time, it was noted that in some instances the errors listed above presented themselves not as a typical form of rule-based errors, but as a manifestation
of the curiosity of users and a lack of generalized rules, or more precisely, metaphors
of interaction in virtual environments. For example, many users were curious from
the outset to see whether they could reach through the walls of the scene, and
tried to do so as soon as the experiment began. This curiosity was independent
of the augmented/virtual reality platform used, and was much more relevant to the
uncertainty associated with the use of such platforms in general. Such errors might
aptly be referred to as lack-of-rule-based errors.
6.4 Ethologically Informed CogInfocom (EtoCom)

There are strong parallels between the evolution of biological systems and the
evolution of ICT. In much the same way that biological systems evolve, the development of technologies and devices in ICT can also be characterized by evolution:
a process that unfolds incrementally, both in terms of form factor and functionality,
based on after-the-fact selection mechanisms imposed by technological and market
trends. If this is accepted as a starting point, the question arises whether an
evolutionary process that targets behavior can also be interpreted, or developed for
ICT applications.
Ethology-based CogInfoCom (EtoCom) is a research direction initiated by
Professor Adam Miklosi and his research group that aims to describe, interpret and
design such a kind of behavioral evolution (Lakatos and Miklosi 2012; Szabo et al.
2012). Besides considering the evolution of form factors and functional capabilities
in ICT devices from an ethology-based perspective, EtoCom aims to do the same
with respect to mutual behaviors directed towards long-term co-evolution. For
example, it is well-known to ethologists that the communication between humans
and dogs has been mutually satisfying for thousands of years, even though dogs
and their owners are by no means co-equal and the patterns of interaction between
them are motivated by concepts such as attachment and separation anxiety instead
of specific, goal-oriented considerations (Lakatos and Miklosi 2012; Miklosi and
Soproni 2006; Viranyi et al. 2004; Miklosi et al. 2003; Topal et al. 1998). In this
context, an important research question is whether the concepts of attachment and
separation anxiety are suitable for use in engineering design; in other words, would
it serve as a benefit to human-ICT co-evolution if ICT devices were capable of
displaying signs of attachment or separation anxiety?
Based on this question, it is clear that EtoCom has strong links to and significant
overlaps with affective computing; however, rather than considering a primarily
emotion-based perspective, it focuses on a broader scope of evolutionary behaviors
with an outlook towards long-term entangled interactions. The abstraction of
81
cognitive capability which characterizes CogInfoCom research is also present in

EtoCom in a strongly embodied sense, allowing for any kind of heterogeneous
cognitive entity to evolve long-term behaviors. In the following subsection, a general
perspective is provided based on which certain features of EtoCom research can be
derived and explained. This is followed by a brief presentation of two application
examples.
6.4.1 CogInfoCom Tanglefaces and Engines

Without the loss of generality, the interaction scheme between cognitive entities can
be conceived as shown in Fig. 6.3. Both entities can be viewed as a complex web of
interrelated action-reaction processes (with both natural processes described at the
level of e.g. neurophysiology or psychology, and artificial processes designed as a
counterpart to natural ones). If environmental effects are disregarded, the entangled
interactions between the two entities form a closed loop.
Even in general terms, such a closed loop of interactions raises important
challenges from various perspectives. For example, from the point of view of
systems and control theory, questions of stability, controllability and observability
naturally arise. Further, it is important to consider various aspects of control
performance, such as how quickly the interactions are stabilized, or what degrees of
perturbation are caused by specific disturbances.
Fig. 6.3 Schematic view of closed-loop interaction between CogInfoCom entities. The meeting
point at which information is exchanged is referred to as the tangleface
82
From the perspective of entangled interactions between cognitive entities, additional levels of complexity are introduced into this picture due to the complex nature
of the entities themselves. Stability, for example, can be interpreted from a celllevelstructural, hormonalparametric or emotionalbehavioral point of view,
depending on the parts the entity most affected by communication and whether those
parts are natural, artificial or a combination of the two. As a result, the following
notions of CogInfoCom tanglefaces and engines can be introduced:
Definition 6.1. CogInfoCom tanglefaces (entangled interfaces) are the meeting
point through which natural and artificial sensors and actuators influence each other
over extended periods of time.
Definition 6.2. CogInfoCom engines are the modules which participate in determining the dynamic properties of communication between CogInfoCom entities.
From the broad perspective of entangled interaction, the dynamic properties
created through CogInfoCom engines can be derived from a variety of different
considerations. By way of example, it is possible to differentiate between the
following kinds of entangled interactions:
If the control performance of interaction is driven by psychological aspects of
mental processes (e.g., perception, memory, mental models, response times), the
entangled interaction can be referred to as psychology based.
If the control performance of interaction is driven by aspects of human capabilities and/or considerations of safety, efficiency and comfort, the entangled
interaction can be referred to as ergonomics based. It is possible to further
distinguish between interactions that are physical ergonomics based, cognitive
ergonomics based, cognitive ergonomics based, social ergonomics based, or
organizational ergonomics based.
If the control performance of interaction is driven by aspects of human and
animal behavior, the entangled interaction can be referred to as ethology-based.
Based on this interpretation, the role and purpose of ethology-based CogInfoCom
(EtoCom) becomes clearer. At the same time, it becomes evident that a wide
range of factors can be considered when developing applications supporting the
communication between cognitive entities.
6.4.2 Examples: The EtoMotor and Its Applications

This section provides an overview of two pilot applications in EtoCom that were
developed at MTA SZTAKI in collaboration with the research groups of Adam
Miklosi at the Eotvos Lorand University of Budapest, and Szilveszter Kovacs at the
University of Miskolc (the backgrounds of these institutions and research groups
in informatics, systems and control theory and ethology have provided strong
interdisciplinary support to the collaboration). The applications demonstrate how
83
Fig. 6.4 Schematic view of the relationship between the abstract EtoMotor and other components
providing links between cognitive entities in ethologically driven applications
the interface between a human user and an artificially cognitive system can be driven
by ethology based concepts relevant to behavioral expressions of emotion (Persa
et al. 2012; Kifor et al. 2011). Both applications make use of the same ethologically
based CogInfoCom engine referred to as the EtoMotor (a schematic view of the
relationships between relevant components is shown in Fig. 6.4). For this reason, the
EtoMotor is presented first, while details on the specific applications are provided
subsequently.
6.4.2.1 The EtoMotor

The CogInfoCom engine which drives the communicative behavior of the applications detailed later is referred to as the EtoMotor, and is represented by an
open-loop discrete-time state-space model that interprets external events through
emotional biases specified via the elementary states of happiness, despair, fear
and anger (Szabo et al. 2012). Although the core of the model is linear so as to
support traceability, it is also extended with a component for saturation (so as to
keep emotional responses in a defined interval) and a component for probabilistic
behavior.
A block diagram of the model is shown in Fig. 6.5. Initially, the model was
formulated to include the effects of both short-term and mid-term emotional states
as follows:
xk C 1 D Axk C Buk
xI k C 1 D Nxk C MxI k
yk D C.xk C xI k/
(6.1)
where x denotes the short-term, while xI denotes the intermediate level emotional
state vector formulated in terms of the basic emotions listed earlier (happiness,
despair, fear and anger). The model was then transformed into a state-space
representation with a single, aggregate state and was further complemented with
a long-term emotional state as follows:
xA k C 1 D AA xA k C BA uk
yk D C.ExA k C d/
(6.2)
84
Fig. 6.5 Block diagram of the EtoMotor proposed in Szabo et al. (2012)
where

x
xI

B
BA D
0

A 0
AA D
NM

ED II
xA D
and I is the identity matrix. Finally, the model was complemented with a probabilistic extension based on a set of variance values capable of modifying the
mean weights originally stored in matrix B. Based on its final form, the model is
capable of:
1. Associating with each salient external event a level of emotional arousal and
valence that is well suited to the current emotional state (this is the emotional
impact of the input stimulus);
2. Combining this emotional impact together with an aggregate of its own internal
states (short-term, mid-term and long-term) to update its state, and to produce
an output signal that is amenable to emotional interpretation (i.e., both the
internal state and output signal are interpreted as a combination of the four basic
emotions).
An example of how the emotional impact of external eventsinterpreted in the
context of interactions between a dog and its ownercan be specified depending on
the state at any given time is shown in Table 6.1 (for more on recent investigations
on modeling human-dog interactions and using them in ICT, readers are referred to
Sect. 6.4.2.3).
85
Table 6.1 Example of how specific environmental events can be mapped onto a dogs basic
emotional states in terms of arousal, valence and variance
Context
Greets Leaving
O
O
Happiness 3"1 3#1
Despair
3#1 3"1
Fear
2#2
Anger
1"3
Touched
by O
2"1
2#2
2#2
1#2
O init.
int.
2"2
2#1
O rej.
int.
2#1
2"1
O not
as exp.
1#2
1"2
1"2
1"3
Ambi.
sit.
2#1
1"2
1"2
1"2
Play
w. h.
2"1
2#1
2#1
2#1
Nov. Unfam. Facing

stim. env.
stranger
1#3
2#2
1"2 2"2
1"2
2"2
1"2
The following abbreviations are used in the table: O (owner), init. int. (initiates interaction),
rej. int. (rejects interaction), not as exp. (does not react as expected), ambi. sit. (ambivalent
social situation), play w. h. (plays with human), nov. stim. (novel environmental stimulus),
unfam. env. (unfamiliar environment)
6.4.2.2 The EtoPhone: An Abstract Emotional Display for Personal

Informatics Devices
The engine described above was complemented by Persa, Kifor and others with
an abstract Emotional Display Agent (EDA) in order to facilitate the development
of emotionally interpretable artificial (feedback) behaviors (Persa et al. 2012;
Kifor et al. 2011). The display was designed with the goal of being neither
anthropomorphic, nor zoomorphic, so as to mitigate usability risks posed by Moris
uncanny valley (Mori 1970). The end result an abstract parametric model based
on a spherical and a square element as shown in Fig. 6.6. In its default state, the
spherical element was given a half-transparent green color, and the square element
was set to spin around it at a pre-determined, relatively low speed. The scaling of
the elements with respect to each other was adjusted so that the edges of the squarelike element protruded slightly. It was demonstrated that a range of emotional states
could be expressed through the model by appropriately modifying various degrees
of freedomsuch as the spinning frequency, spinning torsion and tilt of the square,
the color and transparency of the sphere, and the position and scale of both elements
(Farago et al. 2011).
The utility of the EDA was demonstrated both in the VirCA system, and as
a separate application (the EtoPhone) developed for the iPhone (Kifor et al.
2011). Through a customization backendshown in Fig. 6.7EtoPhone allows
for flexible interfacing between the EtoMotor and a pre-specified set of behaviors,
part of which can be specified with respect to the EDA. Other behaviors include
the playback of appropriate audio clips and phone vibrations. Screenshots of the
iPhone app, demonstrating its on-device configuration options as well as its output
are shown in Fig. 6.8.
86
Fig. 6.6 Visual appearance of the Emotional Display Agent used both in VirCA and in the
EtoPhone application. The bottom part of the figure lists the degrees of freedom through which
the emotional display is capable of modifying its behavior
6.4.2.3 An Adaptation of Ainsworths Strange Situation Test

to Ethology and Robotics
Ainsworths Strange Situation Test is a behavioral model and an associated experiment designed to evaluate attachment in infants towards their mother (Ainsworth
and Bell 1970). Since its original appearance in the literature, the test has been
adapted to a variety of new situations; for example, Topl and his colleagues used it
87
Fig. 6.7 Screenshots of the advanced backend for configuring the EtoPhone application. For each
emotional category that is specified, a different set of emotional sub-ranges can be configured
(top left subfigure). Different behaviors can then be associated with each of those sub-ranges (top
right subfigure). The behaviors that are available are configured separately (bottom two subfigures).
Device-specific settings (such as what events are bound to what emotional states) are configured
separately after the application is deployed (Fig. 6.8)
to evaluate attachment behaviors in various dog breeds, and showed that differences
in individual results could be categorized along the scale of insecure to secure
attachment in the terminology used by Ainsworth (Topal et al. 1998). In this section,
a recent adaptation is described for the evaluation of physical-virtual agents through
their behavior in a VirCA-oriented experimental scenario (Vincze et al. 2012).
88
Fig. 6.8 Screenshots of the EtoPhone application running on an iPhone device. The first two
subfigures show the settings screen that allows users to bind input events with emotional states.
The remaining subfigures from top left to right, and top to bottom are static images of the EDA in
the happy, despairing, fearful and angry state
The Test Procedure

In a way similar to the original test, the authors of both later works cited above
distinguish between seven phases (Ainsworth originally described eight phases, but
the first one consisted merely of the mother and stranger bringing the infant into the
89
room, and the stranger leaving; a condition which was satisfied in the later works to
begin with). The phases were the following:
1. The owner (O) and the dog (D) were in the room, with O passive for the most
part, only beginning to play with D towards the end of the phase.
2. The stranger (S) entered the room, and started to play with D.
3. O unobtrusively leaves the room, leaving D and S alone. S tries to play with D.
4. O returns to the room, and S eventually leaves.
5. O leaves as well, leaving D alone.
6. S returns and tries to engage D.
7. O returns and S leaves. O tries to engage D.
Each phase lasted for 2 min (slight deviations from Ainsworths original study
were allowed in this aspect as well, as in the original test phases had variable lengths
ranging from 1 to 3 min).
Ethological Behaviors Based on Fuzzy Automaton and Fuzzy Rule Interpolation

In the most recent adaptation, the goal of Vincze, Kovacs et al. was to reproduce the
canine behaviors described earlier in artificial agents (Vincze et al. 2012; Kovcs
et al. 2009; Topal et al. 1998). Given that ethological results are generally formulated
verbally rather than through mathematical formulae, the authors constructed a
fuzzy automaton for the construction of behaviors based on verbal descriptions.
In a way similar to finite state automata, the automaton was specified through the
following six parameters (the difference compared to classical automata being that
the transition rules were fuzzy rather than crisp):
A finite set of multidimensional states: Q D fq1 ; q2 ; : : : qQ g

A finite set of input symbols: D fs1 ; : : : sS g
A (potentially fuzzy) starting state R
A finite set of output symbols: Z D fz1 ; : : : zZ g
A fuzzy state transition function: W Q Q ! 0; 1
An output function mapping a (fuzzy) state to an output: ! W Q ! Z
Some examples of the kinds of fuzzy rules introduced into the system include
(Kovcs et al. 2009):
If Owner is not in the room, Then Dog increasingly misses Owner;
If Owner-to-Dog distance is small, And Stranger-to-Dog distance is large Then
Dog anxiety level decreases;
If Owner not in room And Dog misses Owner very much, Then Dog going to
door very likely;
If Dog is anxious Then willingness to explore the room is low;
If the room is unknown Then willingness to explore the room is low.
Although fuzzy techniques are well suited for capturing functionality through
linguistic rules, in many cases their applicability is hindered by either a lack or
over-abundance in datapoints. In such cases, fuzzy rule interpolation can be used
90
Fig. 6.9 Fuzzy automaton based architecture for interpolated rule-based behaviors
as a technique to either compensate for, or increase the sparsity of the rule base. In
the current application, the fact that the exponential input space could have easily
led to intractability prompted the authors to apply a fuzzy rule interpolation based
method. The architecture of the applied model is shown in Fig. 6.9.
6.5 CogInfoCom-Aided Industrial Engineering
91
Fig. 6.10 Simulation of Ainsworths experiment adapted to human-dog interactions based on

fuzzy rule interpolation between observations made by ethologists
Interfacing the Model with VirCA

The fuzzy automaton, together with a fuzzy rule base constructed based on earlier
findings of Topal et al. (1998) and fuzzy rule interpolation based approaches,
were integrated into a simulator program running on Matlab. The program was
then connected to VirCA through the Robot Technology Middleware, enabling
the simulated behaviors to be implemented on Cyber Devices within the platform
(Fig. 6.10). It was suggested by the authors that the resulting VirCA capability
could be useful for helping include ethological perspectives in the design of virtual
avatars. Through the inter-disciplinary evaluation of resulting behaviors, the process
of knowledge transfer can be supported.

Several conference sessions and tracks, as well as journal special issues were
organized by Professors Bjorn and Wei Deng Solvang from Narvik University
College with the goal of supporting the transfer of ideas and concepts between
CogInfoCom and industrially oriented engineering sciences. Given that this goal can
be, and has been applied to a wide range of application areas, a detailed discussion
of proposed results lies outside of the scope of this section. However, some of the
key application areas that have been addressed are briefly highlighted in order to
guide readers interested in more specific results to the corresponding publications.
Wherever appropriate, common perspectives are also highlighted.
92
6.5.1 Design Issues Relevant to Cognitive Networks

As detailed earlier in Sect. 4.6, tendencies relevant to the expansion of cognitive
content space in modern networks are leading to the emerging concept of cognitive
networks. In this context, challenges include finding design methodologies that are
appropriate for communicating novel concepts, as well as designing architectural
solutions that are suitable for the accommodation of novel communication patterns.
With respect to content-related issues, contributions have dealt with applications
including:
flexible creation and manipulation of visual contente.g. a multi-layered shape
description model was proposed in Li et al. (2013), and a tool for interaction with
digital holography was proposed in Mikhaylov et al. (2013)
augmented search capabilities through BCI (Ousterhout and Dyrholm 2013) and
3D visualization (Koltay et al. 2013)
remote multi-user collaboration through 3D augmented-virtual objects (Galambos and Baranyi 2011b; Galambos et al. 2012b; Pontonnier et al. 2014; Duval
et al. 2013)
A variety of tools, analyses and solutions relevant to architectural challenges
behind cognitive networks have also been proposed at the various CogInfoCom fora.
Topics addressed include:
multipath communication with devices equipped with multiple addressable
sensors (Almasi 2013)
multi-component addressing schemes based on contextual information (Kalmar
et al. 2013)
fixed-bandwidth communication in ad hoc networks (Tapolcai et al. 2013)
issues of interference in body area networks (Sayrafian-Pour et al. 2013)
suitability of different network protocols for vehicle-to-vehicle and vehicle-toroadside unit communication in intelligent traffic systems (Jamsa et al. 2014)
6.5.2 Industrial Robotics and Production Management

As pointed out by several authors, CogInfoCom research has relevance to applications in industrial robotics and industrial production by supporting novel types
of inter-cognitive communication among designers, engineers, management and
industrial machines. In many cases, this communication occurs through brain-inthe-loop type interactions, in which the role of the human operator is to make
small, yet indispensable and thus extremely important contributions to otherwise
essentially automated processes at just the right times. Such brain-in-the-loop
communication is increasingly important not only in the operation, but also in
the design of complex industrial systems. The overarching point is that there are
certain decisions that can only be made by the human operator, designer, engineer
93
or stakeholderand even if the time required to make these decisions is 2 % of

the complete process (with 98 % of the process being automated), their functional
impact is crucial. It is important for industrial technologies to be able to merge
the intelligence and high-level decision capabilities of the human brain with such
processes in the most convenient, effective and ergonomically tractable way. At
the same time, the creation of new cognitive entities through the human actors
involved in the processes together with the ICT network surrounding them is equally
important. Only by addressing both of these challenges can the progress of longterm industrial goals can be most effectively supported.
One of the key research areas within this domain focuses on flexible robot
programming and control. Several authors have noted that most industrial systems
require some degree of reconfiguration, and that this can pose challenges especially
for small and medium-sized enterprises with interest in small series production.
To address these challenges, several approaches for using CogInfoCom channels in
human-robot communication, as well as for programming industrial robots based on
heterogeneous input sources (including CAD models and human motions captured
through 3D sensor systems) have been proposed (Aryania et al. 2012; Solvang and
Sziebig 2012; Thomessen and Kosicki 2011; Thomessen and Niitsuma 2013; Pieska
et al. 2014).
A further area of interest is the augmentation of human and/or institutional
capabilities for proactive innovation. Several important ideas were introduced and
summarized in recent works by Pieska, Ericson and others (Pieska et al. 2014;
Ericson et al. 2014).
6.5.3 Industrial Capabilities of VirCA

VirCA provides an extensible VR environment for users and developers implementing the paradigm of augmented virtuality by synchronizing physically existing
entities (e.g., robots, fixtures, machine tools, workpieces) with corresponding virtual
models. In this way, real objects can be added to a virtual environment, facilitating hardware-in-the-loop tests, simulations and virtual commissioning. At one
extremity of possible applications, tracking systems can be used to follow specific
objects (e.g., specific workpieces in manufacturing, unmanned aerial vehicles, etc.)
and periodically refresh their pose and state in the virtual scene. At the other
extremity, the interfacing of physical devices with VirCA can support applications
in which low-level gateways are implemented for vendor-specific modules, allowing
for increasingly sophisticated bidirectional data exchange between VirCA and thirdparty physical devices.
The primary motivation here is that certain tasks are more easily performed in
the physical world (even with the availability of virtual interactions), while others
are more easily performed through virtual representations. For example, it is timeconsuming and extremely difficult to implement real-world physical simulations
(especially of complex dynamic processes that are still difficult to virtualize) in
94
virtual worlds; thus it is best, whenever possible, to allow physical processes to

run their due course and to simply update their status in the virtual world from
time to time through suitably refined sensor measurements. On the other hand,
it is often useful to keep the number of physical measurements performed to a
bare minimum. Performing an excessive amount of such measurements can quickly
become a burden, as new and often costly sensors are deployed in increasing
numbers within the same, limited physical space. Thus, as far as detailed physical
measurements are concerned, it is best to obtain as much of them as possible through
computations derived from state records available in the virtual world. As a case
in point, it is especially trivialat least in the virtual worldto derive a large
number of conclusions from a limited set of joint state recordings in industrial
robot arms and in mobile robots. Such data is generally easily accessible through
the API of the robot equipment; and through a virtual reconstruction of changes
in geometric configurations due to past kinetic movements, it becomes possible to
measure (rather, compute) distances without having recourse to third-party sensor
measurements. For example, in industrial settings measuring distances between one
or more robot arms, workpieces and the operators within and near a robot cell is
an important part of guaranteeing safe operation (in terms of protecting the robots,
humans and workpieces from injury), and if such distances can be computed rather
than measured, this can yield significant practical advantages. In such cases of
virtual sensor measurements, and from a systems control perspective, the virtual
world can be seen as a kind of general observer that is capable of reconstructing
relatively detailed information from a limited number of physical measurements
i.e. the minimum that is required for the reconstruction of the overall state of the
system under observation.
It should be noted, however, that despite these theoretical possibilities, the
implementation of practically viable augmented virtual sensing-decision-actuation
triplets in industry is often easier said than done. The realization of sensing,
decision and actuation is conventionally based on closed-domain subsystems that
are highly integrated (i.e., characterized by a large number of mutual dependencies
with other system components). In contrast, real-life applications (especially in
industrial and military domains) often lead to a preference for heterogeneous
solutions. Over a certain degree of complexity, such an integrated system can
become unwieldy and thus hard to modify or enhance with new features. As a result,
synergies created by recent technological developments are leading to the logical
and/or spatial separation of information gathering (sensing), decision making and
intervention-related (actuation) modules and subsystems. One substantial benefit of
this separation is that the concrete source of information (including the operating
principle of the sensors, method of signal processing, etc.) becomes irrelevant from
the point of view of control and actuation. To support this kind of logico-spatial
separation, VirCA provides a high-level information pool in which information
relevant to the systems state can be collected, maintained and possibly predicted.
A possible list of entities (taken from the context of industrial robotics) that can
connect to this information pool may include:
6.6 Mathability
95
Data sources (generic sensors), which help maintain the validity of the representation, e.g. in terms of configuration of manipulators, position of workpieces,
grasping forces, welding currents, and positions of human workers within the
robots workspace;
Orchestration modules (high-level process controllers), which are responsible
for macro-level process control, e.g. with respect to production scheduling and
cell-level PLCs;
Task-level controllers, which are responsible for specific tasks such as path
planning for autonomous forklifts, SLAM functionalities, etc.;
Actuator-level controllers, which include all manner of components capable
of modifying physical states in the environment. Elementary examples include
servo motion controllers, temperature controllers, etc.
It is important to note that depending on the concrete application, not every
level will necessarily be present. In some cases, low-level control cannot be realized
according this pattern, due to particular design considerations, e.g., when the applied
communication technology does not allow for sufficiently fast sampling, or a robot
controller does not offer low-level interfaces. The system designer has to choose the
abstraction level that is appropriate from the technical, theoretical and economical
point of view.
VirCA implements this idea of networked sensing, decision and actuation at a
proof-of-concept maturity level. The VR engine and the corresponding databases
represent the portion of reality that is relevant with respect to the given process. Each
connecting sensor module pushes the gathered information into the information
pool while the control modules work with the high-level, source independent
representation that is accessible in VirCA.
6.6 Mathability
Mathability was initiated by Professor Attila Gilanyi, and defined at the CogInfoCom 2013 conference as a research direction that investigates artificial and natural
[as well as combined] cognitive capabilities relevant to mathematics ranging from
low-level arithmetic operations to high-level symbolic reasoning (Baranyi and
Gilanyi 2013; Borus and Gilanyi 2013; Torok et al. 2013). Importantly, in much
the same way that the focus of speechability (described earlier in Sect. 5.2) extends
further than speech capabilities, mathability focuses not only on human and artificial
mathematical capabilities, but also on the mathematical capabilities of humans and
ICT together with the heterogeneous cognitive entities they give rise to. Thus, one
of the key questions behind mathability is whether mathematical capability can
be understoodand abstractions of it createdso as to facilitate the design and
strengthening of mathematical capabilities in emergent cognitive entities.
An important motivation behind mathability lies in the observation that in
the past decades, even the notion of what qualifies as a proper solution to a
96
mathematical problem has changed. While a few decades ago, only analytically
closed formulae would have been accepted, today it is not uncommon for so-called
granular (i.e. numerical, sub-symbolic) formulations to be seen as equally useful
and acceptable. Although initially contested by many, by today this tendency is
seen as natural as increasingly complex problems need to be addressed in both the
engineering and social sciences (as the importance of this paradigm shift cannot be
stressed enough, further discussions can be found in Sect. 6.6.1). However, despite
these changes, the human capacity to think in numerical terms remains limited, and
analytically tractable deductive methodologies are preferred. The challenge, then, is
how to bridge between these two worlds of analytical and numerical approaches in
a way that is suitable for the problems at hand to be tackled together by cognitive
entities (Baranyi and Gilanyi 2013; Torok et al. 2013). Ideally, ICT devices to be
able to guide users through solutions to mathematical problems, letting users know
how they are able to help along the way (the outlines as well as some characteristics
of such a process are shown in Fig. 6.11). If such processes were possible, humans
involved would be able to make analytical decisions on what deduction route to
further pursue whenever necessary, while the ICT components involved would be
focused on applying the numerical tools that are most suited to the given context.
A simple example in which the solution sets to functional equations are found
through human intervention is provided in Sect. 6.6.2.
Fig. 6.11 As entangled processes within and between human-artificial cognitive entities are
applied to problems in mathematics, the relative strengths of the entities involved are expected
to be relied upon at different times and for different subtasks. An important consequence is that
the problem spaces that can be handled in such a way are qualitatively different from problems
involving only analytical or only numerical approaches; i.e. instead of merely strengthening
the analytical or numerical capabilities of the entities involved, such processes create a new
dimensionindicated in the figure as a rectangular areathat is created through the combination
of such capabilities
6.6 Mathability
97
Another aspect relevant to mathabilityand also specifically to the motivation

considered just nowis the question of how mathematical capabilities (now treated
in an abstract sense that applies not only to humans) can be graded. To consider
a simple example: if someone learns that a schoolchild finishing the 5th grade
received a C in mathematics, he or she will pretty much understand the level at
which the child can be spoken to about mathematics. Importantly, this level is not
(only) defined by any particular subject matter, but much more so by the capacity
of the child to understand certain concepts, and his or her the lack of capacity to
understand others. The question is whether such a grading system could be created
for ICT devices, and ultimately, to emergent cognitive entities as well in a way that is
useful in practical scenarios. Based on such a grading system, it would be possible
to understand how complex tangleactions could be directed at solving real-world
problems using mathematically sound approaches.
6.6.1 Granular and Analytical Representations

In the past few decades, various new representations (so-called granular representations) of dynamic models have emerged in the natural and technical sciences.
The origins of this paradigm shift can be traced back to the famous speech given
by Hilbert in Paris, in 1900. Hilbert listed 23 problems which at the time had not
yet been solved, and which he believed would prove to be the most challenging
problems in the twentieth century (Gray 2000). In addition to listing these problems,
Hilbert also made public his own conjectures about what the solution to these
problems would be (Gray 2000). Hilberts 13th conjecture stated that mathematicians would find examples of continuous multivariate functions which cannot be
decomposed as the finite superposition of lower-dimensional continuous functions
(i.e. functions having a smaller number of variables). In 1957, Arnold proved that
on the contrary, all multivariate functions can be decomposed in such a way (Arnold
1957). In the same year, Kolmogorov formulated his general representation theorem,
along with a constructive proof, which allows for the decomposition of multivariate
functions into a superposition of single-variable functions (Kolmogorov 1957) (see
also Sprecher 1965; Lorentz 1966). Thus, Kolmogorovs theorem essentially proved
the existence of universal approximators. Based on these early results, researchers
were able to prove, starting from the 1980s, that universal approximators can be
created using computational tools such as biologically inspired neural networks,
non-binary logic inspired fuzzy systems and evolution-inspired genetic algorithms
(Castro 1995; Cybenko 1989). As a result of such breakthroughs, such granular
representations are widely applied to model the behavior of dynamic systems in
modern systems theory.
The identification models listed above (i.e., neural networks, fuzzy systems and
genetic algorithms) are referred to as granular representations because instead of
representing their functionality in a closed, analytical form (i.e., in the form of
mathematical formulae), they use distributed representations in which individual
98
components have no meaning in and of themselves. Although it is difficult to

analyze the mathematical properties of granular representations using formally
rigorous tools, it has been shown that granular representations are powerful in
capturing the functionality of highly parametric systems with complex dynamics.
Granular representations play an increasingly significant role in biologically
inspired engineering systems because:
The motivation behind granular forms often stems from natural sciences (e.g.,
neural networks are inspired by biology), and Kolmogorov proved the applicability of this class of representations as universal approximators. These
circumstances render granular representations well-suited to solve biologicallyinspired problems.
In contrast, the closed, analytical forms commonly used in the engineering
fields are generally not available in biologically inspired engineering models.
The sheer complexity of biological systems (in terms of number of inputs,
number of outputs and number of functionalities in the general case), and our
relative lack of knowledge about their general mechanisms make it difficult
to use closed, analytical forms (while much detail is known about individual
subsystems, but the overall, global mechanisms in biological systems are still not
well understood).
Although it is often difficult, if not impossible to compare two granular representations to each other, and to convert between different granular representations,
the advantages listed above far outweigh these disadvantages in many engineering
applications. The challenge, however, is how to create representations that are
granular, but at the same time still retain some of the properties which will allow
for formal reasoning to at least some degree. More specifically from the point
of view of mathability the ability to bridge between the analytical and numerical
worlds is important for the development and strengthening of emergent, entangled
mathematical capabilities.
6.6.2 Example Application: Human-Aided Resolution

of Linear Functional Equations
The example presented here is based on recent work by Borus and Gilanyi (2013).
The foundations of these works were laid down in Czirbusz (2012), Hzy (2004),
Bajk and Ples (2009), and Gilnyi (1998). The example focuses on finding
solutions to systems of a specific class of functional equations (i.e. equations in
which the variables are functions). The problem domain has relevance in a variety
of fields including economics, however, it is shown through relevant theorems that
a unique solution does not always exist.
6.6 Mathability
99
6.6.2.1 Problem Description

In the example, the following class of systems of functional equations is considered:
nC1
X
fi .pi x C qi y/ D 0
.x; y 2 X/
(6.3)
iD0
where n is a positive integer, p0 ; : : : ; pnC1 and q0 ; : : : ; qnC1 are rational numbers,

X; Y are linear spaces over the rationals and f0 ; : : : ; fnC1 W X ! Y are unknown
functions.
It is easy to see that several well-known functional equations belong to this class.
For example, writing n D 1, p0 D 1, q0 D 1, p1 D 1, q1 D 0, p2 D 0, q2 D 1 and
f2 D f1 D f0 in (6.3), the well-known Cauchy equation is obtained:
f .x C y/ D f .x/ C f .y/
.x; y 2 X/:
It is obvious that the so-called Pexider equation, as well as the square-norm equation
(also known as the Jordanvon Neumann, or parallelogram equation) and the Jensen
equation are also special cases:
f .x C y/ D g.x/ C h.y/
.x; y 2 X/:
f .x C y/ C f .x y/ D 2f .x/ C 2f .y/

f
xCy
2

D
f .x/ C f .y/
2
.x; y 2 X/I
.x; y 2 X/
The functional equations above have several applications in various branches of

sciences. Concerning their usage in economics, readers are referred to Eichhorn
(1978).
6.6.2.2 Key Concepts and Theorems

Before presenting the main theorems which serve as a basis for the solution to the
above presented problem, it is necessary to introduce the concepts of polynomial
and monomial functions.
Let X and Y be linear normed spaces, and let f W X ! Y be a function. Further,
let us define:
1y f .x/ D f .x C y/ f .x/
.x; y 2 X/
100
such that, for any positive integer n,

nC1
f .x/ D 1y ny f .x/
y
.x; y 2 X/:
It can be proved by induction that for an arbitrary positive integer n:

!
n
X
n
nk n
y f .x/ D
.1/
f .x C ky/
.x; y 2 X/:
k
kD0
(6.4)
A function f W X ! Y is called a polynomial function of degree n if it satisfies

the functional equation:
f .x/ D 0
nC1
y
.x; y 2 X/:
(6.5)
A function f W X ! Y is said to be a monomial function of degree n if it satisfies

the functional equation:
ny f .x/ nf .y/ D 0
.x; y 2 X/:
(6.6)
It is easy to prove that whenever X D Y D R, the classical polynomials of the

form
f .x/ D an xn C C a0
.x 2 R/
(6.7)
with real constants a0 ; : : : ; an satisfy Eq. (6.5). Similarly, with a c 2 R, the classical
monomial
f .x/ D cxn
.x 2 R/
(6.8)
satisfies (6.6).
It is also well-known, however, that both classes of functional equations
[i.e. those of Eqs. (6.5) and (6.6)] have solutions that are different from (6.7)
and (6.8): hence, the concepts of polynomial and monomial functions are a
generalization of the concepts of polynomials and monomials. Further, it can
be shown based on Eq. (6.4) that the polynomial equations (6.5) and the monomial
equations (6.6) are in turn special cases of the class of equations given in (6.3). This
means that systems of equations of the type introduced in Eq. (6.3) potentially have
polynomial and/or monomial solutions.
A necessary and sufficient condition for all solutions to a functional equation
of form (6.3) to be available as a sum of monomials can be obtained from two of
Szkelyhidis theorems, which are re-iterated here:
Theorem 6.1 (L. Szkelyhidi). Let X and Y be linear spaces over the field of the
rationals, p0 ; : : : ; pnC1 , q0 ; : : : ; qnC1 be rational numbers and let
.i/
Mk W X ! Y
6.6 Mathability
101
be monomial functions of degree k for i D 0; : : : ; n C 1 and k D 0; : : : ; n. The

functions f0 ; : : : ; fnC1 W X ! Y,
fi .x/ D
n
X
.i/
Mk .x/
.x 2 X; i D 0; : : : ; n C 1/
(6.9)
kD0
solve the functional equation

nC1
X
.x; y 2 X/
iD0
.i/
if and only if the monomial functions Mk W X ! Y above fulfill the equations

nC1
X
j kj
.i/
pi qi Mk .x/ D 0
(6.10)
iD0
for all x 2 X, k D 0; : : : ; n and j D 0; : : : ; k. (We use the convention 00 D 1 here.)

Theorem 6.2 (L. Szkelyhidi). Let X and Y be linear spaces over the field of the
rationals and let p0 ; : : : ; pnC1 , q0 ; : : : ; qnC1 be rational numbers which satisfy
pi qj 6D pj qi
.i; j D 0; : : : ; n C 1; i 6D j/:
(6.11)
The functions f0 ; : : : ; fnC1 W X ! Y solve the functional equation

nC1
X
.x; y 2 X/
iD0
if and only if they have the form

fi .x/ D
n
X
.i/
Mk .x/
.x 2 X; i D 0; : : : ; n C 1/
(6.12)
kD0
where
.i/
Mk W X ! Y;
.i D 0; : : : ; n C 1; k D 0; : : : ; n/
are monomial functions of degree k fulfilling the equations

nC1
X
j kj
.i/
pi qi Mk .x/ D 0
(6.13)
iD0
for all x 2 X, k D 0; : : : ; n and j D 0; : : : ; k. (We use the convention 00 D 1 here.)
102
The interpretation of the two theorems is that if the condition given in Eq. (6.11)
is satisfied, then all solutions to the functional equation can be expressed as sums
of monomial terms. However, if the condition is not satisfied, then the exclusivity
of the solutions expressed as sums of monomial terms is not guaranteed. This has
consequences when different equations in a system of equations have different sets
of solutions and appropriate intersections have to found, as demonstrated below.
6.6.2.3 Finding Solutions to Sets of Functional Equations

A package was developed by Gilnyi and his colleagues for solving such systems
of equations in the Maple environment. According to Theorem 6.2, in cases where
the inequalities in (6.11) are satisfied, the program determines the complete set of
solutions of the equation considered. If the conditions above are not valid then
(based on Theorem 6.1) the program computes those solutions of the equation
investigated, which can be written as a sum of monomial functions.
The program can easily handle systems of equations in which a subset of the
equations does not satisfy the condition given in Eq. (6.11). For example, let us
consider the following system of equations:
f .x C y/ D f .x/ C f .y/
f .x/ D g.x/
.x; y 2 X/
.x 2 X/:
(6.14)
(6.15)
Here the coefficients in the first equation satisfy (6.11), while those of the second
one do not. If the program received as input the second equation alone, it would
yield only monomial solutions of degree 0 (i.e. constant solutions). However, just
by looking at the system of equations, even a human reader could easily tell that the
second equation imposes no real restriction on the general solution: any function
that satisfies the first equation will also satisfy the second one. Thus, the developed
program will signal to the user that there is an ambiguity, and will allow the user to
choose the solution that is to his or her liking.
Part III
CogInfoCom Channels
CogInfoCom channels have been proposed as a generic theoretical framework that

supports the design of multi-sensory communication patterns between cognitive
entities. This part provides an overview of the considerations behind the framework,
as well as of the framework itself and the various tools that have been developed to
support its use in application-oriented research and development.
In Chap. 7, the motivations behind CogInfoCom channels are discussed, and as
a first step towards their formal development, a unified view is provided of the
syntactic elements that have been used in interfaces designed for various (human)
sensory systems. It is demonstrated that not only are these syntactic elements
analogous to each other across sensory systems (and therefore amenable to a
conceptual unification), but that they can also be extended by analogy to the artificial
modalities of any kind of cognitive entity in general.
In Chap. 8, a holistic view of CogInfoCom channels is provided that unifies
syntactic and semantic aspects. CogInfoCom channels are defined as a structured
unification of icons and messages with perceptual interpretations of meaning. This
definition uncovers previously overlooked features of the concept of modality. It
is demonstrated that the parametric structure behind CogInfoCom channels can be
represented in a canonical tensor-algebraic form. It is argued that this representation
can be advantageous in that it allows researchers working on similar problems to
share and reproduce each others results. Several tools developed in later chapters
are as a result based on this representation.
In Chap. 9, various techniques are developed for the mapping of semantic
meaning onto CogInfoCom channels. The mapping task is formulated in terms of
finding appropriate generation parameter types and specifying relationships among
their values based on attribute features of the semantic concepts to be represented.
In this way, the chapter develops a set of tools that can support the development of
human interpretable CogInfoCom channels.
In Chap. 10, possibilities for the customization of CogInfoCom channels are
investigated. It is argued that this is important due to the specificities of the
CogInfoCom modality that is used, especially with respect to the input device and
the noise level characteristic of the transfer medium. However, the task of creating
104
III
CogInfoCom Channels
such a model is rendered difficult due to the fact that the function which links all
possible combinations of generation parameter values to perceptual qualities is both
difficult to compute and also practically impossible to invert. One possible solution
to this challenge is to apply a tuning model that allows users to interactively explore
the parametric space used to generate CogInfoCom messages. The chapter proposes
the spiral discovery method (SDM)a tuning model that fulfills these requirements
and also empirically aims to support flexibility and interpretability.
Finally, in Chap. 11, the possible evolution of CogInfoCom channels is considered based on various dynamical perspectives of biological and human communication. As this subject area has more relevance to the long-term co-existence of
humans and ICT than to episodic interactions, its details are relatively less well
understood. Nevertheless, the chapter aims to highlight some of the more important
challenges to be addressed by future research, and introduces some concepts which
may help guide future work.
Chapter 7
Structure of Sensory Signals: Icons

and Messages
In this chapter, the motivations behind CogInfoCom channels are discussed. As

a first step towards their formal development, a unified view is provided of the
structural elements that have in the past been used in interfaces designed for various
(human) sensory systems. It is demonstrated that not only are these structural
elements analogous to each other, and therefore amenable to conceptual unification,
but that their interpretation can also be extended to the artificial modalities of any
kind of cognitive entity in general.
The chapter is structured as follows. Section 7.1 discusses the motivations behind
the development of the unified concept of CogInfoCom channels. Section 7.2 lists
some of the difficulties that have in the past been associated with the unified treatment of communication across sensory systems. In Sect. 7.3, a broad description is
provided of iconic and message-like forms of communication, and how they can be
interpreted across sensory systems irrespective of the kind of cognitive entity that
is being considered. The concepts developed in the chapter are illuminated through
examples.
7.1 Motivations Behind CogInfoCom Channels

Communication between cognitive entities occurs in a variety of situations and
at different levels. The defining attributes of communicationwhich we may
collectively refer to as the context of communicationcan be seen as a function
of interactions between a number of factors, including:
Mode of communicationwhether communication occurs between two endpoints with different (inter-cognitive) or equivalent (intra-cognitive) cognitive
capabilities, as discussed earlier in Sect. 2.2.1

DOI 10.1007/978-3-319-19608-4_7
105
106
7 Structure of Sensory Signals: Icons and Messages
Type of communicationwhether the same sensory modality and representation is used on the two ends of communication (hence the terms sensorsharing, sensor-bridging, representation-sharing and representation-bridging, as
described earlier in Sect. 2.2.2
Directness of communicationwhether or not the cognitive entity knows that
it is communicating, and whether the act of communication requires its active
involvement in terms of the dedication of cognitive resources
Voluntariness of communicationwhether communication is initiated because
the cognitive entity wants to accomplish a task, or because another entity, such
as a user, application, device or network makes a query
Temporal reference of communicationwhether the message that is communicated pertains to the past, present, or future.
Some of these dimensions are currently much more relevant to human experience
than to any technological consideration, while others can be equally relevant to
humans and artificial cognitive entities. For example, the level of directness and
voluntariness of communication would be of particular interest to users, for whom
too many unsolicited queries could easily be perceived as a nuisance. At the
same time, the type of communication can be relevant irrespective of the kind of
cognitive entity under consideration. For example, representation-sharing could be
preferred over representation-bridging in networked applications where low power
consumption is important.
While the list of factors provided above can by no means be considered
exhaustive, it can be reasonably expected, or has otherwise been made clear in past
research, that all of them contribute to user experience and usability in significant
ways. Further, it can be expected that the subtleness of variation within these factors
of communicative context can add to users appreciation of the cognitive capabilities
with which they are communicating.
On the other hand, it is important to observe that in many cases, the above
descriptions of relevant factors require further clarificationeither because of the
ambiguity of the terms used, or as a result of their multi-faceted qualities. For
example, the following questions arise naturally:
What is meant by level of cognitive capability. What determines whether two sets
of cognitive capabilities are equivalent to each other? Although in Chap. 2 it was
highlighted that this may be dependent on the context and the application, the
added value of distinguishing between such cases is not always apparent [a few
attempts have been made to show either that intra-cognitive communication can
be useful in crowdsensing applications, or that a partitioning of users based on
cognitive capability can be useful (Balata et al. 2014; Blum et al. 2014)].
What is meant by sensory modality? If a user is reading through touch
(as in Braille writing), does communication occur through the visual or tactile
modality? If a user receives text information in once instance, and graphical
information (as in a bar chart or pie chart) in another, are both instances of
7.2 The Need for a Unified View of Communication: Overlaps, Ambiguities: : :
107
communication visual, or should the characterization of the modality be further

qualified? What factors play a role in defining and differentiating between
artificial modalities?
In the case of directness, what is meant by active and passive involvement?
If an entity communicates information at times and locations that are a priori
unknown, but the fact of communication, as well as its subject is specified and
agreed upon beforehand, does that qualify as direct or indirect communication?
Is communication more or less direct if a different sensory system or representation is used for the same task? For example, if a user is required to monitor time
series data in real time, would visual monitoring be more direct than auditory
monitoringe.g. based on a sonified graph?
If a pre-configured setting causes information to be queried at a later (but
potentially inconvenient) time, does that qualify as voluntary or involuntary?
What happens when the configuration of information exchange is voluntary, but
the specific instance of communication is involuntary?
In general, can these factors be quantified in addition to labeled in qualitative
terms?
Ideally, questions such as these should be answered in a way that helps explain
differences in the way users experience communication, or in some objective measure of the relative complexity associated with the development and maintenance
of ICT applications. The goal behind the framework of CogInfoCom channels
is to understand these differences, and to use this understanding to develop a
set of design methodologies that can be effectively applied to the synthesis of
communication patterns between cognitive entities.
7.2 The Need for a Unified View of Communication:

Overlaps, Ambiguities, and Contradictions
In this section, we argue that in order for a coherent view of CogInfoCom
channels to emerge, the terminologies and research methodologies of relevant fields
require some unificationthrough the strengthening of certain concepts and the
re-interpretation of others. The following list provides examples that support this
view:
Icons in cognitive psychology refer to visual information, while in multimodal
interactions, they refer to any sensory stimuli that have uniquely recognizable
connotations. However, even in the context of interaction technologies, the
term icon has been defined rather specifically as an image, picture or symbol
representing a concept (Shneiderman 1998).
The haptic modality in particular gives rise to disparate terminologies, as illustrated in McGee (2002). Some terms, which are often used interchangeably, or
otherwise without clear delineation, include proprioceptive feedback, kinesthetic
108
feedback, cutaneous feedback, tactile feedback, and force feedback. Because

there are both significant differences and overlaps in the way these terms are
used, it is justified to clarify their relationships in CogInfoCom research.
The structure of naming conventions and the analogies they create are sometimes
contradictory even within the same field. For example, in multimodal interactions, auditory icons refer to iconic, and earcons refer to abstract auditory
signals (); while the terms haptic icon and hapticon, or tactile icon and tacton
(which are ethymologically analogous to auditory icons and earcons) are used
interchangeably, with references to both iconic and message-like propertiesas
further described in Sect. 7.3.
The ontological frameworks used in multimodal interaction research for the
design of sensory communication signals often impose unnecessary constraints,
making synthesized applications difficult. One example is the dichotomy
between auditory icons and earcons in the auditory domain, which are
often seen as mutually exclusive elements of interface design. Such a view
can lead to artificial limitations in application development, by encouraging
designers to choose between one or the other even in cases where such a
choice would otherwise be premature. Another example is the distinction
between data-driven communication and concept-driven communication (e.g. as
in audification/sonification and auditory icons/earcons within the auditory
domain). The possibility of applying data-driven approaches to the design
of iconic or abstract representations, or of incorporating iconic elements
in otherwise data-driven solutions reveals that the distinction between these
terminologies is not always clear-cut and does not always translate to specific
design solutions.
In many research fields, the concept of modality is ambiguous. For example,
some works adopt the view that a modality reflects the sensory system that is used
to perceive information, while others focus on the artificial modalityi.e., the
output devicethat is used to present information (see e.g. Jokinen 2008). Still
others distinguish between stimuli even in cases where the former two aspects
coincide based on how the messages are encoded (Vernier and Nigay 2001).
Based on the above, an important prerequisite to the formulation of a generic
framework for multi-sensory communication is the unification of terminologies that
are either similar but are used to refer to different concepts, or are different from
each other but are used to refer to similar concepts. It is also worth considering
whether syntax can be separated from semantics, so that conceptual overlaps are
reduced between structural elements and the ways in which those elements are
generated. The framework of CogInfoCom channels addresses these challenges in
a way that is independent from both technology and modality, allowing for flexible
design choices in the development of structured communication signals.
7.3 Unified Structure of Sensory Signals: Streams, Icons and Messages
109
7.3 Unified Structure of Sensory Signals: Streams,

Icons and Messages
As noted earlier, the fields dealing with sensory interfaces between users and ICT
often create a duality between iconic and more abstract forms of communication.
This is evident from terms such as auditory icons and earcons in auditory interface
design, or olfactory icons and smicons (derived from the word smell) in olfactory
interface design (Gaver 1986, 1989; Blattner et al. 1989; Kaye 2004). In both cases,
the former of the two types of signals are iconic, while the latter rely on abstract
(oftentimes: learned) associations between signal and signified.
Examples in the literature can also be found in which an analogous distinction
is suggested through the naming conventions used, but is not clearly reflected in
later use of the relevant terms. Thus, while MacLean and Enriquez suggest that
haptic icons are conceptually closer to earcons than auditory icons in that they
share more philosophically with the former approach (Maclean and Enriquez
2003), the same authors in a different work write that haptic icons, or hapticons,
[are] brief programmed forces applied to a user through a haptic interface, with
the role of communicating a simple idea in manner similar to visual or auditory
icons (Enriquez and MacLean 2003). Later, haptic phonemes are introduced as
the basic building blocks of haptic communication as an analogy with human
speech (Enriquez et al. 2006). With respect to tactile interfaces, Brewster and Brown
write that Tactons, or tactile icons, are structured, abstract messages that can be
used to communicate messages non-visually (Brewster and Brown 2004). Even in
the auditory domain, the seminal paper on earcons stated that earcons are the aural
counterparts of icons (Blattner et al. 1989)an interpretation that has clearly lost
its appeal after auditory icons and earcons co-existed for so long in the literature
(while Blattner et al. originally distinguished between representational earcons
and abstract earcons, today the term earcon is used to refer to the latter category).
In domains where the distinction between iconic and abstract communication has
been made explicit, much research has been dedicated to the goal of formulating
rules for when to use which. This has strengthened the dichotomy that underlied
these concepts from the outset (see e.g. Blattner et al. 1989; Lemmens et al. 2001;
Pirhonen 2006). However, in terms of the flexibility available to application designers, such a dichotomy can at times be seen as unnecessarily artificial.1 For example,
the point can be made that depending on the environment, signals that are abstract
and learned explicitly can evolve into implicitly interpretable representations that
require increasingly less focused attention (the distinction between implicit and
explicit knowledge, as well as the flexibility of transition between the two is a wellresearched subject, as described in e.g. Sun et al. 2001). As a result, some authors
have highlighted the need for unification: for example, in auditory interface design,
Perhaps this explains why some researchers have not allowed the ethymological structure of their
terminologies to influence their interpretation.
110
it has been suggested that auditory icons and earcons might be used in conjunction
to create more effective interfaces (Gaver 1997; Hearst 1997; Mustonen 2008). As
an extension to this idea, a compositional relationship, such that abstract earcons are
composed of auditory icons at lower levels of granularity, has been suggested in e.g.
Hermann and Ritter (1999), Csapo and Baranyi (2012d), and .
A second level of unification has emerged between conceptually driven and
data-driven communication. An example of the latter in the auditory domain is
sonification, in which data is used to create reproducible patterns of auditory signals.
Hermann defines sonification as the data-dependent generation of sound, if the
transformation is objective, systematic, reproducible and can intentionally be used
with different data (Hermann 2008; Hermann et al. 2011). It is important to note in
this regard that the key distinction between sonification and auditory earcons/icons
is made in terms of whether the signal is data or concept-generated, and not in
terms of any kind of strictly defined perceptual consideration; as noted in Hermann
(2008), a sonification can include iconic and message-like elements if at least part
of the data is discrete and/or conceptual.
In the framework of CogInfoCom channels, these two levels of unification are
reflected in the modality-independent concepts of CogInfoCom streams, CogInfoCom icons and CogInfoCom messages, as well as in a hierarchical relationship
between the three. Additionally, questions of how data or information is mapped
onto these communicative elements are treated as a separate question from the
decision to use one or the other. More specifically, the conceptual associations
underlying CogInfoCom channel based communication are chosen without regard
to whether those associations will be encapsulated in icons or abstract messages.
This distinction arguably helps to more clearly delineate the tasks that need to be
performed when designing and implementing new communication patterns between
cognitive entities. The remainder of this chapter focuses on the structural aspects
of CogInfoCom channels, while the mapping of data/information is considered in
Chap. 9.
7.3.1 CogInfoCom Streams

Definition 7.1. CogInfoCom streams are temporal sequences of sensory signals
that can be related to the state of an environment, and which are stable through
time. If these two criteria are met, it is said that the sensory signals are ecologically
valid.
Remark 7.1. The criterion that signals within CogInfoCom streams are stable
through time means that if the state of the environment (to which the signals are
related) does not change, then the signals also remain the same.
It should be noted that in no way does the definition of CogInfoCom streams
entail that the streams are actually interpreted by a cognitive entity. However, it
is important that streams can at least in theory be used as a basis for meaning
111
construction: this is the purpose of the condition of ecological validity. As described

in later sections of this chapter, segments or parts of streams will often be involved
in the emergence of CogInfoCom icons and messages, both of which are syntactical
structures associated with meaning with respect to the given environment.
7.3.2 CogInfoCom Icons

Definition 7.2. CogInfoCom icons are sensory percepts that give rise to immediate
and unique semantic interpretations. Compound CogInfoCom icons are multimodal percepts that combine the use of several sensory systems and give rise to
immediate and unique semantic interpretations.
Remark 7.2. The condition that the interpretation of icons should be immediate
refers not to a specific timespan, but rather to the requirement that the interpretation
of an icon should not have to based on any kind of contextual information. In other
words, immediate can be taken to mean in itself, or without any context. Of
course, determining what elements belong to an icon and what qualifies as their
context leads to a kind of figure-ground problem that is not amenable to formal
definition. As described earlier in Sect. 5.4, the answer to this question also depends
on the cognitive entity which interprets the sensory signals, as well as on the
intended meaning of the signals.
Remark 7.3. It should be noted that the way in which the terms sensory percept
and sensory system are used deliberately leaves open the question of what
underlying architecture is used to sense and process the signals. This means that
an icon can be any sort of signal in generalas long as it leads to unambiguous and
immediate interpretationsirrespective of whether the interpreter is a human being
or an artificial cognitive entity. This point of view is unavoidable once the merging
between humans and ICT is acknowledged. What is certain is that in the case of
icons, representation and meaning coincide with respect to the conceptual notions
of the interpreting entity.
As a result of the technology-agnostic nature of the definition of CogInfoCom
icons, a wide range of signals can be considered as such. In the following, examples
are provided of both human-oriented and artificial icons. The primary reason why
the examples are divided into these two categories is to make it clear that both
natural and artificial cognitive entities can and do rely on the use of CogInfoCom
icons. However, due to the merging process that is occurring between humans and
ICT, it can be expected that there may eventually be no real cause to distinguish
between certain human and artificial icons. For example, as vision-based artificial
pattern recognition evolves, visual icons that carry meaning for humans and artificial
cognitive entities can easily overlap. Bearing this in mind, some examples of humanoriented icons include:
112
Visual icons: often abbreviated as icons, these basic components of graphical

user interfaces were first invented and used by David Canfield Smith as interface
objects that could be directly manipulated in a system (Smith 1975). Later,
Shneiderman defined the concept of icon as an image, picture or symbol
representing a concept (Shneiderman 1998).
Auditory icons: defined by Gaver in the context of everyday listening as
caricatures of everyday sounds (Gaver 1986, 1989, 1988). Gaver was the first
to generalize Canfield-Smiths icon concept to modalities other than vision, by
deriving the theory of auditory icons through a distinction between everyday
listening and musical listening. In this context, musical listening pertains to the
interpretation-less appreciation of the quality of a sound, while everyday listening
pertains to the interpretation of sounds in terms of e.g. physical/mental states or
events.
Olfactory icons: scent outputs used to convey information, where the scent is
environmental and semantically related to the information to be conveyed (Kaye
2004). The condition that the scent in question should be environmental reflects
Gavers concept of everyday listening in the auditory domain.
Haptic icons: brief computer-generated signals, displayed to a user through
force or tactile feedback to convey information such as event notification, identity,
content or state (Maclean and Enriquez 2003). Haptic icons can be categorized
as compound CogInfoCom icons due to the composite nature of the haptic sense
[which involves proprioceptive, kinesthetic as well as cutaneous components
(Voisin et al. 2002; Lederman 2004)]. The fact that some works have interpreted
haptic icons as somewhat message-like (Maclean and Enriquez 2003), while
others at the same time have suggested that a haptic icon communicates a simple
idea in manner similar to visual or auditory icons (Enriquez and MacLean
2003) also clearly suggests that compound icons are neither iconic, nor messagelike, but in a sense lie between the two layers.
Dynamic icons: Sensory signals which represent dynamic processes in iconic
ways (Baranyi et al. 2012). The term dynamic icon differs from the naming
convention used in previous examples, as it alludes to the content of the icons
rather than to the sensory system that is used to perceive them. There is
no contradiction here: signals categorized as dynamic icons can give rise to
immediate and unique semantic interpretations irrespective of whether they are
perceived through the visual, auditory, or any other sense (as a result, dynamic
icons can be often regarded as compound icons). The unifying feature among
these icons, then, is that they provide interpretations about dynamic processes,
that is, any kind of temporal processes that include some form of memory
(provided that this memory has an effect on their output) (Baranyi et al. 2012).
Due to the novelty of the concept of dynamic icons, some further discussion is in
order here.
The motivation behind dynamic icons can be highlighted through the following points:
113
Humans are capable of sensing, understanding and interpreting a wide

range of dynamic phenomena, including phenomena pertaining to spatiotemporal processes of motion (e.g., rising, falling, drifting, crashing, exploding, vibrating, swinging, accelerating, decelerating phenomena) as well as
spatio-temporal properties which are inherent to everyday materials (e.g.,
compliance, resistance, roughness, stickiness). This remarkable human sensibility is well reflected by the fact that we are not only capable of understanding
and interpreting, but also of anticipating such phenomena and of taking their
anticipated effects into consideration while performing actions (e.g., as in
the case of various precision sports such as archery, bowling, darts, curling,
shooting sports, etc.) (Wolpert and Ghahramani 2000; Flanagan et al. 2003,
2006).
Humans are capable of learning to understand, interpret and anticipate
dynamic phenomena irrespective of whether they are linear or non-linear;
there is strong evidence from the cognitive sciences that humans can quickly
and efficiently learn to predict and execute dynamic behaviors even when
those behaviors are governed by high-dimensional sets of constraints and
significant non-linearities (Wolpert and Kawato 1998; Todorov 2004).
The same dynamic phenomena can be perceived and understood in various representations across a variety of sensory modalities. Thus, the same
phenomena can be conveyed through vision, audition, the tactile sense and
other modalities. Further, in many cases humans are capable of understanding
dynamic phenomena through abstract concepts; e.g., a mathematician or an
engineer can look at a set of differential equations and immediately gain a
general understanding of the key characteristics of a dynamic behavior. The
various concepts developed to characterize electric networks (voltage, current,
etc.) have become so universal that despite their abstract nature they can be
readily used to give experts a deep understanding of underlying dynamics.
Based on the above, humans have a strong disposition to sense, understand,
interpret and anticipate a wide range of dynamic phenomena in a variety of
representations. Key to the subject of dynamic icons (and later, dynamicons)
is the idea that the perception and anticipation of these dynamic phenomena
can also be artificially synthesized through illusion. An archetypical example
that supports this approach is the illusion of self-motione.g. the sensation one
gets when sitting in a stationary vehicle and another vehicle beside it begins to
move (Berthoz et al. 1975). Based on this example alone, dynamic icons have
been developedand complemented with othersin vehicle simulators, where
a common requirement is for users to have a strong experience of presence and
self-motion (see e.g. Pinto et al. 2004; Brunger-Koch et al. 2006; De Groot et al.
2011; Riecke et al. 2005)
Further examples of icons not strictly unique to humans might include:
Emotional icons: iconic representations that convey emotional information
or otherwise influence emotion (in other words, are interpreted by the
114
emotional sense). Classical emoticonsgraphical representations of

emotionare examples of emotional icons in the CogInfoCom terminology,
as they provide concise representations of emotional state. However, emotional
icons can also be compound in nature, so that their scope is not limited to
graphical, but can also involve vocal and other representations.
Kinesthetic icons: iconic representations provided through the kinesthetic sense,
that is, the sense that keeps track of muscle movements. Any physically embodied
cognitive entity could in theory be capable of learning to interpret perceptually
salient and ecologically meaningful kinesthetic icons as reflecting events in the
state of their physical surroundings.
Body icons: physical body movements that can be interpreted immediately and
unambiguously. If the interpretation is unambiguous, body icons can be used as
gestures to influence the operation of the cognitive entity on the other endpoint
of communication.
Network packet icons: network packets that can be filtered and unambiguously
interpreted by network routersor an offline analysis toolwithout reference to
contextual information. For example, if a packet from a blacklisted IP-address
is received, then the packet can be interpreted as an iconic representation of
the network activity of a specific host. Future examples might be motivated by
requirements towards e.g. limited power consumption and dedicated transmission rates in potentially ad hoc and mobile networks.
It should be noted that the latter examples of icons are tentative definitions, and
will undoubtedly be further extended and/or refined as researchers focusing, e.g. on
emotional cognition, body gestures and networked sensor technologies gain a deeper
understanding of the cognitive bases relevant to these domains. The final example
was included to demonstrate that any sort of signal in general can be regarded as a
CogInfoCom icon, as long as it is amenable to immediate and unique interpretation.
7.3.3 CogInfoCom Messages

Definition 7.3. CogInfoCom messages are sensory percepts with abstract meaning
that are often sequentially composed of (compound) CogInfoCom icons.
Remark 7.4. In contrast to CogInfoCom icons, messages are sequential (messagelike) juxtapositions of lower-level structural elements. A major consequence
behind this distinction is that CogInfoCom messages can be seen as contextsensitive representations. Further, if representation and meaning coincide in the
case of icons, these two aspects are separated in messages. A message can
convey meaning that arises either as a superposition of the original interpretations
associated with its constituent icons, or as an abstract interpretation of lower-level
symbols.
115
Remark 7.5. Although many researchers have pointed out that message-like representations often require training, two points can be made to show that this is by no
means a general rule:
It has been demonstrated, for example in the domain of auditory interfaces,
that messages structured appropriately and taught to users through suitable
training methods could often be just as easily retained as update icons (for an
overview of such studies, see Csapo and Wersenyi 2014). Further, when messages
carry information that is based on individual interpretations of its constituent
icons, understanding the intended meaning can translate to understanding the
constituent icons.
Even if complex training suites were required to teach users the meaning of
a message, the software and technology related challenges associated with the
processing of messages in artificial cognitive entities will not necessarily be as
significant.
From the point of view of CogInfoCom, it makes sense to regard CogInfoCom messages as being comprised of more elementary icons, because this
view encourages the joint use of the two classes of signals without any kind of
restriction. A degenerate CogInfoCom message with no icons does not contradict
these definitions, and neither does the use of separate CogInfoCom icons which do
not perceptually and/or conceptually integrate into a single message.
In the same way as was the case with CogInfoCom icons, due to the sensory
system independent and technology-agnostic point of view adopted in the definition,
a virtually infinite range of signals can be cited as examples of CogInfoCom
messages as long as they create abstract associations to some kind of relevant
information. Some examples relevant to human perception include:
Eyecons: visual messages used to convey information that has an abstract
relationship with the message structure. Although this term has not been defined
outside of the scope of CogInfoCom, it arises as a generalization of earcons,
smicons and other message-like elements to the sensory modality of vision.
Earcons: non-verbal audio messages used in the user-computer interface to
provide information to the user about some computer object, operation, or
interaction (Blattner et al. 1989). Contrary to the original approach of Blattner
et al., who treated both representational earcons and abstract earcons as earcons,
today the term is used exclusively to refer to message-like communication.
Smicons: scents used to convey information that has only an abstract relationship with the data it represents (Kaye 2004).
Hapticons: Enriquez and MacLeans terms of haptic icons and hapticons are
separated into two different levels in CogInfoCom. As described earlier, the term
haptic icon is used when referring to haptic signals that are iconic, and the term
hapticon is used when referring to haptic messages that convey information that
has an abstract relationship with the hapticon itself. By adopting this view, it can
be seen that Brewsters tactons can be considered as specific kinds of hapticons
for the tactile sense (Brewster and Brown 2004).
116
Dynamicons: sensory signals that are composed of any number of dynamic

icons and are message-like in their interpretation (Baranyi et al. 2012).
Further examples of messages not strictly unique to humans might include:
Emoticons: message-like emotional representations that convey information
through abstract association. It is worth noting that emoticons in the traditional
sensei.e., graphical representations of emotional facial expressionscan be
more aptly categorized as (graphical) emotional icons, provided that they can
be interpreted without context. An emoticon in the CogInfoCom terminology
would instead be a message-like communication signal that uses emotional
expressions to convey its meaning. For example, Chernoff faces are graphical
emotional representations that correspond to any kind of independent (qualitative
or quantitative) data (Chernoff 1973). The key point is that the interpretation
of a Chernoff face requires a special kind of indirection, an encoding-decoding
scheme that allows the interpreter to decipher meaning. Similarly, emotional
representations that change through longer periods of time and therefore cannot
be interpreted based on a single impression would qualify as emoticons.
Kinestheticons: message-like kinesthetic representations that convey information
through abstract association. For example, in a flight simulator, a series of
shakes and jolts effected on the users body could convey the message that the
aircraft is ascending while experiencing heavy turbulence. Similarly, a futuristic
humanoid robot designed for karate training purposes could interpret a series of
shakes and jolts effected on its physical embodiment as sending the message that
the user is beginning to develop muscle fatigue.
Bodicons: message-like gestural representations that convey information through
abstract association. A series of gestures in human interaction could send
messages such as the interlocutor is losing interest, or that the interlocutor
finds these ideas interesting. Similarly, an artificial cognitive entity with a high
level of social intelligence could be expected to make sense out of bodicons.
Network packetcons: message-like streams of network packets that can be
interpreted by a network routeror an offline analysis toolas conveying
abstract meaning regarding relevant to network conditions.
As earlier in the case of CogInfoCom icons, several of the definitions proposed
here are tentative and can be expected to undergo extension and refinement in future
research. The final example was included to demonstrate that any sort of signal in
general can be regarded as a CogInfoCom message, as long as it is context-sensitive
and abstract in its interpretation.
7.4 Icons and Messages: A Summary of Key Points

The following points were made in this chapter:
7.4 Icons and Messages: A Summary of Key Points
117
Iconic and message-like forms of communication can be interpreted both with

respect to natural and artificial sensory systems.
Iconic elements combine representation and meaning into a single unit, allowing
for strong conceptual associations that exist to begin with to be applied in novel
communication settings.
Examples of icons types include visual icons, auditory icons, olfactory icons,
haptic icons, dynamic icons and emotional icons.
Messages are higher-level elements, potentially comprising any number of iconic
constituents, that separate representation from meaning. A message can be used
to convey any kind of abstract meaning, even if that meaning has no conceptual
relationship with its representation (other than the relationship that is implicitly
created through the design of the message).
At the same time, the general separation between representation and meaning
created by messages does not mean that the two cannot coincide in specific cases:
if designed appropriately, a message can be used to convey information that arises
as a superposition of interpretations provided by its constituent icons. In the latter
case, the iconic interpretations have strong relevance to the interpretation of the
message.
Examples of message types include eyecons, earcons, smicons, hapticons,
dynamicons and emoticons.
Chapter 8
CogInfoCom Channels: Perceptual Unification

of Representation and Meaning
Based on the preliminaries of the previous chapter, this chapter provides a holistic
view of CogInfoCom channels that unifies syntactic and semantic aspects. CogInfoCom channels are defined as a structured unification of icons and messages
with perceptual interpretations of meaning. This definition uncovers previously
overlooked features of the concept of modality. It is demonstrated in the chapter
that the parametric structure behind CogInfoCom channels can be represented in a
tensor-algebraic form that is canonical and interpretable. It is argued that the use
of this representation can be advantageous in that it allows researchers working on
similar problems to share and reproduce each others results. Some of the tools for
working with CogInfoCom channels developed in later chapters are also based on
this representation.
The chapter is structured as follows. In Sect. 8.1, the concepts of icons and messages are extended based on their applicability towards communicating semantic
meaning. The concepts developed in the section are used to formulate a definition
of CogInfoCom channels, as well as to provide a definition of modality which
takes into consideration both the technology-agnostic perspective of CogInfoCom,
as well as the fact that communication is a multi-lateral rather than a uni-lateral
issue. Section 8.2 introduces a tensor algebra based canonical and interpretable
representation of CogInfoCom channels. Finally, the results of the chapter are
briefly summarized in Sect. 8.3.

DOI 10.1007/978-3-319-19608-4_8
119
120
8 CogInfoCom Channels: Perceptual Unification of Representation and Meaning
8.1 CogInfoCom Channels and Relevant Concepts

In the previous chapter, it was shown that icons and messages are central components of communication across sensory systems, irrespective of whether they are
artificial, biological or a combination of the two. It was also demonstrated that icons
and messages can be unified and therefore do not contradict, but rather complement
each other in practical use.
In this section, the two concepts of icons and messages are extended based on
their applicability towards communicating classes of semantic meaning. This will
provide us with a definition of CogInfoCom channels, as well as a reinterpreted
concept of modality that respects earlier definitions, but at the same time also
extends them to suit the requirements of the CogInfoCom perspective.
8.1.1 CogInfoCom Message Generated Perceptual Concepts

Communication exists at least in part because it provides an effective means of
sharing interpretations of meaning. This is the primary goal behind the design of
CogInfoCom channels as well. However, to be able to do this, it is important to
explore the link between the perception and interpretation of icons and messages.
Especially in cases where completely new concepts are to be communicated (as in
a futuristic virtual environment), or when messages are conveyed through a sensory
system that is different from the one that is normally used for the same task, bridging
gaps between perception and interpretation can pose significant challenges.
Such considerations motivate the definition of CogInfoCom message generated
perceptual concepts:
Definition 8.1. CogInfoCom message generated perceptual concepts or perceptual concepts are semantic concepts that can be used to describe the perceptual
quality of CogInfoCom messages.
Remark 8.1. If the set of CogInfoCom messages considered are related to a specific
sensory system, it is possible to speak of perceptual concepts that are particular to
that sensory system. Thus, examples of earcon generated perceptual concepts are
smoothness, raspiness, consonance and dissonance, because audio messages can be
described using such concepts. Examples of smicon generated perceptual concepts
would include sweetness, perfume and chocolate.
Remark 8.2. The notion of perceptual concept has links with Gavers notion of
everyday listening in the sense that the concepts used to describe the perceptual
qualities of a sound would qualify as perceptual concepts (Gaver 1986).
Remark 8.3. The relationship between icons, messages and perceptual concepts is
shown in Fig. 8.1. A formal definition of CogInfoCom message generated perceptual
concepts will be given in Sect. 9.1.
8.1 CogInfoCom Channels and Relevant Concepts
121
Fig. 8.1 Hierarchical relationship between icons, compound icons, messages and messagegenerated concepts in CogInfoCom
8.1.2 Definition of CogInfoCom Channels

Based on the above, CogInfoCom channels can be defined as follows:
Definition 8.2. CogInfoCom channels are ordered sets of CogInfoCom messages
that provide information through message generated perceptual concepts.
Remark 8.4. CogInfoCom messages can belong to the same CogInfoCom channel if:
They give rise to orderable perceptual qualities.
The perceptual qualities along which they can be ordered give rise to associations
with one or more high-level concept (e.g., degree of roughness, degree of
surprise, degree of comfort, etc.), so that changes in perceptual gradation can
be associated with changes in intended interpretation.
Remark 8.5. The concept of CogInfoCom channels arises somewhat as an analogy
to earcon families, which are groups of earcons constructed based on related
motives, that is, sequences of pitches that create a short, distinctive audio pattern
related through transformation, combination or inheritance (Blattner et al. 1989).
The analogy holds as long as motives are considered to be related if they give rise
to information about the same concept through perceptual variation, and unrelated
if they are unsuitable for communication about the same concept.
Based on the above, the key design challenge behind CogInfoCom channels
involves finding sets of icons and messages that are suitable for communication
122
on the semantic concept of interest. In engineering design, it is generally useful if

the icons and messages used are parametric, so that they can be reproduced more
easily as well as systematically modified and tested for perceptual orderability (this
can be of interest to any cognitive entity in general that is to interpret the channel, as
perceptual capability generally depends on a variety of factors, including among
others human psychology, hardware capabilities, network delays and noise).
The challenges associated the design of CogInfoCom channels can be summarized as follows:
1. Finding a perceptual concept that allows for an associative link to be created with
the intended meaning of the channel (to be addressed in Chap. 9).
2. Finding a synthesis method which can be used to generate the selected perceptual
concept using suitable parameter values (to be addressed in Chaps. 9 and 10).
In practical cases, this step is often regarded as trivial: on the one hand, the
perceptual concept that is chosen often gives strong clues as to the generation
parameter types that are to be used; while on the other hand, if a sufficiently
generic synthesis method is chosen, the challenge of finding salient generation
parameters can be reduced to solving the problem outlined in the following point.
3. Finding a suitable set of parameter values based on which a perceptually
orderable set of messages can be synthesized, resulting in a CogInfoCom channel
(to be addressed in Chap. 10).
8.1.3 A Novel Interpretation of the Concept of Modality

Interestingly a novel viewpoint of modality emerges as a consequence of the
definition of CogInfoCom channels. As discussed earlier, the concept of modality
has been defined in many ways, depending on whether the natural or artificial
sensory system used for communication is seen as more important, or if both are
considered along with the encoding (language) used between them (Jokinen 2008;
Vernier and Nigay 2001). The CogInfoCom interpretation clearly favors the third
approach, but it also supersedes all three by removing the need to speak about
human modalities as separate from artificial modalities. Hence, the following
definition is proposed:
Definition 8.3. A CogInfoCom modality is specified by a CogInfoCom channel
associated with:
a group of generation parameter types,
a method for creating CogInfoCom messages based on those parameter types
(synthesis method),
an output device for communicating the resulting CogInfoCom messages,
an input device for sensing the messages and differentiating among them, and
a noise level characteristic of the transfer medium, which directly influences the
ease with which messages can be perceived.
8.2 Formal Representation of CogInfoCom Channel Parameters
123
Based on this definition, it is clear that the proposed concept of modality reflects
a specific relationship between communicating entities. It reflects a relationship
because, through the input and output device, it takes into consideration the sensory
systems of both entities, as well as the encoding of messages that are transferred
between them. It also reflects a specific relationship because the context of the
specific instance of communication, as reflected by the noise level in the channel, is
also taken into consideration.
As a result, if different parameter values (or even different parameter types)
are used to generate two sets of CogInfoCom messages, then for all practical
purposes, they can be regarded as belonging to separate modalities. This point
of view acknowledges that an application can be multimodal even if it only uses
e.g. audio feedback, given that the feedback messages can nevertheless be multifaceted, requiring high levels of attention or processing power.
Similarly, two applications can be regarded as using different modalities if they
use different output or input devices, or if the noise conditions of communication are
different. This point of view acknowledges that various measures of effectiveness
and efficiency (including test suites, empirical usability tests, etc.) can no longer be
automatically seen as valid if any of these devices are changed for different ones,
or if the environment is different. These points highlight the fact that no amount
of testing will produce results that are generalizable to environments with different
parameters.
8.2 Formal Representation of CogInfoCom

Channel Parameters
In this section, a formal representation of CogInfoCom channels is proposed
based on multi-dimensional tensor algebra. The representation creates a quantitative
link between the perceptual qualities of CogInfoCom icons/messages, and the
parameters through which these syntactic structures are generated. The quantitative
characterization of sensory percepts is important because the primary purpose of
any CogInfoCom channel is to encapsulate iconic and message-like structures that
can be ordered based on the semantic meaning they convey. The fact that the
representation includes a parametric description of CogInfoCom channels is also
essential for the formal development of any kind of manipulation that can serve to
modify perceptual/semantic qualities.
8.2.1 Nomenclature and Notations

In the general case, CogInfoCom messages are multi-dimensional in the sense
that they can be ordered along multiple perceptual dimensions. For example, in
the auditory domain, earcons can be compared based on how loud they are, how
124
rough they are, based on their level of tonality or atonality, etc. The results of these
comparisons will generally be different depending on the perceptual dimension that
is considered. For example, if there are three earcons (A, B and C), then earcon A
may be louder than earcon B, which in turn may be louder than earcon C; while
earcon B may be perceptually more rough than earcon C, which may in turn be
more rough than earcon A. It may also be the case that the three earcons cannot
be reliably ordered along a given perceptual dimension, as demonstrated in Barrass
(1998). This observation is general and carries on to other sensory systems.
A significant challenge in the design of CogInfoCom channels is posed by the
fact that the generation parameters used to synthesize CogInfoCom messages have
no directly specified, general relationship with perceptual qualities. In other words,
there will generally be no analytical formula available to specify this relationship.
This will be true even if, for example, the parameter types and synthesis method are
fixed, due to differences in other aspects of the modality that is used. Nevertheless, a
link between generation and perception will always exist (even if it is hidden to the
application designer), that is, we can at least assume that for any given CogInfoCom
modality, there exists a relationship that maps perceptual orderings onto orderings of
parameter sets. To describe these orderings and relationships, we use the following
terminology:
Let the parameters used to generate individual CogInfoCom icons in CogInfoCom channels be represented by an H-dimensional generation vector of real
values, f 2 RH .
Let the perceptual parameters that describe a given CogInfoCom icon be
represented by an N-dimensional perceptual gradation vector of natural values
p 2 NN .
Let the relationship between the perceptual vector and the generation vector be
represented by a parameter-generating function F W NN ! RH , which takes
as its input a perceptual gradation vector, and returns a generation vector that
can be used to synthesize a CogInfoCom icon in the CogInfoCom message that
represents the perceptual qualities contained in the perceptual gradation vector.
The primary challenge in designing CogInfoCom channels is how to specify
and represent the parameter-generating function. If the function could somehow
be specified, that would allow cognitive entities to communicate by encoding the
desired perceptual gradation into a set of parameters which can be in turn used
to generate appropriate messages. Based on the definition of modality, the same
parameter-generating function could be used as long as the input and output devices,
as well as the environment of communication does not change.
In order to address such challenges, however, it is necessary that the parametergenerating function in question should have a formal representation. Such a representation would ideally allow for design choices to be shared among researchers and
developers, and for existing channels to be manipulated so as to suit the modality
that is used for communication.
8.2 Formal Representation of CogInfoCom Channel Parameters
125
8.2.2 Mathematical Background

Before the formal representation of parameter-generating functions is presented,
the definitions of a set of function classes are provided. These will be useful in
characterizing parameter-generation functions:
Definition 8.4. Let F.x; y/ 2 RH be a multivariate vector function, where x 2
PM (with the stipulation that P is a partially ordered set), y 2 NL , M C L D N,
and H; N < 1 (x is an element of a partially ordered, and y is an element of a
discrete subspace of the domain of function F). If M 0 and L 0, then F is an
L-mode (partially) ordered multivariate function. If L D N, then F is an ordered
multivariate function.
Remark 8.6. The definition states that even if some input dimensions of a multivariate function are only partially ordered, the value of at least one input dimension must
be taken from the set of natural numbers in order for the function to be considered an
L-mode ordered multivariate function. This amounts to saying that the input vectors
supplied to the function are ordered along at least one of the input dimensions with
values taken from the set of natural numbers. If a multivariate function is ordered,
then all input vector dimensions can be ordered.
Remark 8.7. In cases where the distinction between partially ordered and ordered
enumerable multivariate functions is not important, the two function types will be
abbreviated in our further discussions as POMFs (partially ordered multivariate
functions).
Definition 8.5. Let F.x; y/ 2 RH be a POMF. The function is a bounded-input
POMF if input vector x is taken from a bounded hypercube of a partially ordered
set P, and input vector y is taken from a bounded hypercube of natural numbers
(x 2 Pa1 ;b1 a2 ;b2 ;:::;aM ;bM , and y 2 Nc1 ;d1 c2 ;d2 ;:::;cL ;dL ).
Definition 8.6. Any bounded-input POMF F can be defined over a discrete, Ndimensional hyper-rectangular grid
G D fgp1 ;p2 ;:::;pN 2 Pa1 ;b1 a2 ;b2 ;:::;aM ;bM Nc1 ;d1 c2 ;d2 ;:::;cL ;dL gPpnnD1
(8.1)
such that Pn is the number of unique discretization points along the nth dimension,
and n D 1::N. A discretized function of this kind can always be represented by an
.N C 1/-dimensional tensor F D.G/ of size P1 : : : PN H
D F.gp1 ;p2 ;:::;pN /
FpD.G/
1 ;::;pN
(8.2)
such that F.gp1 ;p2 ;:::;pN / is the output vector of F corresponding to the input vector in
D.G/
point gp1 ;p2 ;::pN on grid G, and Fp1 ;::;pN is the subtensor of tensor F D.G/ obtained by
fixing the index of the first i dimensions (i D 1::N) to the pi th gradation along the ith
dimension of discrete grid G. F D.G/ is referred to as a discretized bounded-input
POMF.
126
In the following section, it will become clear that the parameter-generating functions associated with CogInfoCom channels can be represented using the concept
of discretized bounded-input partially ordered multivariate functions (DBIPOMFs).
Later, in Sect. 10.2, it will also be shown that such functions can be transformed
into a canonical representation that allows for the interpretation of important tensor
algebraic concepts, such as rank and condition number. More importantly, it will
also allow for the comparison and manipulation of existing solutions using tensor
algebraic operators.
8.2.3 Generative Model of CogInfoCom Channels

This section introduces a formal representation of CogInfoCom channels. The
representation makes explicit the relationship between the parameters used to
generate messages and the perceptual qualities thereof. The concepts introduced
in Sect. 8.2.1 (i.e., the perceptual gradation vector, the generation vector and the
parameter generation function) are referred back to in this section.
8.2.3.1 CogInfoCom Icons: From Perception to Generation

It is clear that if the values in the perceptual gradation vector are bounded, then the
parameter-generating function F is a discretized, bounded-input ordered function.
Thus, based on Eq. (8.2), the parameter-generating function can be represented
using a multi-dimensional tensor as follows:
D F.gp1 ;p2 ;:::;pN /
FpD.G/
1 ;::;pN
(8.3)
where F.gp1 ;p2 ;:::;pN / is the output vector of F corresponding to the generation
vector of H dimensions in point gp1 ;p2 ;::pN on grid G, and the generation vectors
obtained as the output of F are represented along the .N C 1/th dimension of
tensor F . Each dimension in the tensor is indexed by the gradation specified along
the corresponding dimension of grid G, save for the last one, which is indexed by
the position of a specific parameter in the generation vector.
The H-dimensional generation vector can be used as input to synthesize a
CogInfoCom icon within the CogInfoCom message that carries the perceptual
values contained in the perceptual gradation vector p.
8.2.3.2 CogInfoCom Channels: From Perception to Generation

Based on the above, CogInfoCom messages can be generated if there are:
R different parameter-generating functions fi D Fi , i D 1::R for R different
CogInfoCom icons included in the message
8.3 Design Challenges Behind CogInfoCom Channels: A Summary of Key Points
127
Fig. 8.2 Structure of a CogInfoCom message in a CogInfoCom channel, based on the perceptual
gradation vector and the generation vector. The synthesis method accepts a set of generation
parameters based upon which the structural and temporal properties of CogInfoCom icons can
be derived (synth refers to the synthesis algorithm used for CogInfoCom icon generation). The
succession of CogInfoCom icons through time creates CogInfoCom messages
a set of orchestration parameters included in generation vector f, which specify

the starting time and ending time of each component CogInfoCom icon in the
CogInfoCom message
These two criteria can be easily satisfied in the design of a CogInfoCom system.
The generation of CogInfoCom messages can then take place as shown in Fig. 8.2.
If one of the input dimensions of parameter-generating function Fthat is, one of
the first N dimensions of tensor F D.G/ corresponds to the degree to which the
message represents a given concept, then the resulting set of messages belong to a
CogInfoCom channel which conveys information on that concept.
8.3 Design Challenges Behind CogInfoCom Channels:

A Summary of Key Points
The following points were made in this chapter:
Besides conveying conceptual information, icons and messages specified with
respect to various sensory systems also give rise to perceptual concepts, referred
to as CogInfoCom message generated perceptual concepts, or perceptual
concepts.
If a set of messages can be ordered based on a perceptual concept, the messages
can be used to define a CogInfoCom channel that conveys information through
associations created by that perceptual concept.
128
Based on the above, the challenges associated the design of CogInfoCom

channels can be summarized as follows:
1. Finding a perceptual concept that allows for an associative link to be created
with the intended meaning of the channel (to be addressed in Chap. 9).
2. Finding a synthesis method which can be used to generate the selected perceptual concept using suitable parameter values (to be addressed in Chaps. 9
and 10).
3. Finding a suitable set of parameter values based on which a perceptually
orderable set of messages can be synthesized, resulting in a CogInfoCom
channel (to be addressed in Chap. 10).
Based on the above, a new concept of modality can be formulated, which reflects
communication as a relationship between entities rather than an issue pertaining
to a single entity. The proposed definition specifies the synthesis method used
to generate a channel, the parameter values supplied to that synthesis method,
the input and output device (sensory system) used for communication, and the
noise parameters of the communication medium as central to the specification of
modality.
A tensor algebraic representation was proposed for the formal treatment of
the parameter-generating function which specifies the relationship between the
parameters which describe perceptual concepts and the parameters which are
used to synthesize the CogInfoCom messages that are used to create a channel.
It will be shown in Chap. 10 that the representation is canonical and opens the
way to the use of tensor algebraic operators in tuning the parameter-generating
function.
Chapter 9
Mapping Meaning onto CogInfoCom Channels
Based on the results of the previous two chapters, this chapter focuses on techniques
that can be used to map semantic meaning onto CogInfoCom channels. This
mapping task is formulated in terms of finding appropriate perceptual concepts
and a corresponding set of generation parameter typesfor the establishment of
useful semantic associations. In this way, the chapter develops a set of tools that can
support the development of readily interpretable CogInfoCom channels.
The chapter is structured as follows. Section 9.1 introduces a denotational
framework that is applied to the formal treatment of concept mapping. Parts of
the framework have been used in the past [i.e., under the name Object-AttributeRelation (OAR) model (Wang 2008; Yao 2009)], while others are provided as
extensions based on definitions in previous chapters. In particular, formal definitions
are provided of CogInfoCom message generated perceptual concepts, perceptual
evaluation functions, and interactive concepts. Based on the framework, Sect. 9.2
introduces various forms of direct and analogical mapping between CogInfoCom
messages and CogInfoCom message generated perceptual concepts. Future perspectives for automated mapping are discussed in Sect. 9.3.
9.1 Concept Algebra Based Background

This section develops a formal background for semantic mapping, based on the
Object Attribute Relation model and on concepts introduced in previous chapters.
For the sake of clarity, those definitions which already exist in the literature are
reiterated in Sect. 9.1.1. Those definitions that are uniquely relevant to conceptual
mapping in CogInfoCom channels are treated subsequently in Sects. 9.1.2 and 9.1.3.

DOI 10.1007/978-3-319-19608-4_9
129
130
9 Mapping Meaning onto CogInfoCom Channels
9.1.1 Key Definitions in the OAR Model

The following definitions are adopted from Ganter and Wille (1999) and Wang
(2008) with minor extensions:
Definition 9.1. A (formal) context is defined as a 3-tuple, consisting of objects,
attributes and a set of relationships between them:
D .O; A; R/
(9.1)
where R is a set of relations between objects and attributes.

R W O ! OjO ! AjA ! OjA ! A
(9.2)
Objects are instantiations of concrete entities in the world and/or abstract

concepts (defined later), while attributes are subconcepts used to characterize the
properties of a given concept (Wang 2008). As regards the terminology developed
in previous chapters, CogInfoCom icons and CogInfoCom messages can be thought
of as objects, while the perceptual labels that are used to describe the properties of
CogInfoCom icons and CogInfoCom messages can be thought of as attributes.
Definition 9.2. An (abstract) concept c in a context can be defined as a 5-tuple:
c D .O; A; Rc ; Ri ; Ro /
(9.3)
where
O O ; A A ; Rc O A; Ri C0 C; Ro C C0
(9.4)
such that O and A are the power sets of O and A, respectively; and Rc , Ri and Ro
are a set of internal, input and output relations, respectively. A concept is said to be
defined over a particular object o, or a set of objects o, if that object or set of objects
is included in the objects of the concept. Formally:
DO.c D .O; A; Rc ; Ri ; Ro /; o/ , 8o 2 o W o 2 O
In case an attribute has a scalar numerical value associated with it, that value is
returned by the fval W A ! V 1; 1 function. This number is referred to as
the value of the attribute. The number of possible values that can be assumed by
an attribute (or in an infinite case, the cardinality of the set from which it can take its
values) is referred to as the (spatial) resolution of the attribute, and is denoted by
sres.a/ for attribute a. The frequency at which the value of an attribute is updated is
referred to as the (temporal) resolution of the attribute, and is denoted by tres.a/
for attribute a.
131
Definition 9.3. The intension of a concept is defined as the intersect of the sets of
attributes of all objects within the concept:
a .c / D a .O; A; Rc ; Ri ; Ro / D
#O
\
.Aoj /
(9.5)
jD1
where #O is the number of elements in O, and

Aoj D faj.oj ; a/ 2 Rc g
(9.6)
that is, Aoj refers to all attributes within the internal relations associated with
object oj . Whenever the intension of a concept is not empty and contains one or
more attributes, we refer to those attributes as the defining attributes of the concept
(this is in recognition of the fact that those attributes can be used to characterize all
objects in the concept). The cardinality of the set of defining attributes of a concept
is referred to as the dimensionality of the concept, and is denoted by dim.c/ for a
concept c.
Definition 9.4. The set of instant attributes of a concept c is defined as the union
of all attributes of all objects within the concept:
A .c / D A .O; A; R ; R ; R / D
c
#O
[
Aoj
(9.7)
jD1
where #O is the number of elements in O, and

Aoj D faj.oj ; a/ 2 Rc g
(9.8)
that is, Aoj refers to all attributes within the internal relations associated with
object oj .
Based on these definitions, a number of relational operators can be defined
in order to formally model conceptual knowledge-based systems. For details,
the reader is referred to Wang (2008), in which a set of operators is defined
to encapsulate human reasoning in a formal concept algebra based denotational
language and to manipulate knowledge in artificial cognitive entities.
9.1.2 Message Generated Perceptual Concepts

in Terms of OAR
Based on the above definitions, CogInfoCom message generated perceptual concepts can be formally defined as follows:
132
Definition 9.5. CogInfoCom message generated perceptual concepts, or perceptual concepts (Cmg ) of a set of CogInfoCom messages X are a set of abstract
concepts that belong to the context defined over those messages as objects, and have
at least one defining attribute:
o
n

Cmg .X/ D c
mg D .X; A; R/; 9a W a 2 a .cmg /
(9.9)
A perceptual concept c is defined over a message msg if that message is contained

in the considered set of CogInfoCom messages:
DO.c; msg/ , c 2 Cmg .X/; msg 2 X
(9.10)
Remark 9.1. As described earlier in Sect. 8.1.1, perceptual concepts can be identified in terms of the specific sensory system to which they pertain. Hence, in the
case of earcon generated concepts (where X is defined as the set of all earcons),
examples would include smoothness, roughness, consonance and dissonance (given
that earcons can be described using such concepts). Similarly, concepts such as
sweetness, perfume, and chocolate could be relevant to the olfactory domain (given
that smicons can be described using such concepts). All objects (i.e., earcons and
smicons) that can be characterized by these concepts can belong to the relevant
formal concept (whether they actually do depends on whether they are considered
at all in a particular implementation).
Remark 9.2. The notion of a perceptual being defined over a message is analogous
to the earlier notion of a concept being defined over a particular object, or set of
objects (cf. Definition 9.2).
In many cases it is advantageous to broaden this definition by ensuring the lowest
possible cardinality for the set of defining attributes (i.e. finding concepts with the
lowest possible dimensionality). In this case, the set of objects (i.e., CogInfoCom
messages) within the concept is augmented until no further reduction in the number
of defining attributes is possible:
Definition 9.6. A CogInfoCom message generated perceptual concept is fully
specified if no augmentation of the set of messages within the concept can lead
to further reduction in its dimensionality:
8
9
D .X; A; R/; 9a W a 2 a .c
mg /; >
>
>
>
>
>
>
>
0
>
6
9x;
9c
.X
[
x/
W
<
=
mg

Cmg .X/ D cmg
>
>
>
dim.c0mg .X [ x// > 0;
>
>
>
>
>
>
:
;
a .c0mg .X [ x// a .c /
(9.11)
133
There are two reasons why it is useful to consider fully specified CogInfoCom
message generated perceptual concepts:
1. Given that the remaining defining attributes characterize all messages within the
concept, and also that this group of messages is as broad as possible, any of the
messages can in theory be used within a CogInfoCom channel associated with
one of the defining attributes.
2. By reducing the cardinality of the intension to a minimal number, the cardinality
of the set of instant attributes of the CogInfoCom concept will simultaneously be
increased to the broadest set of attributes possible. This will have consequences
in high-level direct mapping, as described later in Sect. 9.2.
Naturally, in practical implementations a number of considerations can lead
to approximate applications of this definition. For instance, the computational
requirements of an application might cause an application designer to settle for
an acceptably small cardinality of defining attributes rather than considering all
possible messages with which the existing concept can be augmented.
Definition 9.7. A perceptual evaluation function is a function that associates a
real number between 0 and 1 with a CogInfoCom message x 2 X and a CogInfoCom
message generated perceptual concept defined over it:
feval W X Cmg .X/ ! 0; 1
(9.12)
such that the output value represents the degree to which the given CogInfoCom
message generated perceptual concept represents the CogInfoCom message. The
output value of a perceptual evaluation function is referred to as the perceptual
value of the message with respect to the given concept.
Remark 9.3. It is important to note that the output of function feval is a characterization of the perceptual relationship between a message and a concept defined
over it, as opposed to the fval function defined earlier, which provides an objective
characterization of attribute values.
Remark 9.4. The output of function feval is equivalent to one dimension of the
perceptual gradation vector defined earlier in Sect. 8.2.1. This means that if it were
possible to invert feval , one could obtain the CogInfoCom message (or set of CogInfoCom messages) which generate the specified perceptual value. Unfortunately,
such an inversion is not possible in general, as argued in the following remark.
Remark 9.5. The output of feval will in general be dependent not only on the given
message and perceptual concept, but also on the cognitive entity performing the
evaluation, as well as on other circumstances reflected in the CogInfoCom modality,
as described in Sect. 8.1.3. This fact shows that even if feval could be specified in
analytic terms, doing so would be impractical for realistic applications. As a result,
134
the inversion of feval as mentioned in the previous remarkis also not possible in
general. This suggests that while the feval function might be useful for empirically
gauging the class of CogInfoCom messages that are suitable to produce certain kinds
of perceptual effects (a task of primary importance!), it will not in general be suitable
when searching for specific CogInfoCom messages with specific perceptual values.
9.1.3 Tools for Analogy: Interactive Concepts

In this subsection, a formal definition of interactive concepts is provided. The notion
of interactive concepts introduced here will be used in the definition of certain kinds
of analogy-based concept mapping techniques in Sect. 9.2.2.
Definition 9.8. The set of interactive concepts (Ciact ) associated with a CogInfoCom message generated perceptual concept cmg .X/ is a set of concepts that
have a defining attribute whose value changes through time to a degree that is
proportional to the perceptual evaluation function of the CogInfoCom message
generated perceptual concept:
Ciact .cmg .X// D
<
c D .O; A; Rc ; Ri ; Ro /; 8x 2 X
: 9a 2 a .c/ W
fval .a/
fval .a/
9
>
=
>
/ feval .x; cmg / ;
(9.13)
Remark 9.6. An example of an interactive concept of the CogInfoCom message

generated perceptual concept of tactile softness (so that cmg .X/ D softness, and
X is a set of tactons) is the concept of finger position (c D .O; A; Rc ; Ri ; Ro / D
finger position). The reason for this is that the concept of finger position has a height
attribute (a D height) that changes in proportion to the perceptual evaluation of
any tactile sensation with respect to the CogInfoCom message generated perceptual
concept of tactile softness (a=a / feval .x; cmg /, 8x 2 X); in other words, the softer
the surface, the greater the change will be in the height of the finger when the surface
is pressed down upon.
9.2 Concept Mapping Techniques as Inference in OAR

Relationships between various message-generated concepts can entail the emergence of a strong association between underlying CogInfoCom messages and icons.
Such mappings can be created in a number of different ways. The purpose of
this section is to summarize these mapping techniques using the concept algebraic
concepts developed earlier in the chapter.
135
9.2.1 Direct Mapping

Conceptual mapping is direct if direct representations of the data are transferred,
in the sense of directness defined by Norman1 (1991). It is possible to distinguish
between low-level and high-level forms of direct mapping.
Definition 9.9. Mapping between a concept c1 and a CogInfoCom message msg is
low-level direct if changes in the value of one of the defining attributes of c1 are
interpreted as a time series that directly controls the perceptual values of msg with
respect to a concept c2 that is defined over msg. Formally:
fval .a/ D ) feval .msg; c2 / D
where a 2 a .c1 / is a defining attribute of the transmitted concept, msg is the
CogInfoCom message onto which the concept is mapped, c2 is a perceptual concept
defined over msg (i.e., DO.c2 ; msg/), is a constant, and is a scaling constant.
Remark 9.7. Versions of this mapping technique have been referred to as the lexical
technique in earlier literature (e.g. Blattner et al. 1994).
Remark 9.8. It should be noted that while the formal definition expresses a linear
relationship, it is linear only in the perceptual sense: the same definition, translated
to the domain of generation parameters would in the general case yield non-linear
relationships in accordance with the general non-linearity of perceptual transfer
functions in cognitive entities. Although CogInfoCom channels generally apply
messages that can be ordered on a perceptually linear scale, non-linear relationships
can also be created if the scaling constant is made to depend on the nominal
change in attribute value reflected in .
Examples of low-level direct mapping include the audification of data in the
auditory domain (Hermann and Ritter 1999), direct force/tactile feedback in the
haptic domain, or the direct transmission of smells using electronic noses and scent
generators.
Definition 9.10. Mapping between a concept c1 and a CogInfoCom message msg is
high-level direct if the defining attribute a of c1 is also an instant attribute associated
with a perceptual concept c2 that is defined over msg. In other words, high-level
direct mapping is possible if:
9a; c2 W DO.c2 ; msg/; a 2 a .c1 /; a 2 A .c2 /
Norman refers to any artificial device that is capable of storing, displaying or modifying
information as a cognitive artifact. Norman writes about different levels of directness with respect
to such devices, to qualify the relationship which exists between a users actions and the changes
these actions create in the stored information (Norman 1991).
136
and if this is the case, then high-level mapping occurs whenever:

fval .a/ D ) feval .msg; c2 / D
One example of high-level direct mapping is the use of auditory granular
synthesis based representations that give rise to direct associations for conveying
information on tactile roughness (Csapo and Baranyi 2012a). Another example
would be to use different kinds of colors (i.e., visual icons) to convey information on
temperature, as colors can be used to generate concepts of coldness and warmth.
High-level direct mapping reflects case-based relationships. For example, not all
outputs of granular synthesis can be used to communicate the concept of roughness.
Similarly, not all colors can be categorized as cold or warm. Nevertheless, the
mapping scheme is direct because the resulting CogInfoCom message gives rise to a
direct interpretation based on the common attribute it shares with a defining attribute
of the transmitted CogInfoCom concept.
9.2.2 Analogy-Based Mapping

Conceptual mapping is analogy-based whenever the necessary associations are
created based on the spatial and temporal aspects of the context in which a concept
is perceived (either generally or in particular cases), rather than the concept itself.
As in the case of direct mapping, it is possible to distinguish between several kinds
of analogy-based mapping.
Definition 9.11. Mapping between a concept c1 and a CogInfoCom message msg
is structural if the dimensionality of c1 is mapped onto the dimensionality of a
perceptual concept c2 that is defined over msg; or if the spatio-temporal resolution
of a defining attribute a1 of c1 is mapped onto similar characteristics of an instant
attribute a2 of a perceptual concept c2 that is defined over msg:
9a1 ; a2 ; c2 W a1 2 a .c1 /; DO.c2 ; msg/; a2 2 A .c2 /
and
dim.c1 / D ) dim.c2 / D , or
sres.a1 / D ) sres.a2 / D , or
tres.a1 / D ) tres.a2 / D
Remark 9.9. Several researchers have in the past referred to the structural mapping
technique as parameter mapping (e.g. Hunt et al. 2004; Madhyastha and Reed
1994; Scaletti 1994). The term structural mapping is used in CogInfoCom because
the structural parameters that are mapped to the CogInfoCom message are not
137
necessarily parameters that are related to the generation of the transmitted concept,
and also do not generally yield perceptual evaluations that would in any way
resemble the original concept.
One example of structural mapping would be the use of graphical triangle
representations to convey information on the height, weight and age of a group
of individuals. A triangle representing a certain person could use a color coding of
age, and the height and skewness of the triangle might reflect the height and weight
of the individual, respectively. Clearly there is no readily available semantic link
between age and color or weight and skewness, however, the dimensionality of the
two representations is the same and can therefore be brought into correspondence.
Another example of structural mapping can be taken from the domain of
vibration-based interaction with mobile devices. As the number of possible types
of vibration is limited by hardware constraints, a bijective relationship between
vibration type and information type is generally impossible to achieve. Hence, while
some information can be conveyed through spatial resolution (e.g., more complex
vibrations represent a more complex information types), generally users are forced
to rely on the temporal resolution of the vibrations. A vibration perceived on an
hourly basis, for example, would be interpreted differently than a vibration that
occurs once a day.
Structural mapping can also be useful in augmenting the sensing capabilities
of artificially cognitive entities. The structural parameters associated with various
activities can for instance allow a mobile device to determine whether the user is
walking, running, reading a book or making coffee, irrespective of semantically
grounded a priori knowledge.
Definition 9.12. Mapping between a concept c1 and two or more CogInfoCom
messages msg1 ; : : : ; msgn is corroborative stimulation (co-stimulation) if association between the concept and messages is created through temporal simultaneity.
Remark 9.10. Corroborative stimulation can rely both on natural associations (e.g.,
this is the case when the contact sounds of a remote probe are fed back to the
teleoperator along with a video stream to provide augmented information on what
is happening in the remote environment), or on associations that are formed through
training (e.g., by simultaneously providing contact sounds and an abstract graphical
representation, in which case the contact sounds would support the user in learning
the meaning of the abstract graphical representation).
Definition 9.13. Mapping between a concept c1 and a CogInfoCom message msg
is scenario-based if the data flow within CogInfoCom messages is specified based
on the structure of some (physical or virtual) interaction.
In the physical case, conceptual mapping can be formally modeled using an
interactive concept ciact of c1 , such that a defining attribute a1 of ciact is mapped
onto a defining attribute a2 of a perceptual concept c2 that is defined over msg:
fval .a.ciact .cmsg1 /// D ) feval .msg2 ; c2 / D
(9.14)
138
where a.ciact .cmsg1 // is a defining attribute of an interactive concept (Ciact ) of the

transmitted concept (cmsg1 ), msg1 is the substituted CogInfoCom message, msg2 is
the transmitted CogInfoCom message, c2 is a perceptual concept defined over msg2
(i.e., DO.c2 ; msg2 /), is a constant, and is a scaling variable.
Remark 9.11. In the physically based form of scenario-based mapping, attributes
of the substituting CogInfoCom messages do not directly reflect the transmitted
concept, but instead reflect an attribute of a concept that changes in accordance with
a certain physical interaction.
Remark 9.12. Special cases of the physically based form of scenario-based mapping have been referred to as scenario-based orchestration, to reflect the fact that
CogInfoCom icons are structured within a message so that the order of icons is
presented based on the order in which a physical interaction occurs (Csapo and
Baranyi 2012a).
Remark 9.13. Model-based sonification (Hermann 2002) can be regarded as a
special case of virtually based scenario-based mapping that pertains to earcons. The
approach is based on a virtual physical model which reflects the structure of the
transmitted data. The virtual model can be stimulated through physical interactions,
so that users are able to link the characteristics of their interactions with the reaction
of the model an thus gain an intuitive appreciation of the structure of the underlying
data.
Scenario-based mapping can be useful if there is no opportunity to create a direct
mapping between a concept and a CogInfoCom channel, or if the transmission of a
number of concepts is required at the same time and it is not clear what order they
should be presented in (in human-ICT interactions it is especially important that the
order of icons be interpretable as an analogy with physical interactions).
9.2.3 A Note on the Joint Use of Mapping Techniques

It should be noted that the various direct and analogy-based mapping techniques
described in this chapter do not contradict each other, and in many applications may
even be used in combination for greater effect. For example, a recent application in
which audio-based CogInfoCom channels were used to provide feedback on tactile
information, high-level direct mapping techniques were combined with scenariobased mapping to model the dimensions of tactile softness, roughness, stickiness
and temperature (Csapo and Baranyi 2012a). Corroborative stimulation in particular
can be used as a generic approach to further strengthen associations created through
direct and other kinds of analogical mapping techniques. The point of view adopted
by CogInfoCom channels that messages can be comprised of any number of icons
allows for the combined use of a variety of representational elements.
9.3 Future Perspectives for Automated Mapping
139
9.3 Future Perspectives for Automated Mapping

The concepts and mapping approaches developed in this chapter provide general
guidelines on how to select perceptual concepts, and corresponding parameters,
which are suitable for information transmission in CogInfoCom channels. The
developed framework makes use of a concept algebraic denotational language that
shows potential for future implementation in automated reasoning systems which,
as a result, could in theory be applied to the computer-aided design of CogInfoCom
channels.
However, it is important to note that the limitations that characterize formal logic
based reasoning systems also apply to concept algebra. It has been argued in the
past that such systems suffer from brittleness with respect to e.g. potential changes
in the world and semantic ambiguities, and that their use is therefore limited to
well-defined domains (Vernon et al. 2007; Valiant 2003, 2013). To solve these
problems, some researchers have applied probability theory or other frameworks
modeling uncertainty to achieve some kind of relaxation in the interpretation of
formal knowledge bases (Halpern 1990). Others have introduced further restrictions
to guarantee e.g. that the concepts being manipulated are interpreted over a limited
set of objects (as an analogy to working memory from psychological models
of human reasoning), and that the rules used within the reasoning process do not
exceed a certain level of complexity (Valiant 1999, 2013). Such restrictions may
also be necessary in the application of the concepts presented in this chapter. For
example, limitations in the number of objects considered when computing the
intension of a perceptual concept may be useful. Such considerations have not been
addressed in the chapter, but nevertheless can be expected to have significance in
future research on conceptual mapping.
Chapter 10
Tunability of CogInfoCom Channels
This chapter investigates the need for designers and users to be able to customize
CogInfoCom channels. It is argued that the availability of tools for this purpose
is important due to the specificities of the CogInfoCom modality that is used
especially with respect to the input device and the noise level characteristic of the
transfer medium. However, the task of creating such a model is rendered difficult
due to the fact that the function which links all possible combinations of generation
parameter values to perceptual qualities (referred to as feval in Chap. 9) is both
difficult to compute and also practically impossible to invert. One possible solution
to this challenge is to apply a tuning model that allows users to interactively
explore the parametric space used to generate CogInfoCom messages. The chapter
introduces the spiral discovery method (SDM)a tuning model that fulfills these
requirements and also empirically aims to support flexibility and interpretability.
10.1 Preliminary Discussions

In this chapter, a generic and interpretable tuning modelcalled the Spiral Discovery Method (SDM)is introduced for CogInfoCom channels. The main purpose
of SDM is to allow users to interactively explore any parametric space used to
generate CogInfoCom messages within CogInfoCom channels based on perceptual
qualities.1 Phrased differently, the primary concern of SDM is to provide a cognitive
artifact that can be used to design perceptually orderable CogInfoCom messages by
supporting an interactive searching process for appropriate generation parameters.
1
Since the feval function introduced in Chap. 9 is not analytically available and also not invertible,
tuning will by necessity resemble a searching process rather than a computation of parameter
values from a quantification of a desired perceptual quality.

DOI 10.1007/978-3-319-19608-4_10
141
142
10 Tunability of CogInfoCom Channels
In certain casesif the number of generation parameters used is small, or if

the perceptual effects of the generation parameters are intuitively clearusers
can easily specify perceptually orderable CogInfoCom messages by modifying
the generation parameters accordingly (Fig. 8.2 gives an overview of how generation parameters are related to perceptual parameters and CogInfoCom channels).
However, in most cases the structure of the parameter space used to generate the
messages is multi-dimensional and contains significant non-linearities, in two senses
of the word:
Small changes in a single parametric dimension can possibly lead to large
variations in the perceptual qualities of the generated message.
In many cases there will be more than a single parametric dimension, and the
effects of changing different parameters will not be perceptually separable.
Such properties of CogInfoCom channels combined with the potentially unlimited number of generation parameters that can be used make it difficult, if not
impossible, for the user to understand what effects their modifications will have on
specific perceptual qualities.2 In the case of CogInfoCom channels, it is not uncommon to use 2530 parameters to generate CogInfoCom messages. Thus, we say that
the generation parameter spaces underlying CogInfoCom channels generally have
high complexity and low interpretability. The goal of SDM is to provide designers
and users with a cognitive artifact that increases the interpretability of the tuning
task, while trading off minimal loss in the complexity of the model.
In general terms, SDM makes use of the fact that in CogInfoCom application
scenarios, it is perfectly suitable to find solutions (i.e., CogInfoCom channels) that
work sufficiently well but are not necessarily optimal.3 SDM, then, allows the user
to perform tuning operations while reverting back and forth between the original
parameter space and a rank-reduced parameter space that is different each time
(albeit in a controlled way). The following operations are available in any order
and an unlimited amount of times during the use of SDM:
An arbitrary succession of CogInfoCom messages can be assembled along
perceptual gradations of a given perceptual dimension (generally, the set of
messages will serve as a starting point for a more refined search so that the
succession of messages can be perceptually smoother).
HOSVD (higher-order singular value decomposition) can be performed on
the tensor representation of the CogInfoCom channel (more specifically, on
the tensor representation on the channels parameter-generating function).
From an engineering perspective, this point can be appreciated if one considers how difficult it
used to be to manually tune a radar locator. Although today this is a task that can be performed
through automation, a few decades ago it was a task for humans to tune the elevation, azimuth and
carrier frequency of the radar locator. This required the use of just three controls; nevertheless, it
was a hugely difficult task that demanded much practice and attention.
It is also important to note that in general it is difficult to specify in the first place what perceptually
optimal CogInfoCom messages would be like.
10.1 Preliminary Discussions
143
This amounts to creating a minimal parameter space and a set of weighting

functions which can be used to reconstruct the original parameters. The minimal
set of weighting functions can itself be modified to change the characteristics of
the CogInfoCom channel.
HOOI (higher-order orthogonal iteration) can be performed to obtain a rankreduced set of parameter dimensions and a corresponding set of weights that will
allow the user to approximate the original CogInfoCom channel and modify it in
a space of reduced dimensionality.
The rank reduction arrived at using HOOI can be compensated for by using SDM
to systematically alter the principal components of the rank-reduced parameter
space. The user, then, is able to explore the original, multidimensional parameter
space using a reduced set of parameters, albeit in a constrained form (the
constraints arise from the fact that the principal components of the rank-reduced
parameter space are changed transparently but systematically by SDM).
Figure 10.1 demonstrates the conceptual background behind the SDM method,
i.e., how the original parameter space to be explored (which can contain any number
of dimensions) can always be reduced to the parameters necessary to define a hyperspiral.4 The figure demonstrates that while SDM does not guarantee in any case that
the whole parameter space can be discovered, it nevertheless provides a structured
means of discovering perceptually meaningful subspaces.
The value of SDM is that it allows the user to trade off the complexity of
the original system and the interpretability of a rank-reduced system. By iterating
through the actions listed above in an interactive way, the user is allowed to either
tune the set of sounds in the original parameter space, in a rank-reduced parameter
space, or in a rank-reduced parameter space in which the principal components are
systematically altered.5 Due to the fact that human perception is not crisp in the
sense that small changes can in many cases be tolerated and adapted to, the SDM
method can serve as a valuable cognitive artifact for tuning CogInfoCom channels.
This chapter is divided into four sections. In Sect. 10.2, the mathematical
background of SDM is detailed. Section 10.3 describes a generic tuning model
based on the representation of CogInfoCom channels previously introduced in
Chap. 8. Section 10.4 introduces the concept of interpretability in tuning, and
provides a formal description of SDM based on the generic tuning model. Finally,
implementation details are provided in Sect. 10.5, and a brief empirical study is
described to demonstrate the applicability of SDM in Sect. 10.6.
From the users perspective, the direction of the hyper-spiral is a transparent parameter, as it will
coincide with the direction of the principal component of a set of control points.
Note that the interactive scheme described here is somewhat relevant to the paradigm of
Interactive Evolutionary Computation, as described by Takagi (2001), in the sense that the users
subjective evaluations are used to guide an iterative search process.
144
path of
discovery
Fig. 10.1 The schematic in this figure shows how SDM, the proposed interpretable tuning model
works. Instead of using the original parameter spacedenoted in this case by p1 ; p2 and p3 (it
should be noted that the number of parameters in general can be much more than 3)it is always
possible to describe a rough approximation of the original parameter space using the parameters of
a discovery spiral. These parameters can be set transparently, without the users intervention (for
example, the direction of the spiral can correspond to the direction of the principal component of
a set of control points already specified by the user; parameters r, d and can be set so that the
hyperspiral covers a representative portion of the parameter space, as required by the application,
etc.). The user, in turn, is only required to manipulate two parameters: a distance parameter in
terms of discrete steps along the hyperspiral (denoted by s in the figure), and a velocity parameter
which specifies the size of the steps (denoted by v in the figure). An important idea behind SDM
is that the users perceptual sensitivity will generally have some tolerance, thus it is sufficient to
discover a structured subspace of the parameter space rather than its entirety
10.2 Mathematical Background

10.2.1 Tensor Product Form of Multivariate Functions
In the following, the definition of various kinds of tensor product functions is
provided in the context of the function types defined earlier in Chap. 8.
Definition 10.1. There exists a subset of all partially ordered multivariate functions
(POMFs) which can be written in the following tensor product form:
M
mD1
lD1
F.x; y/ D S wm .xm / ul yl
(10.1)
10.2 Mathematical Background
145
where:
S 2 RI1 :::IN H is a core tensor of finite dimensions
Each wm .xm / D wm;1 .xm /; wm;2 .xm /; : : : ; wm;Im .xm / is a vector of univariate
weighting functions defined over a partially ordered set
Each ul yl D ul;1 yl ; ul;2 yl ; : : : ; ul;Il yl is a vector of univariate weighting
functions defined over an ordered set
xm denotes the input in the mth partially ordered input dimension
yl denotes the input in the lth ordered input dimension
POMFs which can be written in this form are referred to as partially ordered
multivariate tensor product (TP) functions (POTPFs).
Property 10.1. Tensor product (TP) functions are multivariate functions which can
be represented by a core tensor S and a set of univariate weighting functions wn .xn /
(Baranyi 2004, 2014; Baranyi et al. 2006). The concept of POTP functions is similar
to this class of functions, but it allows certain input dimensions to be partially
ordered. Hence, we can say that the original concept of TP functions is a special
case of POTP functions, in which all input dimensions can be ordered.
Definition 10.2. Discretized partially ordered TP functions are partially ordered
multivariate functions (POMFs) that are discretized along an N-dimensional grid,
G D fgp1 ;p2 ;:::;pN 2 Ra1 ;b1 a2 ;b2 ;:::;aM ;bM Nc1 ;d1 c2 ;d2 ;:::;cL ;dL gPpnnD1 , and can be
expressed in the following tensor product form:
M
mD1
lD1
D.G/
Ul
F D.G/ D S WD.G/
m
(10.2)
where
S 2 RI1 :::IN H is a core tensor of finite dimensions
D.G/
Wm D wm;pm .xm / is a matrix in which the pm th row represents the pm th vector
of univariate coefficient functions along the mth continuous input dimension
Each coefficient function in the pm th row of the matrix is evaluated at the pm th
value of the mth discretized dimension (xm )
D.G/
Ul
D wl;pl yl is a matrix in which the pl th row represents the pl th vector of
univariate coefficient functions along the lth discrete input dimension
Each coefficient function in the pl th row of the matrix is evaluated at the pl th
value of the lth discretized dimension (yl )
10.2.2 HOSVD-Based Canonical Form of Discretized

Multivariate Functions
Theorem 10.1 (Higher-Order Singular Value Based Canonical Form of Discretized Multivariate Functions). Every discretized partially ordered multivariate function, F D.G/ can be written as the product:
146
D.G/

M
L
NC1
D S Wm Ul NC1 ULC1 D S Xn
mD1
lD1
nD1
(10.3)
in which:
.m/
.m/
1. Wm D .w1 ; : : : ; wIm /, m D 1::M is a orthonormed matrix of size .Pm Im /

.l/
.l/
2. Ul D .u1 ; : : : ; uIl /, l D 1::L is a orthonormed matrix of size .Pl Il /

.LC1/
.LC1/
3. ULC1 D .u1
; : : : ; uILC1 / is a orthonormed matrix of size .H ILC1 /
4. S is a real tensor of size I1 : : : IN H, the subtensors Sin D of which have
the following properties:
all-orthogonality: any pair of the subtensors of S are orthogonal, i.e. for all
possible values of n; and subject to :
< Sin D ; Sin D >D 0
(10.4)
ordering: All of the subtensors of S along any given dimension n are ordered
according to their Frobenius norm, i.e. 8n D 1::N C 1:
jjSin D1 jj jjSin D2 jj : : : jjSin DIn jj 0
(10.5)
Proof. The HOSVD of any N-dimensional tensor with real values was introduced
by de Lathauwer et al. (2000). The fact that discretized multivariate functions can
be stored in such tensors, as demonstrated in Eq. (8.2), proves the theorem.
10.3 Generic Tuning Model for CogInfoCom Channels

In Chap. 8 it was established that the parameters for icons and messages can be
stored in an .N C1/-dimensional tensor, denoted by F D.G/ . Using the compact form
of higher-order singular value decomposition (CHOSVD) in which singular values
that are negligible or 0 are removed from the system, it is possible to decompose the
tensor in the following form (de Lathauwer et al. 2000):
NC1
F D.G/ D S Xn
(10.6)
nD1
where refers to the tensor product operation defined in de Lathauwer et al. (2000),
and:
.n/
.n/
1. Xn D .x1 ; : : : ; xIn /, n D 1::N is a orthonormed matrix of size .Pn In /

.NC1/
.NC1/
; : : : ; xINC1 / is a orthonormed matrix of size .H INC1 /

2. XNC1 D .x1
3. S is a real tensor of size I1 : : : IN INC1 , the subtensors Sin D of which
have the following properties:
10.4 The Spiral Discovery Method (SDM)
147
all-orthogonality: any pair of the subtensors of S are orthogonal, i.e. for all
possible values of n; and subject to :
< Sin D ; Sin D >D 0
(10.7)
ordering: All of the subtensors of S along any given dimension n are ordered
according to their Frobenius norm, i.e. 8n D 1::N C 1:
jjSin D1 jj jjSin D2 jj : : : jjSin DIn jj 0
(10.8)
The values I1 ; : : : INC1 are the ranks of the system along each of the input
dimensions. Tensor S is referred to as the core tensor, and matrices Xi are referred
to as weighting matrices. If the values in just the pk th row of any Xk in Eq. (10.6) are
modified, then only those output values of function F D.G/ will be changed which
belong to the pk th perceptual gradation along the kth dimension of hyper-rectangular
grid G. This can be easily seen if we express a single element of F D.G/ as follows:
NC1
F D.G/ .gp1 ;:::;pN / D S xn;pn k xk;pk

nD1;
nk
(10.9)
where the pk th row of matrix Xk is denoted by xk;pk (tensor multiplication with this
term can be written separately due to the commutativity of tensor multiplication).
It is obvious that if any point on hyper-rectangular grid G is chosen in which the
value of the kth dimension is not the pk th discretization point, then the output value
of the function will remain unchanged. For this reason, the manipulation of vector
xk;pk is referred to as the local tuning of the CogInfoCom channel along the kth input
dimension. The values in vector xk;pk in turn are referred to as tuning weights.

The local tuning model provided in Eq. (10.9) operates on a minimal-rank orthogonal system within the parameter space of the CogInfoCom channel. The minimalrank tuning weights could in theory be used for perceptual interpolation between
pairs of CogInfoCom messages, resulting in ordered sets of CogInfoCom messages.
However, the cognitive load associated with keeping track of the perceptual effects
of each of the weights may be significant depending on the rank of the system. The
ease with which the user can keep track of these perceptual effects can be referred
to as the interpretability of the tuning model. Using rank reduction techniques it
is possible to increase the interpretability of the tuning model, but in exchange
the system would suffer loss of complexity (i.e., only a subspace of the original
parameter space would be available for tuning using a rank-reduced system of tuning
weights). In addition, due to the fact that the spatial configuration of the parameter
148
space does not necessarily have much to do with perceptual relationships between
the CogInfoCom messages that are generated, a mere rank reduction of the tuning
model will not necessarily result in perceptually justified tuning controls.
Thus, it is important to relax the tradeoff between interpretability and complexity.
While the user can only concentrate on the perceptual effects of changes made to
tuning weights if the number of weights is small, it is equally important that the user
be able to explore the parameter space of RH to the fullest extent possible. To this
end, a novel approach is proposed that combines rank reduction and the adaptive
modification of the basis vectors in the rank-reduced space so that the full-rank
space can be explored in a restricted, but systematic way. Such an approach can be
achieved through the following steps:
1. Assuming that the user would like to tune the system in the kth dimension, the
first step is to reduce its rank from Ik to 1. This can be achieved using HigherOrder Orthogonal Iteration (HOOI)a method which is proven to yield optimal
rank-reduction (Ishteva et al. 2008)to obtain:
argmin .F D.G/ FO D.G/ /
NC1
O n gnD1
SO;fX
(10.10)
where
NC1
On
FO D.G/ D SO X
nD1
(10.11)
such that IOk D 1; IOn D In ; 8n k, and IOk is the rank along the kth dimension of
the new system.
2. Having obtained this approximation, the goal is to increase the rank of the
vector space which can be controlled by the single tuning parameter remaining
O n; n D
in the kth dimension. In order to achieve this, weighting matrices X
O
1::.N C 1/, as well as the core tensor, S are expanded such that the obtained
Q n and core tensor SQ can be used to reconstruct the tensor
weighting matrices X
representation of the original parameter-generating function:
NC1
Qn
F D.G/ D SQ X
(10.12)
nD1
Q n ; n D 1::.N C 1/ has a rank of Pn (the number of discretization

where each X
points along the given dimension, which is equivalent to saying that there are at
least Pn columns in the nth matrix), and SQ is augmented appropriately. After this
Q k ) are still
step, even if the single tuning parameters (i.e., in the first column of X
only be used to control a single dimension within the system, the space that can
be reconstructed by the system as a whole is RH once again (in other words, the
original system can be fully reconstructed if the values in the augmented part of
SQ are chosen appropriately).
149
3. By systematically modifying certain elements in the augmented part of the new

Q k is modified by the user, it becomes
core tensor, SQ, as the first column of X
possible for the user to traverse a subspace V RH such that V is also
H-dimensional.
In order to demonstrate how Steps 2 and 3 can be achieved (the fact that the
original function can be reconstructed in Eq. (10.12), and that if certain elements of
SQ are systematically modified, then a single tuning weight can be used to control
an H-dimensional subspace of RH ), the following lemma and theorem is provided.
Lemma 10.1. Let us consider the HOSVD of a discretized multivariate function
N
nD1
nD1
O n denote the rank-reduced instance

F D.G/ D S Xn , and let FO D.G/ D SO X
of the same function obtained using HOOI. The original tensor, F .D.G// can be
reconstructed if:
O n ; n D 1::.N C 1/ is augmented with new columns so that rank.X
Q n / D Pn
1. each X
O
2. each dimension of S is augmented so that the length of the nth dimension is the
Qn
same as the number of columns in X
Proof. It is trivial that if the lemma holds true, then the new core tensor, SQ can be
expressed from Eq. (10.12) as:
NC1
QC
SQ D F D.G/ X
n
nD1
(10.13)
The pseudoinverse of the augmented weighting matrices always exists, and

Q n / D Pn , and the rank along the nth dimension of the original tensor,
because
X

rank.
D.G/
rank .F
/.n/ D In Pn , the original tensor can be reconstructed (F.n/ denotes
the n-mode unfolding of tensor F , as described in de Lathauwer et al. 2000).
Q k (denoted by wi ; i D 1::Pk )
Theorem 10.2. The weights in the first column of X
define a hyperline in the H-dimensional space RH :
N
Q NC1 jwpk 2 Rg
HL D fSQ xQ n;pn k wpk NC1 X
nD1
nk
(10.14)
The projection of this hyperline onto output dimensions q1 ; q2 ; H; q1 q2 has

a slope which depends only on the values in the first subtensor of SQ along the kth
dimension, BQ D SQpk D1 :
N
.BQ xQ n;pn / NC1 xQ .NC1/;q2

slopeq1 q2 D
nD1
nk
N
.BQ xQ n;pn / NC1 xQ .NC1/;q1

nD1
nk
(10.15)
150
Proof. Expanding Eq. (10.12), we obtain:

N
Q NC1 /C
FpD.G/
Dwpk ..BQ xQ n;pn / NC1 X
1 ;:::;pN
nD1
nk
(10.16)
Q NC1 / k rpk
..BQ c xQ n;pn / NC1 X
nD1
nk
Q BQ c contains all subtensors of SQ except the first along

where the complement of B,
Q
Q k ), wpk is the pk th element
the kth dimension, Xk D .wjR/ (w is the first column of X
of w, and rpk is the pk th row of R. In simplified form, we may write:
0 1
f1
Bf C
B 2C
B C
B:C
Q Cc
B C D wpk m.B/
B:C
B C
@:A
fH
(10.17)
where
1
0
N
Q D @.BQ xQ n;pn / NC1 X

Q NC1 A
m.B/
nD1
nk
(10.18)
.NC1/
is a column vector obtained as the .N C 1/-mode unfolding of the tensor on the

right-hand side, as defined in de Lathauwer et al. (2000).
If we consider just a single weight, wpk , each element of the H-dimensional
parameter vector is broken up into a sum of two values: the first of these is a variable
Q while
(the weight itself) that is scaled by a term that is dependent on the values of B,
the second can be regarded as a constant. Based on this interpretation, it is clear that
the factor of the weight represents the slope of an H-dimensional hyperline. The
projection of the hyperline onto the output coordinates defined by dimension q1 ; q2
has the slope stated in the theorem.
It is clear that by augmenting SO in dimensions other than k, the number of
values that are scaled by each wpk are increased. If a certain part of these values
is systematically and periodically altered, the slope of the hyperline controlled by
the tuning model can be periodically changed.
10.5 Implementation of SDM
151
10.5 Implementation of SDM

An example of the implementation of Step 2 in the previous section is shown
in Fig. 10.2. In the proposed implementation, only two weighting matrices are
augmented, and accordingly, the core tensor, SO is only augmented along two of
its dimensions. These two dimensions are the kth and the .N C 1/th dimensions.
Augmenting just the kth dimension would he sufficient due to the fact that rank
reduction is only performed in this dimension, but augmenting the .N C 1/th
weighting matrix with an identity matrix is useful for controlling the slope of
the hyperline that is traversed when the tuning weights are manipulated. More
specifically, when calculating the ith parameter in output parameter vector f, the
core tensor is only multiplied by the ith row of XNC1 along its .N C 1/th dimension,
Fig. 10.2 Proposed augmentation method used to compensate for reduced interpretability after the
tuning model is rank-reduced, assuming that k D 1 and N D 2. The light (yellow) shades contain
fix values, and the dark (mauve and green) shades contain variables which can be chosen so as to
compensate for the rank-reduction as well as to alter the slope of the hyperline that is traversed
within RH during tuning
152
and due to the fact that this weighting matrix is augmented by an identity matrix,
only a single and unique subtensor of K will affect the slope of the hyperline.
The key idea behind the implementation is that while the user changes the
weights in the first column of the weight matrix which belongs to dimension k (in
Fig. 10.2, this is denoted by XO1 ), the values of K are periodically and cyclically
modified between two extremes.
It should be noted that while the generic tuning model described in Sect. 10.3
guarantees that the effects of changing a tuning weight will be local, this property
of locality does not transfer to SDM. The reason for this is that as subtensor K
of the core tensor is modified, all gradations of the dimension which is tuned will
be affected. Nevertheless, in practical cases the effects of tuning operations can be
made to be local using a simple implementation workaround: if the user finishes
manipulating a given gradation, and wishes to save the changes made, the local
parameters can be updated in the original tensor representation of F D.G/ , and the
first two steps of SDM (i.e., rank reduction and re-augmentation) can be performed
once again. These operations are not cost-inhibitive, and in any case only need to
be effected if the user wishes to save a set of manipulations with respect to a given
gradation.
10.6 Usability of SDM

The Spiral Discovery Method was implemented as part of a framework for creating
audio-based CogInfoCom channels (Csapo and Baranyi 2012c). Screenshots of
the graphical user interface used to tune messages can be seen in Fig. 10.3. As
the user changes the distance parameter value in SDM (bottom left-hand side
of Fig. 10.3), corresponding changes to the original generation parameters are
displayed immediately (bottom right-hand side of Fig. 10.3). In this way, it is clear
to users that by modifying a single parameter, they are able to discover a large
variety of generation parameter configurations.6 An example is shown in Figs. 10.4
and 10.5. In the former figure, the horizontal axis contains different generation
parameters, and each polyline connects generation parameter values that were active
at the same time. In the latter figure, a selection of generation parameter values
are plotted as a function of time. The figures show clearly how a large variety of
parameter combinations can be explored using just a single parameter in SDM, and
they also clearly demonstrate the fact that the path of discovery is indeed a spiral.
It is also clear that a large portion of the parameter space was traversed while using
only a single tuning parameter.
In order to empirically test the applicability of SDM, a generic model was
created to generate audio-based CogInfoCom messages. The synthesis model
The velocity parameter of SDM is set transparently to the user, as it influences the sensitivity
of the distance parameter, but does not influence the direction of parameter discovery.
153
Fig. 10.3 The GUI used for tuning an audio-based CogInfoCom channel. Controls on the righthand side can be used to directly control generation parameters. Controls on the left-hand side
of the top window represent parameters of the generic tuning model, while the single control
on the left-hand side of the bottom window represents the distance tuning parameter of SDM
(cf. Fig. 10.1). As the parameter of SDM is moved by the user, corresponding changes in the
original generation parameters are updated automatically
was created using 24 parameters, hence, the output dimension of the parametergenerating function contained 24 dimensions. The key point was that it is difficult
for any user to understand both the individual effects of 24 tuning parameters, as
well as the effects that they may have on each other. To further complicate the task,
the goal of the tests was to create a CogInfoCom channel that consists of 20 different
CogInfoCom messages.
The test was performed by ten subjects. Each subject was given as much time
as he/she needed to use SDM in order to find 20 different CogInfoCom messages
that could be ordered along any arbitrary perceptual quality, chosen by the user
according to his or her own liking. Following the tuning process, test subjects were
allowed a few minutes to practice, after which they were presented with 30 test
questions to measure recall precision.
154

1
Weights
0.8
0.6
0.4
0.2
rm
2
ha
rm
ha
ur
rm
ha
ho
ns
gd
pc
de
tc
h
pi
ur
gd
ur
3
gd
ho
pc
ho
h1
h3
pc
pc
pc
ns
de
ns
in
ou
de
fa
d
fa
d
vo
fadin
1
0.5
0
fadout
1
0.5
0
pch1
1
0.5
0
pch3
1
0.5
0
gdur1
1
0.5
0
gdur3
1
0.5
0
dens
1
0.5
0
gdur
1
0.5
0
harm1
1
0.5
0
harm2
1
0.5
0
harm3
Fig. 10.4 As users alter the value of the distance parameter in SDM, all of the generation
parameters change together in accordance with the tuning model. The figure shows various
configurations in the generation vector space which were discovered by a user of SDM. The
horizontal axis refers to different dimensions in the generation vector, and the vertical axis refers
to the normalized value of the corresponding dimension. Line segments with the same color refer
to parameter configurations which were active at the same time
1
0.5
0
20
40
60
80
100
time
120
140
160
180
200
20
40
60
80
100
time
120
140
160
180
200
20
40
60
80
100
time
120
140
160
180
200
20
40
60
80
100
time
120
140
160
180
200
20
40
60
80
100
time
120
140
160
180
200
20
40
60
80
100
time
120
140
160
180
200
20
40
60
80
100
time
120
140
160
180
200
20
40
60
80
100
time
120
140
160
180
200
20
40
60
80
100
time
120
140
160
180
200
20
40
60
80
100
time
120
140
160
180
200
20
40
60
80
100
time
120
140
160
180
200
Fig. 10.5 As users alter the value of the distance parameter in SDM, all of the generation
parameters change together in accordance with the tuning model. The figure shows various
configurations in the generation vector space which were discovered by a user of SDM, in terms
of the values of a selection of individual dimensions as they change through time
155
14
Error (out of 20 gradations)
12
10
8
6
4
2
0
1
Arbitrary dimension
Fig. 10.6 The boxplot shows error sizescounted in number of gradationsalong a single
arbitrary dimension, aggregated over ten test subjects. The red, horizontal line marks the median
error, the edges of the blue box mark the 25th and 75th percentile of error, the whisker marks errors
that lie outside of these percentiles, while the red crosses mark outliers
Figure 10.6 shows results from the tests. The figure shows that 75 % of all test
questions resulted in an error of at most two gradations. These can be regarded as
good results given that there were 20 gradations in a high-dimensional parameter
space. When it was suggested to test subjects that they could also tune the
CogInfoCom channels using the 24 original parameters (which were displayed
simultaneously with the SDM parameters), they agreed that the task would be so
difficult that it would not be even worth trying. Comparing this with the test results,
it can be seen that SDM is a viable and potentially useful approach to user-based
tunability (Fig. 10.7).
156
Response gradation
Arbitrary dimension Confusion Matrix

1
13
4.3%
3
1.0%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
76.5%
0.0% 23.5%
0
0.0%
8
2.7%
3
1.0%
2
0.7%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
57.1%
0.0% 42.9%
0
0.0%
4
1.3%
10
3.3%
9
3.0%
4
1.3%
2
0.7%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
33.3%
0.0% 66.7%
0
0.0%
1
0.3%
0
0.0%
6
2.0%
2
0.7%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
54.5%
0.0% 45.5%
0
0.0%
0
0.0%
1
0.3%
1
0.3%
7
2.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
77.8%
0.0% 22.2%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
3
1.0%
8
2.7%
2
0.7%
1
0.3%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
2
0.7%
0
0.0%
1
0.3%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
0
40.0%
0.0% 60.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
7
2.3%
1
0.3%
0
0.0%
1
0.3%
1
0.3%
0
0.0%
0
0.0%
1
0.3%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
53.8%
0.0% 46.2%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
8
2.7%
1
0.3%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
72.7%
0.0% 27.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
7
2.3%
2
0.7%
0
0.0%
3
1.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
0
50.0%
0.0% 50.0%
10
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
2
0.7%
7
2.3%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
63.6%
0.0% 36.4%
11
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
4
1.3%
3
1.0%
1
0.3%
3
1.0%
2
0.7%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
28.6%
0.0% 71.4%
12
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
1
0.3%
0
0.0%
2
0.7%
0
0.0%
3
1.0%
3
1.0%
2
0.7%
0
0.0%
0
0.0%
1
0.3%
2
0.7%
0
0.0%
0
20.0%
0.0% 80.0%
13
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
1
0.3%
7
2.3%
6
2.0%
1
0.3%
0
0.0%
3
1.0%
1
0.3%
0
0.0%
0
35.0%
0.0% 65.0%
14
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
2
0.7%
4
1.3%
6
2.0%
3
1.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
37.5%
0.0% 62.5%
15
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
1
0.3%
2
0.7%
1
0.3%
9
3.0%
2
0.7%
1
0.3%
1
0.3%
0
0.0%
0
47.4%
0.0% 52.6%
16
0
0.0%
2
0.7%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
2
0.7%
0
0.0%
0
0.0%
0
0.0%
4
1.3%
0
0.0%
0
0.0%
2
0.7%
0
36.4%
0.0% 63.6%
17
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
4
1.3%
10
3.3%
2
0.7%
2
0.7%
3
43.5%
1.0% 56.5%
18
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
3
1.0%
5
1.7%
1
0.3%
4
33.3%
1.3% 66.7%
19
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
1
0.3%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
5
1.7%
5
1.7%
1
38.5%
0.3% 61.5%
20
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
0
0.0%
1
0.3%
1
0.3%
2
50.0%
0.7% 50.0%
100% 44.4% 62.5% 31.6% 38.9% 72.7% 46.7% 66.7% 58.3% 46.7% 50.0% 16.7% 36.8% 30.0% 50.0% 36.4% 52.6% 29.4% 45.5% 20.0% 45.3%
0.0% 55.6% 37.5% 68.4% 61.1% 27.3% 53.3% 33.3% 41.7% 53.3% 50.0% 83.3% 63.2% 70.0% 50.0% 63.6% 47.4% 70.6% 54.5% 80.0% 54.7%
9 10 11 12 13 14 15 16 17 18 19 20
Target gradation
Fig. 10.7 Confusion matrices for the validation of SDM with arbitrary dimensions (10 test
subjects, 30 test questions)
Chapter 11
Temporal Aspects of CogInfoCom

Channel Design
Earlier chapters have described an approach to defining icons and messages for the
design of structured multi-sensory signals capable of providing information on highlevel concepts. Various mapping techniques between generating parameters and
meaning have been presented, and the tunability of the resulting channels through
complexity-preserving, but still interpretable operations has also been discussed.
This chapter focuses on dynamic aspects of communicationi.e., aspects which
pertain to the development of communication through time. As this subject area has
more relevance to the long-term co-existence of humans and ICT than to episodic
interactions, its details are relatively less well understood. For this reason, the
chapter cannot aim to provide definitive answers; instead, its goal is to provide
an introduction to the topic by at least highlighting some of the more important
challenges and by presenting initial ideas that have been proposed for further
elaboration.
The chapter is structured as follows. Section 11.1 provides a brief introduction to
the dynamic aspects of communication considered in the chapter, and highlights two
key questions that motivate this research. This is followed by a discussion on recent
ideas obtained from low-level (i.e. non-human) communication (in Sect. 11.2)
and high-level human-human communication (in Sect. 11.3). A general outline on
how these concepts could be applied towards the design of adaptive CogInfoCom
channels is provided in Sect. 11.4.
11.1 Challenges Behind Temporal Aspects

According to their original definition, CogInfoCom channels represent a very
specific kind of communication that occurs at an explicit and conscious level. This
is in keeping with the broader context of engineering design, where communication
has primarily been interpreted as a set of behaviors that are planned for with
DOI 10.1007/978-3-319-19608-4_11
157
158
11 Temporal Aspects of CogInfoCom Channel Design
a concrete use case in mind and implemented prior to deployment. This is a

viable approach as long as the information requirements of the application, and the
messages to be conveyed so as to fulfill those requirements are known in advance.
The fact that the approach removes all spontaneity from communication is also
far from problematic; on the contrary, lack of spontaneity is seen as a crucial
element in the design of dependable systems (it is interesting to note that one of the
initial criticisms brought up against affective computingas discussed in Picard
(2003a)1was also motivated by the perceived dangers of spontaneity vis--vis
dependability).
In this chapter, it is argued that a reconsideration of these ideals would be
worthwhile for at least three reasons:
Through the entanglement between humans and ICT, the scope of both personal
and social ICT applications is expected to grow towards areas where longterm co-evolvability will become increasingly important. Thus, while individual
interactions (as in the user pressing a button or giving a voice command) may
still be held up to the criterion of dependability, the effectiveness of long-term
co-evolution will be judged based on different criteria, such as the ability of
human-ICT communication to remain spontaneous, interesting and useful.
As outlined in later parts of this chapter, it is clear that natural communication
consists of a rich variety of behaviors that cannot be described through explicitly specified communicational exchanges. For example, it is well-known that
communication between humans has both explicit and implicit aspects. From
the perspective of semantic meaning, explicit communication is generally more
precise, but also requires relatively more cognitive resources and can quickly
grow into a burden if continued for excessive periods of time. On the other
hand, implicit communication is less precise but also requires less attention;
it can be disregarded or otherwise handled with more flexibility. As a result,
different forms of communication exist, and it turns out that different forms of
communication can be effective in different situations depending on a number of
aspects, including the novelty, urgency and usefulness of the conveyed meaning,
as well as on the personality of the interlocutor and what he or she is doing at the
time of communication.
Natural communication can also be seen as an adaptive process that evolves with
respect to the requirements of a changing environment. Two people who know
each other well communicate differently and with different expectations than
people who have only just met. Communication also evolves to be different as the
topic that is focused on becomes more familiar: as shared knowledge increases
and relevant conversational limits are negotiated, communication gradually
becomes less direct and less formal. Despite the fact that personal devices and
personal communication agents are gaining increasing relevance (as is also
1
Emotions and passions tend to be more problematic than helpful in human-human interaction.
There is no need to contaminate purely logical computers with emotional reactiveness (Picard
2003a).
11.2 Aspects of Low-Level Biological Communication
159
clear from the Gartner hype cycles of the past years, such as those shown in
Fig. 1.3), such aspects of communication are almost entirely missing from the
considerations applied in mainstream engineering practices.
Based on these considerations, it is becoming increasingly important for the
engineering sciences to take the spontaneity, variability and progression of natural
communication into account when designing systems for human-ICT co-evolution.
While many authors have in the past advocated the incorporation of individual
aspects of natural communication into engineering design, these three factors have
been rarely addressed together. In our view, answering the following two questions
will be key to making further progress:
1. How does communication emerge in the first place: what phases does a series of
interactions have to go through for it to be considered as communication?
2. Once communication is established, how is it maintained, and how does it
evolve in order to remain relevant, even as the cognitive beings involved
form increasingly detailed representations of each others habits, attitudes and
motives?
Although a definitive answer to these questions currently seems elusive even in
the biological and social sciences, we believe that being aware of their existence
is already half the solution. A greater awareness of how potential answers to these
questions could contribute to engineering design may eventually lead to important
results. In the remainder of this chapter, some ideas from low-level biological
communication and high-level human communication to support future research
are highlighted.

Two concepts which emerge in biology and which have no well-defined parallel
in engineering design are cues and signals (Scott-Phillips et al. 2012; Ruxton
and Schaefer 2011; Carazo and Font 2010). Cues are behaviors which do not in
themselves qualify as a form of communication per se, but which can evolve
through a process referred to as ritualizationinto purposefully generated signals
if they are perceived as effective in eliciting predictable and useful responses from
other individuals in the population. One often cited example is the tendency of
several species of animals to mark their territory through urination: what was
originally a natural response caused by fear grew into a reliable cue of an animals
presence; later, the effectiveness of this cue resulted in the same behavior being
produced without the presence of fear, primarily with the goal-oriented purpose of
averting conflicts (Scott-Phillips et al. 2012).
When considering possible relationships between the existing framework of
CogInfoCom channels and biological cues and signals, it seems viable to view
CogInfoCom channels as a higher-level extension to the latter two concepts. While
160
the concept of CogInfoCom channels in this interpretation still retains its semantically oriented, message-like nature, it is also seen as a form of communication that
evolves from repetitive variations in lower-level signals (in much the same way as
signals are in turn built on the foundation of cues). Before providing a tentative
definition of the concepts of cues and signals, and describing their relationship
with icons and messages, a brief summary of considerations supporting such a
hierarchical approach is provided as follows:
It may be useful to model the biological transition between cues and signals in
ICT systems, as this would allow for a transitioning from interaction modes with
no implicit communication (i.e. characterized only by goal-oriented exchanges)
to interaction modes with meaningful implicit communication (characterized by
contextually supported implicit interpretations besides goal-oriented exchanges).
CogInfoCom channels can be explicitly designed and taught to users (or programmed into artificial systems), but it is unlikely that they can emerge spontaneously and still be effective. If an alternative to explicit design and training
is considered, it is likely that such channels will be formed through extended
periods of prior interaction that leads to both cognitive entities picking up on
the kinds of subtleties that can form the basis of differentiation among alternative
interpretations.
Cues, signals and channels are different and complement each other well in
the sense that their perception and interpretation is founded on different levels
of goal-directedness and also requires different levels of attention. Cues are
unique in that they emerge primarily through goalless behaviors not targeted
at communication; signals are unique in that they emerge primarily through
goal-directed behaviors without explicit shared meaning; and finally, channels
are unique in that they are produced in goal-oriented ways based on previously
established conventions of shared meaning.
11.2.1 CogInfoCom Cues and Signals

Based on the preliminary remarks above, CogInfoCom cues can be defined as
follows:
Definition 11.1. CogInfoCom cues are sensory percepts that give rise to basic
semantic associations.
It is important to note the distinction between immediate and unique semantic
interpretations (as in the definition of CogInfoCom icons in Sect. 7.3) and basic
semantic associations. The distinction between basic association and unique
interpretation lies in the fact that cues often provide binary, on-off information
on e.g. the existence/non-existence of an entity or the occurrence/non-occurrence
of an event, whereas icons can (and generally do) give rise to more complex
interpretations. Also noteworthy is the fact that whereas CogInfoCom icons give
161
SIGNALS
CUES
Fig. 11.1 Structural and interactive aspects of communication. Cues and icons have direct
meaning, whereas messages have abstract meaning. In some cases, cues are also icons at the same
time, while in others, cuesunlike iconsdo not give rise to immediate and unique associations
rise to immediate and unique interpretations, it is not necessary for CogInfoCom

cues to be immediate or unique: a cue is a by-product of the autonomous behavior
of a system (rather than a reflection of its intention to communicate) that creates
possibilities for semantic association as a result of a relatively longer history of
interactions. An interpreter of a cue need not be correct in associating towards a
certain meaningyet, if the association is stable and useful then it can still evolve
into a ritualized signal. Thus, while cues and icons are not necessarily mutually
exclusive, cues are generally simpler and less crystallized in their meaning. Based on
the above, it can be concluded that the class of CogInfoCom cues partially overlaps
with the class of CogInfoCom icons, so that some (but not all) CogInfoCom icons
are also CogInfoCom cues (Fig. 11.1).
In contrast to cues, the concept of signal is implicit in CogInfoCom messages,
so long as those messages are evolved from cues. Therefore, the terms signal
and message can be used interchangeably if such an evolutionary background is
implicitly understood:
Definition 11.2. CogInfoCom signals are CogInfoCom messages which emerge
through the repeated availability and successful interpretation of a set of CogInfoCom cues.
In the remainder of these discussions, the term message is preferred as it reflects
the already established terminology behind CogInfoCom channels. This causes no
ambiguity as all signals are messages; the set of signals are contained in the set of
messages (Fig. 11.1).
162
11.2.2 Cues, Signals and Messages in Past Research

In the fields of HCI and multimodal interactions, terms such as (multimodal)
feedback, feedback cues, feedback signals, feedback information, etc. are used
quite liberally and often interchangeably. In some cases, cues are used in the sense
described here; e.g., in Sharmin (2003), the authors use ternary haptic cues to
provide users feedback on whether they are in contact with or are close to the
border of a drawing path; and in Marentakis and Brewster (2005), the authors
apply cues to guide the movement of users in terms of direction and orientation
during a search task. In both of these cases, the use of cues has stronger links to the
behavior of the system than to the specific content that is being created, represented
or manipulated (i.e., the specific shape of the drawing path, or the specific objects
which are to be searched for). At the same time, they are also sufficiently simple
so that users can learn them without any kind of training in particular. In other
cases, the interpretation of cues (or at least, multi-sensory signals referred to as
cues) is not so clear-cut and cannot be abstracted away from the particular usecase scenario; for example, in Granstrom et al. (2002), feedback cues are generally
interpreted as sensory signals which generate not binary, but graded interpretations
in the affirmative/negative scale, as well as complex interpretations on e.g. the level
of prominence of a word or phrase (in this case, the semantic content of the word
or phrase is also important, hence it cannot be said that the feedback is used only to
characterize the low-level behavior of the system).
All of this suggests that the terminology of CogInfoCom cues, messages and
channels creates a structured approach to making finer distinctions between cases
that have been described in equivalent terms in the past. Cues are defined as
sensory percepts which can lead to basic (most often binary) associations, but which
can evolve (through a process analogous to ritualization in biology) into richer
message-based communication through a history of interactions. Messages, in turn,
have crystallized interpretations and are eventually used to communicate refined
gradations of feedback information to users on high-level concepts, as shown in
Fig. 11.2.
11.3 Aspects of High-Level Human Communication

Based on the above, evolution from cues through signals to channels can be regarded
as a natural tendency in CogInfoCom. However, the question still remains: how can
this tendency be implemented in a way that could unfold in the communication
between humans and cognitive entities, irrespective of the specific details of the
interaction and the application domain?
The challenges that lie behind this question are significant. If the starting point of
the designer is to consider solely the definition of cues, signals and channels, then
enabling an artificially cognitive system to develop its own signals and channels
163
Fig. 11.2 The emergence of messages and channels from cues in CogInfoCom. Communicational
exchanges leading to specific interpretations are represented by arrows, while increasing resolutions in the dashes represent increasing resolutions in interpretation
would require the designer to also implement some way for the system to recognize
what external events (e.g., human behaviors) are useful to it and which of its
cues were successful in eliciting those events. Further, it would require mapping
those cues to appropriate modalities so as to support the emergence of semantic
associations in the most effective way possible. Clearly, just the definitions of these
stages of communication will not help engineers in making good design choices.2
Instead, finding a set of characteristics which transcend the notions of cues, signals
and channels, but which can nevertheless be brought into connection with these
categories may lead to a viable solution. More specifically, if there exist a set of
communicational features that can be associated with cues, signals and channels,
but are also distinct in each of these phases, then they can be taken as a basis for
CogInfoCom design.
In this section, three such communicational features are outlined in terms of
volition, directness and temporal reference; their possible relevance to cues and
signals is outlined in Sect. 11.4. Much inspiration for these features was obtained
from Nass and Yen (2010), in which strong arguments are presented for users
aptitude towards implicitly developing mental and emotional relationships with
machines that are analogous to relationships in human-human communication
(Csapo 2014; Csapo and Baranyi 2013). An important abstraction underlying these
discussions is the notion of triggers; although the term has been used in more than
For example, an application that requires constant feedback from users as to whether or not an
interaction was useful to them would quickly grow tedious; and finding appropriate modalities
through design would lead back to the original problem of designing CogInofCom channels.
164
one way in the literaturefrom workflow modeling to the modeling of negotiations

and social behaviors, as in e.g. Joosten (1994), Blake (2000), and Nass and Yen
(2010)it is used here as a set of conditions and actions that give rise to a
communicational exchange. In this sense, a trigger is activated when the conditions
associated with it are fulfilled (including actions performed). More specifically:
Definition 11.3. A CogInfoCom trigger is a set of conditions and actions that
are to be fulfilled and performed for a communicational exchange to occur. If the
relevant conditions and actions are fulfilled and performed, it is said that the trigger
is activated. Triggers are often characterized by direction: one or more triggering
endpoints activate the trigger, and one or more receiving endpoints begin to send
communicational messages based on this activation.
The notion of triggers will be useful in further discussions because the conditions
and actions associated with the activation of a trigger, and for this activation to
seem natural strongly depends on the context and on the past history of interactions.
In other words, triggers have the potential to be useful in providing indirect
characterizations of the evolution from cues, through signals to channels.
11.3.1 Volition
The dimension of volition concerns the level of consciousnessor, in terms more
suited to artificial cognition: the level of explicitnessrequired for the activation
of a trigger. Thus, it can be made clear that not all forms of communication are
founded on an explicit desire, or functional requirement to communicate. Based on
this notion, voluntary and involuntary triggers are defined as follows:
Definition 11.4. A trigger is voluntary if explicit/conscious actions are required
for its activation.
Definition 11.5. A trigger is involuntary if implicit/unconscious actions are sufficient for its activation.
Responses to voluntary and involuntary triggers can be characterized as episodic,
periodic or aperiodic as described below. Further, depending on whether a trigger
is voluntary or involuntary, the temporal properties of associated response(s) can
be further qualified based on a unique set of criteria. Thus, the following response
types are defined:
Definition 11.6. The response given to a trigger is episodic if it consists of a single
response provided at a unique point in time, and aperiodic/periodic if it consists
of multiple communicational exchanges initiated by the receiving endpoints of the
trigger at varying (in the case of aperiodic responses) or regular (in the case of
periodic responses) periods of time.
165
Definition 11.7. The response given to a voluntary trigger is proactive if the

receiving endpoints of the trigger react based on the prediction that the trigger will
be activated, and reactive if they react only after the trigger has been activated.
Definition 11.8. A response given to an involuntary trigger is pre hoc/ad hoc/post
hoc when the information it communicates is relevant to events in the future, present
or past, respectively.
The temporally relevant distinctions between response types given for voluntary
and involuntary triggers are different in the sense that in the case of voluntary
triggers, it is the act of triggering which is emphasized most, whereas in the
case of involuntary triggers, no explicit triggering takes place and therefore the
response itselfincluding its informational propertiesis what merits most attention. Phrased differently, the most important feature of voluntary triggers is that the
triggering endpoints show explicit interest in receiving some kind of feedback or
in accessing some kind of functionality, in which case the scope of the response
should serve as no surprise. In contrast, the most important feature of involuntary
triggers is that the receivers of the trigger intend to send messages irrespective of
any explicit request from triggering endpoints; hence, the scope and content of the
information transfer gains relative importance. Example scenarios which highlight
the importance of these concepts are provided in Sect. 11.3.3.
11.3.2 Directness
Another important aspect of trigger-based communication is its level of directness.
Earlier in Sect. 7.1, directness was specified as a factor that refers to whether
the cognitive entity knows that it is communicating, and whether the act of
communication requires active involvement in terms of the dedication of cognitive
resources. Untangling from this notion those aspects that are common with volition,
an important feature which emerges is the element of unexpectedness and/or
discomfort associated with a trigger. If the trigger comes as a surprise to (or is
unexpected by) the receiving endpoints; or if the responses of those endpoints are
surprising (or unexpected) to the triggering endpoints, it can be said that the source
of information based on which the endpoints are communicating is unbalanced.
Thus, directness and indirectness can be defined as follows:
Definition 11.9. Knowledge with respect to a given information is shared
between a set of cognitive entities if all entities know with respect to each of the
other entities whether or not they are in possession of the information.
Definition 11.10. A trigger is indirect if knowledge of its occurrence is shared
between the triggering and receiving endpoints, and direct otherwise. Similarly, a
response is indirect if knowledge of its content is shared between the triggering and
receiving endpoints, and direct otherwise.
166
11.3.3 Discussion
The rationale behind these definitions can be supported by the following examples:
In human-human communication, if a stranger asks someone an unexpected
favor, the communicative exchange that ensues is caused by a voluntary and
direct trigger: the favor is asked for purposefully, and the knowledge of its
occurrence is not shared by the interlocutors. In fact, the request may come as
quite a surprise to the receiver of the exchange, prompting a direct need to address
it as well as a direct response (acceptance or rejection) that is perhaps relatively
less tactful than usual.
Conversely, if a favor is asked in friendly settings, the request is perceived as
less direct. If the receiver of the request is accustomed to receiving this request
from the same person, the trigger is indirect. These two examples highlight the
fact that the transition from direct to indirect communication is not necessarily
binary: various levels of indirectness are conceivable based on the degree to
which relevant knowledge is shared.
In human-device communication, the pressing of a button generally corresponds
to a voluntary and indirect trigger: the button is pressed purposefully, and the
user also knows that the device is capable of responding, while the button on the
device was explicitly designed to trigger the requested operations. If the same
button is pressed accidentally, the trigger is involuntary; and if the ability of the
device to respond comes as a surprise to the user, the response will be direct.
If the device performs the operation even before the user presses the button, the
response is proactiveand reactive otherwise.
If a given context (such as the users location, surrounding noise levels and other
sensor measurements) causes an ICT system to remind the user of a past event,
the trigger of the communication is involuntary and the associated response is
post hoc. If, on the contrary, the involuntary trigger causes the system to alert the
users attention to a potential future event, the response is pre hoc. Finally, if the
response refers to the current moment or otherwise has no temporal implication,
then the response is ad hoc.
The kinds of trigger and response types which have been described in this section
are summarized in Fig. 11.3.
11.4 Putting It All Together: Progressive CogInfoCom

Channels
It is clear based on earlier discussions in this chapter that communication is an
emergent phenomenon subject to biological evolution, and that it is also sensitive to
a variety of factors, including the explicitness of the context in which it is triggered,
11.4 Putting It All Together: Progressive CogInfoCom Channels
167
Fig. 11.3 The figure provides a color coded reference on trigger and response types. Mutually
exclusive conceptssuch as voluntary and involuntary triggers, or episodic, aperiodic and periodic
responses are shown in the same color. Concepts related to level of directness are both red and blue
because both triggers and responses can be direct or indirect
its reference to past, presence and future events, and how much of the underlying
context is shared by the communicational endpoints involved in it. Two key points
are suggested by these observations:
In a way that is analogous to the emergence of biological communication, it may
be useful for long-term, entangled interactions if the patterns of communication
among cognitive entities could evolve through time;
In order for users to perceive communication as natural, it should apply different
trigger and response types at different times, as dictated by the context.
It is important to realize that there are also strong parallels between these two
points: interactions in terms of cues, messages and channels are characterized by
different kinds of triggers. For example, a link between biological concepts of
communication and volition/directness of associated triggers is shown in Fig. 11.4.
Although the figure shows tentative conclusions that should be the subject of further
investigations, it nevertheless gives a strong indicationwhich is also intuitively
clearthat:
Cueswhich are a natural by-product of autonomous behaviorscannot be
voluntarily triggered, and cannot be characterized by a priori knowledge (cues
are not communicational elements in the formal sense: therefore, all knowledge
on their occurrencewhich is in fact no knowledgeis trivially shared);
Messageswhich are emitted because they are perceived as useful based on past
interactionsare generally involuntary from the perspective of the entity whose
behavior elicits them, and at the same time rest on the shared knowledge of their
effects (otherwise there would be no purpose for their existence);
168
Fig. 11.4 The emergence of messages and channels from cues in CogInfoCom in the special case
of human-ICT communication. The figure suggests that the ritualization and differentiation process
progresses as the user and system gradually learn to understand and make increasing use of each
others communicative capabilities (although the figure depicts the users perspective, the ability
of the system to learn is also implicitly understood). In this process, what begins as a kind of trialand-error set of exchanges with increasing directness evolves into a process where increasingly
explicit control is achieved through increasingly indirect interactions
Channels are used voluntarily (i.e. for evolved functional purposes) and are also
generally indirect (given that their evolution requires shared knowledge).
Besides highlighting the above points, Fig. 11.4 also suggests a way to solve
the problem of how to achieve cue ritualization and signal differentiation in
CogInfoCom settings without requiring constant, tedious feedback from users.
Specifically, it is suggested that triggers asking for feedback should initially be
indirect, gaining directness only once the functional need for interaction becomes
apparent; whereas later the same triggers should be increasingly less direct, as signal
differentiation is expected to occur autonomously through practice. Directness, in
this case, is seen as a strongly multi-modal concept that can manifest itself in a
large variety of phenomena including word usage in spoken commands as well as
meta-communicational cues and signals.
In any case, the natural progression described here seems to be attainable if both
the humans and ICT devices involved in communication are capable of improving
the effectiveness of their communicational behavior through different forms of
learning. When considering the ways in which a channel might learn representations, it becomes clear that several forms of learning are possible depending on who
11.4 Putting It All Together: Progressive CogInfoCom Channels
169
(or what) is the teacher, and who (or what) adapts during the process. Thus, it is
possible to distinguish among the following forms of learning in the communication
between two entities:
regressive learning: one of the two entities provides learning samples, and one
of the two entities adapts based on those samples
progressive learning: both entities provide learning samples, and both entities
adapt
Regressive learning encapsulates the experience of updating ones representations of an environment based on a set of samples (i.e., experiences) that
are either entirely self-induced, or are passively experienced under comparable
circumstances. This terminology is in keeping with the term regression as it is used
in machine learning, i.e., as a kind of categorization that occurs over a continuum
of potential categories. Progressive learningas opposed to regressive learningis
characterized by bilateral interactions in which both parties participate actively in
the generation of samples during the learning process. In the context of CogInfoCom
channels, this can be beneficial in the sense that the users decisions and actions are
supported by a broader set of feedback behaviors.
The development of suitable algorithms for the progressive evolution of CogInfoCom channels is an important challenge for the future. Based on the discussions
in this chapter, it is suggested that this will require representing and monitoring
the evolution of communicational exchanges, and also the systematic actuation
of communicational behaviors in a way that is suitable to the frameworks of cue
ritualization/signal differentiation and CogInfoCom trigger/response types.
Part IV
Future Perspectives of CogInfoCom
This part outlines several future perspectives relevant to CogInfoCom as derived

from the evolution of the technologies behind the Internet and the synergies between
infocommunications and the cognitive sciences.
In Chap. 12, the concept of Future Internetincluding Internet of Things, 3D
Internet and CogInfoComis formalized with an outlook of 510 years into the
future. Through the discussions in the chapter, a vision of a network aiming at
service and resources awareness emerges, and key clusters of relevant research
issues are identified.
In Chap. 13, the longer term vision of a Cognitive Internet Ecosystem is outlined,
which considers the societal opportunities and impacts that can be derived from the
interaction of human and artificial cognitive capabilities within a Global Networked
Society concept.
Chapter 12
Cognitive Capabilities in the Future Internet
This chapter collects the challenges of the todays Internet, and outlines the
visionary capabilities of the next generation Internet, including cognitive capabilities, virtual collaborations, 3D Internet, Internet of Things and smart applications.
Objects, devices provided by sensors and connected to the Internet become smart,
cognitive things, which can be virtually represented and collaborate with other
virtual or real cognitive things and human beings in a composed virtual 3D space.
The challenge is that billions of cognitive things are to be interconnected (Fig. 12.1),
the concept of Internet of Things was deployed (Smith 2012; Vermesan and Friess
2013). The next generation Internet, called Future Internet (FI) is intensively
investigated (Tselentis et al. 2010; Domingue et al. 2011; Alvarez et al. 2012;
Galis and Gavras 2013); the deployment of the future networked knowledge society
is prospectively based on FI technologies and applications, including advanced
community applications. This chapter also defines the research goals and sorts the
relevant research themes into clusters, pointing out the position of CogInfoCom,
in general the handling of cognitive entities, i.e. cognitive things and beings
(Sallai 2013c).
12.1 Challenges of Todays Internet

The Internet concept was born with different conditions and goals by the fathers
of the Internet, Vinton G. Cerf and Robert E. Kahn in the 70s (Cerf 2009). Since
then Internet became a huge global network and we are speaking on Internet age.
The original concept of Internet (TCP/IPv4) has limitations and there are new
societal requirements (mobility, security, energy saving), while the technological
development provides new opportunities and solutions for the challenges.

DOI 10.1007/978-3-319-19608-4_12
173
174
12 Cognitive Capabilities in the Future Internet
Fig. 12.1 Evolution of networking
Limitations of todays Internet are (Tselentis et al. 2010; Domingue et al. 2011;
Alvarez et al. 2012; Galis and Gavras 2013; Nguyen et al. 2011; Zahariadis et al.
2011; Papadimitriou et al. 2012; Ziegler et al. 2013):
the limited identification capacity, the lack of IPv4 address space;
the essentially private wireline network concept, the lack of an inherent mobile
centric network architecture and a scalable efficient network and mobility
management;
the best effort solutions, the lack of guaranteed and differentiable quality of
services and security;
the growth rate in energy consumption and carbon dioxide emissions are not
sustainable;
the application development is inefficient and inflexible for handling large
diversity of use cases and requirements, etc.
Growth of societal role of the Internet gives rise demands as (Domingue et al.
2011; Alvarez et al. 2012; Galis and Gavras 2013; European Commission 2013b;
Karnouskos and Skarmeta 2013; WEF 2007; CogInfoCom 2012):
anywhere, anytime access (always on);
interconnection of devices, objects, sensors (networked 20100 billion things);
12.2 Future Internet Visions
175
expansion of content space with 3D and cognitive contents (gestures, emotion);

scalable and customized data and knowledge engineering;
a lot of human-centric, secure smart applications.
New technological opportunities are for managing limitations (Tselentis et al.
2010; Domingue et al. 2011; Alvarez et al. 2012; Galis and Gavras 2013; de Meer
et al. 2012; Pallot et al. 2012; WEF 2007; Papadimitriou et al. 2012):
advanced wireless/mobile technologies;

broadband optical solutions;
huge storage capacity, storage efficiency;
innovations in material and manufacturing technology, especially in the technology of sensors, CPUs, memories and energy sources;
potential opportunities deriving from nanotechnology and biotechnology.
We can conclude that:
we were witnesses of the radical increase of Internet in size and complexity;
there is a great technical potential and societal need for significant expansion of
applications; and
the penetration of Internet has fundamental impact on lifestyle and human
relations.

The challenges of the todays Internet, the tangible and potential demands, and the
technical opportunities determine the critical research issues, research objectives,
and need the reconsideration of the classic Internet concept and the construction of
the vision of the Future Internet (Sallai 2014).
Japans National Institute of Information and Communications Technology
(NICT) prepared the first vision for New-Generation Networks (NWGN) in 2008
focusing on the technology requirements for solving social problems (energy
shortage, medical care, crime prevention, technology gap, etc.) and creating new
values to achieving a future knowledge society, using the NWGN (Nishinaga 2010).
Two pillars (Internet of People and Internet of Things) and five network targets were
identified for an NWGN R&D strategy. This scheme was used for the Future Internet
and continuously evolved on the Future Internet Assemblies (FIA 2011 in Budapest
and Poznan, FIA 2012 in Aalborg, FIA 2013 in Dublin, FIA 2014 in Athens) aiming
at an intelligent, sustainable world, an innovative, secure society as a generic goal
of the Future Internet (Domingue et al. 2011; Alvarez et al. 2012; Galis and Gavras
2013; Wainwright and Papanikolaou 2012).
A Future Internet vision is based on the standardization work of the Future
Networks (FNs) performed by International Telecommunication Union Telecommunication Standardization Sector (ITU-T). The ITU-T has started the standardization
of FNs as networking systems to be deployed in the 20152020 timeframe.
176
Fig. 12.2 Future Internet vision based on NWGN and FN visions
FN standardization combined two complementary approaches: a top down method

starting from objectives and design goals, and a bottom up method starting from
relatively matured candidate technologies. Recommendations ITU-T Y.3001, 3011,
3021 and 3031 identify four essential objectives which were not concerned enough
in designing current networks (ITU 2011a,b, 2012a,b; Matsubara et al. 2013).
The Future Network is described like Future Internet as a unified infrastructure
of communication, computing and storage resources (together called network
resources) which connects and orchestrates the Future Internet of people, devices,
content, computers, clouds and things.
Figure 12.2 suggests a vision for the Future Internet, combining the research
oriented NWGN vision scheme of the NICT and the standardization oriented FN
vision of the ITU-T, taking also into account the achievements of the Future Internet
Assemblies. Figure 12.2 demonstrates three concepts as pillars, shows five strategic
objectives and indicates the smart, high-value applications provided as services in
the Cloud. The pillars are Internet of People, Internet of Things and their unification
and holistic extension for all cognitive entities, Internet of Everything. The strategic
objectives are based on the ones of the NWGN and FN visions, as follows:
Scalable, service aware network refers to scalable network architecture with
functional flexibility, which can accommodate wide range of services with
diverse and evolving requirements. Future network architectures are expected
to support not only current services such e-mail and web browsing, but also
emerging services, including mission critical services, by providing additional
functionality without drastic increases in network resources, deployment and
operational costs. Furthermore the network architecture should support advanced
177
mobility features, enhanced reliability and security as well as low latency

requirements. Fifth generation (5G) scalable, super real-time, holistic network
infrastructures are investigated. For scalability the replacement of IPv4 communication protocol seems to be inevitable (probable by IPv6). For service
awareness the emerging paradigm, cloud computing and networking, i.e. provision of resources as a service (IaaSInfrastructure as a Service, PaaSPlatform
as a Service, SaaSSoftware as a Service, etc.) is enjoying an increased
scientific and business interest (ITU 2011a; Petcu and Galis 2013; CloudNet
2014).
Virtual, resource aware network refers to the virtualization of network resources
and network functions, as well as a unified efficient network resources management. Network virtualization means the process of partitioning of the network
resources, abstracting the partitions as virtual network resources, and combining
virtual network resources and network functionsusing them according to
necessityinto logically isolated software defined networks (SDN). Such a
way, flexible usage of the virtualized resources and functions is allowed by
the services, and multiple virtual networks can be created in a single physical
network (Galis et al. 2004; ITU 2011b).
Data and content awareness embraces goals on efficient transportation, access,
analytics and usage of huge amount of various data sets generated by the Internet
of Things (IoT) and the media content sources including 3D and cognitive
content. Scientific disciplines on Big Data technologies, media mining, content
centric network architectures, CogInfoCom, etc. are emerged for responding
objectives. Communications in current networks is based on the globally unique
location identity (ID) and location based routing. If identical contents (data,
information) are placed in multiple data locations, then they may have the same
content ID and the content can be accessed via a nearest location using content
ID based routing (Zahariadis et al. 2010; ITU 2012b).
Sustainability, environmental awareness refers to energy awareness, efficient
spectrum usage and any other ecological aspects. The enormous increase in
Internet traffic means increase in energy consumption; hence energy awareness is
a key objective (Green Internet). To save energy we should optimize the network
to reduce the effective traffic loads and the network capacity in operation taking
into account the traffic dynamics, as well as improve the energy efficiency using
lower power electronic technology and dynamic control techniques (ITU 2012a;
Zeller et al. 2013).
Intelligent, innovative and secure society, as a generic target of Future Internet,
comprises the societal objectives and aspects (social networks, social interaction,
augmented reality, etc.). Internet becomes essential infrastructure utility in the
Digital Ecosystem; the right to have access to a global network will be one
of the fundamental rights in the future. Future Internet should consider socioeconomic objectives as governance issues (e.g. the barrier to enter the market, the
lifecycle cost for operation and deployment, service universalization), Internetbased innovation, the information security, personal data protection (ITU 2011a;
Wainwright and Papanikolaou 2012).
178
The FI-based applications are in the centre of the vision for a smarter world
involving applications as smart cities, smart homes and offices, smart health,
education and entertainment, smart energy, smart transportation and logistics, smart
manufacturing and agriculture, smart governance, etc. The FI applications are
software solutions for a specific task, hence they are expected to be seamlessly
adjusting to the users needs and context, while in parallel hiding the complexity of
the FI infrastructure and interactions with other applications (Alvarez et al. 2012).
In the frame of the FI PPP (public-private partnership) programme FI-ware cloudbased generic application enabler platform has been created, which is completed by
specific modules for use cases (e.g. FI-Content for social connected TV, smart city
services and augmented reality based games, FI-Space for business collaboration
networks in agri-food, transport and logistics, FI-Star for healthcare, FITMAN for
manufacturing industries, FINESCE for smart energy) (ECFI 2014).
12.3 Recent Future Internet Capabilities

Recently there is no accepted definition for Future Internet, rather it is described by
some capabilities, which do not exist in, or are not typical of the todays Internet.
Some new functions were created and certain features became obvious in the last
some years, which are relevant criterions to separate todays and next generation
Internet. In the followings eight functions and four features associated to Future
Internet are listed. The lists as a matter of course are open; recently one or more
functions and some of the features are characteristic of the Future Internet solutions.
A recent list of relevant Future Internet functions is (Domingue et al. 2011;
Alvarez et al. 2012; Galis and Gavras 2013; Daras and Alvarez 2009; Nishinaga
2010; de Meer et al. 2012; Karnouskos and Skarmeta 2013; Stratogiannis et al.
2013; Vermesan and Friess 2013; European Commission 2013b; CloudNet 2014):
1. Identification and interconnection of things, devices, sensors, actuators (Internet
of Things);
2. Network architecture intrinsically handling mobility, anywhere, anytime data
collection, tracking and tracing;
3. Programmability of networks: virtualized, software-defined networks;
4. Distributed data centers: access to and management of huge data sets, multimedia
contents;
5. Content-aware technologies: content selection, mining, networking and management;
6. Communicating and managing 3D and cognitive contents, virtual and augmented
world;
7. Cloud computing and networking: infrastructure, platform, software, etc. are
provided as a service;
8. Remote collaboration, monitoring and control of physical processes (Tactile
Internet).
12.4 Clustering Future Internet Research Themes
179
A list of relevant Future Internet features is (Domingue et al. 2011; Alvarez et al.
2012; Galis and Gavras 2013; Matsubara et al. 2013; Papadimitriou et al. 2012;
Wainwright and Papanikolaou 2012):
1.
2.
3.
4.
Inherent information security, personal data protection;

Managed quality, flexible service and application creation;
Enforcement of energy awareness in network planning and operation;
Personalization: customized solutions and presentation profiles.
Recent capabilities are in good coincidence with the visionary ones, what shows
the reality of the vision and the awareness of the research governance. The Future
Internet application research is focusing on smart cities (Alvarez et al. 2012), green
energy (Nguyen et al. 2011) and advanced humanoid applications (CogInfoCom
2012, 2013, 2014) especially on smart health and ambient assisted living (Minutolo
et al. 2012), safe vehicular communications (Papadimitratos et al. 2009) and
driver assistance systems (Betaille and Toledo-Moreo 2010), as well as interactive
cognitive mobile applications (Csapo and Baranyi 2010). CogInfoCom research
works and some applications, as crowdsourcing and crowdsensing community
applications, augmented reality, 3D virtual collaboration and massive open online
courses (MOOCs) are closely related to Future Internet function 6 and support the
holistic concept of Internet of all cognitive entities.

The research goals and spheres can be combined into three levels:
Basic research topics related to the fundamentals of Future Internet, embracing
mathematical modelling of large scale networks (Network Science), theory of
computation and computer architectures (Computer Science), enabling quantum
technology, cryptography as the theory of security, cognitive and socio-economic
characteristics and environmental aspects, legislation and governance principles,
etc. are collected in Internet Science (European Commission 2013b).
Engineering research issues represent the backbone of the Internet applied
research, called Internet Engineering (CogInfoCom 2013). This sphere of
research comprises creation and elaboration of Future Internet technologies,
network architectures, data and content management methods, including Internet
of Things and 3D Internet architectures, methods for managing cognitive
capabilities.
Future Internet experimental research and innovation actions aim at the development of FI-based solutions, smart industrial and community applications,
customizable content services, involving their experimentation, demonstration
and standardisation issues, shortly referred to as Internet Practice (Domingue
et al. 2011; Alvarez et al. 2012; Galis and Gavras 2013; European Commission
2013b).
180
Fig. 12.3 Future Internet research clusters and research spheres
Studying the research themes on the Future Internet in the literature, in particularly on FIAs (Tselentis et al. 2010; Domingue et al. 2011; Alvarez et al. 2012;
Galis and Gavras 2013; Wainwright and Papanikolaou 2012), Working Programme
of Horizon2020 (European Commission 2013b) as well as the Hungarian Future
Internet research activity (Sallai 2013b; CogInfoCom 2013; Bakonyi and Sallai
2014a,b), the relevant research topics have been identified and clusterized (Sallai
2013a, 2014). The main research clusters of Future Internet research activity
has been arranged in a layered model from basic research on Internet Science
through Internet Engineering up to Internet Practice. Figure 12.3 shows the nine
clusters and their main research goals and spheres. The Internet Science, the basic
research is Cluster 1, the Internet Engineering applied research is divided into five
clusters (Cluster 2 : : : 6), the Internet Practice, the applications and experiments
are comprised in three clusters (Cluster 7, 8 and 9). Figure 12.3 also indicates the
possible relations of the clusters to other research goals and spheres.
In the following, the research clusters and their relevant topics, the elements
of clusters are presented. Each cluster has 57 elements, and 54 relevant topics
are defined in total. Relevant research topics on cognitive sciences, cognitive
infocommunications or related applications are indicated by asterisks.
181
12.4.1 Cluster 1: Internet Basic Research (Internet Science)

Internet Science aims at an integrated and interdisciplinary scientific understanding
of Internet networks and their co-evolution with society, embracing all disciplines
which study Internet from any technological or humanistic perspective. Relevant
research topics as elements of clusters are (European Commission 2013b; Newman
et al. 2006; Lewis 2009; Mihalydeak and Csajbok 2013; Alpcan et al. 2010; HirataKohno and Petho 2013; Danyadi et al. 2012; Foldesi and Botzheim 2012; Trinh et al.
2011; WEF 2007; Liu 2013):
A.
B.
C.
D.
E.
Network science: modelling and investigating large scale networks;

Computer science, from computational theory to computer architecture;
Basic enabling technologies, as quantum and nanotechnologies, etc.;
Cryptography, cyber security;
*Human aspects: social and behavioural attributes, trust and aversion, cognitive
processes, cognitive biases, social networks;
F. Network economics, game theory;
G. Legislation and governance (privacy, data protection, net neutrality).
12.4.2 Cluster 2: Future Internet Modeling, Analysis

and Design
These research topics target the modelling of Future Internet enabling infocommunication systems, network concepts and technologies, for analysing their performance,
scalability, stability, availability, resilience, quality of service, etc. as well as the
elaboration of novel modelling and design paradigms. Relevant topics are as (Biro
et al. 2010; Tapolcai et al. 2012; Van Do et al. 2013; Stratogiannis et al. 2013;
European Commission 2013b; CloudNet 2014):
A. Network modelling and performance analysis;
B. Queuing systems: traffic analysis and design;
C. Communication systems: advanced modulation, coding, access, spectrum
usage;
D. Datacenters, resource allocation and optimization methods;
E. Investigating networked media services: video streaming, VoIP, IPTV;
F. Survivability techniques, monitoring, failure exploration.
12.4.3 Cluster 3: Future Internet Network Architectures

Novel architectural solutions are needed to meet the societal challenges, which the
todays Internet may not be able to support sufficiently. This central research area
182
focuses on new network architectures and protocols, mechanism enhancements,

virtualization of resources, as (Tselentis et al. 2010; Domingue et al. 2011; Alvarez
et al. 2012; Galis and Gavras 2013; European Commission 2010, 2013b; CloudNet
2014; Galis et al. 2004; Zahariadis et al. 2010, 2011; de Meer et al. 2012;
Papadimitriou et al. 2012; Matsubara et al. 2013; Petcu and Galis 2013):
A. Future Internet network requirements (scalability, manageability, flexibility,
reliability, resilience, robustness, security, latency, simplicity, etc.) and concepts
(Identifier/ID spaces, network virtualization, cloud services, etc.)
B. Future Internet routing methods, e.g. transport protocols, path selections;
C. Mobile/holistic network architectures: optimal data access, intrinsic mobility
handling, fixed-mobile convergence (FMC), 5G super real-time network infrastructures;
D. Future media networks, content-aware networks, content-centric and content
delivery networks (CCN, CDN);
E. Network computing: ubiquitous, grid, cloud computing;
F. Virtual networks, software defined networks (SDN), network function virtualization (NFV);
G. Cloud networking, cloud infocommunications: NaaS (Network as a Service),
IaaS, PaaS, SaaS, CaaS, etc.
12.4.4 Cluster 4: Data and Content Technologies

The huge volumes of data and multimedia content are essential driving forces for
the evolution of the Future Internet. These research topics aim at extracting meaning
from data and supporting the collection, searching, managing and publishing of
information in any form, as (Domingue et al. 2011; Alvarez et al. 2012; Boiko
2005; Prekopcsak et al. 2011; Weikum et al. 2011; Garzo et al. 2013; Szucs 2013;
European Commission 2013b):
A.
B.
C.
D.
E.
F.
Data, text and media mining;

Big Data challenges (volume, velocity, variety) and solutions;
Semantic multimedia search methods, knowledge discovery;
Presentation, visualisation;
Digital library functions (archives, name spaces, etc.);
Multimedia content processing, information management.
183
12.4.5 Cluster 5: 3D Internet and Cognitive

Infocommunications
The three dimensional (3D) communications can embrace our cognitive systems,
including not only hearing and vision, but touch, gestures, emotion, smell, etc.
Along with its enormous potential 3D Internet opens many research challenges to
expand cognitive capabilities and interactions, realize augmented and virtual spaces,
as (CogInfoCom 2010, 2011, 2012, 2013, 2014; Alpcan et al. 2007; Daras and
Alvarez 2009; Prekopcsak et al. 2008; Csapo and Baranyi 2010, 2012d; Galambos
et al. 2012a; Toth et al. 2012):
A. *3D Internet architecture and content technology (data collection from augmented/virtual environment, 3D media analytics, processing and presentation);
B. *Multimodal human-computer interactions, human-ICT systems;
C. *Cognitive interactions: cognitive infocommunications channels and sensor
bridging;
D. *Virtual and augmented reality, 3D Internet based control and communications,
virtual remote collaboration;
E. *Devices and procedures of 3D and 4D content creation and presentation;
F. *Cognitive capabilities, cognitive entities, cognitive augmented social intelligence, speechability, mathability, etc.
12.4.6 Cluster 6: Internet of Things (IoT)

Internet of Things is seen as a key part of the Future Internet vision, which targets
real-time complex interactions and collaborations of billions of heterogeneous
devices (objects, sensors, actuators), and requires tackling numerous technological,
connectivity and interoperability issues (Tselentis et al. 2010; Domingue et al. 2011;
Alvarez et al. 2012; Galis and Gavras 2013; Nishinaga 2010; Karnouskos and
Skarmeta 2013; Smith 2012; Vermesan and Friess 2013; Ziegler et al. 2013; Gal
et al. 2014; Future RFID 2014):
A. *IoT enabling technologies (RFID, NFC, etc.), sensors and actuators, energy
and spectrum awareness;
B. Identification: naming, addressing, privacy awareness, IPv6-based IoT;
C. Communication architectures for constrained devices, IoT infrastructures, selfaware and organizing networks;
D. Data management, software solutions, security technologies: self-adaptive security mechanism and protocols, self-managed secure IoT;
E. Services support platforms, heterogeneous data collection and processing infrastructures.
184
12.4.7 Cluster 7: Cyber-Physical Systems and Applications

Cyber-physical systems refer to next generation embedded ICT systems that are
interconnected and collaborating through the Internet of Things, and providing a
wide range of innovative applications and services. Relevant research directions are
(Galis and Gavras 2013; Papadimitratos et al. 2009; Nguyen et al. 2011; Lee and
Seshia 2011; Smith 2012; Vermesan and Friess 2013; Fischer et al. 2013):
A. Embedded and intelligent engineering systems;
B. Intelligent production applications, measuring and controlling physical processes;
C. Intelligent transport, cars and logistics;
D. Smart agriculture-food applications,
E. Smart energy systems, Green Internet.
12.4.8 Cluster 8: Future Internet Based Community

Applications
Users demand always on access to cheap, easy-to-use, secure, mobile, personalized
and context-aware applications, which are to be realized over highly interconnected,
increasingly complex infrastructures. Internet of Things is implicated for smart
environments and smart spaces, and the active collaboration of the users can be
also involved through crowdsourcing and -sensing techniques. Cross-disciplinary
research challenges should be addressed (Tselentis et al. 2010; Domingue et al.
2011; Alvarez et al. 2012; Galis and Gavras 2013; CogInfoCom 2012; Smith 2012;
Vermesan and Friess 2013; Minutolo et al. 2012; Zeller et al. 2013):
A.
B.
C.
D.
E.
F.
G.
*Crowdsourcing and crowdsensing platforms and functions;

*Smart home and office applications;
*Smart health and well-being applications;
Smart business applications;
Smart governance applications;
*Smart city community applications;
*Other smart and cognitive community applications.
12.4.9 Cluster 9: Experimentation, Standardization,

Regulation
Themes of this practical chapter embrace the requirements and design of comprehensive test facilities, the federation of test-beds, the technical and social
experiments, the standardization activity as well as the emerging complex regulatory
185
issues. The cluster elements are defined as (Domingue et al. 2011; Alvarez et al.
2012; Galis and Gavras 2013; ITU 2011a,b, 2012a,b; Sales et al. 2012; Matsubara
et al. 2013; Liu 2013; Vermesan and Friess 2013):
A.
B.
C.
D.
*Experimental systems, test-beds, including virtual test spaces;

*Experimental methods, demonstrations, field results;
Socio-economic studies, business models;
Technical standards, recommendations, standardization issues, e.g.: identification, communications, virtualization, interoperability, security;
E. Technical, economic and content regulatory issues.
Clustering the Future Internet challenges and opportunities, the cognitive capabilities, the research issues of CogInfoCom have been found to be deeply embedded.
Cluster 5 focuses on CogInfoCom, and further CogInfoCom-related basic and
practical research issues are also allocated in Clusters 1, 6, 8 and 9. Deployment
of CogInfoCom is closely related to the evolution of Internet of Things and 3D
Internet.
Chapter 13
Towards a Cognitive Internet Ecosystem
In the previous chapter, the concept of Future Internet was formalized through the
perspective of the next 510 years. This formalization included a vision of a future
network that aims at service and resource awareness. In this chapter, a longer term
and broader vision is outlined, which covers not only technological aspects, but
also aspects relevant to the impact of Future Internet on the physical and societal
environment. Through the convergence of TIM technologies with the cognitive
sciences and a novel concept of Global Networked Society, it is argued, customers
and their devices together will be increasingly viewed as cognitive entities capable
of bringing about unprecedented functional interactions.
The chapter first describes the ways in which the original Digital Ecosystem
conceptformulated in 2007is being incrementally reshaped through emergent
technological-social-cognitive phenomena. This is followed by a brief overview of
the newly emerging Cognitive Internet Ecosystem. It is important to clarify that
the goal of this chapter is not to provide a detailed account of definite facts, but
rather to describe the authors views on a possible long-term evolution of networked
CogInfoCom systems. Due to the rapid development of technologies and synergies,
the discussion here can by no means serve as a definite characterization of the future,
and should therefore be approached with critical caution.
13.1 Reshaping Digital Ecosystems

An ecosystem is an interdependent and dynamic network of living organisms
and their physical environment. The term Digital Ecosystem (DES) was defined
in 2007 by the World Economic Forum as the ecosystem that is produced by the
convergence of the telecommunications, IT and media (TIM) sectors and composed
of users, companies, governments and civil society, as well as digital infrastructure

DOI 10.1007/978-3-319-19608-4_13
187
188
13 Towards a Cognitive Internet Ecosystem
(WEF 2007). At the same event, possible scenarios were formulated for the
evolution of DES into the year 2015 based on answers to the following key
questions:
Will social and economic value creation be industry controlled and led, or organic
and community-led?
Will the digital business environment evolve toward a more open or closed
system?
Guided by these questions, the following three scenarios emerged for the Digital
Ecosystem:
Safe Havens scenario (industry led value creation and closed business environment) describes a digital world in which online security issues create a need
for virtual safe havens. The industry would respond through vertical integration
by creating secure walled environments that provide all digital services, but
nevertheless lend a sense of constriction to user interaction.
Middle Kingdoms scenario (industry led value creation and open business environment) describes a digital world dominated by intermediaries that effectively
connect users to like-minded individuals and to the highly specialized suppliers
that can best meet their needs. At the center of the space between consumers and
suppliers would lie the kingdoms where the power would be concentrated.
Youniverse scenario (community-led value creation and open business environment) describes a digital world characterized by the rise of new organizational
structures, the change of economic value creation and traditional business
thinking, and by highly personalized digital experiences. To survive uncertainty
and changes, companies would have to find ways to capitalize on this distributed
innovation landscape.
During the past few years many aspects of the DES have materialized. Thus,
the current time period is often referred to as the Digital Age, in which open
access to information and knowledge is fundamental. Communication platforms
are widely seen as resources which much remain open for all in order to ensure
equitable access to the Internet. Through this vision, a reassessment of the current
regulatory framework has also begun. Digital users are starting to take control of
when, where and how they can consume digital content; hence, they are no longer
merely consumers, but also participate in the DES in a variety of other ways, e.g.
by contributing to online communities and taking part in the creation/distribution
of digital content and services. Communities are also being created around the
development of new infrastructure, as when members of a community agree to share
their wireless internet access (Yoonaidharma et al. 2014).
At the same time, the current manifestation of digital convergence rests primarily
on Internet technology; the Internet is considered as a global platform for accelerating the flow of information that is pushing many earlier forms of information
and knowledge sharing into obsolescence. Most traditional communications media,
including telephone, music, film, and television are being reshaped or redefined
so as to suit the Internet, giving birth to new services such as Voice over Internet
13.1 Reshaping Digital Ecosystems
189
Protocol and Internet Protocol TeleVision. The Internet has become a global system,
an international network of networks that consists of millions of public and private
packet switched networks, linked by a wide range of networking technologies.
Therefore, the Digital Ecosystem concept is increasingly referred to as an Internet
Ecosystem (IES). Accordingly, the term Digital Age is being replaced by Internet
Age, so as to adequately reflect changes in the essence of the technology used in
the infrastructure that enables digital interactions; as well as to express the global
character and the principle of collectiveness realized in todays Digital Ecosystem.
Although both the Digital Ecosystem and the Internet Ecosystem are founded on
the infocommunication or ICT infrastructure (see also Chap. 3), a new wave of both
opportunities and threats has emerged, which is described by Gartner Inc. (2014):
an exponential pace of technological innovation;

the Internet of Things (IoT);
combinatorial value creation by combining different platforms and applications;
the appearance of innovative business models and changes in playing rules.
As a consequence, rapid technological breakthroughs have been achieved, leading to radical changes in lifestyle and business landscape (the latter also through an
underestimation of the digital environment). Companies hoping to achieve success
in the DES/IES are required to find digital synergies, and more generally to adopt
appropriate digital business strategies based on a suitable analysis of the digital
environment, leading to the identification of digital opportunities and threats as
well as a better understanding of digital organizational capabilities, including digital
strengths and weaknesses.
At the same time, the advanced digital TIM technology has penetrated into
customer devices, and has integrated customers through a large variety of interaction modes, leading to an extension of relevant content space with sensory and
cognitive content. Applications can increasingly make use of and modify the context
which surrounds customerse.g. via crowdsourcing and crowdsensing community
applications. Inasmuch as this is the case, the DES/IES is progressively widening
so as to comprise cognitive infocommunication networks, the Internet of Things,
Big Data tools and solutions, 3D Internet and community applications, as well
as all of the devices, tools and opportunities that are in general provided by
the Future Internet. Expressed more primarily, the Digital/Internet Ecosystem is
radically being enlarged and changed in character through the sudden involvement
of billions of smart devices. The term Internet of Things Ecosystem is used to
express this important motivating feature (IDC 2014). Further, the holistic term
Internet of Everything Ecosystem becomes viable when the Future Internet vision
is interpreted for bringing together people and things (Kurian and Anand 2014).
190
13 Towards a Cognitive Internet Ecosystem
13.2 A Long-Term Vision: Cognitive Internet Ecosystem

Beyond the substantial quantitative changes brought about by the formation of
new interconnections between masses of smart things, even more dramatic changes
can be expected from the emergence of new cognitive capabilities in the Internet
Ecosystem:
On the one hand, cognitive beings and cognitive things are becoming increasingly
connected to a global collectiveness, through which the elements of a higherorder world of integrated cognitive entities is being formed as human and
artificial capabilities influence each other in functionally effective directions.
In broad terms, such functionally effective interaction patterns can be seen as
emergent cognitive capabilities (Baranyi et al. 2014)
On the other hand, Future Internet networks already have (and will increasingly
have) self-management capabilities, including capabilities for self-reflective
analysis, context awareness and adaptive reconfiguration in order to optimize the
both network performance and customers quality of experience. Such a smart
networkreferred to as a cognitive network in Sect. 4.6will not only be
expected to ensure connections between cognitive entities, but also to manage the
expanding content space resulting from their interactions, and provide a basis for
new synergies between human and artificial systems as both become increasingly
entangled with smart networked services (Baranyi et al. 2014).
Based on these changes, the long-term vision of a Cognitive Internet Ecosystem
(CIES) can be conceived. The CIES would provide a materialized manifestation
of the convergence of TIM technologies with the cognitive sciences through the
CogInfoCom concept, and would involve the co-evolution of cognitive entities (i.e.
users, governments, civil societies, companies together with smart things) on global
Future Internet infrastructures with cognitive network capabilities. Not only would
human users and artificial systems be active participants in the CIES, they would
also have an active involvement in creating newly emergent, synergic cognitive
capabilities. This can be seen as a natural continuation of the merging of capabilities
described in earlier chapters throughout the book, resulting in e.g. the augmentation
of social capabilities in speechability and socio-cognitive ICT, the augmentation
of situation awareness through adaptive CogInfoCom channels, and an increasingly
successful co-evolution of capabilities for solving mathematical problems or making
unbiased decisions.
As has been detailed throughout the book, the appearance of collective data,
knowledge and capabilities is bringing about radically new possibilities in applications characterized as relevant to socio-cognitive ICT. The authors believe that a
strong argument can be made for the view that this is in turn leading the way toward
the formation of new kinds of informational (rather than purely physical) sensory
modalities, and to the emergence of generation CE, a new generation growing up in
this environment of cognitive entities.
13.2 A Long-Term Vision: Cognitive Internet Ecosystem
191
As recently highlighted by Auvray and her colleagues, attempts made of

providing a definition of sensory modalities remain controversial; however, a
number of criteria can be useful in characterizing human modalities (Auvray and
Myin 2009). Examples include:
type of sensory organ

physical manifestations of stimuli
conceptual properties to which perception gives rise
behavioral effects of perception
dedication of sensory organs to a given form of perception from an evolutionary
perspective.
Although it is mentioned in that paper any single one of these (and other)
criteria can be (and are) used to delineate sensory modalities, many of them can be
equally applied to the kinds of informational sensory modalities which are appearing
today. Through the analysis of real-time feeds of data and information, it will
someday be possible to obtain a virtual sensation of the general psychological
atmosphere in a city, the level of collective awareness on a specific issue,
or the overall conditions of health in a region. As capabilities for obtaining
such information are increasingly used and increasingly grow to become a part of
the standardized (canonical) capabilities of the CIES, they will increasingly be
viewed as (augmented virtual) sensory modalities that can be concentrated into a
single concept. Based on the services of such artificial modalities, new augmented
capabilities for the manipulation of previously inaccessible information types will
also become possible through integrated/augmented virtual realities. Such are the
perspectives which await members of generation CE.
References
Abuczki A (2014) On the disambiguation of multifunctional discourse markers in multimodal

interaction. J Multimodal User Interfaces 8(2):121134
Adams R, Hannaford B (1999) Stable haptic interaction with virtual environments. IEEE Trans
Robot Autom 15(3):465474
Ainsworth MDS, Bell SM (1970) Attachment, exploration, and separation: illustrated by the
behavior of one-year-olds in a strange situation. Child Dev 41:4967
AIST RTM (2010) AIST, OpenRTM-aist. http://www.openrtm.org
Almasi B (2013) Multipath communication: a new basis for the future Internet cognitive
infocommunication. In: 4th IEEE international conference on cognitive infocommunications,
Budapest, pp 201204
Alpcan T, Bauckhage C, Kotsovinos E (2007) Towards 3D Internet: why, what, and how?
In: International conference on cyberworlds, 2007 (CW07), pp 9599
Alpcan T, Buttyan L, Baras JS (2010) Decision and game theory for security. Lecture notes in
computer science, vol 6442. Springer, Berlin
Alvarez F et al (eds) (2012) The future internet - from promises to reality. In: Future Internet
assembly, Aalborg. Springer, Heidelberg
Ambady N, Rosenthal R (1992) Thin slices of expressive behavior as predictors of interpersonal
consequences: a meta-analysis. Psychol Bull 111(2):256
Ando N, Suehiro T, Kitagaki K, Kotoku T, Yoon WK (2005) RT-middleware: distributed
component middleware for RT (robot technology). In: 2005 IEEE/RSJ international conference
on intelligent robots and systems, 2005 (IROS 2005). IEEE, pp 39333938
Arend M (2002) SEAMATE: socio-economic analysis and macro-modeling of adapting to
information technology in Europe. Technical Report IST-2000-31104, Information Society
Technologies
Arnold V (1957) On functions of three variables. Dokl Akad Nauk USSR 114:679681
Aryania A, Daniel B, Thomessen T, Sziebig G (2012) New trends in industrial robot controller user
interfaces. In: 3rd IEEE international conference on cognitive infocommunications, Kosice,
pp 365369
Auvray M, Myin E (2009) Perception with compensatory devices: from sensory substitution to
sensorimotor extension. Cognit Sci 33:10361058
Bach-y Rita P, Tyler M, Kaczmarek K (2003) Seeing with the brain. Int J Hum Comput Interact
15(2):285295
Bajk S, Ples Z (2009) Computer aided solution of the invariance equation for two-variable Gini
means. Comput Math Appl 58:334340. doi:10.1016/j.camwa.2009.03.107

DOI 10.1007/978-3-319-19608-4
193
194
References
Bakonyi P, Sallai G (2014a) Future Internet National Research Program - JINKA 2.4 (in
Hungarian), Budapest, p 60
Bakonyi P, Sallai G (eds) (2014b) Future Internet research in Hungary - FIRCC Report 2014,
Debrecen, p 128. ISBN: 978-963-473-718-6
Balata J, Franc J, Mikovec Z, Slavik P (2014) Collaborative navigation of visually impaired.
J Multimodal User Interfaces 8:175185
Baranyi P (2004) TP model transformation as a way to LMI based controller design. IEEE Trans
Ind Electron 51(2):387400
Baranyi P (2014) The generalized TP model transformation for TS fuzzy model manipulation and
generalized stability verification. IEEE Trans Fuzzy Syst 22(4):934948. ISSN: 1063-6706
Baranyi P, Csapo A (2010) Cognitive infocommunications: CogInfoCom. In: 2010 11th international symposium on computational intelligence and informatics (CINTI), Budapest,
pp 141146
Baranyi P, Csapo A (2012) Definition and synergies of cognitive infocommunications. Acta
Polytech Hung 9:6783
Baranyi P, Gilanyi A (2013) Mathability: emulating and enhancing human mathematical
capabilities. In: 2013 IEEE 4th international conference on cognitive infocommunications
(CogInfoCom). IEEE, pp 555558
Baranyi P, Yam Y, Varlaki P (2006) Tensor product model transformation in polytopic model-based
control. CRC/Taylor & Francis, Boca Raton/London
Baranyi P, Galambos P, Csapo A, Varlaki P (2012) Stabilization and synchronization of dynamicons through CogInfoCom channels. In: 2012 IEEE 3rd international conference on cognitive
infocommunications (CogInfoCom), pp 3336
Baranyi P, Csapo A, Varlaki P (2014) An overview of research trends in CogInfoCom. In: 2014
18th international conference on intelligent engineering systems (INES). IEEE, pp 181186
Barrass S (1998) Auditory information design. Ph.D. thesis, Australian National University
Benford S, Bowers J, Fahlen LE, Greenhalgh C, Snowdon D (1997) Embodiments, avatars, clones
and agents for multi-user, multi-sensory virtual worlds. Multimedia Systems 5(2):93104
Benus S (2013) Cognitive aspects of communicating information with conversational fillers
in slovak. In: 4th IEEE international conference on cognitive infocommunications. IEEE,
pp 271276
Benus S, Levitan R, Hirschberg J (2012) Entrainment in spontaneous speech: the case of
filled pauses in supreme court hearings. In: 3rd IEEE international conference on cognitive
infocommunications, pp 793797
Benus S, Gravano A, Levitan R, Levitan SI, Willson L, Hirschberg J (2014a)
Entrainment, dominance and alliance in supreme court hearings. Knowl-Based Syst.
doi:10.1016/j.knosys.2014.05.020
Benus S, Levitan R, Hirschberg J, Gravano A, Darjaa S (2014b) Entrainment in slovak collaborative dialogues. In: 5th IEEE international conference on cognitive infocommunications, Vietri
sul Mare, pp 309313
Berthelon F, Sander P (2013a) Emotion ontology for context awareness. In: 4th IEEE international
conference on cognitive infocommunications, Budapest, pp 5964
Berthelon F, Sander P (2013b) Regression algorithm for emotion detection. In: 4th IEEE
international conference on cognitive infocommunications, Budapest, pp 9196
Berthoz A, Pavard B, Young LR (1975) Perception of linear horizontal self-motion induced by
peripheral vision (linearvection) basic characteristics and visual-vestibular interactions. Exp
Brain Res 23(5):471489
Betaille D, Toledo-Moreo R (2010) Creating enhanced maps for lane-level vehicle navigation.
IEEE Trans Intell Transp Syst 11(4):786798
Bhatt GD (2001) Knowledge management in organizations: examining the interaction between
technologies, techniques, and people. J Knowl Manag 5(1):6875
Biocca F, Kim J, Choi Y (2001) Visual touch in virtual environments: an exploratory study of
presence, multimodal interfaces and cross-modal sensory illusions. Presence Teleoperators
Virtual Environ 10(3):247265
References
195
Biocca F, Inoue Y, Polinsky H, Lee A, Tang A (2002) Visual cues and virtual touch: role of visual
stimuli and intersensory integration in cross-modal haptic illusions and the sense of presence.
In: Gouveia F (ed) Proceedings of presence, Porto
Biro J, Heszberger Z, Martinecz M, Kis SM (2010) Novel equivalent capacity approximation
through asymptotic loss analysis. Comput Commun 33(1):S152S156
Blake R (2000) Computer mediated communication: a window on l2 Spanish interlanguage. Lang
Learn Technol 4(1):120136
Blattner M, Sumikawa D, Greenberg R (1989) Earcons and icons: their structure and common
design principles. Hum Comput Interact 4(1):1144
Blattner MM, Papp A, Glinert E (1994) Sonic enhancement of two-dimensional graphics displays.
In: Kramer G (ed) Auditory display: sonification, audification and auditory interfaces. Addison
Wesley, Reading, pp 447470
Blum JR, Eichhorn A, Smith S, Sterle-Contala M, Cooperstock JR (2014) Real-time emergency
response: improved management of real-time information during crisis situations. J Multimodal
User Interfaces 8:161173
Boiko B (2005) Content management bible. Wiley, Indianapolis
Bonin F, Campbell N, Vogel C (2014a) Time for laughter. Knowl-Based Syst 71:1524
Bonin F, Vogel C, Campbell K (2014b) Social sequence analysis: temporal sequences in interactional conversations. In: 5th IEEE international conference on cognitive infocommunications,
Vietri sul Mare, pp 403406
Borus G, Gilanyi A (2013) Solving systems of linear functional equations with computer. In: 4th
IEEE international conference on cognitive infocommunications (CogInfoCom), pp 559562
Brewster S, Brown L (2004) Tactons: structured tactile messages for non-visual information
display. In: Proceedings of the 5th conference on Australasian user interface (AUIC04), vol
28. Dunedin, pp 1523
Brunger-Koch M, Briest S, Vollrath M (2006) Virtual driving with different motion characteristics:
braking manoeuvre analysis and validation. In: Proceedings of the driving simulation conference, pp 6978
Campbell N (2012) Social aspects and speechability in CogInfoCom systems. In: 3rd IEEE
international conference on cognitive infocommunications, Kosice
Carazo P, Font E (2010) Putting information back into biological communication. J Evol Biol
23(4):661669
Card SK, Moran TP, Newell A (1983) The psychology of human-computer interaction. Lawrence
Erlbaum Associates, Hillsdale, NJ, USA
Cassell J, Bickmore T, Billinghurst M, Campbell L, Chang K, Vilhjalmsson H, Yan H (1999)
Embodiment in conversational interfaces: REA. In: Proceedings of the SIGCHI conference on
human factors in computing systems, pp 520527
Castro J (1995) Fuzzy logic controllers are universal approximators. IEEE Trans Syst Man Cybern
25:629635
Castrucci M, Priscoli FD, Pietrabissa A, Suraci V (2011) A cognitive future Internet architecture.
Springer, Berlin
Cerf VG (2009) The day the Internet age began. Nature 461(7268):12021203
Chaffee SH, Metzger MJ (2001) The end of mass communication? Mass Commun Soc 4(4):
365379
Chen M, Gonzalez S, Vasilakos A, Cao H, Leung V (2011) Body area networks: a survey. Mobile
Netw Appl 16(2):171193
Cheng DS, Salamin H, Salvagnini P, Cristani M, Vinciarelli A, Murino V (2014) Predicting
online lecture ratings based on gesturing and vocal behavior. J Multimodal User Interfaces
8(2):151160
Chernoff H (1973) The use of faces to represent points in k-dimensional space graphically. J Am
Stat Assoc 68(342):361368
ChuDuc H, NguyenPhan K, NguyenViet D (2013) A review of heart rate variability and its
applications. APCBEE Procedia 7:8085
CloudNet 2014 (2014) Third IEEE international conference on cloud networking
196
References
CogInfoCom (2010) First international workshop on cognitive infocommunications

CogInfoCom (2011) Second international conference on cognitive infocommunications
CogInfoCom (2012) Third IEEE international conference on cognitive infocommunications
CogInfoCom (2013) Fourth IEEE international conference on cognitive infocommunications
CogInfoCom (2014) Fifth IEEE international conference on cognitive infocommunications
Csapo A (2014) Outlines of a framework for progressive representation-bridging coginfocom
channels. In: 5th IEEE international conference on cognitive infocommunications, Vietri sul
Mare, pp 321326
Csapo A, Baranyi P (2010) An interaction-based model for auditory subsitution of tactile percepts.
In: 14th IEEE international conference on intelligent engineering systems (INES), Gran
Canaria, pp 271276
Csapo A, Baranyi P (2011) Perceptual interpolation and open-ended exploration of auditory icons
and earcons. In: 17th international conference on auditory display, international community for
auditory display, Budapest
Csapo A, Baranyi P (2012a) A conceptual framework for the design of audio based cognitive infocommunication channels. In: Recent advances in intelligent engineering systems.
Studies in computational intelligence, vol 368. Springer, Berlin/Heidelberg, pp 261281.
doi:10.1007/978-3-642-23229-9_12
Csapo A, Baranyi P (2012b) CogInfoCom channels and related definitions revisited. In: 2012 IEEE
10th jubilee international symposium on intelligent systems and informatics (SISY). Subotica,
pp 7378
Csapo A, Baranyi P (2012c) The spiral discovery method: an interpretable tuning model for
CogInfoCom channels. J Adv Comput Intell Intell Inform 16(2):358367
Csapo A, Baranyi P (2012d) A unified terminology for the structure and semantics of CogInfoCom
channels. Acta Polytech Hung 9(1):85105
Csapo A, Baranyi P (2013) The evolving nature of human-device communication: lessons learned
from an example use-case scenario. Infocommun J 5(4):2732
Csapo A, Wersenyi G (2014) Overview of auditory representations in human-machine interfaces.
ACM Comput Surv 46(2):123
Csapo A, Israel JH, Belaifa O (2013) Oversketching and associated audio-based feedback
channels for a virtual sketching application. In: 4th IEEE international conference on cognitive
infocommunications, Budapest, pp 509513
Cybenko G (1989) Approximation by superposition of sigmoidal functions. Math Control Signals
Syst 2:303314
Czirbusz S (2012) Testing regularity of functional equations with computer. Aequationes Math
84(3):271283. doi:10.1007/s00010-012-0157-7
Dahlbom B (1996) The new informatics. Scand J Inf Syst 8(2):3
Danyadi Z, Foldesi P, Koczy L (2012) Fuzzy search space for correction of cognitive biases
in constructing mathematical models. In: 3rd IEEE international conference on cognitive
infocommunications, Kosice, pp 585589
Daras P, Alvarez F (2009) A future perspective on the 3D media Internet. In: Tselentis G et al
(eds) Towards the future Internet - a European research perspective. IOS Press, Amsterdam, pp
303312
De Groot S, De Winter JCF, Mulder M, Wieringa PA (2011) Nonvestibular motion cueing in a
fixed-base driving simulator: effects on driver braking and cornering performance. Presence
Teleoperators Virtual Environ 20(2):117142
de Meer H, Hummel KA, Basmadjian R (2012) Future Internet services and architectures: trends
and visions. Telecommun Syst (Special Issue) 51(4):219303
De Vos M, Debener S (2014) Mobile EEG: towards brain activity monitoring during natural action
and cognition. Int J Psychophysiol 91(1):12
Deacon TW (2013) Incomplete nature: how mind emerged from matter. WW Norton & Company
New York, NY, USA. ISBN: 978-0393343908
Dobelle W (2000) Artificial vision for the blind by connecting a television camera to the visual
cortex. ASAIO J 46(1):39
References
197
Domingue J et al (eds) (2011) The Future Internet - achievements and technological promises. In:
Future Internet assembly, Budapest. Springer, Heidelberg
Dropuljic B, Popovic S, Petrinovic D, Cosic K (2013) Estimation of emotional states enhanced by
a priori knowledge. In: 4th IEEE international conference on cognitive infocommunications,
Budapest, pp 481486
Drouin M, Kaiser DH, Miller DA (2012) Phantom vibrations among undergraduates: prevalence
and associated psychological characteristics. Comput Hum Behav 28(4):14901496
Duval T, Nguyen TTH, Fleury C, Chauffaut A, Dumont G, Gouranton V (2014) Improving awareness for 3D virtual collaboration by embedding the features of users physical environments and
by augmenting interaction tools with cognitive feedback tools. J Multimodal User Interfaces
8(2):187197
ECFI (2014) First European conference on future Internet, Brussels, 2-4 April 2014. Future Internet
PPP. www.ecfi.eu/brussels2014
Eichhorn W (1978) Functional equations in economics. Addison-Wesley, Reading
EITO (2014) European information technology observatory. ICT Market Report, Yearbooks from
2001 to 2014. ISSN: 097-4862
Ekman P (2003) Darwin, deception, and facial expression. Ann N Y Acad Sci 1000(1):205221
Ellis S (1991) Nature and origins of virtual environments: a bibliographical essay. Comput Syst
Eng 2(4):321347
Endsley M, Garland D (2000) Situation awareness: analysis and measurement. Routledge,
New York
Enriquez M, MacLean K (2003) The hapticon editor: a tool in support of haptic communication
research. In: Proceedings of the 11th symposium on haptic interfaces for virtual environment
and teleoperator systems (HAPTICS03). IEEE Computer Society, Los Angeles, pp 356362
Enriquez M, Maclean K, Chita C (2006) Haptic phonemes: basic building blocks of haptic
communication. In: Proceedings of the 8th international conference on multimodal interfaces
(ICMI 2006). ACM Press, Banff, pp 302309
Ericson A, Johansson C, Nergard H (2014) Manufacturing knowledge: going from production of
things to designing value in use. Intell Decis Technol 9(1):7989
ETSI (2011) About ETSI, 2011. European Telecommunications Standards Institute, SophiaAntipolis. http://www.etsi.org/website/aboutetsi/aboutetsi.aspx
European Commission (1997) Green paper on the convergence of the telecommunications, media
and information technology sectors, and implications for regulation. Towards an Information
Society Approach, COM, 3 December 1997
European Commission (2010) Future media networks - research challenges 2010. Future media
networks cluster of networked media systems FP7 projects. doi:10.2759/37178
European Commission (2013a) HORIZON 2020 - The Framework Programme for Research
and Innovation. Work Programme (2014-2020) 5.i. Leadership in enabling and industrial
technologies: Information and communication technologies. Annex 6 to Decision, December
2013, p 107
European Commission (2013b) International conference on Internet science. The FP7 European
Network of Excellence in Internet Science, Brussels, 9-11 April 2013. http://internet-science.
eu
Farago T, Konok V, Gacsi M (2011) Humans are able to recognize emotions by behavioural
expressions of an abstract artificial agent. In: 2nd international conference on cognitive
infocommunications
Fekete K, Csorba K (2014) Power extraction of human behavior in mobile environment. In: 5th
IEEE international conference on cognitive infocommunications, Vietri sul Mare, pp 481486
Fenn J, Raskino M (2008) Mastering the hype cycle: how to choose the right innovation at the right
time. Harvard Business Press Review, Boston
Fischer A, Beck MT, De Meer H (2013) An approach to energy-efficient virtual network
embeddings. In: 2013 IFIP/IEEE international symposium on integrated network management
(IM 2013), pp 11421147
198
References
Flanagan JR, Vetter P, Johansson RS, Wolpert DM (2003) Prediction precedes control in motor
learning. Curr Biol 13(2):146150
Flanagan JR, Bowman MC, Johansson RS (2006) Control strategies in object manipulation tasks.
Curr Opin Neurobiol 16(6):650659
Foldesi P, Botzheim J (2012) Computational method for corrective mechanism of cognitive
decision-making biases. In: 2012 IEEE 3rd international conference on cognitive infocommunications (CogInfoCom), pp 211215
Fortuna C, Mohorcic M (2009) Trends in the development of communication networks: cognitive
networks. Comput Netw 53(9):13541376
Fransman M (2002) Mapping the evolving telecoms industry: the uses and shortcomings of the
layer model. Telecommun Policy 26(9):473483
Freeman C (1994) The diffusion of information and communication technology in the world
economy in the 1990s. In: The management of information and communication technologies:
emerging patterns of control. Aslib, London, pp 841
Fuchs S, Hale K, Axellson P (2007) Augmented cognition can increase human performance in
the control room. In: 2007 IEEE 8th human factors and power plants and HPRCT 13th annual
meeting, Monterey, pp 128132
FutureRFID (2014) First international conference on future RFID technologies, Eger, 6-7 November 2014. www.futurerfid.ektf.hu
Gal Z, Almasi B, Daboczi T, Vida R, Oniga S, Baran S, Farkas I (2014) Internet of things:
application areas and research results of the FIRST project. Infocommun J 6(3):3744
Galambos P (2012) Vibrotactile feedback for haptics and telemanipulation: survey, concept and
experiment. Acta Polytech Hung 9(1):4165
Galambos P, Baranyi P (2011a) Vibrotactile force feedback for telemanipulation: concept and
applications. In: 2011 2nd international conference on cognitive infocommunications (CogInfoCom). IEEE, Budapest, pp 16
Galambos P, Baranyi P (2011b) VirCA as virtual intelligent space for RT-middleware. In: 2011
IEEE/ASME international conference on advanced intelligent mechatronics (AIM), Budapest,
pp 140145. doi:10.1109/AIM.2011.6027133
Galambos P, Weidig C, Baranyi P, Aurich JC, Hammann B, Kreylos O (2012a) VirCA NET:
a case study for collaboration in shared virtual space. In: 3rd IEEE international conference
on cognitive infocommunications. IEEE, Kosice, pp 273277
Galambos P, Weidig C, Zentay P, Csapo d, Baranyi P, Aurich JC, Hammann B, Kreylos O
(2012b) VirCA NET: a collaborative use case scenario on factory layout planning. In: 3rd IEEE
international conference on cognitive infocommunications. IEEE, Kosice, pp 467468
Galambos P, Csapo d, Zentay P, Fulop IM, Haidegger T, Baranyi P, Rudas IJ (2015) Design,
programming and orchestration of heterogeneous manufacturing systems through VR-powered
remote collaboration. Robot Comput Integr Manuf 33:6877
Galanis D, Karabetsos S, Koutsombogera M, Papageorgiou H, Esposito A, Riviello MT (2013)
Classification of emotional speech units in call centre interactions. In: 4th IEEE international
conference on cognitive infocommunications, pp 403406
Galis A, Gavras A (eds) (2013) The future Internet validated results and new horizons. In: Future
Internet assembly, Dublin. Springer, Heidelberg
Galis A, Denazis S et al. (eds) (2004) Programmable networks for IP service deployment. Artech
House Books, Norwood
Ganter B, Wille R (1999) Formal concept analysis. Springer, Berlin
Garriott R (1985) Ultima IV: quest of the avatar. Origin Systems US Copyright Office PA317504
Gartner Inc. (2014) Lead and Transform Your enterprise into the Digital Age, presented by L.
Weldon, Budapest
Garzo A, Benczur AA, Sidlo CI, Tahara D, Wyatt EF (2013) Real-time streaming mobility
analytics. In: 2013 IEEE international conference on big data, pp 697702
Gaver W (1986) Auditory icons: using sound in computer interfaces. Hum Comput Interact
Elsevier Science, Amsterdam 2(2):167177
References
199
Gaver W (1988) Everyday listening and auditory icons. Ph.D. thesis, University of California, San
Diego
Gaver W (1989) The SonicFinder: an interface that uses auditory icons. Hum Comput Interact
4(1):6794
Gaver W (1997) Auditory interfaces. In: Helander HG, Landauer TK, Prabhu P (eds) Handbook of
human-computer interaction. Elsevier, Amsterdam
Gazebo (2012) GAZEBO robot simulation environment. http://www.gazebosim.org/
Gilnyi A (1998) Solving linear functional equations with computer. Math Pannonica 9(1):5770
Giles J (2009) What is ICT? Michalsons. http://www.michalsons.co.za/what-is-ict/2525
Gilovich T, Griffin D, Kahneman D (2002) Heuristics and biases: the psychology of intuitive
judgement. Cambridge University Press, Cambridge
Granstrom B, House D, Swerts M (2002) Multimodal feedback cues in human-machine interactions. In: International conference on speech prosody 2002
Gray J (2000) The Hilbert problems 19902000. Newsletter 36:1013
Greitzer FL, Griffith D (2006) A human-information interaction perspective on augmented
cognition. In: Abstract submitted to augmented cognition international, CA[PNNL-SA-49657],
San Francisco
Griffith D, Greitzer FL (2007) Neo-symbiosis: the next stage in the evolution of human information
interaction. Int J Cogn Inform Nat Intell 1(1):3952
Gripenberg P (2011) The walking video interview (WVI) as a potential technique to tap into the
everyday experiences of ICTs. In: Information systems research seminar in Scandinavia (IRIS),
Turku, pp 139155
Gurkok H, Nijholt A (2012) Brain-computer interfaces for multimodal interaction: a survey and
principles. Int J Hum Comput Interact 28(5):292307
Hale KS, Fuchs S, Berka C (2008) Driving EEG cognitive assessment using eye fixations. In: 2nd
international conference on applied human factors and ergonomics, Las Vegas
Halpern JY (1990) An analysis of first-order logics of probability. Artif Intell 46(3):311350
Hangya V, Farkas R (2013) Target-oriented opinion mining from tweets. In: 4th IEEE international
Hanson MA, Powell HC Jr, Barth AT, Ringgenberg K, Calhoun BH, Aylor JH, Lach J (2009) Body
area sensor networks: challenges and opportunities. Computer 42(1):5865
Hzy A (2004) Solving linear two variable functional equations with computer. Aequationes Math
67(12):4762. doi:10.1007/s00010-003-2703-9
Hearst MA (1997) Dissonance on audio interfaces. IEEE Expert 12(5):1016
Hecht D, Reiner M (2009) Sensory dominance in combinations of audio, visual and haptic stimuli.
Exp Brain Res 193:307314
Heder M (2014) Emergence and tacit knowledge in machines. Ph.D. thesis. http://www.omikk.
bme.hu/collections/phd/Gazdasag_es_Tarsadalomtudomanyi_Kar/2014/Heder_Mihaly/tezis_
eng.pdf
Hemingway CJ, Gough TG (1998) A socio-cognitive theory of information systems. Research
Report Series, University of Leeds, School of Computer Studies (25):140
Henten A, Samarajiva R, Melody W (2003) Designing next generation telecom regulation: ICT
convergence or multi-sector utility? Info 5(1):2633
Hercegfi K (2011a) Heart rate variability monitoring during human-computer interaction. Acta
Polytech Hung 8(5):205224
Hercegfi K (2011b) Improved temporal resolution heart rate variability monitoring-pilot results
of non-laboratory experiments targeting future assessment of human-computer interaction.
Int J Occup Saf Ergon 17(2):105117
Hermann T (2002) Sonification for exploratory data analysis. Ph.D. thesis, University of Bielefeld
Hermann T (2008) Taxonomy and definitions for sonification and auditory display. In: 14th
international conference on auditory display, pp 18
Hermann T, Ritter H (1999) Listen to your data: model-based sonification for data analysis. In:
Lasker GE (ed) Advances in intelligent computing and multimedia systems. The International
Institute for Advanced Studies in System Research and Cybernetics, Baden-Baden, pp 189194
200
References
Hermann T, Hunt A, Neuhoff J (2011) The sonification handbook. Logos Verlag, Berlin
Hewes D (1995) The cognitive bases of interpersonal communication. Routledge, New York
Hirata-Kohno N, Petho A (2013) On a key exchange protocol based on diophantine equations.
Infocommun J 5(3):1721
Hochberg L, Donoghue J (2006) Sensors for brain-computer interfaces. IEEE Eng Med Biol Mag
25(5):3238
Hodder I (2012) Entangled: An Archaeology of the Relationships between Humans and Things.
Blackwell Publishing Wiley-Blackwell, Chichester, UK. ISBN: 978-0470-67211-2
Hodder I (2014) The entanglements of humans and things: a long-term view. New Lit Hist
45(1):1936
Hokanson B, Hooper S (2000) Computers as cognitive media: examining the potential of
computers in education. Comput Hum Behav 16(5):537552
Hollan J, Hutchins E, Kirsh D (2000) Distributed cognition: toward a new foundation for humancomputer interaction research. ACM Trans Comput Hum Interact 7(2):174196
Hunt A, Hermann T, Pauletto S (2004) Interacting with sonification systems: closing the loop.
In: Banissi E, Brner K (eds) IV04: proceedings of the 8th international conference on
information visualisation (IV04). IEEE Computer Society, London, pp 879884
Hurley S, Noe A (2003) Neural plasticity and consciousness. Biol Philos 18:131168
IDC 2014 (2014) International data corporation: European Internet of things ecosystem and trends.
http://www.idc.com/getdoc.jsp?containerId=IDC_P29863
IDATE (2009) DigiWorld Yearbook 2009, Montpellier. ISBN:978-2-84822-143-4
Ishteva M, Lathauwer L, Absil P, van Huffel S (2008) Dimensionality reduction for higher-order
tensors: algorithms and applications. Int J Pure Appl Math 42(3):337343
ITU (1999) Convergence and regulation. Volume of trends in telecommunication reform. International Telecommunication Union, Geneva
ITU (2009) Measuring the information society: the ICT development index. International Telecommunication Union, Geneva
ITU (2011a) ITU-T recommendation Y.3001: future network vision objectives and design goals.
International Telecommunication Union, Geneva
ITU (2011b) ITU-T recommendation Y.3011: framework of network virtualization for future
networks. International Telecommunication Union, Geneva
ITU (2012a) ITU-T recommendation Y.3021: framework of energy saving for future networks.
International Telecommunication Union, Geneva
ITU (2012b) ITU-T recommendation Y.3031: identification framework in future networks. International Telecommunication Union, Geneva
Jamsa J, Sukuvaara T, Luimula M (2014) Vehicle in a cognitive network. Intell Decis Technol
9(1):1727
Jokinen K (2008) User interaction in mobile navigation applications. In: Meng L, Zipf A, Winter S
(eds) Map-based mobile services. Lecture Notes in geoinformation and cartography. Springer,
Berlin/Heidelberg, pp 168197
Joosten S (1994) Trigger modelling for workflow analysis. In: Proceedings of CON94: workflow
management, challenges, paradigms and products, pp 236247
Kahneman D (2011) Thinking, fast and slow. Farrar, Straus and Giroux, New York, NY, USA
Kalmar A, Vida R, Maliosz M (2013) Context-aware addressing in the Internet of things
using bloom filters. In: 4th IEEE international conference on cognitive infocommunications,
pp 487492
Kapahnke P, Liedtke P, Nesbigall S, Warwas S, Klusch M (2010) An open platform for semanticbased 3D simulations in the 3D Internet. Lect Notes Comput Sci 6497:161176
Karnouskos S, Skarmeta A (2013) The future Internet of things. Introduction to chapter on Internet
of things. In: FIA 2013. Springer, Heidelberg, pp xxvxxvii
Kaye J (2004) Making scents: aromatic output for HCI. Interactions 11:4861.
doi:http://doi.acm.org/10.1145/962342.964333
References
201
Keszei B, Dull A, Logo E, Hamornik BP, Koles M, Tovolgyi S, Hercegfi K (2014) Visual attention
and spatial behavior in VR environment: an environmental psychology approach. In: 2014 5th
IEEE conference on cognitive infocommunications (CogInfoCom). IEEE, pp 247250
Kifor T, Gottdank T, Hajnal A, Baranyi P, Korondi B, Korondi P (2011) Smartphone emotions
based on human-dog interaction. In: 2011 2nd international conference on cognitive infocommunications (CogInfoCom), pp 16
Kimura M, Uwano H, Ohira M, Matsumoto KI (2009) Toward constructing an electroencephalogram measurement method for usability evaluation. In: Jacko JA (ed) Human-computer
interaction. New trends. Springer, Berlin, pp 95104
Kiss G, Vicsi K (2014) Physiological and cognitive status monitoring on the base of acousticphonetic speech parameters. In: Besacier L, Dediu A-H, Martin-Vide C (eds) Statistical
language and speech processing. Springer, Berlin, pp 120131
Kiss G, Sztaho D, Vicsi K, Golemis A (2014) Connection between body condition and speech
parameters - especially in the case of hypoxia. In: 5th IEEE international conference on
cognitive infocommunications, Vietri sul Mare, pp 333336
Kitagawa M, Dokko D, Okamura A, Yuh D (2005) Effect of sensory substitution on suturemanipulation forces for robotic surgical systems. J Thorac Cardiovasc Surg 129(1):151158
Koles M, Hercegfi K, Logo E, Tovolgyi S, Szabo B, Hamornik BP, Komlodi A, Baranyi
PZ, Galambos P, Persa G (2014) Collaboration experience in immersive VR environment
in the frame of the NeuroCogSpace project. In: 2014 5th IEEE conference on cognitive
infocommunications (CogInfoCom). IEEE, pp 373376
Kolmogorov A (1957) On the representation of continuous functions of many variables by
superposition of continuous functions of one variable and addition. Dokl Akad Nauk USSR
114:953956
Koltay K, Borbely M, Dani E, Balazs L, Csernoch M, Bujdoso G (2013) Users retrieval skills in
library environment: ways to measure and improve them. In: 4th IEEE international conference
on cognitive infocommunications, Budapest, pp 319323
Kondor D, Csabai I, Dobos L, Szule J, Barankai N, Hanyecz T, Sebok T, Kallus Z, Vattay G (2013)
Using robust PCA to estimate regional characteristics of language use from geo-tagged twitter
messages. In: 4th IEEE international conference on cognitive infocommunications, pp 393398
Kovcs S, Vincze D, Gcsi M, Miklosi A, Korondi P (2009) Interpolation based fuzzy automaton
for human-robot interaction. In: Preprints of the 9th international symposium on robot control
(SYROCO09). The International Federation of Automatic Control (IFAC), Nagaragawa
Convention Center, Gifu, pp 451456
Kovesdan G, Asztalos M, Lengyel L (2014) Fast android application development with component
modeling. In: 5th IEEE international conference on cognitive infocommunications, Vietri sul
Mare, pp 515520
Kozma R (1991) Learning with media. Rev Educ Res 61(2):179212
Krafft J (2003) Vertical structure of the industry and competition: an analysis of the evolution of
the info-communications industry. Telecommun Policy 27(8):625649
Krafft J (2010) Profiting in the info-coms industry in the age of broadband: lessons and new
considerations. Technol Forecast Soc Chang 77(2):265278
Kurian G, Anand M (2014) GENKEY - 2400 - The Internet of everything ecosystem - bringing IT
and OT together with the Internet of things, San Francisco. https://www.ciscolive.com/online/
connect/sessionDetail.ww?SESSION_ID=78188
Lakatos G, Miklosi A (2012) How can the ethological study of dog-human companionship inform
social robotics? In: Birke L, Hockenhull J (eds) Crossing boundaries: investigating humananimal relationships. Brill, Leiden, pp 187208
de Lathauwer L, Moor BD, Vandewalle J (2000) A multi linear singular value decomposition.
SIAM J Matrix Anal Appl 21(4):12531278
Latre B, Braem B, Moerman I, Blondia C, Demeester P (2011) A survey on wireless body area
networks. Wirel Netw 17(1):118
Lederman S (2004) Haptic identification of common objects: effects of constraining the manual
exploration process. Percept Psychophys 66(4):618628
202
References
Lee EA, Seshia SA (2011) Introduction to embedded systems - a cyber-physical systems approach.
LuLu, Berkeley. ISBN:978-1312-42740-2. http://LeeSeshia.org
Lee H, Lee J, Seo S (2009) Brain response to good and bad design. In: Jacko JA (ed) Humancomputer interaction. New trends. Springer, Berlin, pp 111120
Lemmens P, Bussemakers M, De Haan A (2001) Effects of auditory icons and earcons on visual
categorization: the bigger picture. In: Proceedings of the international conference on auditory
display (ICAD01), Helsinki, pp 117125
Leung WK, Yuen KW, Wong KH, Meng H (2013) Development of text-to-audiovisual speech
synthesis to support interactive language learning on a mobile device. In: 4th IEEE international
Lewis TG (2009) Network science: theory and applications. Wiley, Hoboken, NJ, USA. ISBN:
978-0470331880
Li Z, Giannini F, Falcidieno B, Pernot JP, Veron P (2013) Towards a new approach for the
description of shapes from multimodal data. In: 4th IEEE international conference on cognitive
Liao Sh (2003) Knowledge management technologies and applications-literature review from 1995
to 2002. Expert Syst Appl 25(2):155164
Liu Yl (2013) Convergence in the digital age. Telecommun Policy 37(8):611614.
doi:10.1016/j.telpol.2013.04.012
Logo E, Hamornik BP, Koles M, Hercegfi K, Tovolgyi S, Komlodi A (2014) Usability related
human errors in a collaborative immersive VR environment. In: 2014 5th IEEE conference
oncognitive infocommunications (CogInfoCom). IEEE, pp 243246
Lorentz G (1966) Approximation of functions. Holt, Reinhard and Winston, New York
Maclean K, Enriquez M (2003) Perceptual design of haptic icons. In: Proceedings of eurohaptics
2003, pp 351363
Madhyastha T, Reed A (1994) A framework for sonification design. In: Kramer G (ed) Auditory
display. Addison-Wesley, Reading
Marentakis G, Brewster SA (2005) A comparison of feedback cues for enhancing pointing
efficiency in interaction with spatial audio displays. In: Proceedings of the 7th international
conference on human computer interaction with mobile devices & services, pp 5562
Massimino M (1992) Sensory substitution for force feedback in space teleoperation. Ph.D. thesis,
MIT, Department of Mechanical Engineering
Matsubara D, Egawa T, Nishinaga N, Shin MK, Kafle VP, Galis A (2013) Open the way to future
networks a viewpoint framework from ITU-T. Springer, Berlin
Matuszka T, Vinceller Z, Laki S (2013) On a keyword-lifecycle model for real-time event detection
in social network data. In: 4th IEEE international conference on cognitive infocommunications,
pp 453458
McGee M (2002) Investigating a multimodal solution for improving force feedback generated
textures. Ph.D. thesis, University of Glasgow
Melody WH (1997) Telecom reform: principles, policies and regulatory practices. Den Private
Ingenirfond, Technical University of Denmark
Mihalydeak T, Csajbok ZE (2013) Membranes with boundaries. In: Membrane computing.
Springer, Berlin, pp 277294
Mikhaylov K, Pitkaaho T, Tervounen J, Niemela M (2013) Wireless sensor glove interface
and its application in digital holography. In: 4th IEEE international conference on cognitive
Miklosi A, Soproni K (2006) A comparative analysis of animals understanding of the human
pointing gesture. Anim Cogn 9(2):8193
Miklosi A, Kubinyi E, Topal J, Gacsi M, Viranyi Z, Csanyi V (2003) A simple reason for a big
difference: wolves do not look back at humans, but dogs do. Curr Biol 13(9):763766
Minutolo A, Esposito M, De Pietro G (2012) Development and customization of individualized
mobile healthcare applications. In: 3rd IEEE international conference on cognitive infocommunications, Kosice, pp 321326
References
203
Mitola J, Maguire G (1999) Cogitive radio: making software radios more personal. IEEE Pers
Commun 6(4):1318
Mori M (1970) Bukimi no tani (Uncanny valley effect). Energy 7(4):3335
Mulas F, Pilloni P, Manca M, Boratto L, Carta S (2013) Linking human-computer interaction with
the social web: a web application to improve motivation in the exercising activity of users.
In: 4th IEEE international conference on cognitive infocommunications, pp 351356
Mustonen M (2008) A review-based conceptual analysis of auditory signs and their design.
In: Proceeding of ICAD
Nannicelli T, Taberham P (eds) (2014) Cognitive media theory, 1st edn. Routledge, New
York/London
Nass CI, Yen C (2010) The man who lied to his laptop: what machines teach us about human
relationships. Current Trade Penguin Group, New York, NY, USA
Navarretta C (2014a) Feedback facial expressions and emotions. J Multimodal User Interfaces
8(2):135141
Navarretta C (2014b) Predicting emotions in facial expressions from the annotations in naturally
occurring first encounters. Knowl-Based Syst 71:3440
Navarretta C (2014c) Speech, emotions and facial expressions in dyadic spontaneous conversations. Intell Decis Technol 8(4):255263
Newman M, Barabasi AL, Watts DJ (2006) The structure and dynamics of networks. Princeton
University Press, Princeton
Nguyen KK, Cheriet M, Lemay M, Arnaud BS, Reijs V, Mackarel A, Minoves P, Pastrama A,
Van Heddeghem W (2011) Renewable energy provisioning for ICT services in a future Internet.
Springer, Berlin
Niitsuma M, Hashimoto H (2009) Observation of human activities based on spatial memory in
intelligent space. J Rob Mechatronics 21(4):515523
Niitsuma M, Hashimoto H, Hashimoto H (2007) Spatial memory as an aid system for human
activity in intelligent space. IEEE Trans Ind Electron 54(2):11221131
Nijholt A, Tan D (2008) Brain-computer interfacing for intelligent systems. IEEE Intell Syst
23(3):7279
Ning H, Wang Z (2011) Future Internet of things architecture: like mankind neural system or social
organization framework? IEEE Commun Lett 15(4):461463
Nishinaga N (2010) NICT new-generation network vision and five network targets. IEICE Trans
Commun 93(3):446449
Norman D (1991) Cognitive artifacts. Cambridge University Press, New York, pp 1738. http://
www.portal.acm.org/citation.cfm?id=120352.120354
OECD (1992) Telecommunications and broadcasting: convergence or collision? Organization for
Economic Cooperation and Development, Paris
OECD (2011) Guide to measuring the information society. Organization for Economic Cooperation and Development, Paris
Origlia A, Galata V, Cutugno F (2014) Introducing context in syllable-based emotion tracking.
In: 5th IEEE international conference on cognitive infocommunications, Vietri sul Mare,
pp 337342
Ousterhout T, Dyrholm M (2013) Cortically coupled computer vision with emotiv headset using
distractor variables. In: 4th IEEE international conference on cognitive infocommunications,
Budapest, pp 245249
Pallot M, Trousse B, Senach B (2012) A tentative design of a future Internet networking domain
landscape. In: The future Internet. Springer, Berlin, pp 237249
Palovics R, Daroczy B, Benczur AA (2013) Temporal prediction of retweet count. In: 4th IEEE
international conference on cognitive infocommunications, pp 267270
Pang ASK (2013) The distraction addiction: getting the information you need and the communication you want, without enraging your family, annoying your colleagues, and destroying your
soul. Hachette, New York
204
References
Papadimitratos P, La Fortelle A, Evenssen K, Brignolo R, Cosenza S (2009) Vehicular communication systems: enabling technologies, applications, and future outlook on intelligent
transportation. IEEE Commun Mag 47(11):8495
Papadimitriou D, Zahariadis T, Martinez-Julia P, Papafili I, Morreale V, Torelli F, Sales B,
Demeester P (2012) Design principles for the future Internet architecture. In: lvarez F et
al (eds) The future Internet. Springer, Berlin, pp 5567
Pavani F, Spence C, Driver J (2000) Visual capture of touch: out-of-the-body experiences with
rubber gloves. Psychol Sci 11(5):353359
Pentland A (2007) Social signal processing. IEEE Signal Process Mag 24(4):108
Pentland A (2008) Honest signals: how they shape our world. MIT Press, London
Perera C, Zaslavsky A, Christen P, Georgakopoulos D (2014) Context aware computing for the
Internet of things: a survey. IEEE Commun Surv Tutorials 16(1):414454
Persa G, Csapo A, Baranyi P (2012) CogInfoCom systems from an interaction perspective a pilot
application for EtoCom. J Adv Comput Intell Intell Inform 16(2):297304
Persa G, Torok A, Galambos P, Sulykos I, Kecskes-Kovacs K, Czigler I, Honbolygo F, Baranyi
P, Csepe V (2014) Experimental framework for spatial cognition research in immersive virtual
space. In: 2014 5th IEEE conference on cognitive infocommunications (CogInfoCom). IEEE,
pp 587593
Petcu D, Galis A (2013) Introduction to chapter on computing and networking clouds. In: The
future Internet: cloud computing, networking and mobility. Springer, Heidelberg, pp xiiixv
Picard RW (1995) Affective computing. The MIT Press, Cambridge
Picard R (1997) Affective computing. The MIT Press, Cambridge
Picard RW (2003a) Affective computing: challenges. Int J Hum Comput Stud 59(1):5564
Picard RW (2003b) What does it mean for a computer to have emotions. In: Trappl R, Petta P,
Payr S (eds) Emotions in humans and artifacts. MIT Press, Cambridge, pp 213235
Pieska S, Kaarela J, Luimula M (2014) Enhancing innovation capability with cognitive infocommunications. Intell Decis Technol 9(1):6778
Pinto M, Cavallo V, Ohimann T, Espie S, Roge J (2004) The perception of longitudinal
accelerations: what factors influence braking manoeuvres in driving simulators? In: Confrence
simulation de conduite, pp 139151
Pirhonen A (2006) Non-speech sounds as elements of a use scenario: a semiotic perspective.
In: Proceedings of the 12th international conference on auditory display (ICAD2006, London
Platzer E, Petrovic O (2011) An experimental deprivation study of mobile phones, Internet and
TV. Comput Technol Appl 2(8):600606
Pontonnier C, Dumont G, Samani A, Madeleine P, Badawi M (2014) Designing and evaluating
a workstation in real and virtual environment: toward virtual reality based ergonomic design
sessions. J Multimodal User Interfaces 8(2):199208
Preece J, Rogers Y, Sharp H, Benyon D, Holland S, Carey T (1994) Human-computer interaction.
Addison-Wesley Longman Ltd, Harlow, UK
Preissl B, Muller J (2006) Governance of communication networks: connecting societies and
markets with IT. Physica, Heidelberg
Prekopcsak Z, Halacsy P, Gaspar-Papanek C (2008) Design and development of an everyday hand
gesture interface. In: Proceedings of the 10th international conference on human computer
interaction with mobile devices and services, pp 479480
Prekopcsak Z, Makrai G, Henk T, Gaspar-Papanek C (2011) Radoop: analyzing big data with
rapidminer and hadoop. In: Proceedings of the 2nd RapidMiner community meeting and
conference (RCOMM 2011), pp 865874
Prinz J (2006) Putting the brakes on enactive perception. Psyche 12:119
Puszta A, Szule J, Laki S (2013) Near real-time thematic clustering of web documents and other
Internet contents. In: 4th IEEE international conference on cognitive infocommunications,
Budapest, pp 307312
Raptis S (2013) Exploring latent structure in expressive speech. In: 4th IEEE international
conference on cognitive infocommunications. IEEE, pp 741746
References
205
Rasmussen J (1982) Human errors: a taxonomy for describing human malfunction in industrial
installations. J Occup Accid 4(2):311333
Reason J (1990) Human error. Cambridge University Press, Cambridge
Recker M, Ram A, Shikano T, Li G, Stasko J (1995) Cognitive media types for multimedia
information access. J Educ Multimedia Hypermedia 4(23):183210
Riecke BE, Schulte-Pelkum J, Caniard F, Bulthoff HH (2005) Towards lean and elegant selfmotion simulation in virtual reality. In: Proceedings of virtual reality, 2005 (VR 2005). IEEE,
pp 131138
Riva G, Davide F (2001) Communications through virtual technologies. Identity, community and
technology in the communication age. IOS Press, Amsterdam, pp 124154
Rochlis J (2002) Human factors and telerobotics: tools and approaches for designing remote
robotics workstation displays. Ph.D. thesis, Massachusetts Institute of Technology
Romportl J, Zackova E, Kelemen J (eds) (2015) Beyond artificial intelligence the disappearing
human-machine divide. Topics in intelligent engineering and informatics, vol 9. Springer
International Publishing, Switzerland
Roschelle J (1996) Designing for cognitive communication: epistemic fidelity or mediating
collaborative inquiry? In: Computers, communication and mental models. Taylor & Francis,
London, UK pp 1527
RTC (2008) Robot technology component specification. http://www.omg.org/spec/RTC/
Ruxton GD, Schaefer HM (2011) Resolving current disagreements and ambiguities in the
terminology of animal communication. J Evol Biol 24(12):25742585
Ryan M et al. (eds) (2003) The EU regulatory framework for electronic communications and
related EU legislation. Handbook. Arnold and Porter, London
Saito T (1994) An evolving scenario of communication network towards b-ISDN. In: Iversen
VB (ed) Integrated broadband communication networks and services, vol 18. North Holland,
Amsterdam
Sales B, Darmois E, Papadimitriou D, Bourse D (2012) A systematic approach for closing the
research to standardization gap. In: lvarez F et al (eds) The future Internet. Springer, Berlin,
pp 1829
Sallai G (2007) Converging information, communication and media technologies. In: Banse G
(ed) Assessing societal implications of converging technological development. Edition Sigma,
Berlin, pp 2543
Sallai G (2012a) The cradle of cognitive infocommunications. Acta Polytech Hung 9(1):171181
Sallai G (2012b) Defining infocommunications and related terms. Acta Polytech Hung 9(6):515
Sallai G (2013a) Chapters of Future Internet Research. In: 2013 IEEE 4th international conference
on cognitive infocommunications (CogInfoCom). IEEE, New York, pp 161166
Sallai G (2013b) The FIRST project and the Future Internet National Research Programme. In:
Future Internet PPP workshop on building an eco-system for delivering innovative future
Internet services and applications, IEEE ICC 2013. IEEE, Budapest
Sallai G (2013c) From telecommunications to cognitive infocommunications and Internet of things
- phases of digital convergence. In: 2013 IEEE 17th international conference on intelligent
engineering systems (INES), pp 1317
Sallai G (2014) Future Internet visions and research clusters. Acta Polytech Hung 11(7):524
Sayrafian-Pour K, Yang WB, Hagedorn J, Terrill J, Yazdandoost KY, Hamaguchi K (2010) Channel
models for medical implant communication. Int J Wireless Inf Networks 17(34):105112
Sayrafian-Pour K, Hagedorn J, Barbi M, Terrill J, Alasti M (2013) A simulation platform to study
inter-BAN interference. In: 4th IEEE international conference on cognitive infocommunications, pp 345350
Scaletti C (1994) Sound synthesis algorithms for auditory data representations. In: Kramer G (ed)
Auditory display. Addison-Wesley, Reading
Schmorrow D, Stanney KM, Wilson G, Young P (2006) Augmented cognition in human-system
interaction. In: Handbook of human factors and ergonomics, 3rd edn. Wiley, New York,
pp 13641383
206
References
Scott-Phillips TC, Blythe RA, Gardner A, West SA (2012) How do communication systems
emerge? Proc R Soc B Biol Sci 279(1735):19431949
Sharmin S (2003) Evaluating non-visual feedback cues for touch input device. In: Evreinov G (ed)
New interaction techniques. University of Tampere, Tampere. pp 39. ISBN:951-44-5188-0
Sharples M, Jeffery N, Du Boulay JBH, Teather D, Teather B, Du Boulay GH (2002) Sociocognitive engineering: a methodology for the design of human-centred technology. Eur J Oper
Res 136(2):310323
Sheridan T (1992) Musings on telepresence and virtual presence. Presence Teleoperators Virtual
Environ 1(1):120126
Sheridan T (1994) Human factors considerations for remote manipulation, In: Proceedings of the
advanced guidance and control aspects in robotics, Lisbon, pp 124 ISBN: 92-835-0751-7
Sheth AP (2009) Citizen sensing, social signals, and enriching human experience. IEEE Internet
Comput 13(4):87
Shneiderman B (1998) Designing the user interface: strategies for effective human-computer
interaction, 3rd edn. Addison-Wesley, Reading
Skinner A, Long L, Vice J, Blitch J, Fidopiastis CM, Berka C (2013) Augmented interaction:
applying the principles of augmented cognition to human-technology and human-human
interactions. In: Foundations of augmented cognition. Springer, Berlin, pp 764773
Smith D (1975) Pygmalion: a computer program to model and stimulate creative thought. Ph.D.
thesis, Stanford University, Department of Computer Science
Smith IG (2012) The Internet of things 2012: new horizons. CASAGRAS2, Halifax
Smith C, Kisiel K, Morrison J (2009) Working through synthetic worlds. Ashgate, London
Solvang B, Sziebig G (2012) On industrial robots and cognitive info-communication. In: 3rd IEEE
international conference on cognitive infocommunications, Kosice, pp 459464
Spitters S, Sanders M, op den Akker R, Bruijnes M (2013) The recognition of acted interpersonal
stance in police interrogations. In: 4th IEEE international conference on cognitive infocommunications, Budapest, pp 6570
Sprecher D (1965) On the structure of continuous functions of several variables. Trans Am Math
Soc 115:340355
St John M, Kobus DA, Morrison JG, Schmorrow D (2004) Overview of the DARPA augmented
cognition technical integration experiment. Int J Hum Comput Interact 17(2):131149
Staal MA, Bolton AE, Yaroush RA, Bourne LE Jr (2008) Cognitive performance and resilience to
stress. In: Lukey B, Tepe V (eds) Biobehavioral resilience to stress. Francis & Taylor, London,
pp 259299
Stanney KM, Schmorrow DD, Johnston M, Fuchs S, Jones D, Hale KS, Ahmad A, Young P (2009)
Augmented cognition: an overview. Rev Hum Factors Ergon 5(1):195224
Stein B, Wallace M, Meredith A (1995) Neural mechanisms mediating attention and orientation to
multisensory cues. In: Gazzaniga M (ed) The cognitive neurosciences. MIT Press, Cambridge,
pp 683702
Stone L (2011) Just breathe: building the case for email apnea (the huffington post). http://www.
huffingtonpost.com/linda-stone/just-breathe-building-the_b_85651.html
Stratogiannis DG, Tsiropoulos GI, Vinel A, Koucheravy Y, Tsiropoulou EE (2013) Special issue
on mobile computing and networking technologies. Telecommun Syst 52(4):17142145
Streitz N, Nixon P (2005) The disappearing computer. Commun ACM 48(3):3235
Sun R, Merrill E, Peterson T (2001) From implicit skills to explicit knowledge: a bottom-up model
of skill learning. Cognit Sci 25(2):203244
Szabo C, Roka A, Farago T, Gacsi M, Miklosi A, Korondi P (2012) Building a human-dog
interaction inspired emotional engine model. In: IECON 2012 38th annual conference on IEEE
industrial electronics society, pp 55165522
Szabo R, Farkas K, Ispany M, Benczur AA, Batfai N, Jeszenszky P, Laki S, Vagner A, Kollar L,
Sidlo C (2013) Framework for smart city applications based on participatory sensing. In: 4th
IEEE international conference on cognitive infocommunications, pp 295300
References
207
Szegletes L, Koles M, Forstner B (2014) The design of a biofeedback framework for dynamic
difficulty adjustment in games. In: 5th IEEE international conference on cognitive infocommunications, Vietri sul Mare, pp 295299
Szucs G (2013) Decision trees and random forest for privacy-preserving data mining. In: Tarnay K,
Imre S, Xu L (eds) Research and development in e-business through service-oriented solutions.
IGI Global, Hershey, pp 7190
Stevenson D (ed) (1997) Information and communications technology in UK schools, an Independent Inquiry. The Independent ICT in Schools Commission, London
Takagi H (2001) Interactive evolutionary computation: fusion of the capabilities of EC optimization and human evaluation. Proc IEEE 89(9):12751296
Tan D, Nijholt A (2010) Brain-computer interfaces: applying our minds to human-computer
interaction. Springer, Berlin
Tanaka H, Sakti S, Neubig G, Toda T, Campbell N, Nakamura S (2012) Non-verbal cognitive skills
and autistic conditions: an analysis and training tool. In: 3rd IEEE international conference on
cognitive infocommunications, Kosice, pp 4146
Tapolcai J, Gulyas A, Heszbergery Z, Biro J, Babarczi P, Trossen D (2012) Stateless multi-stage
dissemination of information: source routing revisited. In: 2012 IEEE Global communications
conference (GLOBECOM), pp 27972802
Tapolcai J, Heszberger Z, Retvari G, Biro J (2013) Reduced information scenario for shared
segment protection. In: 4th IEEE international conference on cognitive infocommunications,
pp 589594
Taylor TL (2002) Living digitally: embodiment in virtual worlds. In: The social life of avatars.
Springer, London, pp 4062
Telpolicy (1994) Special issue on competition and convergence. Telecommun Policy 18(8)
Thomas RW, Friend DH, Dasilva LA, Mackenzie AB (2006) Cognitive networks: adaptation and
learning to achieve end-to-end performance objectives. IEEE Commun Mag 44(12):5157
Thomessen T, Kosicki T (2011) Cognitive audio-visual infocommunication applied in remote
support for industrial robot systems. In: 2011 2nd international conference on cognitive
infocommunications (CogInfoCom), pp 15
Thomessen T, Niitsuma M (2013) Cognitive human-machine interface with multi-modal manmachine communication. In: 4th IEEE international conference on cognitive infocommunications, Budapest, pp 873876
Todorov E (2004) Optimality principles in sensorimotor control. Nat Neurosci 7(9):907915
Topal J, Miklosi A, Csanyi V, Doka A (1998) Attachment behavior in dogs (canis familiaris): a new
application of Ainsworths (1969) strange situation test. J Comp Psychol 112(3):219
Torok M, Kosa Z (2015) Association in knowledge management technologies. In: 2015 IEEE
13th international symposium on applied machine intelligence and informatics (SAMI). IEEE,
pp 301306
Torok M, Toth MJ, Szollosi A (2013) Foundations and perspectives of mathability in relation to the
CogInfoCom domain. In: 4th IEEE international conference on cognitive infocommunications,
Budapest, pp 869872
Torok A, Sulykos I, Kecskes-Kovacs K, Persa G, Galambos P, Kobor A, Czigler I, Csepe V,
Baranyi P, Honbolygo F (2014) Comparison between wireless and wired EEG recordings in a
virtual reality lab: case report. In: 2014 5th IEEE conference on cognitive infocommunications
(CogInfoCom). IEEE, pp 599603
Toth B, Nagy P, Nemeth G (2012) New features in the voxaid communication aid for speech
impaired people. Springer, Berlin
Trinh TA, Gyarmati L, Sallai G (2011) Understanding the impact of loyal user behaviour on
Internet access pricing: a game-theoretic framework. Telecommun Syst 48(12):4361
Trung DN, Jung JJ, Kiss A (2013) Towards modeling fuzzy propagation for sentiment analysis in
online social networks: a case study on TweetScope. In: 4th IEEE international conference on
cognitive infocommunications, pp 331338
Tselentis G et al (eds) (2010) Towards the future Internet: emerging trends from European research.
In: Future Internet assembly, Valencia. IOS Press, Amsterdam
208
References
Uckelmann D, Harrisson M, Michahelles F (eds) (2011) Architecting the Internet of things.

Springer, Berlin
Ullah S, Higgins H, Braem B, Latre B, Blondia C, Moerman I, Saleem S, Rahman Z, Kwak KS
(2012) A comprehensive survey of wireless body area networks. J Med Syst 36(3):10651094
Valiant LG (1999) Robust logics. In: Proceedings of the 31st annual ACM symposium on theory
of computing, pp 642651. http://www.dl.acm.org/citation.cfm?id=301425
Valiant LG (2003) Three problems in computer science. J ACM 50(1):9699. http://www.dl.acm.
org/citation.cfm?id=602410
Valiant LG (2013) Probably approximately correct: natures algorithms for learning and prospering
in a complex world. Basic Books, New York, NY, USA
Valtonen TP (2001) Governmental visions for future info-communication-a survey of the European
union, the United States and Japan. Technical report no. 425, Turku Centre for Computer
Sciences
Van Do T, Chakka R, Sztrik J (2013) Spectral expansion solution methodology for QBD-M
processes and applications in future Internet engineering. In: Nguyen NT, Do TV, Le Thi
HA (eds) Advanced computational methods for knowledge engineering. Springer, Heidelberg,
pp 131142
Vermesan O, Friess P (2013) Internet of things: converging technologies for smart environments
and integrated ecosystems. River Publishers, Aalborg
Vernier F, Nigay L (2001) A framework for the combination and characterization of output modalities. In: Palanque P, Fabio P (eds) Interactive systems design, specification,
and verification. Lecture notes in computer science, vol 1946. Springer, Berlin/Heidelberg,
pp 3550
Verner L, Okamura A (2006) Sensor/actuator asymmetries in telemanipulators: implications of
partial force feedback. In: Proceedings of 14th symposium on haptic interfaces for virtual
environments and teleoperator systems, Arlington, pp 309314
Vernon D, Metta G, Sandini G (2007) A survey of artificial cognitive systems: implications for
the autonomous development of mental capabilities in computational agents. IEEE Trans Evol
Comput 11(2):151179
Vidal J (1973) Toward direct brain-computer communication. Ann Rev Biophys Bioeng 2:157180
Vilhjalmsson HH, Cassell J (1998) Bodychat: autonomous communicative behaviors in avatars.
In: Proceedings of the 2nd international conference on autonomous agents, pp 269276
Vinciarelli A, Pantic M, Bourlard H, Pentland A (2008) Social signals, their function, and automatic analysis: a survey. In: Proceedings of the 10th international conference on multimodal
interfaces, pp 6168
Vinciarelli A, Pantic M, Bourlard H (2009) Social signal processing: survey of an emerging
domain. Image Vision Comput 27(12):17431759
Vincze D, Kovacs S, Gacsi M, Korondi P, Miklosi A, Baranyi P (2012) A novel application of the
3d virca environment: modeling a standard ethological test of dog-human interactions. Acta
Polytech Hung 9(1):107120
Viranyi Z, Topal J, Gacsi M, Miklosi A, Csanyi V (2004) Dogs respond appropriately to cues of
humans-attentional focus. Behav Processes 66(2):161172
Vogel C (2013) Multimodal conformity of expression between blog names and content. In: 2013
IEEE 4th international conference on cognitive infocommunications (CogInfoCom). IEEE,
pp 2328
Voisin J, Lamarre Y, Chapman C (2002) Haptic discrimination of object shape in humans:
contribution of cutaneous and proprioceptive inputs. Exp Brain Res 145(2):251260
Waggoner Z (2009) My avatar, my self: identity in video role-playing games. McFarland, Jefferson
Wainwright N, Papanikolaou N (2012) Introduction: the FIA research roadmap, priorities for future
Internet research. In: The future Internet. Springer, Berlin, pp 15
Wang Y (2002) On cognitive informatics (keynote speech). In: 1st IEEE international conference
on cognitive informatics, Calgary, pp 3442
Wang Y (2008) On concept algebra: a denotational mathematical structure for knowledge and
software modeling. Int J Cognit Inform Nat Intell 2(2):119
References
209
Wang Y, Kinsner W (2006) Recent advances in cognitive informatics. IEEE Trans Syst Man
Cybern 36(2):121123
Watabe T, Niitsuma M (2013) Mental map generation assistance tool using relative pitch difference
and angular information for visually impaired people. In: 4th IEEE international conference on
cognitive infocommunications, pp 255260
WEF (2007) World Economic Forum: digital ecosystem - convergence between IT, Telecoms,
Media and Entertainment: Scenarios to 2015. World Scenario Series
Weikum G, Ntarmos N, Spaniol M, Triantafillou P, Benczr A et al (2011) Longitudinal analytics
on web archive data: its about time! 5th Biennial Conference on Innovative Data Systems
Research (CIDR 2011), 912 January 2011, pp 199202. Asilomar
Welch R, Warren D (1986) Intersensory interactions. In: Boff L, Thomas J (eds) Handbook of
perception and human performance, vol 1. Wiley, New York, pp 2536
Wilson L (2003) Interactivity or interpassivity: a question of agency in digital display. In:
Proceedings of the 5th international digital arts and culture conference, Melbourne, pp 14
Wolpert DM, Ghahramani Z (2000) Computational principles of movement neuroscience. Nat
Neurosci 3:12121217
Wolpert DM, Kawato M (1998) Multiple paired forward and inverse models for motor control.
Neural Netw 11(7):13171329
WUN CogCom (2008) Worldwide universities network cognitive communications consortium established. http://www.wun-cogcom.org/
Yang GZ (2014) Body sensor networks, 2nd edn. Springer, Berlin
Yang WB, Sayrafian-Pour K (2012) Interference mitigation using adaptive schemes in body area
networks. Int J Wireless Inf Netw 19(3):193200
Yao Y (2009) Interpreting concept learning in cognitive informatics and granular computing. IEEE
Trans Syst Man Cybern 39(4):855866
Yoonaidharma S, Kilenthong W, Bunaramrueang P, Kesavayuth D (2014) Special issue on moving
forward with future technologies: opening a platform for all. Telecommun Policy 38(89):
659850
Yuce MR (2010) Implementation of wireless body area networks for healthcare systems. Sensors
Actuators A Phys 162(1):116129. doi:10.1016/j.sna.2010.06.004
Zahariadis T, Daras P, Bouwen J, Niebert N, Griffin D, lvarez F et al (2010) Towards a contentcentric Internet. In: Tselentis G et al (eds) Towards the future Internet - emerging trends from
European Research. IOS Press, Amsterdam, pp 227236
Zahariadis T, Papadimitriou D, Tschofenig H, Haller S, Daras P, Stamoulis GD, Hauswirth M
(2011) Towards a future Internet architecture. Springer, Berlin
Zander TO, Kothe C (2011) Towards passive brain-computer interfaces: applying brain-computer
interface technology to human-machine systems in general. J Neural Eng 8(2):025005
Zeller D, Olsson M, Blume O, Fehske A, Ferling D, Tomaselli W, Godor I (2013) Sustainable
wireless broadband access to the future Internet-the earth project. Springer, Berlin
Zhang D, Guo B, Yu Z (2011) The emergence of social and community intelligence. Computer
44(7):2128
Ziegler S, Crettaz C, Ladid L, Krco S, Pokric B, Skarmeta AF, Jara A, Kastner W, Jung M (2013)
IoT6 moving to an IPv6-based future IoT. In: FIA. Springer, Berlin
Zimmerman T (1996) Personal area networks: near-field intrabody communication. IBM Syst
J 35(3):609617
Zimmerman TG (1999) Wireless networked digital devices: a new paradigm for computing and
communication. IBM Syst J 38(4):566574
Glossary
This glossary summarizes those terms that were either newly introduced, or
considered from a novel perspective in the book. The numbers listed next to each of
the items refer to the sections which may be consulted for further details (in some
cases, more than one section is indicated if the same term is discussed in different
places from different perspectives).
3D Internet 12.4
CogInfoCom channels 5.1, 8.1.2
Cognitive capabilities 2.3.2
Cognitive entity 2.3.2
Cognitive Internet Ecosystem 13.2
Cognitive networks 3.2.3, 4.6
Conceptual mapping (direct/analogy-based) 9.2
Cues 11.2
Data and content technologies 12.4
Digital convergence 3.1, 3.3
Digital Ecosystem 3.1, 3.3, 13.1
Electronic communications 3.2.2
Ethologically inspired CogInfoCom (EtoCom) 6.4
Future Internet 12
FI PPP (Future Internet Public-Private Partnership) 12.2
Generation CE 2.3.2

DOI 10.1007/978-3-319-19608-4
211
212
Glossary
Human-ICT entanglement 2.3.1

Icons 5.4, 7.3
Infocommunications 3.2.3
Information and Communications(s) Technology (ICT) 3.3
Intra-cognitive/inter-cognitive modes 2.2.1
Internet Ecosystem 13.1
Internet of Everything 12.2
Internet of Things 3.2.4, 12.4
Internet Science/Engineering/Practice 12.4
Mathability 6.6
Messages 5.4, 7.3
Modality 8.1.3
TIM (telecommunications, information and media) convergence 3.1, 3.2.3
Triggers (voluntary/involuntary, direct/indirect) 11.3
Representation-sharing/Representation-bridging types 2.2.2
Ritualization 11.2
Sensor-sharing/Sensor-bridging types 2.2.2
Signals 11.2
Socio-cognitive ICT 5.3
Speechability 5.2
Spiral Discovery Method (SDM) 10.1
Streams 7.3
Tangleface/Tangleaction 2.3.1, 6.4.1
Index
A
Abuczki, A., 65
Affective computing, 37, 4041
Ainsworths strange situation test
Fuzzy automaton and Fuzzy rule
interpolation, 8991
human-dog interactions, 91
test procedure, 8889
Analogy-based mapping
CogInfoCom channels, 138
corroborative stimulation, 137
parameter mapping, 136
scenario-based orchestration, 138
structural mapping, 136137
Arnold, V., 97
Auditory icons, 108, 112
Augmented cognition (AugCog), 37, 4142
Augmented reality, 179
Auvray, M., 50
B
Bach-y-Rita, P., 50, 51
Bajk, S., 98
Baranyi, P., 13, 69, 110
Barrass, S., 124
Benus, S., 60, 67
Berthelon, F., 69
Bjorn Solvang, 91
Blake, R., 164
Blattner, M., 115
Blum, J.R., 70
Body area networks (BANs), 37, 43
Body icons, 114, 116
Bonin, F., 64
Borus, G., 98
Brain computer interfaces (BCI), 37, 44

Brewster, S., 109
Brewster, S.A., 162
Brown, L., 109
C
Campbell, N., 13, 60
Cerf, V.G., 173
Cheng, D.S., 67
Chernoff, H., 116
CogInfoCom channels
bounded-input POMF, 125
channel differentiation, 58
context-based approaches and applications
in multimodal feedback, 69
in verbal signals, 6768
in written text, 68
DBIPOMFs, 125
definition, 121122
discretized bounded-input POMF, 125
generation vector, 124
generative model, 126127
high-level human communication
CogInfoCom trigger, 164
definitions, 166
directness, 165
mental and emotional relationships, 163
trigger and response types, 166, 167
volition, 164165
human-ICT communication, 158
icon-based approaches and applications
(see Icons)
interaction and communications, 57
low-level biological communication

DOI 10.1007/978-3-319-19608-4
213
214
cues, 159162, 167168
messages, 162, 163, 167168
ritualization, 159
signals, 159, 161, 162, 167168
mapping techniques
analogy-based mapping, 136138
automated reasoning systems,
139
corroborative stimulation, 138
direct mapping, 135136
high-level direct mapping
techniques, 138
modality, 122123
natural communication, 158159
nomenclature and notations, 123124
ORA model (see Object attribute relation
(ORA) model)
parameter-generating function, 124
pattern-based approaches and applications,
6970
perceptual concepts, 120121
perceptual gradation vector, 124
progressive learning, 169
regressive learning, 169
ritualization, 58
socio-cognitive ICT, 6061
speechability, 5960
spiral discovery method, 58
structural and semantic elements, 58
tunability
generic tuning model, 146147
HOSVD-based canonical form,
145146
POTPFs, 145
SDM (see Spiral discovery method
(SDM) method)
tensor product functions, 144145
Cognitive being, 20
Cognitive capabilities, 173
Cognitive capability, 106
Cognitive entity, 20
Cognitive infocommunications
(CogInfoCom), 4, 14
channels (see CogInfoCom channels)
cognitive capability, 4
cues, 159162, 167168
definition, 4
defnition, 14
design issues relevant, 92
engines, 8182
functionality, 4
generic perspective, 4
humanoid applications, 179
Index
icons
auditory icons, 112
body icons, 114
dynamic icons, 112113
emotional icons, 113114
haptic icons, 112
kinesthetic icons, 114
network packet icons, 114
olfactory icons, 112
visual icons, 112
implicit and explicit assumptions, 1
emerging and entanglement,
1618
industrial robotics and production
management, 9293
messages, 110, 162, 163,
167168
qualitative and quantitative analysis,
4
scientific priming effects (see Scientific
priming effects)
signals, 159, 161, 162, 167168
streams, 110111
synergic contribution, 40
tanglefaces, 8182
transition, 2122
trigger, 164
VirCA, industrial capabilities, 9395
Cognitive informatics (CI), 44
Cognitive Internet Ecosystem (CIES)
DES (see Digital Ecosystem (DES))
IES, (see Internet Ecosystem (IES))
Internet of Things, 189
long-term vision, 190191
3D Internet and community
applications, 189
Cognitive networks, 30, 190
definition, 45
Future Internet
cognitive content space, 45
Internet of Things (IoT), 4647
3D Internet (3DI), 46, 47
Cognitive thing, 20
Compact form of higher-order singular value
decomposition (CHOSVD), 146
Corroborative stimulation, 137
Crowdsourcing and crowdsensing community
applications, 179
Csapo, A., 13, 69, 110
Csorba, K., 69
Cyber devices (CDs), 74
Czirbusz, S., 98
Index
D
Dahlbom, B., 16
DBIPOMFs. See Discretized bounded-input
partially ordered multivariate
functions (DBIPOMFs)
Deference thesis, 50
Digital Age, 188189
Digital convergence, 1
Internet, 25
Digital convergence prism, 34
Digital ecosystem (DES), 35
Digital Age, 188189
middle kingdoms scenario, 188
safe havens scenario, 188
youniverse scenario,
188
Direct mapping, 135136
Discretized bounded-input partially ordered
multivariate functions (DBIPOMFs),
126
Dominance thesis, 50
Dropuljic, B., 68
Dynamic icons, 112113, 115116
E
Earcons, 115
e-content, 29
Eichhorn, W., 99
Electronic communications, 2728
Emergent cognitive capabilities,
190
Emoticons, 116
Emotional icons, 113114
Enriquez, M., 109, 115
Ericson, A., 93
Esposito, A., 60
Ethologically informed CogInfocom
(EtoCom)
Ainsworths strange situation test
Fuzzy automaton and Fuzzy rule
interpolation, 8991
human-dog interactions, 91
test procedure, 8889
CogInfoCom tanglefaces and engines,
8182
EtoMotor, 8385
EtoPhone, 8587
human-ICT co-evolution, 80
EtoMotor, 8385
EtoPhone, 8587
Eyecons, 115
215
F
Fekete, K., 69
FI public-private partnership (PPP), 178
Fixed and mobile telephony convergence
(FMC), 27
Foldesi, P., 42
Future Internet (FI), 37
cognitive capabilities, 173
cognitive content space, 45
features, 179
functions, 178179
Internet of Things (IoT), 4647, 173, 175
limitations, 174175
networking, evolution of, 173174
recent capabilities, 179
research goals and spheres, 179180
community applications, 184
cyber-physical systems and
applications, 184
data and content technologies, 182
experimentation, standardization,
regulation, 184185
Internet of Things (IoT), 183
Internet Science, 181
modeling, analysis and design, 181
network architectures, 181182
3D internet and cognitive
infocommunications, 183
smart applications, 173
technological opportunities, 175
3D Internet (3DI), 46, 47, 173
Vinton G. Cerf and Robert E. Kahn (fathers of
the Internet), 173
virtual collaborations, 173
visions
data and content awareness, 177
FI-based applications, 178
intelligent, innovative and secure society,
177
ITU-T, 175176
NICT, 175176
NWGN, 175176
scalable, service aware network, 176177
sustainability, environmental awareness,
177
virtual, resource aware network, 177
G
Galanis, D., 67
Ganter, B., 130
Garriott, R., 52
216
Index
Gaver, W., 112

Gazebo, 75
Gedeon, T., 13
generation CE, 20
Generic tuning model, 146147, 152153
Gilanyi, A., 95, 98, 102
Gough, T.G., 61
Granstrom, B., 162
Gripenberg, P., 16
International Telecommunication Union

Telecommunication Standardization
Sector (ITU-T), 175
Internet Ecosystem (IES), 189
Internet of Everything Ecosystem, 189
Internet of Things, 173
Internet of Things Ecosystem, 189
Internet service providers (ISPs), 29
Intra-cognitive communication, 14
H
Haptic icons, 108, 112, 115
Hashimoto, H., 13
Hzy, A., 98
Hemingway, C.J., 61
Hercegfi, K., 78
Hermann, T., 110
Higher-order singular value decomposition
(HOSVD), 142143
High-level direct mapping, 135136
Hodder, I., 17
Human-computer interactions (HCI), 4748
intersensory integration, 49
sensory dominance, 49
sensory modalities, 49
Human-device communication, 166
Human-human communication, 166
J
Jokinen, K., 13
Joosten, S., 164
I
Icons
auditory icons, 108
body icons, 114
haptic icons, 108, 112
in multimodal feedback, 6566
in physiological signals, 63
tactile icon, 108
in verbal signals, 6465
visual icons, 112
in written text, 65
Infocommunications (Infocom), 33
Information and communications technology
(ICT), 1
Information Society Technologies (IST), 33
Inter-cognitive communication, 1415
International Telecommunication Union (ITU),
28, 34
L
Lathauwer, L., 146, 149, 150
Lee, J.-H., 13
Linear functional equations, 9899
Li, Z., 92
Logo, E., 79
Low-level direct mapping, 135
K
Kaczmarek, K., 50
Kahn, R.E., 173
Kato, T., 13
Kifor, T., 85
Kinesthetic icons, 114, 116
Kiss, G., 64
Knowledge-based errors, 79
Kolmogorov, A., 97
Komlodi, A., 78
Kondor, D., 68
Kotoku, T., 13
Kovacs, S., 82, 89
M
MacLean, K., 109, 115
Maguire, G., 6
Magyar, G., 13
Marentakis, G., 162
Massive open online courses (MOOCs), 179
Mathability
grading system, 97
granular and analytical representations,
9798
granular formulations, 96
ICT devices, 96
Linear functional equations, 9899
mathematical capabilities, 95
Index
Mathability (cont.)
proper solution, 9596
sets of functional equations, 102
Theorems, 99102
McGee, M., 107
Meng, H., 13
Miklosi, A., 80, 82
Mitola, J., 6
Myin, E., 50
N
Nass, C.I., 163, 164
National Institute of Information and
Communications Technology
(NICT), 175
Nmeth, G., 14
Network packet icons, 114, 116
New-generation networks (NWGN), 175
Next generation Internet. See Future
Internet (FI)
Niitsuma, M., 14, 47
Nixon, P., 48
Norman, D., 135
O
Object attribute relation (ORA) model
definitions, 130131
interactive concepts, 134
message generated perceptual concepts
fully specified, 132133
interactive concepts, 134
perceptual concepts, 132
perceptual evaluation function, 133134
Olfactory icons, 112
ORA model. See Object attribute relation
(ORA) model
Origlia, A., 67
P
Ples, Z., 98
Plovics, R., 68
Pang, A.S.K., 16, 17
Parameter mapping, 136137
Partially ordered multivariate (POMFs)
functions, 125
Partially ordered multivariate tensor product
(TP) functions (POTPFs), 145
Perceptual evaluation function, 133134
Persa, G., 85
Personal area networks (PANs), 43
Picard, R., 40
217
Pieska, S., 93
POMFs functions. See Partially ordered
multivariate (POMFs) functions
Progressive learning, 169
Puszta, A., 68
R
Raptis, S., 64
Rasmussen/Reason framework errors, 79
Regressive learning, 169
Representation-bridging communication, 15
Representation-sharing communication, 15
Ritter, H., 110
Romportl, J., 16
RT-Middleware (RTM), 7475
Rule-based errors, 7980
S
Sallai, G., 7, 14
Scenario-based mapping, 137138
Schmorrow, D., 41
Scientific priming effects
cognitive communication, 6
cognitive informatics, 6
cognitive media, 5
convergence process, 7
infocommunication, 4, 78
media informatics and media
communications, 5
Sensor-bridging communication, 15
Sensor-sharing communication, 15
Sensory dominance, 5051
Sensory modality, 106
Sensory signals structure
auditory icons and earcons, 109
CogInfoCom icons
auditory icons, 112
body icons, 114
haptic icons, 112
visual icons, 112
CogInfoCom messages, 110, 114116
CogInfoCom streams, 110111
context of communication
directness, 106
mode, 105
218
temporal reference, 106
type, 106
voluntariness, 106
modality, 108
olfactory icons and smicons, 109
sensory modality, 106
sonification, 110
Sharmin, S., 162
Sharples, M., 61
Shneiderman, B., 112
Skill-based errors, 79
Smart applications, 173
Smicons, 115
Smith, D.C., 112
Social signal processing (SPS), 37, 5152
Socio-cognitive ICT, 37
Solvang, W.D., 91
Speechability, 37
Spiral discovery method (SDM) method,
58, 144
audio-based CogInfoCom channels,
152153
CogInfoCom application scenarios, 142
cognitive artifact, 142
confusion matrices, 155156
error sizes, 155
generate CogInfoCom messages, 141
generation parameter configurations, 152,
154
generic model, 152153
HOOI, 143
HOSVD, 142143
implementation, 151152
interpretability and complexity, 147148
lemma and theorem, 149150
minimal-rank orthogonal system, 147
original parameter-generating function, 148
yield optimal rank-reduction, 148
Spitters, S., 70
Streitz, N., 48
Structural mapping technique, 136137
Szabo, C., 84
Szegletes, L., 63
T
Tactile icon, 108
Technological priming effects
discrete gestures, 10
elementary capabilities, 9
enlightenment, 8
inflated expectations, 8
plateau of productivity, 9
Index
R&D initiatives, 10
speech utterances/facial information, 10
technological hype cycle, 8
technology trigger, 8
trough of disillusionment, 8
Telecommunications
cognitive infocommunications, 3132
communication sectors, 2627
content space, 3132
electronic communications, 2728
infocommunications, 2830
ISP, 29
Three dimensional (3D) Internet, 31, 173
3D virtual collaboration, 179
Topl, J., 86, 91
U
Uckelmann, D., 46
V
Vinciarelli, A., 51, 69
Vincze, D., 89
Virtual and augmented avatars, 37
definition, 52
episodic interactions, 53
identity, 53
metacommunicational fidelity, 53
objects and information, 53
presence, 53
qualifications, 52
Virtual and augmented reality, 37
communication capabilities, of human, 54
encumberment, 54
sensory capabilities, 55
situation awareness, 54
Virtual Collaboration Arena (VirCA) platform
CDs, 74
CogInfoCommediated neuroscientific
analysis, 77
experiment descriptor, 78
industrial capabilities, 9395
MTA SZTAKI, 74
multi-user scenarios, 7576
neuroscience inspired tools, 77
new synergies, development of, 7677
Rasmussen/Reason framework errors,
79
RTM, 7475
structure, 75
wired and wireless EEG configurations,
78
Index
219
Virtual collaborations, 173

Visual icons, 112
Wille, R., 130

Wilson, L., 52
W
Wang, Y., 130, 131
Wersenyi, G., 110
Y
Yen, C., 163, 164

Cognitive Infocommunications

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Cognitive Infocommunications

Transféré par

Droits d'auteur :

Formats disponibles

PterBaranyi AdamCsapo

Cognitive Infocommunications (CogInfoCom)

Pter Baranyi Adam Csapo Gyula Sallai

Szchenyi Istvn University

ISBN 978-3-319-19608-4 (eBook)

Library of Congress Control Number: 2015953263

HCI D computational linguistics

Emergence of CogInfoCom in Science and Technology . . . . . . . . . . . . . . . .

Definitions, Concepts and Assumptions . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

ICT-Based Foundations of CogInfoCom .. . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

Research Directions Under CogInfoCom

Synergies Between CogInfoCom and Other Fields .. . . . . . . . . . . . . . . . . . . .

CogInfoCom-Driven Research Areas.. . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

Novel Research Initiatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

Structure of Sensory Signals: Icons and Messages. .. . . . . . . . . . . . . . . . . . . .

10 Tunability of CogInfoCom Channels . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .

10.5 Implementation of SDM . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 151

Future Perspectives of CogInfoCom

12 Cognitive Capabilities in the Future Internet . . . . . . .. . . . . . . . . . . . . . . . . . . .

13 Towards a Cognitive Internet Ecosystem . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 187

About the Authors

Pter Baranyi is a professor at the Multidisciplinary Doctoral School of the

About the Book

This book describes the theoretical foundations of cognitive infocommunications

This part presents the scope and goal of cognitive infocommunications

Emergence of CogInfoCom in Science

In this chapter, an overview is given of the scientific and technological triggers

Springer International Publishing Switzerland 2015

1 Emergence of CogInfoCom in Science and Technology

1.1 What Is CogInfoCom?

1.2 Scientific Priming Effects

1.2 Scientific Priming Effects

1.2.1 Convergence from a General Perspective

1 Emergence of CogInfoCom in Science and Technology

1.2 Scientific Priming Effects

1.2.2 Convergence from an Infocommunications Perspective

1 Emergence of CogInfoCom in Science and Technology

1.3 Technological Priming Effects

1.3 Technological Priming Effects

1 Emergence of CogInfoCom in Science and Technology

technologies can also be assessed relative to one another, or alternatively in groups

1.3 Technological Priming Effects

Fig. 1.3 (continued)

1 Emergence of CogInfoCom in Science and Technology

Definitions, Concepts and Assumptions

2.1 Defining CogInfoCom

Springer International Publishing Switzerland 2015

2 Definitions, Concepts and Assumptions

2.2 Concepts Emerging from CogInfoCom

2.2.1 Mode of Communication

2.2 Concepts Emerging from CogInfoCom

In accordance with the introductory remarks to this section, future developments

2.2.2 Type of Communication

2 Definitions, Concepts and Assumptions

2.3 Implicit and Explicit Assumptions

2.3.1 Levels of Merging and Entanglement

2.3 Implicit and Explicit Assumptions

2 Definitions, Concepts and Assumptions

2.3.2 Levels of Cognitive Capability

2.3 Implicit and Explicit Assumptions

2 Definitions, Concepts and Assumptions

newer use cases

starting point in the design of CogInfoCom systems, especially when combined

2.3 Implicit and Explicit Assumptions