Vous êtes sur la page 1sur 393

Multisensory Development

This page intentionally left blank


Multisensory
Development
Edited by

Andrew J. Bremner
Goldsmiths, University of London

David J. Lewkowicz
Florida Atlantic University
and

Charles Spence
University of Oxford

1
1
Great Clarendon Street, Oxford OX2 6DP,
United Kingdom
Oxford University Press is a department of the University of Oxford.
It furthers the University’s objective of excellence in research, scholarship,
and education by publishing worldwide. Oxford is a registered trade mark of
Oxford University Press in the UK and in certain other countries
© Oxford University Press 2012
The moral rights of the authors have been asserted
First Edition published in 2012
Impression: 1
All rights reserved. No part of this publication may be reproduced, stored in
a retrieval system, or transmitted, in any form or by any means, without the
prior permission in writing of Oxford University Press, or as expressly permitted
by law, by licence or under terms agreed with the appropriate reprographics
rights organization. Enquiries concerning reproduction outside the scope of the
above should be sent to the Rights Department, Oxford University Press, at the
address above
You must not circulate this work in any other form
and you must impose this same condition on any acquirer
British Library Cataloguing in Publication Data
Data available
Library of Congress Cataloging in Publication Data
Data available
ISBN 978–0–19–958605–9
Printed and bound by
CPI Group (UK) Ltd, Croydon, CR0 4YY
Whilst every effort has been made to ensure that the contents of this work
are as complete, accurate and-up-to-date as possible at the date of writing,
Oxford University Press is not able to give any guarantee or assurance that
such is the case. Readers are urged to take appropriately qualified medical
advice in all cases. The information in this work is intended to be useful to the
general reader, but should not be used as a means of self-diagnosis or for the
prescription of medication
Links to third party websites are provided by Oxford in good faith and
for information only. Oxford disclaims any responsibility for the materials
contained in any third party website referenced in this work.
Foreword

Andy and Charles first thought up the idea of producing this book some years ago over a few pints
of Sam Smith’s Nut Brown Ale in the Three Goats Heads in Oxford (a rich multisensory experi-
ence if ever there was one). After starting on a book proposal they quickly realised that the project
would not be possible without David’s help. Throughout the process of putting this volume
together we have been very lucky to have been able to attract such a fantastic and willing set of
contributors. We are very proud of the addition that this book will make to the literature and we
have the many authors who have contributed to thank for that. We would also like to acknowl-
edge the support that we have received from Martin Baum and Charlotte Green at OUP, and
also colleagues at Goldsmiths, Oxford, and Florida Atlantic. Fran Knight, JJ Begum, Madeleine
Miller-Bottome, Conor Glover, and Jenn Hiester bear particular mention. Nut Brown Ales all
around!
AJB was supported by European Research Council Grant No. 241242 (European Commission
Framework Programme 7) and DJL was supported by NSF grant BCS-0751888 and NIH grant
D057116 during the preparation of this volume.
This page intentionally left blank
Contents

Contributors ix
1 The multisensory approach to development 1
Andrew J. Bremner, David J. Lewkowicz, and Charles Spence

Part A Typical development of multisensory processes from


early gestation to old age
2 The role of olfaction in human multisensory development 29
Benoist Schaal and Karine Durand
3 The development and decline of multisensory flavour perception:
Assessing the role of visual (colour) cues on the perception of taste
and flavour 63
Charles Spence
4 Crossmodal interactions in the human newborn: New answers to
Molyneux’s question 88
Arlette Streri
5 The development of multisensory representations of the body and of
the space around the body 113
Andrew J. Bremner, Nicholas P. Holmes, and Charles Spence
6 The development of multisensory balance, locomotion, orientation,
and navigation 137
Marko Nardini and Dorothy Cowie
7 The unexpected effects of experience on the development of
multisensory perception in primates 159
David J. Lewkowicz
8 The role of intersensory redundancy in early perceptual, cognitive,
and social development 183
Lorraine E. Bahrick and Robert Lickliter
9 The development of audiovisual speech perception 207
Salvador Soto-Faraco, Marco Calabresi, Jordi Navarra,
Janet F. Werker, and David J. Lewkowicz
10 Infant synaesthesia: New insights into the development of
multisensory perception 229
Daphne Maurer, Laura C. Gibson, and Ferrinne Spector
11 Multisensory processes in old age 251
Paul J. Laurienti and Christina E. Hugenschmidt
viii CONTENTS

Part B Atypical multisensory development


12 Developmental disorders and multisensory perception 273
Elisabeth L. Hill, Laura Crane, and Andrew J. Bremner
13 Sensory deprivation and the development of multisensory integration 301
Brigitte Röder

Part C Neural, computational, and evolutionary mechanisms


in multisensory development
14 Development of multisensory integration in subcortical and cortical
brain networks 325
Mark T. Wallace, Dipanwita Ghose, Aaron R. Nidiffer, Matthew C. Fister,
and Juliane Krueger Fister
15 In search of the mechanisms of multisensory development 342
Denis Mareschal, Gert Westermann, and Nadja Althaus
16 The evolution of multisensory vocal communication in primates and the
influence of developmental timing 360
Asif A. Ghazanfar

Author index 373


Subject index 375
Contributors

Nadja Althaus Asif A. Ghazanfar


Department of Psychology, Neuroscience Institute,
Oxford Brookes University, Departments of Psychology and Ecology and
Headington Campus, Evolutionary Biology,
Gipsy Lane, Oxford OX3 0BP, UK Princeton University,
Lorraine E. Bahrick Princeton, NJ 08540, US
Department of Psychology, Dipanwita Ghose
Florida International University, Department of Psychology,
Miami, FL33199, US Vanderbilt University, 465 21st Avenue
Andrew J. Bremner South,
Sensorimotor Development Research Unit, MRB III, Suite 7110, Nashville,
Department of Psychology, TN 37232–8548, US
Goldsmiths, University of London, Laura C. Gibson
New Cross, London, SE14 6NW, UK Department of Psychology,
Marco Calabresi Neuroscience and Behavior,
Department of Information and McMaster University,
Communication Technologies, 1280 Main Street West, Hamilton,
Universitat Pompeu Fabra, Ontario L8S4L8, Canada
Room 55.108, c/Roc Boronat 138, Elisabeth L. Hill
08018 Barcelona, Spain Sensorimotor Development Research Unit,
Dorothy Cowie Department of Psychology,
Sensorimotor Development Research Unit, Goldsmiths, University of London,
Department of Psychology, Goldsmiths, New Cross, London, SE14 6NW, UK
University of London, Nicholas P. Holmes
New Cross, London, SE14 6NW, UK Department of Psychology,
Laura Crane School of Psychology and Clinical Language
Department of Psychology, Sciences,
Goldsmiths, University of Reading, Reading, RG6 6AL, UK
University of London, Christina E. Hugenschmidt
New Cross, London, SE14 6NW, UK Center for Human Genomics,
Karine Durand Wake Forest University School of Medicine,
Developmental Ethology and Cognitive Winston-Salem NC, 27157, US
Psychology Group, Juliane Krueger Fister
Centre for Taste and Smell Science, Neuroscience Graduate Program,
CNRS (UMR 6265), Université de Bourgogne, Vanderbilt University,
15 rue Picardet, 21000 Dijon, France 465 21st Avenue South, MRB III,
Matthew C. Fister Suite 7110, Nashville, TN 37232–8548, US
Vanderbilt University Institute of
Imaging Science, 1161 21st Avenue South,
Medical Center North, AA-1105,
Nashville, TN 37232-2310, US
x CONTRIBUTORS

Paul J. Laurienti Benoist Schaal


Department of Radiology, Developmental Ethology and Cognitive
Wake Forest University School of Medicine, Psychology Group,
Winston-Salem NC, 27157, US Centre for Taste and Smell Science,
David J. Lewkowicz CNRS (UMR 6265),
Department of Psychology, Université de Bourgogne, 15 rue Picardet,
Florida Atlantic University, 21000 Dijon, France
777 Glades Road, Boca Raton, FL 33431, US Salvador Soto-Faraco
Robert Lickliter Department of Information and
Department of Psychology, Communication Technologies,
Florida International University, Universitat Pompeu Fabra, Room 55.108,
Miami, FL33199, US c/Roc Boronat 138, 08018 Barcelona, Spain

Denis Mareschal Ferrinne Spector


Centre for Brain and Cognitive Development, Department of Psychology,
School of Psychological Sciences, Birkbeck, Neuroscience and Behavior,
University of London, Malet Street, McMaster University,
London, WC1E 7HX, UK 1280 Main Street West, Hamilton, Ontario
L8S4L8, Canada
Daphne Maurer
Department of Psychology, Charles Spence
Neuroscience and Behavior, Crossmodal Research Laboratory,
McMaster University, Department of Experimental Psychology,
1280 Main Street West, Hamilton, University of Oxford,
Ontario L8S4L8, Canada South Parks Road, Oxford, OX1 3UD, UK

Marko Nardini Arlette Streri


Department of Visual Neuroscience, Paris Descartes University
UCL Institute of Ophthalmology, Institut Universitaire de France
11–43 Bath Street, Laboratoire Psychologie de la Perception
London EC1V 9EL, UK CNRS UMR 8158
rue des Saints Pères, 45
Jordi Navarra 75006 Paris, France
Fundació Sant Joan de Déu,
Sant Boi de Llobregat, Mark T. Wallace
Hospital Sant Joan de Déu, Vanderbilt Brain Institute,
Edifici Docent, 4th floor, Departments of Hearing and Speech Sciences,
C/Santa Rosa, 39–57, 08950 Psychology and Psychiatry,
Esplugues del Llobregat, Vanderbilt University,
Barcelona, Spain 465 21st Avenue South, MRB III,
Suite 7110, Nashville, TN 37232–8548, US
Aaron R. Nidiffer
Department of Hearing and Speech Sciences, Janet F. Werker
Vanderbilt University, Department of Psychology,
465 21st Avenue South, MRB III, Suite 7110, University of British Columbia,
Nashville, TN 37232–8548, US 2136 West Mall, Vancouver,
V6T 1Z4, Canada
Brigitte Röder
Biological Psychology and Gert Westermann
Neuropsychology, Department of Psychology,
University of Hamburg, Von-Melle-Park 11, Lancaster University, Fylde College,
D-20146 Hamburg, Germany Lancaster, LA1 4YF, UK
Chapter 1

The multisensory approach


to development
Andrew J. Bremner, David J. Lewkowicz, and
Charles Spence

1.1 Introduction
The question of how humans and other species come to be able to deal with multiple sources of
sensory information concerning the world around them is of great interest currently, but has also
has been on the minds of philosophers and psychologists alike for centuries (e.g. Berkeley 1709;
Locke 1690). Nonetheless, it has only been in the last two decades or so that we have witnessed a
dramatic progress in our understanding of how the brain actually integrates the information
available to the different senses in the mature adult (e.g. Calvert et al. 2004; Dodd and Campbell
1987; Lewkowicz and Lickliter 1994; Naumer and Kaiser 2010; Spence and Driver 2004; Stein and
Meredith 1993). This progress has resulted, in part, from the proliferation of new methodologies
(functional imaging, transcranial magnetic stimulation, and so on). Such methods have stimu-
lated a veritable explosion of knowledge about multisensory processes at various levels of neural
organization in humans and other species, in human behavioural performance, and in various
neurally-compromised clinical populations. We are now standing at a point where this growth in
the knowledge-base about mature multisensory functioning, and in the availability of newly-
developed experimental techniques, is now beginning to be applied to developmental questions
concerning the emergence of multisensory processes at a neural level (e.g. see Chapter 14 by
Wallace et al. for work on the development of the superior colliculus (SC) in the animal model),
and at a behavioural level in human infants and children.
However, there is, of course, a great tradition of thinking concerning the development of
multisensory processes, which began well before the advent of modern cognitive neuroscience.
The central question posed by this tradition (and indeed by this volume and all of its contribu-
tors) concerns how it is that we come to be able to process the information conveyed by our
multiple senses such that we can perceive the world (and ourselves), thereby enabling us to
function sensibly within it. As has been acknowledged by philosophers and early psychologists
alike, despite the fact that adults are able to achieve this feat in a seemingly effortless manner, we
cannot assume that the same is true for others who have had different degrees, or qualities, of
experience as compared to typically developed adults. From Molyneaux’s famous question to
Locke (as reported by Locke 1690) about whether a person who had been blind from birth could,
on the restoration of their vision, see and recognize an object (see Held et al. 2011, for the latest
evidence on this fascinating topic), to William James’s (1890, p. 488) assertion that the newborn
experiences a ‘blooming buzzing confusion’, the question of how multisensory development comes
about has been put forward time and again as one of seminal importance to developmental
psychology.
2 THE MULTISENSORY APPROACH TO DEVELOPMENT

Developmental scientists have by no means been idle when it comes to addressing topics
concerning multisensory development (see Lewkowicz and Lickliter 1994). While the literature
on multisensory integration in adult humans has only recently become a topic of substantive
theoretical interest within the fields of experimental psychology and cognitive neuroscience,
theory and research in the field of multisensory development has been a more constant theme in
developmental psychology over the last 50 years or so (Birch and Lefford 1963, 1967; Bryant 1974;
Bryant et al. 1972; Gibson 1969; Meltzoff and Borton 1979; Piaget 1952; for more recent approaches
see Chapter 8 by Bahrick and Lickliter and Chapter 7 by Lewkowicz, as well as Lickliter and
Bahrick 2000 and Lewkowicz 2000).
However, despite the considerable attention that has been devoted to the development of mul-
tisensory integration, we would argue that it still remains essentially a niche consideration among
developmental psychologists (see Fig. 1.1). Indeed, many of the most influential current theo-
retical accounts of development, and indeed empirical investigations, particularly of cognitive
abilities in infants and children, are still addressed within a theoretical framework that is largely
unimodal (e.g. see Baillargeon 2004; Johnson and de Haan 2010; Mareschal et al. 2007; Spelke
2007). This is not to claim that developmental psychology as a field is devoting too much atten-
tion to the study of individual sensory modalities (although it is quite clear that the focus of
research has clearly been on visual and auditory processing at the expense of other channels;
indeed, researchers have argued that some of the other senses are profoundly neglected both in
study and stimulation; e.g. Field 2003, 2010; Field et al. 2006; see Fig. 1.1), but rather that
little attention has been paid to the role of the multisensory nature of sensory experience in
mainstream accounts of development.
The provision of multiple sensory modalities bestows both important adaptive benefits, and
challenges for ontogenetic development. If we are to fully understand how development progresses
and the constraints placed upon it by the biological reality of our bodies and the neural apparatus
contained therein, the multisensory nature of our perceptual world across the course of develop-
ment will have to be characterized more fully. First, it is our contention that researchers too fre-
quently treat the sensory experience of their participants as being all of a piece, with little regard
to both the subtle and more obvious differences in the kinds of information provided by the dif-
ferent sensory channels. Different sensory channels complement each other by providing differ-
ent kinds of information (see Section 1.2 for a discussion of this matter). Understanding these
differences, and how individuals at various stages in the lifespan are able to benefit from these
various forms of information, is central to understanding the development of information
processing in humans.
However a more central consideration of the current volume concerns how it is that we come
to be able to use our senses in concert. There are both adaptive benefits and challenges to achiev-
ing this. By way of an illustration of the adaptive benefits, suppose that in a given instance, the
provision of several sensory inputs at one and the same time enhances an infant’s ability to make
a perceptual discrimination (e.g. imagine the situation in which a face, the sound of a voice,
the way someone touches us, and even the person’s distinctive smell, provide cues both to their
identity and gender; see Smith et al. 2007). If we consider an infant’s ability to recognize a person
from the point of view of just a single sensory input (as is frequently done in face recognition
research), then we may well be underplaying their social cognitive abilities at any given develop-
mental stage (on this theme, see also Gallace and Spence 2010).
But the problem is not as straightforward as it might at first seem; as researchers studying
multisensory development have been at pains to point out, the senses do not interact in a homo-
geneous way across the course of development (e.g. see Chapter 9 by Lewkowicz and Chapter 14
by Wallace et al.). So, if a given perceptual skill, which in adults relies on more than one sensory
INTRODUCTION 3

1000 Multisensory
Multisensory AND Development
900
Visual AND Development
Number of articles retrieved on MEDLINE

Auditory AND Development


800
Tactile AND Development
Olfactory AND Development
700
Executive function AND Development

600

500

400

300

200

100

0
1

1
96

96

97

97

98

98

99

99

00

00

01
–1

–1

–1

–1

–1

–1

–1

–1

–2

–2

–2
57

62

67

72

77

82

87

92

97

02

07
19

19

19

19

19

19

19

19

19

20

20
Fig. 1.1 The number of articles retrieved by MEDLINE for searches restricted to 5-year time-windows
from 1957–2011. In order to retrieve as many publications related to multisensory development as
possible, we searched for articles with the words ‘multisensory’, ‘multimodal’, ‘crossmodal’,
‘intermodal’, ‘intersensory’ in conjunction with the word ‘development’ (this is represented by the
line labelled ‘Multisensory AND Development’). Articles addressing multisensory development have
undergone a recent increase since 2001, more than double the number published in any 5-year
period before that year. Nonetheless, this number is still dwarfed by research on auditory and visual
perceptual development (despite the fact that we conducted more conservative searches for such
articles by searching for ‘Visual perception’ or ‘Auditory perception’ in conjunction with
‘Development’). Research on the development of visual and auditory perception underwent
something of a resurgence of interest in the early 1970s, likely due to the popularity of Piaget’s
sensorimotor theory of development and the advent of new methods for investigating infants’
and children’s abilities in vision and audition (Fantz 1961, 1964). Research addressing olfactory
and tactile development is particularly under-represented (when searching for studies on the
development of Tactile perception AND Development we also included the search terms
‘Haptic perception’ and ‘Tactual perception’. We provide the results of a search for ‘Executive
function’ AND ‘Development’ as a point of reference with a topic of current interest. Research
using the words ‘multisensory’, ‘multimodal’, ‘crossmodal’, ‘intermodal’, and/or ‘intersensory’
has increased exponentially over the last 50 years (5610 entries are retrieved between 2007
and 2011).

modality, requires a form of multisensory integration which the child has not yet attained, then a
unisensory experiment may well provide a misleading picture of the child’s competence.
In a recent discussion of cognitive development, Neuroconstructivism (Mareschal et al. 2007), a
persuasive argument was put forward for the necessity of considering the constraints that are
placed on development at all levels of functioning of the organism. Such levels of functioning
4 THE MULTISENSORY APPROACH TO DEVELOPMENT

include the organism’s genetic inheritance, its brain and cellular machinery, its body and physical
interactions with the environment, and, of course, the environment (which can itself influence
the organism at these various different levels of functioning). Whilst many approaches to devel-
opment have assumed that it is possible to understand ontogeny via changes in the interactions
occurring at one or two levels of functioning (e.g. changes in the organism’s behaviour in relation
to its environment), Mareschal et al. argued that developmental change can only be fully under-
stood if all the factors that influence change are taken into account; they particularly emphasize
the role of neural and more large-scale brain architecture and also the biomechanics of the body
in its environment.
We consider this kind of approach to be the ‘state-of-the-art’ in terms of developmental theory.
However, we also believe that Neuroconstructivism has a glaring omission; it does not consider in
detail the constraints placed on development of the multisensory apparatus we possess
(A.J. Bremner and Spence 2008; although see Chapter 15 by Mareschal et al. and Sirois et al.
2008). As will be made clear in the next section, the multiple sensory inputs that humans (and
other species) are bestowed with provide a rich variety of different kinds of sensory information
and these interact in complex and adaptive ways (See Fig. 1.2 for a classification of multisensory
interactions). If there is one overarching lesson from the recent explosion of research into multi-
sensory processes in neurologically-normal mature adults, it is that multisensory interactions are
the rule rather than the exception. However, these interactions are constrained by the nature of
our multisensory apparatus and the way in which that apparatus is embedded in our bodies and
nervous systems. Importantly, they are also constrained by unfolding changes in the ways in
which our multisensory abilities develop (Lewkowicz 2011a; Occelli et al. 2011; Turkewitz 1994;
Turkewitz and Kenny 1982). We would thus like to argue here that a full picture of perceptual,
cognitive, and social development will only emerge once diverse ecological multisensory
constraints are considered and studied.

Separate components Multisensory composite signal


Signal Percept Signal Percept Category

a a+b Equivalence
(intensity unchanged)
Redundancy
b a+b Enhancement
(intensity increased)

a+b and Independence


a a+b
Non-redundancy Dominance
b a+b (or ) Modulation

a+b Emergence

Single signal a a and Synaesthesia

Fig. 1.2 A classification of multisensory interactions (from Science, 283, Sarah Partan and Peter Marler,
Communication Goes Multimodal, pp. 1272–1273 © February 1999, The American Association for
the Advancement of Science. Reprinted with permission from AAAS). The distinction between
redundancy and non-redundancy is often compared to the distinction drawn between amodal and
arbitrary correspondences in the developmental literature (see, e.g. Walker-Andrews 1994). Note,
however, that there is some variance in the literature with regard to what exactly constitutes an
amodal relation (see Spence 2011a for discussion). As there has been significant interest in
multisensory synaesthetic percepts since the publication of Partan and Marler’s (1999) paper (e.g. see
Chapter 10 by Maurer et al.), we have included an additional section on their figure in parentheses.
HOW DO MULTISENSORY PROCESSES CONSTRAIN AND ENRICH DEVELOPMENT? 5

1.2 How do multisensory processes constrain and enrich


development?
Despite the fact that there is still quite some uncertainty over how many senses we possess, and
indeed how we should define sensory modalities in the first place (e.g. Durie 2005; Macpherson
2011), we can safely state that we perceive our environment through multiple sensory systems.
Vision, touch, hearing, taste, smell, proprioception, whilst being most frequently considered, are
just some of the modalities available to humans. Humans have likely evolved the organs necessary
for these multiple channels of information processing for a wide variety of adaptive functions.
If we do not attempt to understand such multisensory benefits in developing individuals we will
likely misrepresent the development of the skills we are interested in (which will typically be mul-
tisensory in our everyday ecological settings). But it is critical to bear in mind that the provision
of these multiple channels also poses significant problems for the developing individual. With all
of this information arriving in different representational formats and neural codes, how does the
child or young animal make sense of the environment from which the information originates? In
this section, as an introduction to the matters considered in this volume, we outline both the
adaptive significance and the challenges posed by the provision of multiple different sensory
modalities to developing humans.

1.2.1 Developmental benefits of multiple senses


The provision of multiple sensory modalities gives us complementary sources of information
about the environment (E.J. Gibson 1969; J.J. Gibson 1966; 1979). The sensory modalities
have likely evolved to make use of the constraints of the physical environment around us.
So, for example, vision is particularly good at transducing spatial information about near and far
space from the visible light spectrum. This helps us to recognize objects, people, and spatial
arrays. In contrast, audition is particularly good at encoding rapid temporal patterns of informa-
tion arising from mechanical disturbances in both near and far space (Bregman 1990), including
from places which are not in a direct line of sight to the body (i.e. for sound sources located
behind the head, or obscured by an occluder), and in darkness. Information from the somatosen-
sory channels (including information about touch from cutaneous receptors and about the
arrangement and movement of our limbs from proprioceptors) helps us to perceive our body and
the environment that is in direct contact with our body. The chemoreceptors (including gusta-
tory and olfactory receptors) provide us with information about chemicals impinging on the
membranes in which they are located, telling us about everything from nutrients and poisons to
social signals in the environment. Such chemicals may arise from objects that happen to be in
contact with our various sensory epithelia, while others may have arisen from more distant
objects (in the case of orthonasal olfaction).
These channels of information are not (always) redundant. ‘Direct’ receptors like taste and
touch do not tell us about the distal environment except in some special situations (e.g. when the
sun’s rays heat one’s skin). Likewise, we cannot hear or see most of the chemicals that make their
way onto our olfactory epithelia. It seems safe to conclude then that we are able to glean more
information about the world by sampling from multiple modalities than we could from just a
single sensory modality. That said, the functional onset and development of the various senses
follow quite different ontogenetic trajectories, with the earliest to begin functioning being touch,
followed by the chemoreceptors, the vestibular sense, and then audition and finally vision (see
Fig. 1.3 and Table 1.1. for a detailed timetable of the developmental emergence of the senses; also
Gottlieb 1971). Not only do the senses develop at different rates, but there are remarkable idio-
syncrasies in the ways in which unisensory abilities develop within them. Indeed, as will be
explained later, some researchers have suggested that the heterochronous emergence of multisensory
6 THE MULTISENSORY APPROACH TO DEVELOPMENT

0 4 8 12 16 20 24 28 32 36 38 weeks

Trigeminal anatomy (touch/chemoreception)

Cutaneous touch anatomy

Olfactory and taste anatomy

Vestibular anatomy

Auditory anatomy

Visual anatomy

Trigeminal function (touch/chemoreception)

Cutaneous tactile function

Olfactory and taste function

Vestibular function

Auditory function

Visual function

Birth
Fig. 1.3 The emergence of the anatomy and function of the multiple sensory systems in humans during
gestation (Reproduced from Moore, K.L., and Persaud, T.V.N., The developing human: Clinically oriented
embryology, 8th Ed. Saunders Elsevier: Philadelphia, PA © Elsevier 2008 with permission). Pink bars indicate
the emergence and maturation of the senses (usually provided by histological evidence). Shading of the
pink bars indicates the time between the first observation of sensory apparatus and its full anatomical
maturation (as gleaned from the incomplete evidence summarized in Table 1.1 and Gottlieb 1971). Green
bars indicate the onset of function of the senses. Shading of the green bars indicates the uncertainty in the
literature concerning the first age at which function is observable (as gleaned from the incomplete evidence
summarized in Table 1.1 and Gottlieb 1971). Arrows pointing beyond birth indicate the continued postnatal
development of anatomy and function. Visual receptors continue to mature beyond birth, possibly due to
the relatively low level of stimulation they have received in utero. Both Gottlieb (1971) and Lickliter (Lickliter
and Bahrick 2000; Lickliter and Banker 1994) have noted that the senses emerge in a strikingly similar order
across vertebrate species (but vary with respect to the onset of functionality). It has been suggested that this
ordering and timing of functionality plays an important epigenetic role in the development of information
processing (see also Turkewitz and Kenny 1982). Reproduced in colour in the colour plate section.
HOW DO MULTISENSORY PROCESSES CONSTRAIN AND ENRICH DEVELOPMENT? 7

Table 1.1 The emergence of anatomy and function of our multiple sensory systems during
gestation (adapted from Gottlieb 1971). Here, we present some of the key pieces of evidence
for prenatal development of the senses. Anatomical development is determined typically from
histological evidence, whereas functional development comes from behavioural and physiological
evidence. Functional development occasionally occurs in advance of anatomical maturation; this is
likely due to partial functionality before full maturation has occurred. Note that the trigeminal nasal
system contributes to both touch and chemosensation as chemicals give rise to tactile stimulation
registered by trigeminal receptors embedded in the olfactory epithelium. The table contains a
number of omissions. Notably proprioception is not mentioned, since it is difficult to determine
functionality in utero (although there are indications of well-developed proprioceptive functioning
at birth; e.g. van der Meer 1997). Similarly, the various subdivisions of the chemosensory receptors
and cutaneous receptors are not discussed either for the same reason.

Sensory channel Anatomical development Functional development


Touch 4–7 weeks’ gestation: Maturation 7 weeks’ gestation: foetus moves if the lips are
of trigeminal and cutaneous touched (Hooker 1952; Humphrey 1964).
receptors (Humphrey 1964).
12 weeks’ gestation: grasp and rooting reflex
responses (Moon and Fifer 2011; Humphrey
1964).
Chemosensation 4 weeks’ gestation: trigeminal 7 weeks’ gestation: foetus responds to touch
neurons mature (Chapter 2 by on trigeminal receptors.
Schaal and Durand).
It is difficult to determine functionality of
11 weeks’ gestation: olfactory chemoreceptors in utero, but it is likely that
receptor neurons mature (Chapter they are functional by at least late gestation
2 by Schaal and Durand). (LeCanuet and Schaal 1996; Schaal et al. 2004;
Chapter 2 by Schaal and Durand).
12–13 weeks’ gestation: taste
buds mature (Beidler 1961).
Vestibular system 10–14 weeks’ gestation: sensory 11–25 weeks’ gestation: foetus will show a
cells are present in semicircular ‘righting’ reflex (see Gottlieb 1971).
canals (Moon and Fifer 2011).
Birth: newborns show ocular nystagmus in
12–16 weeks’ gestation: Lateral response to rotation (Galebsky 1927).
vestibular nucleus present (Gottlieb
1971).
Auditory system 9 weeks’ gestation: the cochlea 24 weeks’ gestation: foetus begins to respond
forms. to sounds (Abrams et al. 1995; Bernard and
Sontag 1947).
24 weeks’ gestation: inner ear has
reached its adult size and shape; 21 weeks’ gestation: auditory cortical evoked
organ of Corti is fully mature responses recorded in an infant born
(Bredberg 1968; Moore and prematurely (Weitzman and Graziani 1968).
Persaud 2008).
Visual system 5 weeks’ gestation: eyes begin to 22–28 weeks’ gestation: earliest age at which
form. Primordial neural retina first premature infants have shown evoked cortical
appears at about 7 weeks, but the responses to light flashes. Behavioural responses
retina is not completely mature such as tracking have also been measured
until around 4 months after birth around this age (Ellingson 1960; Engel 1964;
(Mann 1964; Moore and Persaud Moon and Fifer 2011; Taylor et al. 1987).
2008). Myelination of retinal
ganglion cells is complete at about
3 months (Moore and Persaud
2008).
(Continued)
8 THE MULTISENSORY APPROACH TO DEVELOPMENT

Table 1.1 (continued ) The emergence of anatomy and function of our multiple sensory systems
during gestation

Sensory channel Anatomical development Functional development


Visual system 28 weeks’ gestation: Laminar Postnatal 3–4 months: Braddick and Atkinson
structure appears in striate cortex, (2011) argue that early visual evoked potentials
but adult-like appearance and (VEPs) indicate that activity reaches cortex, but
connections continue to develop not that cortical neurons are responding to
postnatally (see Burkhalter et al. that. VEPs that respond to binocular constrasts
1993). are their preferred marker and appear
postnatally at 3–4 months. Visual acuity is
thought to reach adult levels soon afterwards
(Boothe et al. 1985).

functioning plays an important role in development (Chapter 16 by Ghazanfar; Chapter 2 by


Schaal and Durand; Lewkowicz 2002; Lewkowicz and Ghazanfar 2009; Lickliter and Bahrick
2000; Turkewitz and Kenny 1982). The more we understand about the trajectory of development
of all of the senses, the more we will get a complete picture of the development of our ability to
make sense of the whole array of environmental information to which we are exposed.
However, to suggest a model in which information gained is related in a very simple
linear manner to the number of channels available would be misleading. The senses have an impact
well beyond their own immediate provision of information. By way of an example consider the
sense of touch. Touch is most typically stimulated by objects in direct contact with the body surface.
This is not a unique provision as we can also, in many cases, see when something is touching us.
Importantly however, this redundancy may have important implications beyond the sense of touch
as it can disambiguate which cues from our distal spatial senses (e.g. vision and audition) specify
aspects of our environment that are within reach. Touch can, as such, ‘tutor’ vision to perceive
depth, or at least the environment that affords immediate action (Berkeley 1709; Gori et al. 2010a;
Gregory 1967; Held et al. 2011). Aside from this classic example, there are numerous ways in which
the senses can inform one another across development in this fashion. It is, for instance, well-known
that when humans have bad (and good) experiences with taste and flavour, that this can help us to
learn which visually perceived foodstuffs are safe to eat (e.g. Bernstein 1978).
Perhaps a more precise way of expressing this particular advantage of multiple redundant sen-
sory inputs is as a facility to use one sensory channel in order to improve the performance of
another in situations when is the latter is used on its own (this is known as ‘crossmodal calibra-
tion’). For some time now, researchers have suggested that the senses are not equal in their ability
to provide accurate information about the environment (e.g. Freides 1974). In particular,
Welch and Warren (1980, 1986) have argued that the superiority of vision in terms of providing
information about the spatial environment can account for the numerous demonstrations that
vision tends to dominate over other modalities in guiding our judgments and responses with
respect to both objects and spatial layouts (e.g. Gibson 1943; Lee 1980; Botvinick and Cohen
1998; Spence 2011b; although see Spence et al., 2001). The fact that the senses are not equal in this
regard can also provide a means of improving performance within the less accurate modality; that
is, the more accurate sense can be used to calibrate, and thereby improve the sensitivity of, the less
accurate sense. This kind of process of calibration has been used to explain developmental
improvements in proprioception in the absence of vision (Chapter 6 by Nardini and Cowie;
Chapter 12 by Hill et al.; Lee and Aronson 1974). It can also explain the poorer balance and
HOW DO MULTISENSORY PROCESSES CONSTRAIN AND ENRICH DEVELOPMENT? 9

haptic judgment of orientation that has been observed in blind participants (Edwards 1946;
Gori et al. 2010b)1.
Redundant multisensory information can also improve the accuracy with which we can make per-
ceptual judgements when more than one sense is available concurrently. As has been extensively dis-
cussed in the recent literature on multisensory integration, mature adults, rather than being dominated
by information from the most reliable modality (Welch and Warren 1980), actually appear to integrate
the information from several available modalities according to their relative reliabilities in the context
of the task at hand (see Ernst and Bülthoff 2004; Trommershäuser et al. 2011). Typically, the senses
appear to be weighted optimally in accordance with the maximum likelihood estimation model (Alais
and Burr 2004; Ernst and Banks 2002; although see Brayanov and Smith 2010 for a recent exception).
Interestingly, as discussed particularly in Chapter 6 by Nardini and Cowie (see also recent work by
Gori et al. 2008, 2010b; Nardini et al. 2008; Nava et al. in press), this is not the case at all stages of per-
ceptual development, with young children apparently weighting the senses in a sub-optimal fashion.
The integration of sensory stimuli across multiple senses has also been shown to speed responses
to it (e.g. Hershenson 1963; Miller 1991; Roland et al. 2007; see also Spence et al. 2004). This
multisensory speeding has also been shown to develop at various stages across early development,
and to continue changing into old age (Chapter 11 by Laurienti and Hugenschmidt; Chapter 9 by
Lewkowicz; Chapter 6 by Nardini and Cowie; Barutchu et al. 2009, 2011; Neil et al. 2006). Clearly,
the development of this kind of multisensory integration, which can lead to more accurate and
faster responses to the stimuli/events in environment, will have important implications for the
development of ecological perception and cognition.
The senses are also combined in particular ways by virtue of the action system in which they are
situated. Going back to the example of touch, two forms of tactile sensation that are frequently
discussed in the literature are passive and active touch (with active touch sometimes being referred
to as ‘haptics’; see Chapter 4 by Streri for a discussion of haptics). Whilst passive touch can, in
some sense, be considered as a purer form of sensory transduction in that it relies solely on pres-
entation of stimuli to cutaneous receptors, haptic exploration of an object involves the combina-
tion of cutaneous sensory inputs with other sensory inputs; specifically, proprioceptive information
concerning the arrangement of our limbs (as the hand is the primary haptic organ, our finger
joints are particularly important here), and efferent motor signals concerning our movement.
This multisensory combination enhances our ability to identify objects through active tactile
exploration (see Klatzky and Lederman 1993; Klatzky et al. 1987).
Up until now, we have discussed how multisensory processes can have an impact at any given
stage in development, and how differences in multisensory processes between these stages have an
important bearing on ecological perception across the lifespan. However, it is particularly impor-
tant to consider how multiple senses can play a role in driving or constraining development from
one stage to the next. One particularly influential recent account of multisensory development
has been offered by the ‘intersensory redundancy hypothesis’ (IRH) proposed by Bahrick and
Lickliter (2000, see also Chapter 8 by Bahrick and Lickliter). According to these researchers,
infants selectively attend to redundant information across the senses and this then enables them

1 Actually, various accounts of developmental crossmodal calibration have been put forward to date. Whilst
some researchers have argued that the principle role of crossmodal calibration is to improve accuracy
(sensitivity) in the calibrated sense (e.g. Lee 1980), Burr and colleagues (Burr et al. in press) have, more
recently, presented a somewhat different perspective. They suggest that crossmodal calibration serves the
purpose of tuning sensory inputs to environmental information to which they do not have ‘direct’ access
(as in the case of actual size perceived via the visual modality, which requires more computational
transformations than does actual size perceived through haptics; see Gori et al. 2008; 2010b).
10 THE MULTISENSORY APPROACH TO DEVELOPMENT

to learn about the cues in the environment that specify unified objects and events. This account
shows how an interaction between the multisensory infant and his/her environment can give rise
to a simple orienting preference, which can elegantly explain a number of the perceptual develop-
ments typically observed in early life (Chapter 8 by Bahrick and Lickliter; see also Lewkowicz
2011b).
Another argument that multiple senses ‘scaffold’ development comes from Piaget, who in this
case (and more widely) was more concerned with the development of knowledge about the envi-
ronment. In fact, the interaction of multiple sensory modalities was a key aspect of Piaget’s con-
structionism. He argued that the initial steps toward objectivity in infants’ representations of their
world were achieved, in part, through the integration of separate, modality-specific schemas
(what he termed the ‘organisation of reciprocal assimilation’, Piaget 1952). Piaget describes a
number of observations in which his young children (at around 4 months of age) first noticed
that an object could be apprehended in more than one modality (or sensory schema) at once.
Piaget goes on to describe this as a new kind of schema in which infants can, for instance, grasp
what they see, and see what they grasp: a reciprocal relationship. Because of this organization, an
object is no longer a thing of looking or a thing of grasping, but an entity existing in a more objec-
tive representation. There are, of course, problems with the view that infants require visually
guided reaching in order to develop with typical intelligence, as there are many instances of blind
children or children without limbs who manage to do just this (e.g. Gouin Décarie 1969).2
Nevertheless, regardless of the specific importance of visually-guided reaching in development,
Piaget’s conception of a reciprocal schema shows us how two initially modality-specific represen-
tations of environmental stimulation can be enriched almost fortuitously by virtue of the spatial
coincidence of the modalities that occurs when an infant, for example, picks up an object.
This gives us another example of an enhancement of development by the provision of multiple
senses.

1.2.2 Developmental challenges of multiple senses


The ease with which we typically accomplish multisensory integration belies the complexity of the
computational processes involved. The senses convey information about the environment using
different neural codes. Spatial codes vary substantially from one spatial channel to the next. For
instance, vision is initially coded in terms of a retinocentric spatial frame of reference, auditory
stimuli in terms of head-centred coordinates, touch in terms of the location of stimulation on the
skin-surface, and proprioception in terms of limb-centred coordinates (Spence and Driver 2004).
This clearly poses a problem for simultaneously locating a stimulus or object within more than
one sense. Do we relate everything to a single sensory coordinate system (e.g. a visual frame of
reference), or do we re-code it all into a multisensory, receptor-independent, coordinate system
(see Avillac et al. 2005; Ma and Pouget 2008)? How do we translate information between sensory
frames of reference? For some consideration of how infants and young children come to be able
to solve these kinds of problems see Chapter 5 by Bremner et al. and Chapter 13 by Röder.
Temporal codes also vary between the senses, causing computational problems in terms of the
perception of synchrony (King 2005; Mauk and Buonomano 2004; Meck 1991; Petrini et al.
2009a, 2009b; Spence and Squire 2003).

2 In fact, there is reason to believe that Piaget considered visually-guided reaching to be merely one example
of how the organization of reciprocal assimilation could occur, and that he viewed evidence concerning
infants without reaching or vision to be irrelevant to this matter (Gouin Décarie and Ricard 1996).
WHERE ARE WE NOW? 11

Despite the multiplicity of the spatial and temporal codes in which information is provided by
the senses, in some ways, once one has settled on the frame of reference which one is going to use,
this problem constitutes a relatively easy hurdle to surmount. If the infant (or foetus) can learn
the relationships between the various sensory spatial frames of reference and temporal codes
(perhaps through the common occurrence of stimuli arising from specific objects or spatial lay-
outs), then the problem is solved (although of course this ignores a sticky issue: when there are
multiple stimuli in each modality, how does one know which stimulus goes with which? This is
known as ‘the crossmodal correspondence problem’ (Spence et al. 2010a; also see Lewkowicz
2011b, for a developmental solution)). However, at least for spatial binding, the problem is much
more difficult than this because the spatial relationship between the sensory frames of reference
frequently changes when, for example, the body changes posture (e.g. when the eyes move in their
sockets, or when the hands move relative to the body; see Pöppel 1973; Spence and Driver 2004).
This difficulty is compounded across development, as, especially in the early years but continuing
right across the lifespan, the relative shapes and sizes of the body, limbs, and head change, as do
the number and variety of postural changes that an individual can readily make.3 These compu-
tational problems prompt an important question which represents the key focus of the present
volume: how is it that we develop the ability to integrate the senses? As many of the chapters in
this volume propose answers to these questions we will not pre-empt them here, apart from
noting that researchers have provided a wide variety of solutions to these problems (see Chapter 8
by Bahrick and Lickliter, Chapter 16 by Ghazanfar, Chapter 7 by Lewkowicz, Chapter 10 by
Maurer et al., and Chapter 14 by Wallace et al.).
Interestingly, some theorists have made a virtue of certain of the challenges posed by multisen-
sory integration to ontogenetic development. For instance, Turkewitz and his colleagues
(Turkewitz 1994; Turkewitz and Kenny 1982; Kenny and Turkewitz 1986) have both argued, and
presented evidence to support the claim, that the particular heterogeneous way in which the
function of the different sensory systems emerge prenatally facilitates perceptual development.
The suggestion is that heterogeneous emergence of multisensory function reduces the amount of
sensory information the developing foetus has to assimilate at early stages in development,
and reduces competition between the separate sensory systems facilitating their functional
development.

1.3 Where are we now?


The question of how multisensory development occurs has been approached in several ways, but
perhaps the clearest delineation has been between theorists who suggest that the senses become
integrated across development and those who suggest that the senses become differentiated across
development. The integration account is perhaps best exemplified by Piaget who, as we have just
explained, considered the senses and their associated action schemas to be separate at birth. The
gradual integration of sensory schemas during the first two years of life was a hallmark of Piaget’s
sensorimotor period of development. On the other hand, following William James’s (1890) asser-
tion that newborn infants perceive a ‘blooming buzzing confusion’ in which the senses are united
and undifferentiated, Eleanor Gibson (1969) proposed that the senses are initially fused, resulting
in unified perceptual experiences. In contrast with James’s position however, Gibson’s argument
did not assume that young infants are confused by this lack of differentiation, but instead sug-
gested that such a unity of the senses allows young infants to more easily pick-up information

3 Developments in the ability to move the body in certain ways are also likely to have an important impact
on senses which are active by nature such as haptic tactile exploration (see Section 1.2.1).
12 THE MULTISENSORY APPROACH TO DEVELOPMENT

from the structure presumably available in the environment. In direct contrast with the Piagetian
position, Gibson argued that as development progresses and as perceptual learning and differen-
tiation permit the gradual pick-up of increasingly finer stimulus structure, infants become capa-
ble of differentiating the senses one from another.
One traditional way to assess the relative merits of these early and opposing theoretical views
was to test whether newborn infants can perceive links between stimulation in different sensory
modalities. At the last major overview of multisensory development in the mid-1990s (Lewkowicz
and Lickliter 1994), an opening chapter by Linda Smith stated that this question had been com-
prehensively answered in the affirmative; infants perceive in a multisensory way from the outset.
Such a claim provided strong support for a differentiation account. However, one of the principle
concerns of Lewkowicz and Lickliter’s volume was to delineate the kinds of multisensory skills
that infants and children have to master. One particular distinction was drawn between amodal
and arbitrary crossmodal correspondences (see also Parise and Spence in press; Spence 2011a).
Amodal correspondences were described as those in which information about the world is sup-
plied redundantly across the senses (e.g. intensity, duration, tempo, rhythm, shape, texture, and
spatial location), whereas arbitrary correspondences are those that are naturally linked in the
environment but which provide different information according to the modality in which they
are delivered (such as that between the colour of a person’s hair and the sound of their voice).
Research has indicated that infants are rather good at detecting amodal information from early on
in life (e.g. Chapter 7 by Lewkowicz; Lewkowicz and Turkewitz 1980; Lewkowicz et al. 2010) but
develop more slowly in their ability to learn about naturally-occurring arbitrary crossmodal
correspondences (Bahrick 1994; Hernandez-Reif and Bahrick 2001).
More recently, the IRH has provided a useful framework with which to explain this develop-
mental precedence of amodal correspondences and at the same time explain how infants learn
about the multisensory environment (see our earlier discussion of this hypothesis). The IRH pro-
poses that infants’ attention is captured early on by amodal correspondences, requiring little or no
integration, and that infants’ attention to these amodal properties of the environment bootstraps
their subsequent ability to perceive arbitrary crossmodal correspondences. In that sense, then, the
IRH represents a reconciliation between Piagetian integration and Gibsonian differentiation.
Since the publication of Lewkowicz and Lickliter’s (1994) seminal volume, there have been
numerous advances in terms of the empirical and theoretical contributions to our understanding
of multisensory development (see Fig. 1.1). At the same time, however, we have also witnessed
a huge increase in research into multisensory processes in mature adults across a range of
disciplines (Stein et al. 2010). This research has pointed to the pervasiveness of multisensory
interactions both at the behavioural (e.g. Spence and Driver 2004) and at the neural levels (e.g.
Calvert et al. 2004; Driver and Noesselt 2008; Ghazanfar and Schroeder 2006; Murray and Wallace
2011; Stein and Meredith 1993; Stein and Stanford 2008). This new knowledge about adult mul-
tisensory functioning from the domain of cognitive neuroscience is now beginning to have an
impact on the field of multisensory development and we hope that the current volume reflects
this fact. Below, we highlight some of the issues that we think are of key current interest.

1.4 Important themes in this book


A number of volumes have been published on the subject of multisensory perception. Typically
these have dealt with multisensory processes from a broad and multimethodological perspective,
but are largely dominated by research into mature multisensory processes in adults (e.g. Calvert
et al. 2004; Spence and Driver 2004). The themes emphasised in the research on mature adults at
the time focused around the problem of spatial and temporal integration across multiple sensory
IMPORTANT THEMES IN THIS BOOK 13

modalities. These themes echo the priorities of a particular model of multisensory integration put
forward by Stein and Meredith (1993), which dealt specifically with the neurophysiological
aspects of multisensory integration within the SC.
More recently, we have witnessed a shift, at least in the adult behavioural literature, towards a
consideration of other factors that modulate multisensory integration such as, for example,
semantic congruency (Doehrmann and Naumer 2008; Hein et al. 2007; Chen and Spence 2010,
2011; Laurienti et al. 2004; Naumer and Kaiser 2010) and synaesthetic congruency (otherwise
known as crossmodal correspondence; see Spence 2011a; Parise and Spence in press). This move
has been motivated, in part, on the basis of the observation that spatial coincidence does not
appear to play anything like as important a role in multisensory integration when it comes to
stimulus identification as opposed to stimulus detection/localization (see Spence in press). There
is also a move towards a consideration of the development of multisensory processes, as it is
increasingly acknowledged that developmental findings are important in validating models of
multisensory functioning in mature adults (e.g. Lewkowicz and Ghazanfar 2009; Spence and
A.J. Bremner 2011). Indeed, some of the new topics of research in the adult literature are particu-
larly well-suited to developmental investigation (especially the role of semantic knowledge and
congruency in multisensory integration).
As should be clear from the title, the focus of the current volume is an examination of how
multisensory perception develops. The closest book in terms of this emphasis is the seminal
edited volume by Lewkowicz and Lickliter (1994). In choosing the chapter coverage offered in the
current volume we have attempted to strike a balance between updating the progress with regard
to the questions that book addressed in 1994, and coverage of new and emerging topics in
multisensory development. Below, we describe some of the particular emphases of the current
volume.

1.4.1 Chemosensory and gustatory information processing:


odours, tastes, textures, and flavours
The chapters in this volume by Schaal and Durand, and Spence address, respectively, the litera-
tures on the development of olfaction and gustation and multisensory flavour perception more
generally (i.e. including vision). Chapter 2 by Schaal and Durand reports on the emergence of
multisensory interactions with olfaction, from the womb through infancy to childhood.
Chapter 3 by Spence focuses on crossmodal interactions between vision and the traditional
flavour senses.
Chemosensory and gustatory information processing is a relatively new area for developmental
research. In fact, the development of the chemical senses was not an issue that was covered in
separate chapters in Lewkowicz and Lickliter’s (1994) earlier volume on multisensory develop-
ment. Indeed, this comes as little surprise given that the chemical senses have typically not received
much interest from developmental psychologists. That said, the reasons for this neglect are
unclear. Is it perhaps due to an implicit assumption that these are ‘lower’ senses, which are in
some way less worthy of study? Likely more relevant here is the difficulty associated with stimulat-
ing the taste buds or olfactory epithelium in a controlled manner (in adults let alone in infants).
Notwithstanding the above, there is a relatively large body of research tracking changes in chemi-
cal sensitivity from birth (and before, see Chapter 2 by Schaal and Durand). However, much of
this research has appeared in the food science and nutrition journals (e.g. journals such as Journal
of Sensory Studies, Chemical Senses, Food Quality, and Preference and Appetite), which are not
typically read by developmental psychologists. Furthermore, much of the research has tended to
investigate development within a unisensory framework (e.g. asking how salt sensitivity changes
over the first few years of life). To date, there has been much less multisensory research concerning
14 THE MULTISENSORY APPROACH TO DEVELOPMENT

the interaction between the chemical senses and vision, audition, and somatosensory channels
(although we must acknowledge that there have been numerous considerations of interactions
between taste, retronasal olfaction, and trigeminal stimulation; i.e. those sensory inputs that
together contribute to the classical definition of flavour; see ISO 1992, 2008; Lundström et al.
2011; Spence et al. 2010b). There is certainly less in terms of a theoretical framework for under-
standing the development of multisensory integration of the chemical senses. We predict that,
given the difficulty of stimulating the chemical senses, that models of multisensory integration
developed on the basis of the study of interactions between audition, vision, and touch will have
an important impact on progress in understanding the chemical senses.
Despite the dearth of prior research, interest in the chemical senses within the more main-
stream developmental psychology literature is unquestionably on the increase (see Fig. 1.1).
As food choice and preferences become an increasingly important social issue, researchers are
beginning to investigate how multisensory processes are involved in the development of these
kinds of behaviours. Spence highlights the importance of expectancy and cultural variation on
multisensory interactions (the latter is an issue that multisensory research has yet to tackle with
any vigour; see also Howes 1991; 2006). Furthermore, as emphasized in Chapter 2 by Schaal and
Durand, the chemical senses are among the most useful for early development, functioning in the
womb well before birth. As these authors elegantly describe, the chemical senses may play a vital
role in setting the sensory canvas for the development of the other senses that become functional
later (Turkewitz 1994; see Table 1.1).

1.4.2 Embodiment
Multisensory representations of the body and of the world from an embodied perspective, play a
fundamental role in all of our mental processes by providing a point of reference to objects in the
external world (this is what is known as ‘embodied cognition’; Bermúdez et al. 1995; Gallagher
2005; Varela et al. 1991). At an even more basic level, multisensory perception of the body is
required if we are to manipulate and move around our environments in a physically competent
and functional manner.
Despite the above, developmental psychology as a field has been rather underwhelming in its
attempt to understand the development of perceptual and cognitive processes from an embodied
perspective. This is perhaps due to the successful proliferation of looking-time techniques, which
have themselves brought a particular bias in the knowledge that has been gathered about early
cognitive development. Because it is difficult to record infants’ looking behaviours whilst they are
manipulating objects, the methods we have available have typically been used to present stimuli
via more indirect modalities (i.e. vision and audition). These limitations have also been inherited
by the more recently developed functional imaging measures of early cognition (see Johnson and
de Haan 2010 for an up-to-date review of neuroscientific studies of perceptual and cognitive
development). In order to better understand perceptual development from an embodied per-
spective we badly need more research that investigates the development of multisensory abilities
that bridge between vision, audition, and the more body-specific sensory channels: touch and
proprioception. As can be seen from Fig. 1.1, research on the development of tactile perception is
hugely underrepresented compared to that on visual and auditory development. Nonetheless, a
number of recent advances have been made (see, e.g., A.J. Bremner et al. 2008; J.G. Bremner et al.
in press; Nardini et al. 2008; Gori et al. 2008; Zmyj et al. 2011)
In this volume two chapters specifically examine the development of infants’ and children’s
sensitivity to, and perception of, tactile stimuli. Streri (Chapter 4) reports the most up-to-date
perspective on her long-standing programme of research into the perception of crossmodal
IMPORTANT THEMES IN THIS BOOK 15

links across vision and haptics in very young infants (some of them newborns). Meanwhile,
Bremner et al. (Chapter 5) describe recent research into tactile orienting responses in infants
and the perception of the position of their limbs. Chapter 6 by Nardini and Cowie covers our
understanding of the development of multisensory processes underlying the employment of the
body in balancing, orienting behaviours, and more large-scale sensorimotor abilities including
locomotion and spatial navigation.
As pointed out by Mareschal and his colleagues (2007), the body plays an important role in
constraining the ways in which environment and inheritance interact to give rise to developmen-
tal change. The way in which the body influences multisensory processes is clearly no exception
to this rule (see above). Without a concrete understanding of the way in which the body struc-
tures multisensory experience at various stages of human development, it will be difficult for us
to understand the emergence of perceptual abilities. A particularly nice example of this is pro-
vided by Bahrick and Lickliter (Chapter 8). They describe developmental robotics research that
suggests that ‘infant-like’ movements can help structure the multisensory canvas in such a way as
to enhance the availability of multisensory cues such as amodal synchrony (e.g. Arsenio and
Fitzpatrick 2005).

1.4.3 Neuroscience
Related to the issue of how we integrate inputs from the sensory apparatuses distributed across
our dynamic body, is how the structural and functional organisation of the brain constrains mul-
tisensory development. The sensory apparatus we possess feeds into the central nervous system at
different points of entry (via the primary cortices). The structural and functional constraints of
the brain on our ability to integrate these inputs together at any given stage in development is, of
course, critical to our understanding of the way in which the developing infant, child, or even
adult can process multisensory information (Johnson 2011; Mareschal et al. 2007). Whilst behav-
ioural studies can, to a certain extent, tell us about the neural processes that underlie behaviours,
research is increasingly indicating that behaviour and associated neural processing do not appear
at the same time in development (e.g. see Elsabbagh and Johnson 2010; Halit et al. 2003). Such
findings highlight the possibility that perceptual abilities develop in advance of the behaviours
which developmentalists have typically used to identify them, and thus provide an additional
reason to consider a neuroscientific approach to understanding multisensory development.
The great bulk of what we now know about the neural basis of multisensory development
comes from research on the development of the multisensory responsiveness of cells in the SC
from recordings in cats and monkeys (Stein et al. 1994; Stein and Stanford 2008; Wallace 2004).
Two chapters in the current volume tackle the development of neural processes linked to the SC
model of multisensory integration. Wallace et al. (Chapter 14) describe research in animal prepa-
rations on the development of and the role of sensory experience in the development of the SC
model of integration, while Laurienti and Hugenschmidt (Chapter 11) examine changes in mul-
tisensory integration during aging, with specific reference to marker tasks of SC function (e.g.
tasks in which participants’ responses to bimodal stimuli are enhanced in terms of their speed or
accuracy over the same responses to unimodal stimuli are typically taken as providing a behav-
ioural marker for multisensory integration in SC; see Spence et al. 2004).
The SC animal model has contributed a great deal to our understanding of multisensory devel-
opment; a particularly important contribution has been to enable the assessment of the role of
specific kinds of sensory experience on developmental change in the neural functions of multi-
sensory processes (Chapter 14 by Wallace et al.; Yu et al. 2010). Nonetheless, it has been rather
difficult to make the leap from these neurophysiological studies in animals to the development of
16 THE MULTISENSORY APPROACH TO DEVELOPMENT

multisensory processing in human infants and children. Primarily, this has been because studies
of SC function in cats and monkeys indicate that the kinds of experiences required to foster adult-
like responsiveness in SC neurons occur early in life and have their impact quickly (although see
Yu et al. 2010). Such findings are rather at odds with recent evidence, which have demonstrated
that changes in multisensory integration in human behaviour continue well into childhood and
early adolescence (Barutchu et al. 2009, 2011; Gori et al. 2008, 2010b; Hillock et al. 2010; Nardini
et al. 2008; Ross et al. 2011). They are also difficult to reconcile with the knowledge that sub-
cortical structures such as the SC are largely mature before birth in humans.
Thus, an important direction for future research will be to understand the neural basis of
changes in multisensory functioning in human infants and children. Functional imaging research
on cortical multisensory processes in developing human populations is just starting to appear in
the literature (e.g. Brandwein et al. 2011; Kushnerenko et al. 2008; Russo et al. 2010). Furthermore,
researchers are now beginning to consider neural models of multisensory development (e.g.
A.J. Bremner et al. 2008). Certainly, as more and more data arise it will become increasingly
important to consider whether our explanations of the development of multisensory processes
are consistent with the development of the central nervous system that subserves them. It will also
be particularly important to consider how the extant animal models can inform our understand-
ing of neural and behavioural development seen in humans.4 Chapter 16 by Ghazanfar presents a
particularly useful contribution to this matter by showing how a comparative perspective on
brain development can help explain why certain developmental processes occur in the way they
do in different species.
Lastly in this section, we mention an area of behavioural research in human infants, young
children, and clinical populations that has been brought to bear on the question of how neural
communications between sensory cortical areas change over the course of human development.
In Chapter 10, Maurer et al. explain how synaesthesia, a clinical condition in which stimuli to one
sensory system give rise to additional (often arbitrarily-related) atypical sensations in the same or
in a different sensory modality, could result from atypical neural development involving connec-
tions between sensory cortical areas (see also Marks and Odgaard 2005; Maurer and Mondloch
2005). Maurer et al. describe different developmental explanations of synaesthesia and cover the
literature concerning behavioural development in typical infants and children and which relates
to the typical developmental process which seems to have gone awry in synaesthetes (see
Lewkowicz 2011a; Walker et al. 2010; Wagner and Dobkins 2011, for recent data and discus-
sion).

1.4.4 Understanding the developmental process (inheritance


and environment)
As we discussed earlier, theory and evidence indicates that multisensory development is strongly
influenced by both context and timing (e.g. Turkewitz 1994). This role for timing and context in
development has been highlighted a great deal in recent times; particularly by theorists
arguing for ‘process-oriented’ approaches to understanding development (e.g. Lewkowicz 2000;
2011a; Lickliter and Bahrick 2000; Karmiloff-Smith 2009; Mareschal et al. 2007) as an alternative
to the more deterministic viewpoints on development, which peaked during the 1990s
(e.g. Barkow et al. 1992; Baron-Cohen 1998; Leslie 1992; Spelke and Kinzler 2007). Unfortunately

4 Indeed, Chapter 14 by Wallace et al. and Chapter 11 by Laurienti and Hugenschmidt tackle these issues
head-on, drawing links between behavioural findings in humans and the implication of the SC model, and
considering the role of SC–cortical interactions.
IMPORTANT THEMES IN THIS BOOK 17

for developmental psychologists, ‘process-oriented’ approaches are complex in that they require
the integration of information from multiple disciplines. That is, they require an understanding
of the ontogeny of behaviour through a detailed analysis of the dynamic interactions between
different levels of organization including the genetic, neural, and environmental.
Although a process-oriented approach to the development of multisensory development is a
complex one, some of the chapters in this volume tackle these matters head-on by discussing the
developmental processes involved in the emergence of multisensory functions. For example,
Ghazanfar (Chapter 16) examines the impact of context and timing on multisensory develop-
ment through a comparative approach, Lewkowicz (Chapter 7) examines the paradoxical effects
of early experience, namely perceptual narrowing, on the development of audiovisual perception,
and Soto-Faraco et al. (Chapter 9) investigate the effects of growing up in a multilingual environ-
ment on audiovisual integration in speech processing (see Pons et al. 2009). Meanwhile, Mareschal
et al. describe their computational modelling approach. This considers how the constraints of
connectionist neural architectures and environment can give rise to developmental change in
cognition and behaviour (see Elman et al. 1996; Mareschal et al. 2007; O’Reilly and Munakata
2000).
Lastly, the chapters by Hill et al. (Chapter 12) and Röder (Chapter 13) specifically address the
question of how multisensory processes might go awry in atypical human development (see also
Chapter 9 by Soto-Faraco et al.). Röder reviews the significant body of research that has now
accrued regarding the influence of the loss of a single sensory modality (vision) on multisensory
processes in the remaining modalities. In Chapter 12, Hill et al. address multisensory processes in
developmental disorders. Given the number of disorders in which multisensory abnormalities are
implicated and, indeed, the number of multisensory therapeutic interventions that are currently
available (e.g. Ayres 1979; see Chapter 12 by Hill et al.), it is odd to note that little attention has
been given to atypical multisensory development prior to the publication of this chapter.
Nonetheless, research on the atypical development of multisensory processes in developmental
disorders such as autism spectrum disorder and developmental dyslexia are certainly on the
increase (e.g. see recent articles by Bahrick 2010; Blau et al. 2009; Facoetti et al. 2010; Foss-Feig
et al. 2010; Klin et al. 2009; Russo et al. 2010; Hairston et al. 2005). It is our hope that this multi-
sensory research will shed new light on these disorders and perhaps offer some validation for
multisensory interventions. Furthermore, as Hill et al. demonstrate, a careful analysis of the role
of multisensory abilities in atypical development (and vice versa) will likely shed additional light
on our understanding of the processes of multisensory development across both atypical and
typical populations.

1.4.5 Development, learning, and aging


As we have already discussed, the task of determining how perceptual processes emerge through
the complex dynamic interactions underlying developmental change is a complex puzzle.
Furthermore, it is perhaps even more difficult to determine what processes underlie multisensory
development than it is to understand unisensory development. This is best illustrated by means
of an example. Developmental changes in the relative weighting of the senses do not necessarily
result from changes in the multisensory integrative process itself but, in line with current thinking
regarding integration, could also result from changes in the reliability of one or more of the contrib-
uting senses across the course of development. Thus, unisensory developments can have subsequent
influences on multisensory processes. One way to help solve this kind of problem is through
carefully designed experimentation in which developmental changes in unisensory and multisen-
sory abilities are investigated in concert/parallel (e.g. Gori et al. 2008 and see Chapter 6 by Nardini
and Cowie). However, another way to approach this general problem of understanding the
18 THE MULTISENSORY APPROACH TO DEVELOPMENT

drivers of developmental change is to consider the wider context of development. Perhaps the
most informed picture of how context affects development is provided by a life-span approach in
which developmental changes are compared across life-periods in which the processes of
development are quite different.
Chapter 11 by Laurienti and Hugenschmidt provides a much needed review of the literature cov-
ering the development of multisensory processes in old age. Chapter 3 by Spence and also Chapter
9 by Soto-Faraco et al. adopt life-span approaches by covering and comparing development at vari-
ous points across the life-span (from infancy to old age). For instance, Spence’s chapter reviewing
the extant literature on multisensory flavour development highlights changes in multisensory inte-
gration in early life, adulthood (on the acquisition of perceptual expertise), and in old age.
But how does this richer context help us solve the problem of determining a developmental
process in multisensory abilities? From the wider literature we can learn much about the context
of developmental change across different periods of life, For instance, it is well known that, during
infancy and early childhood, unisensory performance improves, whereas in old age unisensory
acuity declines. Interestingly, researchers have reported increases in multisensory integration
across infancy, childhood, and old age (Chapter 11 by Laurienti and Hugenschmidt; Barutchu
et al. 2009; Gori et al. 2008; Nardini et al. 2008; Laurienti et al. 2006; Neil et al. 2006). Others have
tentatively suggested that multisensory integration can help to hide the consequences of unisen-
sory decline (see Laurienti et al. 2006). Thus we can conclude that developmental change in mul-
tisensory integration is occurring in the same direction across quite different contexts of
development in terms of unisensory functioning (improvement in infancy and childhood, decline
in old age). This points either to an independence of multisensory and unisensory developmental
processes, or toward there being qualitatively different kinds of changes in multisensory function-
ing at play across the different periods of the lifespan. Taking a life-span approach to development
therefore enriches our understanding of the factors involved in developmental processes.

1.5 Summary
Multisensory processes are at play in almost everything we do. It has taken mainstream psycholo-
gists (developmental and otherwise) some time to appreciate this fact, and we still think that more
needs to be done in order to integrate our understanding of multisensory perceptual functioning
into the developmental literature. Recently, however, there has been dramatic progress in our
understanding of multisensory development, as witnessed by the rich set of contributions in this
volume. As is often the case, the more we learn about multisensory development, the more com-
plex the picture becomes and the more questions are raised. Nonetheless it is our conviction (and
we hope that, after reading this book, you will agree with us) that a full picture of perceptual,
cognitive, and social development will only emerge once we consider the fact that all of these
processes depend crucially on multisensory interactions. We hope that this volume goes some
way toward documenting this fact and, even more importantly, that it spurs the next generation
of researchers to plunge head on into the complex world of multisensory interaction, for ulti-
mately it is at this level of organization that accurate explanations of behaviour lie.

Acknowledgements
AJB is supported by European Research Council Grant No. 241242 (European Commission
Framework Programme 7) and DJL was supported by NSF grant BCS-0751888 and NIH grant
D057116 during the work on this volume.
REFERENCES 19

References
Abrams, R.M., Gerhardt, K.J., and Peters, A.J.M. (1995). Transmission of sound and vibration to the fetus.
In Fetal development: A psychobiological perspective (eds. J. LeCanuet, W. Fifer, N. Krasnegor, and
W. Smotherman), pp. 315–30. Lawrence Erlbaum Associates, Hillsdale NJ.
Alais, D., and Burr, D. (2004). The ventriloquist effect results from near-optimal cross-modal integration.
Current Biology, 14, 257–62.
Arsenio, A., and Fitzpatrick, P. (2005). Exploiting amodal cues for robot perception. International Journal
of Humanoid Robotics, 2, 125–143.
Avillac, M., Deneve, S., Olivier, E., Pouget, A., and Duhamel, J.-R. (2005). Reference frames for
representing visual and tactile locations in parietal cortex. Nature Neuroscience, 8, 941–49.
Ayres, A.J. (1979). Sensory integration and the child. Western Psychological Services, Los Angeles, CA.
Baillargeon, R. (2004). Infants’ reasoning about hidden objects: evidence for event-general and
event-specific expectations. Developmental Science, 7, 391–414.
Bahrick, L.E. (1994). The development of infants’ sensitivity to arbitrary intermodal relations. Ecological
Psychology, 6, 111–23.
Bahrick, L.E. (2010). Intermodal perception and selective attention to intersensory redundancy:
implications for typical social developmental and autism. In The Wiley-Blackwell handbook of infant
development, 2nd edn. (eds. J.G. Bremner, and T.D. Wachs), pp. 120–66. Wiley-Blackwell, Oxford, UK.
Bahrick, L.E., and Licklier, R. (2000). Intersensory redundancy guides attentional selectivity and perceptual
learning in infancy. Developmental Psychology, 36, 190–201.
Barkow, J.H., Cosmides, L., and Tooby, J. (eds.) (1992). The adapted mind: Evolutionary psychology and the
generation of culture. Oxford University Press, New York.
Baron-Cohen, S. (1998). Does the study of autism justify minimalist innate modularity? Learning and
Individual Differences, 10, 179–91.
Barutchu, A., Crewther, D.P., and Crewther, S.G. (2009). The race that precedes co-activation:
Development of multisensory facilitation in children. Developmental Science, 12, 464–73.
Barutchu, A., Crewther, S.G., Fifer, J., et al. (2011). The relationship between multisensory integration and
IQ in children. Developmental Psychology, 47, 877–85.
Beidler, l.M. (1961). Taste receptor stimulation. Progress in Biophysics and Biophysical Chemistry, 12,
107–32.
Berkeley, G. (1948). An essay toward a new theory of vision (originally published in 1709). In Readings in
the history of psychology (ed. W. Dennis), pp. 69–80. Appleton-Century-Crofts, East Norwalk, CT.
Bermudez, J.L., Marcel, A.J., and Eilan, N. (1995). The body and the self. MIT Press, Cambridge, MA.
Bernard, J., and Sontag, L.W. (1947). Fetal reactivity to tonal stimulation: a preliminary report. Journal of
Genetic Psychology, 70, 205–210.
Bernstein, I.L. (1978). Learned taste aversion in children receiving chemotherapy. Science, 200, 1302–1303.
Birch, H.G., and Lefford, A. (1963). Intersensory development in children. Monographs of the Society for
Research in Child Development, 28(5), 1–48.
Birch, H.G., and Lefford, A. (1967). Visual differentiation, intersensory integration, and voluntary motor
control. Monographs of the Society for Research in Child Development, 32(2), 1–87.
Blau, V., van Atteveldt, N., Ekkebus, M., Goebel, R., and Blomert, L. (2009). Reduced neural integration of
letters and speech sounds links phonological and reading deficits in adult dyslexia. Current Biology, 19,
503–508.
Boothe, R.G., Dobson, V., and Teller, D.Y. (1985). Postnatal development of vision in human and
nonhuman primates. Annual Review of Neuroscience, 8, 495–545.
Botvinick, M., and Cohen, J. (1998). Rubber hands ‘feel’ touch that eyes see. Nature, 391, 756.
Braddick, O., and Atkinson, J. (2011). Development of human visual function. Vision Research, 51,
1588–1609.
20 THE MULTISENSORY APPROACH TO DEVELOPMENT

Brandwein, A.B., Foxe, J.J., Russo, N.N., Altschuler, T.S., Gomes, H., and Molholm, S. (2011). The
development of audiovisual multisensory integration across childhood and early adolescence:
a high-density electrical mapping study. Cerebral Cortex, 21, 1042–55.
Brayanov, J.B., and Smith, M.A. (2010). Bayesian and ‘anti-Bayesian’ biases in sensory integration for
action and perception in the size-weight illusion. Journal of Neurophysiology, 103, 1518–31.
Bredberg, G. (1968). Cellular pattern and nerve supply of the human organ of Corti. Acta Oto-
Laryngologica, Suppl. 236, 1–135.
Bregman, A.S. (1990). Auditory scene analysis: The perceptual organization of sound. MIT Press,
Cambridge, MA.
Bremner, A.J., and Spence, C. (2008). Unimodal experience constrains while multisensory experiences
enrich cognitive construction. Behavioral and Brain Sciences, 31, 335–36.
Bremner, A.J., Holmes, N.P., and Spence, C. (2008). Infants lost in (peripersonal) space? Trends in
Cognitive Sciences, 12, 298–305.
Bremner, J.G., Hatton, F., Foster, K.A., and Mason, U. (2011). The contribution of visual and vestibular
information to spatial orientation by 6- to 14-month-old infants and adults. Developmental Science 14,
1033–1045.
Bryant, P.E. (1974). Perception and understanding in young children. Methuen, London.
Bryant, P.E., Jones, P., Claxton, V., and Perkins, G.H. (1972). Recognition of shapes across modalities by
infants. Nature, 240, 303–304.
Burkhalter, A., Bernardo, K.L., and Charles, V. (1993). Development of local circuits in human visual
cortex. Journal of Neuroscience, 13, 1916–31.
Burr, D., Binda, P., and Gori, M. (2011). Combining vision with audition and touch, in adults and in
children. In Sensory cue integration (eds. J. Trommershäuser, M.S. Landy, and K.P. Körding).
Oxford University Press, New York.
Calvert, G.A., Spence, C., and Stein, B.E. (eds.) (2004). The handbook of multisensory processes. MIT Press,
Cambridge, MA.
Chen, Y.-C., and Spence, C. (2010). When hearing the bark helps to identify the dog: semantically-
congruent sounds modulate the identification of masked pictures. Cognition, 114, 389–404.
Chen, Y.-C., & Spence, C. (2011). Crossmodal semantic priming by naturalistic sounds and spoken words
enhances visual sensitivity. Journal of Experimental Psychology: Human Perception and Performance 37,
1554–1568.
Dodd, B., and R. Campbell (eds.) (1987). Hearing by eye: the psychology of lip-reading. Lawrence Erlbaum
Associates, Hillsdale, NJ.
Doehrmann, O., and Naumer, M.J. (2008). Semantics and the multisensory brain: How meaning
modulates processes of audio-visual integration. Brain Research, 1242, 136–50.
Driver, J., and Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on
‘sensory-specific’ brain regions, neural responses, and judgments. Neuron, 57, 11–23.
Durie, B. (2005). Future sense. New Scientist, 2484, 33–36.
Edwards, A. (1946). Body sway and vision. Journal of Experimental Psychology, 36, 526–35.
Ellingson, R.J. (1960). Cortical electrical responses to visual stimulation in the human infant.
Electroencephalography and Clinical Neurophysiology, 12, 663–77.
Elman, J.L., Bates, E., Johnson, M.H., Karmiloff-Smith, A., Parisi, D., and Plunkett, K. (1996). Rethinking
innateness: a connectionist perspective on development. MIT Press, Cambridge, MA.
Elsabbagh, M., and Johnson, M.H. (2010). Getting answers from babies about autism. Trends in Cognitive
Science, 14, 81–87.
Engel, R. (1964). Electroencephalographic responses to photic stimulation, and their correlation with
maturation. Annals of the New York Academy of Science, 12, 663–77.
Ernst, M.O., and Banks, M.S. (2002). Humans integrate visual and haptic information in a statistically
optimal fashion. Nature, 415, 429–33.
REFERENCES 21

Ernst, M.O., and Bülthoff, H.H. (2004). Merging the senses into a robust percept. Trends in Cognitive
Sciences, 8, 162–69.
Facoetti, A., Tussardi, A.N., Ruffino, M., et al. (2010). Multisensory spatial attention deficits are predictive
of phonological decoding skills in developmental dyslexia. Journal of Cognitive Neuroscience, 22,
1011–25.
Fantz, R. (1961). The origin of form perception. Scientific American, 204(5), 66–72.
Fantz, R. (1964). Visual experience in infants: Decreased attention to familiar patterns relative to novel
ones. Science, 146, 668–70.
Field, T. (2003). Touch. MIT Press, Cambridge, MA.
Field, T. (2010). Massage therapy facilitates weight gain in preterm infants. Current Directions in
Psychological Science, 10, 51–54.
Field, T., Diego, M.A., Hernandez-Reif, M., Deeds, O., and Figuereido, B. (2006). Moderate versus light
pressure massage therapy leads to greater weight gain in preterm infants. Infant Behavior and
Development, 29, 574–78.
Foss-Feig, J.H., Kwakye, L.D., Cascio, C.J., et al. (2010). An extended multisensory temporal binding
window in autism spectrum disorders. Experimental Brain Research, 203, 381–89.
Freides, D. (1974). Human information processing and sensory modality: Cross-modal functions,
information complexity, memory, and deficit. Psychological Bulletin, 81, 284–310.
Galebsky, A. (1927). Vestibular nystagmus in new-born infants. Acta Oto-Laryngologica, 11, 409–23.
Gallace, A., and Spence, C. (2010). The science of interpersonal touch: An overview. Neuroscience and
Biobehavioral Reviews, 34, 246–59.
Gallagher, S. (2005). How the body shapes the mind. Oxford, UK: Oxford University Press.
Ghazanfar, A.A., and Schroeder, C.E. (2006). Is neocortex essentially multisensory? Trends in Cognitive
Sciences, 10, 278–85.
Gibson, E.J. (1969). Principles of perceptual learning and development. Appleton-Century-Crofts,
East Norfolk, CT.
Gibson, J.J. (1943). Adaptation, after-effect and contrast in the perception of curved lines. Journal of
Experimental Psychology, 16, 1–31.
Gibson, J.J. (1966). The senses considered as perceptual systems. Houghton Mifflin, Boston, MA.
Gibson, J.J. (1979). The ecological approach to visual perception. Lawrence Erlbaum Associates, Hillsdale NJ.
Gori, M., Del Viva, M., Sandini, G., and Burr, D.C. (2008). Young children do not integrate visual and
haptic information. Current Biology, 18, 694–98.
Gori, M., Giuliana, L., Alessandra, S., Sandini, G., and Burr, D. (2010a). Calibration of the visual by the
haptic system during development. Journal of Vision, 10, Article 470.
Gori, M., Sandini, G., Martinoli, C., and Burr, D. (2010b). Poor haptic orientation discrimination in
nonsighted children may reflect disruption of cross-sensory calibration. Current Biology, 20, 223–25.
Gottlieb, G. (1971). Ontogenesis of sensory function in birds and mammals. In The biopsychology of
development (eds. E. Tobach, L.R. Aronson, and E. Shaw), pp. 67–128. Academic Press, New York.
Gouin Décarie, T. (1969). A study of the mental and emotional development of the thalidomide child. In
Determinants of infant behavior, Vol. 4 (ed. B.M. Foss), pp. 167–87. Barnes and Noble, New York.
Gouin Décarie, T., and Ricard, M. (1996). Revisiting Piaget revisted, or the vulnerability of Piaget’s infancy
theory in the 1990s. In Developmental and vulnerability in close relationships (eds. G.G. Noam, and
K.W. Fischer), pp. 113–34. Lawrence Erlbaum Associates, Mahwah, NJ.
Gregory, R.L. (1967). Origin of eyes and brains. Nature, 213, 369–72.
Hairston, W.D., Burdette, J.H., Flowers, D.L., Wood, F.B., and Wallace, M.T. (2005). Altered temporal
profile of visual-auditory multisensory interactions in dyslexia. Experimental Brain Research, 166,
474–80.
Halit, H., de Haan, M., and Johnson, M.H. (2003). Cortical specialisation for face processing: face-sensitive
event-related potential components in 3 and 12 month-old infants. NeuroImage, 19, 1180–93.
22 THE MULTISENSORY APPROACH TO DEVELOPMENT

Hein, G., Doehrmann, O., Müller, N.G., Kaiser, J., Muckli, L., and Naumer, M.J. (2007). Object familiarity
and semantic congruency modulate responses in cortical audiovisual integration areas. Journal of
Neuroscience, 27, 7881–87.
Held, R., Ostrovsky, Y., de Gelder, B., et al. (2011). The newly sighted fail to match seen with felt. Nature
Neuroscience, 14, 551–53.
Hernandez-Reif, M. and Bahrick, L.E. (2001). The development of visual-tactile perception of objects:
Amodal relations provide the basis for learning arbitrary relations. Infancy, 2, 51–72.
Hershenson, M. (1963). Reaction time as a measure of intersensory facilitation. Journal of Experimental
Psychology, 63, 289–93.
Hillock, A.R., Powers, A.R., and Wallace, M.T. (2010). Binding of sights and sounds: age-related changes
in multisensory temporal processing. Neuropsychologia, 49, 461–67.
Hooker, D. (1952). The prenatal origin of behavior. University of Kansas Press, Lawrence, KS.
Howes, D. (ed.) (1991). The varieties of sensory experience: A sourcebook in the anthropology of the senses.
University of Toronto Press, Toronto.
Howes, D. (2006). Cross-talk between the senses. The Senses and Society, 1, 381–90.
Humphrey, T. (1964). Some correlations between the appearance of human fetal reflexes and the
development of the nervous system. Progress in Brain Research, 4, 93–135.
ISO (1992). Standard 5492: Terms relating to sensory analysis. International Organization for
Standardization.
ISO (2008). Standard 5492: Terms relating to sensory analysis. International Organization for
Standardization.
James, W. (1890). Principles of psychology, Vol. 1. Henry Holt, New York.
Johnson, M.H. (2011). Interactive specialization: A domain-general framework for human functional brain
development? Developmental Cognitive Neuroscience, 1, 7–21.
Johnson, M.H., and de Haan, M. (2010). Developmental cognitive neuroscience, 3rd edn. Wiley-Blackwell,
Oxford.
Karmiloff-Smith, A. (2009). Nativism versus neuroconstructivism: rethinking the study of developmental
disorders. Developmental Psychology, 45, 56–63.
Kenny, P., and Turkewitz, G. (1986). Effects of unusually early visual stimulation on the development of
homing behaviour in the rat pup. Developmental Psychobiology, 19, 57–66.
King, A.J. (2005). Multisensory integration: strategies for synchronization. Current Biology, 15, R339–41.
Klatzky, R.L., and Lederman, S.J. (1993). Toward a computational model of constraint-driven exploration
and haptic object identification. Perception, 22, 597–621.
Klatzky, R.L., Lederman, S.J., and Reed, C. (1987). There’s more to touch than meets the eye: the salience of
object attributes for haptic with and without vision. Journal of Experimental Psychology: General, 116,
356–69.
Klin, A., Lin, D.J., Gorrindo, P., Ramsay, G., and Jones, W. (2009). Two-year-olds with autism orient to
non-social contingencies rather than biological motion. Nature, 459, 257–61.
Kushnerenko, E., Teinonen, T., Volein, A., and Csibra, G. (2008). Electrophysiological evidence of illusory
audiovisual speech percept in human infants. Proceedings of the National Academy of Sciences U.S.A.,
105, 11442–45.
Laurienti, P.J., Kraft, R.A., Maldjian, J.A., Burdette, J.H., and Wallace, M.T. (2004). Semantic congruence is
a critical factor in multisensory behavioral performance. Experimental Brain Research, 158, 405–414.
Laurienti, P.J., Burdette, J.H., Maldjian, J.A., and Wallace, M.T. (2006). Enhanced multisensory integration
in older adults. Neurobiology of Aging, 27, 1155–63.
LeCanuet, J.P., and Schaal, B. (1996). Fetal sensory competences. European Journal of Obstetrics,
Gynaecology, and Reproductive Biology, 68, 1–23.
Lee, D.N. (1980). The optic flow field: the foundation of vision. Philosophical Transactions of the Royal
Society London B, 290, 169–79.
REFERENCES 23

Lee, D.N., and Aronson, E. (1974). Visual proprioceptive control of standing in human infants. Perception
and Psychophysics, 15, 529–32.
Leslie, A.M. (1992). Pretence, autism, and the theory-of-mind-module. Current Directions in Psychological
Science, 1, 18–21.
Lewkowicz, D.J. (2000). The development of intersensory temporal perception: an epigenetic systems/
limitations view. Psychological Bulletin, 126, 281–308.
Lewkowicz, D.J. (2002). Heterogeneity and heterochrony in the development of intersensory perception.
Cognitive Brain Research, 14, 41–63.
Lewkowicz, D.J. (2011a). The biological implausibility of the nature-nurture dichotomy and what it means
for the study of infancy. Infancy, 16, 1–37.
Lewkowicz, D.J. (2011b). The development of multisensory temporal perception. In Frontiers in the neural
bases of multisensory processes (eds. M.M. Murray, and M.T. Wallace). Taylor and Francis, New York.
Lewkowicz, D.J., and Ghazanfar, A.A. (2009). The emergence of multisensory systems through perceptual
narrowing. Trends in Cognitive Sciences, 13, 470–78.
Lewkowicz, D.J., and Lickliter, R. (eds.) (1994). The development of intersensory perception: Comparative
perspectives. Lawrence Erlbaum Associates, Hillsdale, NJ.
Lewkowicz, D.J., and Turkewitz, G. (1980). Cross-modal equivalence in early infancy: auditory-visual
intensity matching. Developmental Psychology, 16, 597–607.
Lewkowicz, D.J., Leo, I., and Simion, F. (2010). Intersensory perception at birth: Newborns match
nonhuman primate faces and voices. Infancy, 15, 46–60.
Lickliter, R., and Bahrick, L.E. (2000). The development of infant intersensory perception: advantages of a
comparative convergent-operations approach. Psychological Bulletin, 126, 260–80.
Lickliter, R., and Banker, H. (1994). Prenatal components of intersensory development in precocial birds.
In The development of intersensory perception: comparative perspectives (eds. D.J. Lewkowicz, and
R. Lickliter), pp. 59–80. Lawrence Erlbaum Associates, Hillsdale, NJ.
Locke, J. (1690). An essay concerning human understanding (ed. P.H. Nidditch 1979). Oxford University
Press, Oxford.
Lundström, J.N., Boesveldt, S., and Albrecht, J. (2011). Central processing of the chemical senses: an
overview. ACS Chemical Neuroscience, 2, 5–16.
Ma, W.J., and Pouget, A. (2008). Linking neurons to behavior in multisensory perception: a computational
review. Brain Research, 1242, 4–12.
Macpherson, F. (ed.) (2011). The senses: classic and contemporary philosophical perspectives. Oxford
University Press, Oxford.
Mann, I. (1964). The development of the human eye, 3rd edn. Grune and Stratton, New York.
Mareschal, D., Johnson, M.H., Sirois, S., Spratling, M., Thomas, M., and Westermann, G. (2007).
Neuroconstructivism, Vol. I: How the brain constructs cognition. Oxford University Press, Oxford.
Marks, L.E., and Odgaard, E.C. (2005). Developmental constraints on theories of synaesthesia. In
Synaesthesia: perspectives from cognitive neuroscience (eds. L. Robertson, and N. Sagiv), pp. 214–36.
Oxford University Press, New York.
Mauk, M.D., and Buonomano, M.D. (2004). The neural basis of temporal processing. Annual Review of
Neuroscience, 27, 307–40.
Maurer, D., and Mondloch, C.J. (2005). Neonatal synaesthesia: a re-evaluation. In Synaesthesia: Perspectives
from cognitive neuroscience (eds. L.C. Robertson, and N. Sagiv), pp. 193–213. Oxford University Press,
Oxford.
Maurer, D., Pathman, T., and Mondloch, C. (2006). The shape of boubas: sound-shape correspondences in
toddlers and adults. Developmental Science, 9, 316–22.
Meck, W.H. (1991). Modality-specific Circadian rhythmicities influence mechanisms of attention and
memory for interval timing. Learning and Motivation, 22, 153–79.
Meltzoff, A.N., and Borton, R.W. (1979). Intermodal matching by human neonates. Nature, 282, 403–404.
24 THE MULTISENSORY APPROACH TO DEVELOPMENT

Miller, J.O. (1991). Channel interaction and the redundant targets effect in bimodal divided attention.
Journal of Experimental Psychology: Human Perception and Performance, 17, 160–69.
Moon, C., and Fifer, W.P. (2011). Prenatal development. In An Introduction to Developmental Psychology,
2nd edn (eds. A. Slater and J.G. Bremner). Wiley-Blackwell, Oxford.
Moore, K.L., and Persaud, T.V.N. (2008). The developing human: clinically oriented embryology, 8th edn.
Saunders Elsevier, Philadelphia, PA.
Murray, M.M., and Wallace, M.T. (eds.) (2011). The neural bases of multisensory processes. Taylor and
Francis, New York.
Naumer, M.J., and Kaiser, J. (eds.) (2010). Multisensory object perception in the primate brain. Springer,
New York.
Nardini, M., Jones, P., Bedford, R., and Braddick, O. (2008). Development of cue integration in human
navigation. Current Biology, 18, 689–93.
Nava, E., Scala, G.G., and Pavani, F. (in press). Changes in sensory preference during childhood:
Converging evidence from the Colavita effect and the sound-induced flash illusion. Child Development.
Neil, P.A., Chee-Ruiter, C., Scheier, C., Lewkowicz, D.J., and Shimojo, S. (2006). Development of
multisensory spatial integration and perception in humans. Developmental Psychology, 9, 454–64.
Occelli, V., Spence, C., and Zampini, M. (2011). Audiotactile interactions in temporal perception.
Psychonomic Bulletin and Review, 18, 429–54.
O’Reilly, R.C., and Munakata, Y. (2000). Computational explorations in cognitive neuroscience:
understanding the mind by simulating the brain. MIT Press, Cambridge, MA.
Partan, S. and Marler, P. (1999). Communication goes multimodal. Science, 283, 1272–73.
Parise, C.V., and Spence, C. (in press). Audiovisual crossmodal correspondences. To appear in Oxford
handbook of synaesthesia (eds. J. Simner, and E. Hubbard). Oxford University Press, Oxford, UK.
Petrini, K., Dahl, S., Rochesson, D., et al. (2009). Multisensory integration of drumming actions: Musical
expertise affects perceived audiovisual asynchrony. Experimental Brain Research, 198, 339–52.
Petrini, K., Russell, M., and Pollick, F. (2009). When knowing can replace seeing in audiovisual integration
of actions. Cognition, 110, 432–39.
Piaget, J. (1952). The origins of intelligence in the child. Routledge and Kegan-Paul, London.
Pons, F., Lewkowicz, D.J., Soto-Faraco, S., and Sebastián-Gallés, N. (2009). Narrowing of intersensory
speech perception in infancy. Proceedings of the National Academy of Sciences U.S.A., 106, 10598–10602.
Pöppel, E. (1973). Comments on ‘Visual system’sf view of acoustic space’. Nature, 243, 231.
Roland, B.A., Quessy, S., Stanford, T.R., and Stein, B.E. (2007). Multisensory integration shortens
physiological response latencies. Journal of Neuroscience, 27, 5879–84.
Ross, L.A., Molholm, S., Blanco, D., Gomez-Ramirez, M., Saint-Amour, D., and Foxe, J.J. (2011).
The development of multisensory speech perception continues into the late childhood years.
European Journal of Neuroscience, 33, 2329–37.
Russo, N., Foxe, J.J., Brandwein, A.B., Altschuler, T., Gomes, H., and Molholm, S. (2010). Multisensory
processing in children with autism: high-density electrical mapping of auditory-somatosensory
integration. Autism Research, 3, 253–67.
Schaal, B., Hummel T., and Soussignan, R. (2004). Olfaction in the fetal and premature infant: functional
status and clinical implications. Clinics in Perinatology, 31, 261–85.
Sirois, S., Spratling, M.W., Thomas, M.S.C., Westermann, G., Mareschal, D., and Johnson, M.H. (2008).
Precis of neuroconstructivism: how the brain constructs cognition. Behavioral and Brain Sciences, 31,
321–56.
Smith, E.L., Grabowecky, M., and Suzuki, S. (2007). Auditory-visual crossmodal integration in perception
of face gender. Current Biology, 17, 1680–85.
Spelke, E.S., and Kinzler, K.D. (2007). Core knowledge. Developmental Science, 10, 89–96.
Spence, C. (2011a). Crossmodal correspondences: A tutorial review. Attention, Perception, and
Psychophysics, 73, 971–95.
REFERENCES 25

Spence, C. (2011b). The multisensory perception of touch. In Art and the sense (eds. F. Bacci, and
D. Mecher), pp. 85–106. Oxford University Press, Oxford, UK.
Spence, C. (in press). Multisensory perception, cognition, and behaviour: Evaluating the factors
modulating multisensory integration. To appear in The new handbook of multisensory processing
(ed. B.E. Stein). MIT Press, Cambridge, MA.
Spence, C. and Bremner, A.J. (2011). Crossmodal interactions in tactile perception. In The handbook of
touch: neuroscience, behavioral, and health perspectives (eds. M. Hertenstein, and S. Weiss), pp. 189–216.
Springer, New York.
Spence, C., and Driver, J. (eds.) (2004). Crossmodal space and crossmodal attention. Oxford University Press,
Oxford.
Spence, C., and Squire, S. (2003). Multisensory integration: Maintaining the perception of synchrony.
Current Biology, 13, R519–21.
Spence, C., Shore, D.I., and Klein, R.M. (2001). Multisensory prior entry. Journal of Experimental
Psychology: General, 130, 799–832.
Spence, C., McDonald, J., and Driver, J. (2004). Exogenous spatial cuing studies of human crossmodal
attention and multisensory integration. In Crossmodal space and crossmodal attention (eds. C. Spence,
and J. Driver), pp. 277–320. Oxford University Press, Oxford.
Spence, C., Ngo, M., Lee, J.-H., and Tan, H. (2010a). Solving the correspondence problem in haptic/
multisensory interface design. In Advances in haptics (ed. M.H. Zadeh), pp. 47–74. In-Teh Publishers,
Vukovar, Croatia.
Spence, C., Levitan, C., Shankar, M. U., and Zampini, M. (2010b). Does food color influence taste and
flavor perception in humans? Chemosensory Perception, 3, 68–84.
Stein B.E. (ed.) (in press). The new handbook of multisensory processes. MIT Press, Cambridge, MA.
Stein, B.E., and Meredith, M.A. (1993). The merging of the senses. MIT Press, Cambridge, MA.
Stein, B.E., and Stanford, T.R. (2008). Multisensory integration: current issues from the perspective of the
single neuron. Nature Reviews Neuroscience, 9, 255–67.
Stein, B.E., Meredith, M.A., and Wallace, M.T. (1994). Development and neural basis of multisensory
integration. In The development of intersensory perception: Comparative perspectives (eds.
D.J. Lewkowicz, and R. Lickliter), pp. 81–107. Lawrence Erlbaum Associates, Hillsdale, NJ.
Stein, B.E., Burr, D., Costantinides, C., et al. (2010). Semantic confusion regarding the development of
multisensory integration: a practical solution. European Journal of Neuroscience, 31, 1713–20.
Taylor, M.J., Menzies, R., MacMillan, L.J., and Whyte, H.E. (1987). VEPs in normal full-term and
premature neonates: longitudinal versus cross-sectional data. Electroencephalography and Clinical
Neurophysiology, 68, 20–27.
Trommershäuser, J., Landy, M.S., and Körding, K.P. (eds.) (2011). Sensory cue integration. Oxford
University Press, New York.
Turkewitz, G. (1994). Sources of order for intersensory functioning. In The development of intersensory
perception: comparative perspectives (eds. D.J. Lewkowicz, and R. Lickliter), pp. 3–18. Lawrence
Erlbaum Associates, Hillsdale, NJ.
Turkewitz, G., and Kenny, P.A. (1982). The role of developmental limitations of sensory input on
perceptual devleopment: a preliminary theoretical statement. Developmental Psychobiology, 15, 357–68.
Van der Meer, A.L. (1997). Keeping the arm in the limelight: Advanced visual control of arm movements in
neonates. European Journal of Paediatric Neurology, 1, 103–108.
Varela, F.J., Thompson, E., and Rosch, E. (1991). The embodied mind: cognitive science and human
experience. MIT Press, Cambridge, MA.
Wagner, K., and Dobkins, K.R. (2011). Synaesethetic associations decrease during infancy. Psychological
Science 22, 1067–1072.
Walker, P., Bremner, J.G., Mason, U., et al. (2010). Preverbal infants’ sensitivity to synaesthetic
cross-modality correspondences. Psychological Science, 21, 21–25.
26 THE MULTISENSORY APPROACH TO DEVELOPMENT

Walker-Andrews, A. (1994). Taxonomy for intermodal relations. In The development of intersensory


perception: comparative perspectives (eds. D.J. Lewkowicz, and R. Lickliter), pp. 39–56. Lawrence
Erlbaum Associates, Hillsdale, NJ.
Wallace, M.T. (2004). The development of multisensory processes. Cognitive Processes, 5, 69–83.
Wallace, M.T. et al. (2004) Visual experience is necessary for the development of multisensory integration.
Journal of Neuroscience, 24, 9580–84.
Weitzman, E.D., and Graziani, L.J. (1968). Maturation and topography of the auditory evoked response of
the prematurely born infant. Developmental Psychobiology, 1, 79–89.
Wallace, M.T., and Stein, B.E. (1997) Development of multisensory neurons and multisensory integration
in cat superior colliculus. Journal of Neuroscience, 17, 2429–44.
Welch, R.B., and Warren, D.H. (1980). Immediate perceptual response to intersensory discrepancy.
Psychological Bulletin, 3, 638–67.
Yu, L., Rowland, B.A., and Stein, B.E. (2010). Initiating the development of multisensory integration by
manipulating sensory experience. Journal of Neuroscience, 30, 4904–13.
Zmyj, N., Jank, J., Schütz-Bosbach, S., and Daum, M.M. (2011). Detection of visual-tactile contingency in
the first year after birth. Cognition, 120, 82–89.
Part A

Typical development of
multisensory processes from
early gestation to old age
This page intentionally left blank
Chapter 2

The role of olfaction in human


multisensory development
Benoist Schaal and Karine Durand

The omission of taste, touch and olfaction. . .is. . .an accurate


reflection of the disproportionate concern which has been
afforded to vision and audition in studies of infancy. . .
Such imbalance would detract from a comprehension of
functioning at any stages of development, but may be
particularly distorting with regard to our understanding of the
world of the infant.
(G. Turkewitz 1979, p. 408)

2.1 Introduction
Olfaction is an unavoidable and ubiquitous source of perceptual experience from the earliest
steps of mammalian development.1 Olfaction is unavoidable because, in all mammals, nasal
chemosensors develop functionally in advance of other sensory systems (with the exception of
somesthesic/kinesthesic sensors), and are thus in a position of neurologically-imposed readiness
to ‘feed’ the brain before the functional inception of hearing and vision, and to bind with their
inputs when these latter sensory systems set on. Olfaction is ubiquitous because nasal chemosen-
sors are in direct contact with stimuli that result from the normal biological functioning of a
mother–infant relationship. In all placental mammals, the foetuses are bathed in an odorous
amniotic fluid (which is also likely to give rise to tastes). All infants are fed flavourful milk and
cared for in the fragrant bubble of a parent’s body, and they are then all introduced to non-milk
foods whose chemosensory properties are essential to the establishment of liking and wanting,
and thereby sorting out the unwanted and the disliked. All through these stages, either as fore-
ground or as background elements, odours co-occur, interact, and merge into percepts elabo-
rated with the other senses.
The present chapter attempts to summarize the current understanding of how olfaction
functions in concert with the other senses during human development, and the various ways early
multisensory effects involving the chemical senses operate. We will survey some of the available
results concerning odour-based intersensory effects. There are several ways for a sense modality

1 In fact, olfaction appears to be universal in the early developmental steps of any animal species, but our
focus here will be restricted to mammals.
30 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

to be caught up in multisensory functioning. As defined broadly, ‘intersensory functioning is


involved whenever there are joint influences of stimulation in more than one modality’ (Turkewitz
1994, p. 3). Such intersensory influences are multiple and change over the course of development
(Turkewitz and Mellon 1989; Tees 1994), involving either:
◆ no effects of one modality on another
◆ facilitatory or inhibitory effects of stimulation in one modality on responsiveness to stimuli in
another modality
◆ the association of stimuli between modalities
◆ the extraction of properties that can be picked up by any sensory modality, therefore desig-
nated as amodal properties (E.J. Gibson 1969).
Transposed into the domain of olfaction, these categories of multisensory effects raise the
following questions:
1) Do odours exist independently from other percepts? More specifically, do some odour
perceptions function uninfluenced by other kinds of sensory information and does this
change over development?
2) How do odours alter what one sees, hears, and feels by touch, kinesthesis, proprioception,
pain, or interoception? For instance, how do odours modulate organismic states (attention,
motivation, emotion, mood) that bias the perception of stimuli in other modalities?
Reciprocally, do stimuli in other sense modalities shape olfactory perception?
3) How do olfactory and proprioceptive/tactile/auditory/visual linkages univocally or reciprocally
grow out of, and depend on, experience with objects, persons, and events? Are there biological
constraints to bind odours more easily with certain stimuli rather than with others?
4) To what extent can odours present information that is redundant with information presented
by coordinated sensory systems? Can properties of objects, events, or states be concurrently
specified by olfaction and other sense modalities? In other words, which amodal properties of
objects or events, such as their location, temporal synchrony, tempo, duration, intensity,
variety, complexity, substance, affective value, or common affordances (Walker-Andrews
1994), can be carried by odours, and through which mechanisms are they acquired?
Taken together, all the above issues lead to two overarching research questions. The first
considers how odours are mutually articulated and bound with other sensory impressions to
become integral attributes of multisensory objects, persons, events, or contexts. The second ques-
tion concerns how multisensory processes involving olfaction affect behaviour and contribute
to emotional and cognitive development.
The reader is warned that these questions can be answered only partially today. Indeed, if ‘our
understanding of intersensory functioning is truly in its infancy’ in the course of ontogenesis
(Turkewitz 1994: p. 15), our developmental appreciation of how the chemical senses participate
in multisensory processing is only in its very initial, ‘embryonic’ stage. Thus, the first step that is
required before we can better understand mechanisms underlying the developmental interactions
between the chemical senses and the other senses is a concentrated collection of descriptive
and normative data (at least in humans). Only after such data are obtained can the focus shift
to underlying processes and mechanisms, both in humans and non-humans. Whenever the avail-
able results make it possible, we will here interweave the patchy data from humans with the more
substantial data from non-human animal studies. These latter studies provide research hypothe-
ses and theoretical scaffolding for the human domain, and enrich explanations of human data.
However, sensory functions, and hence the different categories of intersensory functions, unques-
tionably depend on the level of maturity of an infant’s sensory modalities. We will therefore
SOME SPECIFICITIES OF OLFACTION IN HUMAN DEVELOPMENT 31

address them at various developmental stages, namely in foetuses, newborns, infants, and
children. Again, however, it is clear that developmental data on the participation of chemorecep-
tion in multisensory processes are not only scant, but they have been unevenly addressed across
these periods of development.

2.2 Some specificities of olfaction in human development


This section briefly summarizes aspects of current biological and psychological knowledge
concerning olfaction in humans. The aim is to provide some functional principles useful to
understand the following exposition on the development of chemosensation in the context of
multisensory processes. At this stage, nasal chemoreception will be considered in combination
with oral chemoreception to highlight their basic functional interconnectedness. But the empha-
sis of the subsequent sections of the chapter will essentially be on olfaction (for taste and flavour,
see the Chapter 3 by Spence).

2.2.1 Chemoreception is anatomically and functionally complex


Chemoreception is a multisensory system from the outset, constituted by the accretion of ana-
tomically-distinct systems of sensors placed close to the aerial and digestive entries (cf. Fig. 2.1).2
In humans, two main chemoreceptive subsystems are located in the nasal cavities and two others
in the mouth (for reviews on these subsystems, see Doty 2003).3 The major structures to establish
nasal chemoreception are the olfactory and the trigeminal subsystems. Olfactory sensory neurons
(OSNs) of the main olfactory system dwell in the roof of the nasal cavity, connecting through
olfactory nerves to the main olfactory bulbs (OBs). Branches of the trigeminal system innervate
the epithelium lining the upper respiratory tract. The olfactory system is tuned to detect innu-
merable compounds carried in low concentrations in the incoming airflow. The trigeminal sys-
tem is mainly sensitive to higher concentrations of chemostimulants bearing irritating properties,
mediating sensations such as stinging, burning, or cooling (Doty and Cometto-Muniz 2003).
These nasal subsystems appear, however, well interconnected, with functional overlaps or syner-
gies (Murphy and Cain 1980). Thus unless stated otherwise, olfaction will be considered here as a
multi-channel, nasal event without further specification of the separate ‘subsystem’ responsible
for corresponding percepts. Oral chemoreception is based on the specific gustatory (or taste)
pathway via the taste buds, and on a tacto-chemical pathway via trigeminal innervation of the oral
cavity. The gustatory subsystem mediates several principal tastes (sweet, sour, salty, bitter, and
savoury or umami) and their combinations, while the oral trigeminal subsystem mediates astrin-
gency, burning, and cooling, in the same time as temperature and ‘mouthfeel’ (Rolls 2005; Simon
et al. 2006). Thus, from the very periphery of the systems, olfaction, taste, and trigeminal sensa-
tion operate concomitantly (and perhaps sequentially) while food is processed, to provide multi-
sensory information to higher brain structures.

2 Chemoreception is also effective through taste receptors present in the gut (duodenum) where it can
mediate associative processes between intestinal chemoreception and post-ingestive interoception (e.g.
Raybould 1998; Sclafani 2007).
3 Two other chemoreceptive subsystems dwell in the nose: the vomeronasal and the terminal subsystems. The
vomeronasal system comprises bilateral invaginations on the nasal septum (the middle-wall of the nose) lined
with sensory-like cells. The terminal system distributes free nerve endings to the anterior part of the nasal
septum. The exact sensory functions of these two subsystems remain unclear in humans (cf. Doty 2003).
32 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

Olfactory bulb Trigeminal


nerve
Olfactory
nerves

Ortho-
nasal
&
retro-
nasal
routes

Olfactory system Trigeminal system

Olfactory Taste bud


receptor Taste
neuron nerves

Taste cells
Taste system

Fig. 2.1 Schematic representation of the human nasal and oral chemosensory systems. Upper left:
the main olfactory system: odorant molecules access the olfactory region at the top of the nasal
cavity by the orthonasal or retronasal pathways. The odorants interact with the receptor proteins
situated on the dendrites of the olfactory sensory neurons that transmit information to the
olfactory bulbs, which then project to primary and secondary areas. Low middle: the gustatory
system: taste buds situated on the tongue send axons to primary and secondary projection areas.
Upper right: the nasal and oral trigeminal subsystem: free nerve endings distributed in the whole
nasal and oral mucosae connect with the brain stem and somesthesis integrative area (© André
Holley, 2012).

2.2.2 The chemoreceptive modalities engender unitary percepts


Although anatomically and functionally dissociable, the chemoreceptive modalities give rise to
unitary percepts. An organism can sample stimuli arising from outside (odours) and from inside
(flavours) the body. Olfaction is thus involved in two perceptual systems, one concerned with
guiding actions in the physical and social world, and the other concerned with savouring food
and beverages, and controlling eating. Odorants can reach the nasal chemoreceptive subsystems
by the orthonasal (when inhaling through the external nares) or the retronasal routes (when
odorants are aspired up from the mouth into the nasal cavities through the choanae; cf. Fig. 2.1).
Ortho- and retronasal pathways elicit different, and sometimes discrepant, perceptions from the
same odour stimulus (Rozin 1982) and they function at different levels of sensitivity (Heilmann
and Hummel 2004). ‘Inside olfaction’ and taste generally work in synchrony, especially when
eating or drinking is concerned, and give rise to odour–taste perceptual mixtures—so-called
flavours—which result from co-operating oro-nasal chemoreception, passive and active (e.g. in
chewing) somesthesis, and audition and vision (Auvray and Spence 2008 ; see Chapter 3
by Spence). Olfaction and taste fuse in a single flavour percept, in which gustatory/olfactory
SOME SPECIFICITIES OF OLFACTION IN HUMAN DEVELOPMENT 33

components are difficult to tell apart spontaneously. Some authors insist on the automatic nature
of the odour–taste association and suggest that it has synesthesia-like properties (Stevenson and
Boakes 2004). Such odour–taste associations are common and do persist lifelong, unlike similar
phenomena involving other sense modalities (Stevenson and Tomiczek 2007). This stability of
odour–taste synesthesia has been suggested to date back to the earliest stages of development,
when oronasal sensations do regularly co-occur through chemical cues carried in amniotic fluid
and milk. Thus, the taste–olfaction synesthesia may in part be explained by the lack of early
pruning of neural interconnections (Spector and Maurer 2009; Verhagen and Engelen 2006),
so that these sense modalities remain more closely interconnected in perception (cf. Chapter 10
by Maurer).

2.2.3 Nasal and oral chemoreceptive subsystems are characterized


by their ontogenetic precocity
Nasal and oral chemoreceptive systems begin their functional activity early along a non-random
temporal sequence of sensory ontogeny, which is well conserved among vertebrates (Gottlieb
1971). Olfactory and gustatory functions begin somewhere between the onset of somesthesis and
that of kinesthesis, and well before audition and vision (Lecanuet and Schaal 1996). In humans,
the olfactory system shows adult-like ciliated OSNs by the 11th gestational week. Taste buds can
be found as early as gestational week 12, and are morphologically mature by gestational week 13
(Beidler 1961). Nasal and oral trigeminal subsystems do support functional activity even earlier
than olfaction and taste, as they appear by gestational week 4 and respond to touch stimulation
by gestational week 7. Thus, human nasal and oral chemoreceptors undergo an anatomical devel-
opment compatible with sensory function from early gestation, although perceptual processing of
sensory inputs may arise later in the last trimester (Schaal et al. 1995b, 2004). Accordingly, from
this age on, informative odour (and taste) cues can theoretically be detected and transduced to
the brain, and can be perceptually bound together, as well as with other sources of sensation.

2.2.4 The nature of information provided by the chemical senses


The chemical senses potentially mediate a wealth of informative cues that are inherent either to
the stimuli—quality, perceived intensity, variety, complexity—or to the properties derived from
an individual’s idiosyncratic interactions with stimuli—hedonic valence (pleasantness/unpleas-
antness), familiarity/novelty, and utility knowledge or affordances (e.g. edibility, stress-buffer-
ing). These properties are interrelated, quality being indeed linked with intensity, and hedonic
valence with intensity and familiarity (Cain and Johnson 1978; Delplanque et al. 2008). Humans’
initial reactions when facing an odorous object or context, whether autonomic, behavioural, or
declarative, pertain to their pleasantness (Schiffman 1974). Such reactions are not symmetric for
pleasant and unpleasant odours or flavours, the latter being treated more rapidly than the former
(Bensafi et al. 2002). Many of the above properties, namely the hedonic and familiarity properties,
have been shown to be attended to keenly from the earliest stages of development (Schaal 1988,
2006; Soussignan et al. 1997; Steiner 1979; Steiner et al. 2001) and even in premature infants
(Pihet et al. 1996, 1997; Schaal et al. 2004). To a certain extent, chemoreception can also carry
spatial cues, although adults are generally considered poor at localizing based on odour alone and
trigeminal sensation seems mainly involved in laboratory experiments (Kobal et al. 1992). As in
adults, trigeminally-driven orientation responses are observed in newborns for offensive stimuli
(Rieser et al. 1976). But ontogenetically adaptive localizing responses based on odour cues may
be proper to the neonatal period. Indeed, infants display reliable spatial performance in localizing
an odour source of low intensity, and this may be improved by bilateral sampling movements
34 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

(rooting behaviour) within odour gradients to eventually bring the nose (and the mouth) in con-
tact with the target breast and nipple (see below, Section 2.4).

2.2.5 The role of the neural architecture of olfaction in


multisensory processing
There is a close structural overlap between brain regions related to olfaction (and taste), and those
regions involved in the affective processing of incoming stimuli. Primary and secondary projec-
tions of olfaction (namely anterior olfactory nucleus, olfactory tubercle, piriform cortex, anterior
nucleus of the amygdala, periamygdalian cortex and entorhinal cortex, hypothalamus, hippoc-
ampus, and orbitofrontal cortex (Price 1990; Carmichael et al. 1994) are indeed known to orches-
trate endocrine, autonomic, visceral, emotional, associative, and memory functions (Gottfried
2006; Rolls 2005). Thus, at any stage in development, odours carry strong intrinsic abilities to
evoke instant affects and to re-evoke affect-laden memories long after they have been encoded
(Yeshurun and Sobel 2010). Olfaction is thus in a position to integrate and associate with inputs
from the visceral field that are related to homeostasis or illness, at the same time as memories
from previous experience, emotion, feelings, and knowledge supported by higher cognitive
competences.

2.2.6 Modularity of the chemosensory systems


The chemosensory systems appear to be ‘modular’, and such functional modules appear to follow
a heterochronous ontogeny. This is exemplified in gustation, where distinct ‘submodalities’ sense
sweet, bitter, sour, savoury, and salty stimuli. All of these prototypical taste modes are operative
since birth, and even before, with the seeming exception of saltiness, which appears to control
responses later in development (Beauchamp et al. 1986). In olfaction, evidence from rodents
suggests that the OBs are built as assemblies of heterogeneous neural subsystems or functional
modules that process distinct chemical compounds (e.g. Johnson and Leon 2002; Kobayakawa
et al. 2007). The developmental dynamics of such modules show that glomeruli in the OBs are
progressively but rapidly specified in early development. In the rat pup, odour activation recruits
only a limited set of bulbar zones during the first 3 postnatal days, but bulbar activation increases
to nearly adult levels by day 15 (Astic and Saucier 1982). Thus, functional modules in the OB are
heterochronous, the earliest being presumably linked with OSNs that become functionally mature
in advance of others. Such early-developing modules may be caught-up in the processing
of stimuli mediating vital responses in newborn organisms. This order of functional onset of
modules in the OB is analogous to the ordered onset of sensory systems noted by Gottlieb (1971),
which Turkewitz and Kenny (1985) theorized as an evolved strategy to open the brain progres-
sively to sensory activity, thereby reducing the nature, amount, and complexity of stimuli availa-
ble to maturing neural tissue and regulating competition between emerging sensory systems. This
same logic may be applied intra-modally: a limited set of early functional processing modules
may prevent informational overload in the modalities that support the completion of actions that
are critical to neonatal adaptation and survival. In summary, apparent sensory specializations in
olfaction might emerge from time-ordered neural development, leading certain chemosensory
stimuli to be more readily engaged than others in the control of behaviour, and in uni- and mul-
tisensory learning.

2.2.7 Early functions of olfaction


The first function of olfaction and taste is to divide the world into entities that should be
approached and those that should be avoided, before and after oral incorporation, respectively.
SOME SPECIFICITIES OF OLFACTION IN HUMAN DEVELOPMENT 35

Chemoperceptual abilities support such discrimination from birth. Some chemosensors are con-
genitally connected with distinctive motor-action patterns in the oral–facial field. For example, in
human newborns, the taste of glucose, sucrose, or umami (monosodium glutamate) elicit suck-
ing and smacking, tongue protrusion, and a relaxed facial configuration with smiling-like actions.
In contrast, bitter or sour stimuli trigger gape and gag responses with grimaces, negative head
turns, and crying (Rosenstein and Oster 1990; Steiner 1979; Stirnimann 1936). These responses
occur even in infants born prematurely, who may not yet have been directly exposed to such
stimuli (Maone et al. 1990; Tatzer et al. 1985). Comparable unconditional reactions seem rarer in
the olfactory domain, although newborns display hedonically-differentiated responses to odor-
ants before they had opportunities for extensive postnatal odour learning (Steiner 1979 ;
Soussignan et al. 1997). One case is the rabbit pup’s response to a single odour compound emitted
in milk by the nursing female. This rabbit ‘mammary pheromone’ is behaviourally active appar-
ently without inductive specification by postnatal or prenatal exposure with the same compound
(Schaal et al. 2003). So far it is not known whether such unconditioned odour stimuli do exist in
humans. Such ‘ready-to-use’ stimulus-response loops have been termed prefunctional (Hogan
1998) or predisposed (Bolhuis 1996), in order to highlight the fact that they work in newly born
organisms in advance of functional demands from the environment and without the intervention
of obvious environmental processes (direct sensory exposure) to induce the stimulus–response
association (Schaal 2012). Interestingly, such predisposed chemosensory stimuli also often act as
potent primary reinforcing agents, which transfer their behavioural effect to any contingent, ini-
tially neutral, stimulus (e.g. Booth 1990; Coureaud et al. 2006).

2.2.8 The effect of environmental exposure on


chemosensory structures
Apart from some notable predisposed mechanisms, chemosensory structures and their func-
tional performances are massively influenced by environmental exposure effects and learning
(reviewed in Schaal 2012). The fine-tuning (through neuronal selection, inter-neuronal connec-
tivity, neoneurogenesis) of the sensory organs, connecting centres, and corresponding motor
loops have been shown to depend strongly on prior sensory impact by deprivation or selective
enrichment experiments. These epigenetic influences may be maximized during sensitive periods
of neurosensory ontogeny. However, it is a general property of olfaction (as well as of taste) to be
extremely susceptible to the local conditions of the environment. This plasticity has been docu-
mented both at the peripheral and central levels, where odour exposure impinges on sensitivity
(Wang et al. 1993) and where learning and expertise ameliorate discrimination (Bende and
Nordin 1997; Rabin 1988). The processes underlying odour learning range from non-associative
familiarization (e.g. Li et al. 2006) through to associative processes, such as those mobilized in
evaluative or classical conditioning (Hermans and Baeyens 2002; Li et al. 2008). As for other brain
functions, the plasticity of olfaction, although generally high, may not be linear during develop-
ment. ‘Sensitive periods’ can indeed augment the influence of environmental exposure to and/or
learning of odours and flavours. Such sensitive periods have been described during the neonatal
period when the acquisition of arbitrary odours appears to be facilitated in both non-human and
human subjects (Delaunay-El Allam et al. 2010; Leon et al. 1987). But, more broadly, early infancy
represents a period when the range of odour/flavour preferences first become established
(Beauchamp and Mennella 1998). Finally, although research on experience-dependent plasticity
has been mainly concerned with odour–odour or odour–taste processes, it is argued that wider
integration of olfaction with other sensory modalities also depends largely on experience
(e.g. Spear and Molina 1987).
36 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

2.2.9 Subliminal processing of odour stimuli


Olfactory thresholds are highly variable among individuals (Cain 1977), but odour stimuli can be
processed even when they are unnoticed or delivered subliminally. Odours can indeed be proc-
essed subconsciously and affect numerous psychobiological variables in adult humans, including
for example endocrine release (e.g. Wyart et al. 2007), mood fluctuations (e.g. Bensafi et al. 2004;
Lundstrom and Olsson 2005), cognitive performance (e.g. Kirk-Smith et al. 1983; Köster et al.
2002; Zucco et al. 2009), visual exploration (e.g. Michael et al. 2003; Seigneuric et al. 2011), or
likeability assessment of others (e.g. Li et al. 2007). It is interesting to note that the influence of
implicit odour stimuli on memory or on social preferences has been found to be more effective
when participants report that they are unaware of any odour stimulus (Degel and Köster 1999).
These results suggest that odours can exert strong memory and behavioural effects outside
of conscious notice, a point that should be borne in mind in order to understand the subtle influ-
ence of olfaction in infants’ and children’s attitudes and behaviour, as discussed below.

2.2.10 Persistence of odours in memory


Once encoded, odours generally appear to be more persistent in memory than cues encoded in
the other sense modalities (Engen 1991; Herz and Engen 1990). Adult olfactory memory is indeed
only slightly influenced by passing time either in the short-term or in the long-term, and this
stands in contrast with what is observed in vision and audition (Engen and Ross 1973; Engen et al.
1973). Furthermore, the acquisition of olfactory associations appears strongly affected by proac-
tive interference, but only negligibly by retroactive interference, leading the organism to give
more weight to experiences encountered for the first time (Lawless and Engen 1977; Engen 1991).
These fundamental properties have great importance in understanding the developmental roles
of the olfactory sensorium. The early functionality of olfactory memory is exemplified in the
neonatal retention of foetal odours (Abate et al. 2000; Hepper 1995; Faas et al. 2000; Mennella
et al. 2001; Schaal et al. 2000), infantile retention of neonatal odour imprints (Delaunay-El Allam
et al. 2010; Mennella and Beauchamp 1998), and in adults’ reliance on early odour memories
during biographic recollections (Chu and Downes 2000; Willander and Larsson 2007). For
instance, adults can name odours they have not re-encountered since childhood (Goldman and
Seamon 1992) and can display aversions to odours that were negatively conditioned in childhood
(Batsell et al. 2002). Indeed, the profile of their olfactory likes and dislikes (Haller et al. 1999;
Teerling et al. 1994) or differential brain responses to olfactory cues (Poncelet et al. 2010) can be
traced back to early exposure to characteristic ethnic flavours. Such autobiographical memories
are typically composed of sensory percepts in a range of modalities, but odour-based representa-
tions are better remembered and for longer than vision- and word-based representations
(Chu and Downes 2002; Herz 2004).

2.2.12 Links of olfactory percepts with language


The links of olfactory percepts with language are weak. Firstly, odours are poorly represented in
languages around the world and, when they are, local terms are mainly devoted to odours bearing
unpleasant connotations or representing potential harm (e.g. Boisson 1997; Mouélé 1977). Thus,
as already noted, the usual way to label an odour percept is first in the realm of binary hedonic
appreciation. When precise identification is required, such as an accurate name, humans per-
form, on average, very poorly (Engen 1987). However, when words or icons are provided as cues,
or when odour–word links are taught, odour identification is greatly improved, suggesting that
the odour-related verbal deficit resides in great part in memory access or retrieval limitation.
However, this lexical deficit regarding odours does not necessarily imply a systematic semantic
CAN ODOURS BE DRAWN INTO MULTISENSORY PROCESSES IN THE FOETUS? 37

deficit. For example, when confronted with a range of odours from hazardous household prod-
ucts, children gave the correct name to only 15% of them but accurately rated their edibility in
79% of cases (de Wijk and Cain 1994). Thus, although linguistic competence facilitates odour
discrimination and categorization (e.g. Möller et al. 2004), there is no absolute necessity to master
language in order to make sense of the odour and flavour worlds. Thus odours, flavours, and
tastes can mediate multiple and sophisticated affordances regarding the environment in preverbal
stages of human development. Odours or flavours are mainly encoded as associative properties
of objects or contexts: they are accordingly verbalized in reference to these objects or contexts,
and not in terms of attributes that are abstracted from objects as would be the case with colours.
This basic associative character of olfaction has already been noted above in relation to the elabo-
ration of biographical memories, and it will be further highlighted below in our discussion of the
pervasive involvement of odours in multisensory learning.
We conclude from the above points that the involvement of the chemosensory modalities in
early sensory, emotional, cognitive, and behavioural development is multi-determined. Although
some odour-specific perceptual specializations may emerge uninfluenced (or minimally influ-
enced) by experience, the bulk of our chemosensory responses are predominantly canalized by
experience from the earliest stages in ontogeny. Such effects of experience are persistent through-
out the lifespan. Thus the young organism can be considered to be both a perceptual specialist,
able to attend to particular chemosensory stimuli that have survival value, and a skilful generalist,
prone to react to and learn any novel odour stimulus in the environment linked with beneficial or
detrimental consequences.

2.3 Can odours be drawn into multisensory processes


in the foetus?
The developmental precocity of the chemical senses makes possible months of epigenetic influ-
ence created by the pregnant mother and the growing foetus itself. Mammalian foetuses, includ-
ing human ones, are indeed exposed to an amniotic ecology that is replete with chemosensory
agents (Schaal et al. 1995; Robinson and Mendéz-Gallardo 2010) that nasal chemosensors can
detect and which connected brain structures can process, as shown by unimodal testing in animal
foetuses (Schaal and Orgeur 1992; Smotherman and Robinson 1987, 1995). It has been estab-
lished that foetuses can process chemosensory information by testing neonates with chemical
compounds they could have encountered only in the womb. In utero, both olfactory and gusta-
tory chemosensors are probably stimulated, but subsequent testing in aerial conditions shows
that the olfactory percept is sufficient to explain detection/retention by the foetal brain. Thus,
when presented with the odour of their amniotic fluid as compared to that of the amniotic fluid
from another foetus, 3-to-4-day-old newborns exhibit a preference for the familiar amniotic fluid
(Schaal et al. 1998). Along similar lines, human foetuses, much like foetuses in other mammalian
species (e.g. Hepper 1988a; Schaal et al. 1995a; Smotherman and Robinson 1995), are able to
extract a variety of odorants transferred in utero from the mother’s diet and which they can retain
in memory for several days, and up to several months, after birth (Hepper 1995; Faas et al. 2000;
Mennella et al. 2001; Schaal et al. 2000; cf. Schaal 2005, for a review).

2.3.1 The associative nature of foetal learning


Do events exist in utero that can create contingent relationships between stimuli captured through
different sense modalities, and is the foetal brain able to detect such crossmodal relationships? Let
us begin with the impact of fortuitous events imposed by experimenters on the foetal acquisition
of co-occurring odour cues. The induction of a visceral malaise is known to create, in a single
38 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

pairing episode, a strong aversion to any associated smell in adult humans and animals (Bernstein
1991), as well as in neonatal animals (Rudy and Cheatle 1977). Accordingly, this associative
ability was used to demonstrate foetal chemoreception in the first place. This demonstration
consisted of causing a digestive malaise in the rat foetus by the intraperitoneal injection of lithium
chloride (LiCl) immediately after a flavouring agent (apple) was infused in utero on the last ges-
tational day. In tests conducted 2 months later, the animals exposed to the apple–LiCl association
as foetuses avoided drinking apple-flavoured water more than did the control groups, who were
exposed to either the malaise alone, the apple flavour alone, or to a sham treatment. This result
demonstrated that the rat foetus can associate an arbitrary chemical cue made contingent on a
state of sickness, and that an enduring aversive representation of the odour can be persistently
stored beyond birth (Smotherman 1982; Stickrod et al. 1982).
The rat foetus is also responsive to psychobiological alterations that constitute everyday occur-
rences in prenatal life, such as transitory fluctuations of internal or external conditions. For
example, the incidental compression of the umbilical cord caused by foetal posture can result in
brief periods of hypoxia, inducing motor activity that contributes to the relief of the mechanical
cause of hypoxia (Smotherman and Robinson 1988). Such physiological variations can engage
rapid differentiation in the salience of coincidental stimuli. This has been demonstrated by
clamping the umbilical cord in order to mimic brain oxygenation upheavals. The results showed
that an odour infused in utero in association with decreasing brain oxygenation induced by
clamping the cord takes on an aversive value, whereas the same odour made contingent with the
release from hypoxia after unclamping the cord takes on an attractive value in the foetal brain.
This differentiation appears stable as it is demonstrable in 12 day-old rats (Hepper 1991a, 1993).
Thus the foetal brain has (in rodent foetuses at least) the ability to olfactorily tag somatic or brain
states through the sensing of undefined interoceptive cues caused by physiological events.
Other sources of intersensory contingencies that might engage the chemical senses are nor-
mally iterative in the typical foetal environment. For example, near-term, human foetuses exhibit
bursts of pseudo-respiratory movements, whereby amniotic fluid is propelled through the foetal
mouth and nasal passageways (Badalian et al. 1993). Continuous recording of these foetal respira-
tory movements using ultrasonic real-time visualization has shown that they are organized in
cycles (Patrick et al. 1980) closely related with maternal meals and subsequent variations in blood
glucose (Patrick et al. 1980, 1982). Maternal glucose transfer to the foetus most probably parallels
the transplacental transfer of chemosensorially-effective metabolites (aromas, flavour agents)
although the relative transfer kinetics of the different compounds is unknown. Thus, at each
of the pregnant mother’s meals, a foetus may be exposed to more or less coincident inflows of
odorous or flavoursome metabolites and nutrients, which at the same time may affect its sensory
processes and its arousal and physiological state. Hence the foetus might be able to learn that such
chemosensorially-efficient metabolites co-occur with some kinds of interoceptive changes in
brain or in somatic states.
Finally, another instance of ecological context promoting the linkage between chemosensory
inputs and inputs from the other senses occurs during the transition from foetal to neonatal
stages of development. Based on the observation that exposure to uterine contractions during
labour was predictive of successful nipple attachment in neonatal rats, Ronca and Alberts (1996)
compared the response to a nipple scented with lemon flavour (citral) in a group of pups exposed
to it concurrently with bodily compression mimicking labour contractions and in a group of pups
exposed to it without compression. The results showed that 88% of compressed pups seized
the scented nipple as compared to only 20 % of non-compressed pups. Thus somesthesic,
kinesthesic, and proprioceptive stimuli that simulate uterine contractions induced the learning
CAN ODOURS BE DRAWN INTO MULTISENSORY PROCESSES IN THE FOETUS? 39

of a contingent odour stimulus that promoted later rat pups’ oral responses. The underlying
processes have been suggested to reside in a state of generalized arousal and neurochemical cor-
relates which are triggered by head compression, and which promote the acquisition of concur-
rent stimuli. Facilitated odour-learning related to the type of birth has also been suggested in the
human case, where the impact of a 30-minute exposure to an odour differs after vaginal delivery
and related contractions, or after caesarean delivery, which take place in very different proprio-
ceptive and tactile conditions (Varendi et al. 2002).
Taken collectively, all above studies suggest that the foetal sensing of an organism’s own physi-
ological state and the consequences for arousal could provide favourable conditions for the
encoding of co-incidental stimuli, namely odours and flavours. Such conditions constitute a
functional ground to shape early intersensory equivalence in terms of psychophysical properties
or affective meaning of temporally coincident stimuli in any sensory modality. For example,
as foods eaten by pregnant women may change in composition each day, foetuses may record
several odour/flavour cues in association with metabolic changes, leading them to learn a range of
stimuli that will elicit equivalent attractiveness when reencountered in the neonatal environment.
Otherwise, studies have suggested that foetuses can acquire cues that are contingent with a mater-
nal behaviour state, such as the distinctive musical tune of a popular television program acquired
while the mother relaxes watching it (Hepper 1988b). As compared to non-exposed foetuses,
those exposed to this music throughout pregnancy display selective motor activation upon its
playback when tested in utero at gestational age 36–37 weeks, and a discriminative heart-rate
and motor response when tested as newborns 2–4 days after delivery (Hepper 1991b). Strong
conclusions cannot be drawn from this research, as it may have multiple interpretations. Foetuses
may have learned the iteratively played auditory stimulus, or they may have associated the
external sound with the relaxed, silent state of the mother; still another possibility involves a
third component: the television-watching mothers were in the habit of ‘settling down with a cup
of tea’ (Hepper 1988b, p. 1348) and hence introduced chemosensory cues to the unfolding
sequence of sensory events whilst watching television. The contingent exposure to regular flavour
(or psychostimulant) cues may serve in making the distinct melody even more salient and
may also have given rise to an expectation of the melody (or conversely, the melody may give
rise to a chemosensory expectation). Although the relative validity of the above interpretations
cannot be determined from the available data, this study points to a potentially suitable ecological
context within which to assess multisensory, including chemoreceptive, learning in the human
foetus.

2.3.2 Can foetuses extract supramodal properties from


odour stimuli?
Another issue concerns whether foetuses can extract supramodal properties, i.e. informative cues
that are common to several sensory modalities. It is generally considered that mammalian
foetuses dwell in an environment where highly intense stimuli are filtered out. Thus, the sensory
flow to which they are ordinarily exposed can be characterized in all sensory modalities as being
low-to-moderate in intensity (Lecanuet and Schaal 1996). Specifically, odorants are probably
presenting at very low concentrations to the developing chemosensors, as attested by the ratio of
the amount of aromas that reach the amniotic fluid to the amount that a pregnant mother takes
in. For example, when mothers ingested capsules filled with a gram-wise quantity of powdered
cumin in the days preceding parturition, only traces of molecular markers of cumin aroma could
be chromatographically detected in the amniotic fluid (Schaal 2005). Nevertheless, such trace
amounts of odorants can be detected by the untrained adult nose (Mennella et al. 1995; Schaal
40 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

2005; Schaal and Marlier 1998) and by the foetus itself, as inferred by the fact, mentioned above,
that neonates appear to remember those odours that they have encountered prenatally. Thus a
general intensity attribute occurring in the womb context might be olfactorily extractable by the
foetal brain. Such exposure to generally low-intensity stimuli during the prenatal formative
period of perceptual processing may, in part, explain the pattern of intensity-based responsive-
ness of neonatal and infantile organisms. Schneirla (1965) proposed that early reactions to stimu-
lation are fundamentally dichotomous. He argued that low-intensity tactile/thermal stimuli elicit
approach responses whereas high-intensity stimuli elicit avoidance responses and corresponding
patterns of autonomic activity. This hypothesis has been empirically addressed for audition and
vision (Lewkowicz and Turkewitz 1980; Lewkowicz 1991) but not yet for olfaction. It may be
predicted that suprathreshold odorants will be more attractive (or less aversive) to neonates
when presented at attenuated intensities. Furthermore, intensity-based equivalences between
modalities could potentially be assessed by contrasting odour stimuli that have been intensity-
matched to stimuli in other modalities versus odours that are mismatched in intensity.

2.4 Intersensory involvement of olfaction in the newborn


2.4.1 Neonatal chemosensory integration in action
The olfactory processes that operated in the foetus are further solicited in the newborn infant,4
binding with a wider range of sensory inputs, response resources, and feeding into growing
memory abilities. Unisensory investigations of olfactory responsiveness in newborns indicate that
mammalian, including human, neonates are born with olfactory abilities that are strongly con-
nected with action systems from birth (see reviews by Alberts 1981, 1987; Blass 1990; Ganshrow
and Mennella 2003; Rosenblatt 1983; Schaal 1988, 2005, 2006).
An infant’s chemosensorially-guided actions indeed constitute an important component of the
future integration of multisensory experience. Olfaction directs sensory-motor performance in a
number of different ways (cf. Fig. 2.2). Firstly, odours are potent regulators of activational states in
inhibiting high-arousal states and in promoting the advent of calm awake states (Doucet et al.
2007; Schaal et al. 1980; Sullivan and Toubas 1998). Such state control is followed by heart rate
stabilization, the reduction of respiratory variability, the mobilization of low amplitude move-
ments, cephalic alignment, and often eye-opening and ‘gazing’, and the activation of oral and
lingual movements. All of these variations are indicative of autonomous orientation, interest,
attention, and attraction. In olfaction, stimulus sampling is expressed as sniffing, i.e. a respiratory
pattern that optimizes the airflow over the nasal mucosa. Sniffing varies with the quality and
intensity of odorants and their hedonic value in adults (Bensafi et al. 2005). Although it does not
yet operate on a volitional basis, three-day-old newborns also adjust their nasal airflow depending
on whether it carries a pleasant/unpleasant odour quality (Soussignan et al. 1997).
Secondly, more active odour-guided responses can be seen in orienting movements of the head.
A two-choice test first developed by Macfarlane (1975) and perfected in subsequent studies
(Cernoch and Porter 1985; Schaal et al. 1980, 1995c, 1998; Schleidt and Genzel 1990) capitalized
upon odour-released head movements towards bilaterally-presented pairs of stimuli in newborns
laying supine in their crib or sitting upright in a seat (cf. Fig. 2.2). Using this kind of test, several
laboratories have analyzed neonatal responsiveness towards odours acquired in the maternal
ecology, either prenatally or postnatally in the context of nursing (see below).

4 The human neonatal period is considered here as corresponding to the first month after birth.
INTERSENSORY INVOLVEMENT OF OLFACTION IN THE NEWBORN 41

A C

B F

Fig. 2.2 Different ways and devices used to analyse infant behavioural responsiveness to odours
presented unimodally. (A) Odorants are sequentially presented on Q-tips to assess differential
oral–facial responses or corresponding autonomous responses (e.g. Steiner 1979; Soussignan et al.
1997; Doucet et al. 2009). (B) Responsiveness of infants directly exposed to the mother’s breast
after different odour-related treatment (e.g. Doucet et al. 2007). (C–D) Paired-odour test devices
allowing researchers to assess infants’ relative head orientation and general motor responses while
laying supine (C, Macfarlane 1975; D, e.g. Schaal et al. 1980). (E) Paired-odour choice test for the
assessment of differential head-turning and oral activation toward either odour stimulus (e.g. Schaal
et al. 1995c; Delaunay-El Allam et al. 2006). (F) Differential rooting or crawling movements of the
infant toward a target odour source (e.g. Varendi and Porter 2001) (Drawings: A–E, © B Schaal; F,
redrawn after Prechtl 1958).

In sum, the action systems and sensorimotor configurations released by olfactory cues may
have multiple, cascading consequences on intersensory processes in newborns. When an infant
turns its head in a given direction in response to an odorant, usually he or she is also exposed
to other kinds of stimuli associated with the target object or person in a given context. This leads
to opportunities to create novel associations or to update old ones, as well as to educate spatial
representation of the body and embodied representation of self, or the actions of self, in space
(Aglioti and Pazzaglia 2010; Bremner et al. 2008; Chapter 5 by Bremner et al.).

2.4.2 Nursing and other social contexts that


‘intersensorialize’ odours
The prototypical context of mother–infant exchanges in any mammalian species is nursing.
The transfer of the multiple biological benefits of milk goes on within a concert of sensory
influences. All kinds of sensory inputs are then available and, in theory, all kinds of multisensory
events can be selected by the neonatal brain. The manifold sensory, motor, and reward-related
42 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

elements of the nursing context provide a repertoire of cues towards the recognition that
certain events recur in a similar context (with the same person) with similar consequences.
Nursing therefore provides a potent contextual basis that pulls sensory stimuli into a multisen-
sory perceptual framework.
This process has been well analyzed in the rat where passive (warmth) and active tactile stimuli
from the nursing female and etho-physiological events linked with milk intake have been shown
to be causal in assigning attractive value to any arbitrarily associated odour stimulus (e.g.
Rosenblatt 1983; Alberts 1987; Blass 1990; Brake 1981). This bimodal correspondence between
tactile and olfactory inputs is acquired even more efficiently when the odour has previously
gained some predictability through prenatal exposure (Pedersen and Blass 1982). Although the
database is currently meagre regarding the human newborn, converging evidence exists to suggest
that the rate of contingent odour-nursing exposure affects the development of odour preference.
A human mother’s breast odour elicits increasingly reliable positive head orientation in breastfed
infants as a function of suckling experience. By six days of age, infants can reliably orient their
head to the mother’s breast odour (Macfarlane 1975; but less demanding behavioural variables
led to discriminative responses from postnatal days 2 or 3—see Schaal et al. 1980; Doucet et al.
2007). Likewise, nursing-related exposure to maternal stimuli can help to explain why it is that
15 day-old breast-fed infants, but not bottle-fed infants, are able to recognize their mother’s
(axillary) odour (Cernoch and Porter 1985). Even arbitrary odorants can be engaged in the
approach and appetitive responses of neonates after their recurring association with nursing
(Schleidt and Genzel 1990; Delaunay-El Allam et al. 2010). After they were exposed to a chamo-
mile odorant painted on the breast at each feed, three-day-old infants oriented their head (nose)
towards the odour more than to a paired scentless control stimulus (using the test method pic-
tured in Fig. 2.2; Delaunay-El Allam et al. 2006). However, when the chamomile odour was paired
with the odour of human milk in this two-choice test, both stimuli proved to be equally attractive,
indicating that different odour qualities associated with the same context of reinforcement can
thereafter share similar hedonic value (Delaunay-El Allam et al. 2006).
Other developmental niches also promote the associative linkage of odorants with stimuli in
other modalities, such as non-suckling contact comfort and, in nonhuman newborns, huddling
with siblings. Regarding olfaction, the influence of such tactile stimulation on olfactory learning
has begun to be characterized. For example, Brunjes and Alberts (1979) demonstrated that in rat
pups odours gain attractive meaning in both suckling and non-suckling interactions with the
mother. They suggested that temperature was instrumental in this process. When human infants
are exposed to episodes of gentle massage (for ten 30-second periods) together with lemon odour
on postnatal day 1, the odour elicits positive head turning responses when presented separately
the following day (Sullivan et al. 1991). Control groups of infants exposed either to just the
massage, just the odour, or to the odour followed by the massage did not exhibit any differential
response to the lemon odour. It is noteworthy that the touch-then-odour contingency
was required for the learning of an unfamiliar odour to occur and that the odour-then-touch
condition was unsuccessful in the one-day-old infant. This order effect of stimulations may be
related to differences in the arousing properties of touch versus olfaction. Touch-related bodily
sensations may be more efficient in mobilizing attention by their alerting and/or pleasurable
properties, whereas an unfamiliar odour may not appear to induce a similar excitatory/hedonic
effect. Thus a hedonically neutral odour can change into an alerting or attractive stimulus after a
short association with pleasurable massage. However, whether a familiar or pleasant odour can
conversely change the meaning of a tactile stimulus remains to be tested.
So far, no experiment has addressed the effect of visual or auditory stimulation on the functioning
of the chemical senses in newborn infants. Of course, this does not mean that there is an absence of
INTERSENSORY INVOLVEMENT OF OLFACTION IN THE NEWBORN 43

interaction. However, there are a number of studies that have documented effects in the converse
direction, namely influences of the chemical senses on other modalities. Taste has been demon-
strated to have strong modulating potency on neonatal responses to painful stimuli (Blass et al.
1990). An ‘analgesic’ effect similar to that of sweet taste has also been observed with olfactory
stimuli. When, on the occasion of a heel-prick procedure, newborns were exposed to the odour
of their mother’s milk (when separated from their mother) or to a non-maternal but familiar
odour, the behavioural manifestations of pain were attenuated as compared to infants receiving
an unfamiliar odorant or water (Rattaz et al. 2005). The effect of an olfactory stimulus on the
reduction of pain reactions is more efficient, however, when it is used additively with tactile con-
tainment of the whole body (achieved by swaddling) (Goubet et al. 2007). The pacifying effect of
a familiar odour has also been noted in premature infants (Goubet et al. 2003), indicating that an
early-developing process of recognition of olfactory recurrence is a key factor in alleviating the
response to negative or noxious stimuli. Interestingly, all sense modalities seem not to be equiva-
lent in their ability to block pain afferents or efferents (e.g. audition: Arditi et al. 2006; vision:
Johnston et al. 2007), and olfaction and taste may bear special properties in this respect because
of their precocious (Johnston et al. 2002) and privileged connections with reward processes
(Anseloni et al. 2005, 2002; Pomonis et al. 2000).
Finally, olfactory stimuli might also modulate visual activity. When exposed to their mother’s
breast odour immediately before nursing, awake and calm newborns display longer episodes of
eye opening as compared to a similar situation where the odour is masked (cf. Fig. 2.3; Doucet
et al. 2007, but this is significant for boys only). The processes underlying such olfactory–visual
interaction may reside in the facts that:
◆ a familiar odour stimulus is arousing and, hence, stimulates all-purpose sensory seeking
activity, including visual orienting;
◆ breast odours are already associated with expectancies for visual/auditory/tactile/taste
reinforcements, hence triggering the multisensory intake of information (e.g. Korner and
Thoman 1970);
◆ in three-day-olds, breast odour is already part of an organized activity pattern mobilizing
vision and touch/temperature sensing to boost the localization of the nipple.
This latter point is backed up by the second aspect of the observed olfactory–visual interaction
in Doucet et al.’s (2007) study, namely that infants exposed to breast odour when their eyes are
open tend to display augmented oral activity (as compared to the situation when olfaction and
vision are not stimulated or are stimulated separately; cf. Fig. 2.3). This finding, as well as the
additive odour–tactile effect on pain response regulation mentioned above (Goubet et al. 2007),
accords with Bahrick and Lickliter’s (2002) intersensory functioning hypothesis, which states that
‘information presented redundantly and in temporal synchrony to two or more sense modalities
recruits infant attention. . .more effectively than does the same information presented to one
sense modality at a time’ (2002, p. 165; see also Chapter 8 by Bahrick and Lickliter). Here, concur-
rent and complementary olfactory and visual inputs recruit more intensive oro-motor actions
than either of these inputs on their own. In this case, the focus of the actions is primarily aimed at
orienting to the mother and grasping of the nipple.
In sum, at the start of postnatal life, olfaction, so far mostly understood unimodally, supports
adaptive responsiveness. One prominent goal of adaptive behaviour in any mammalian newborn
organism is to acquire milk and to rapidly increase its ability to acquire it at the minimum of cost.
There is a considerable urge for human newborns to ingest colostrum and milk to counteract the
dangers of bacterial predation (Edmond et al. 2007a, b). Thus, any perceptual means that can help
speed-up neonatal performance in the maternal environment can only be beneficial for survival.
44 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

Relative duration of eye opening Duration of oral activation (sec.)


0.6 14
B C
12

0.5 10
8
6
0.4
4
2
0.3 0
Breast odour No odour Eyes Open Closed Open Closed

Odour Present Masked


Fig. 2.3 Odour–vision interaction at the breast. (A) The testing situation: awake 3–4 day-old infants
held in a cradle prior to a feed are exposed to their mother’s breast without contact; the breast is
either uncovered or covered with a plastic film to mask its odour (Photograph: Sébastien Doucet).
(B) Relative duration of eye opening of infants when exposed to the breast as a function of breast
odour availability (in the no-odour condition, the breast odour was masked). The duration of eye
opening was longer when the infants were facing the odorous breast as compared with the odour-
masked breast. (C) Duration (sec) of oral activation (rooting, licking, sucking) as a function of
olfactory (breast odour present or masked) and visual inputs (eyes open or closed). Longer oral
activation was noted when infants were simultaneously exposed to the sight and odour of the
breast as compared to the other conditions, which are equivalent (Adapted from Sébastien Doucet,
Robert Soussignan, Paul Sagot,and Benoist Schaal, The 'smellscape' of mother's breast: Effects of
odor masking and selective unmasking on neonatal arousal, oral, and visual responses,
Developmental Psychobiology, 49 (2), pp. 129–38, (c) 2007, John Wiley and Sons, with permission.).

Multisensory integration is certainly an essential mechanism in the rapid improvement of neona-


tal performance (Rosenblatt 1983). The observed facts suggest that ‘olfaction may be important
in early behavioural development precisely because it is particularly well suited to mediate the
transition from neonatal responses based upon the intensity characteristics of tactile and thermal
stimulation to those based upon stimulus meaning’ (Rosenblatt 1983, p. 363). This proposal
raised for mammalian altricial (rat, cat, dog) newborns certainly has general validity for other
mammalian newborns regardless of their state of natal maturation.

2.5 Intersensory binding of olfaction in infancy


and early childhood
During the months following the neonatal period, olfaction will be subject to further multisen-
sory optimization with respect to maturing motivation, action, and cognitive systems. During the
first year of life, infants progressively categorize objects, differentiate self from others, delineate
selective attachments, and become capable of producing increasingly complex and intentional
INTERSENSORY BINDING OF OLFACTION IN INFANCY AND EARLY CHILDHOOD 45

actions to explore the physical and biological world, and to communicate with their social
surroundings. What roles does olfaction play in the development of knowledge about the physical
and social environment? How and when is it involved in the dynamic processes that underlie
adaptive cognition, in terms of attention, motivation, learning, memory, preferences, abstraction,
and predictive abilities?

2.5.1 Odours as foreground cues to multisensory events


A small set of experiments indicates that infants and children are adept learners of
multisensory correspondences involving olfaction. Fernandez and Bahrick (1994) studied the
ability of 4-month-old children to pair an arbitrary odorant—cherry—with an arbitrary object.
Following a period during which an object (A) was systematically linked with the odour,
the infants were given a preference test between objects A and B, with and without the
odour. After familiarizing the infants with the appearance of both objects, the previously
odorized object and the control object were presented alternately for two 30-second trials each.
In the test session, the infants looked more at object A in the presence of the cherry odour than
in its absence, showing that they were able to associate an object with a distinctive odour. It
may be noted, however, that this capacity was observed only in female infants, pointing to
the possibility of early sex-related differentiation in the ability to detect contingences between,
or to bind, odours and stimuli in other sensory modes. Using a similar, well-counterbalanced
paradigm, Reardon and Bushnell (1988) served 7-month-old children with apple sauce, flavoured
either sour or sweet and presented in red or blue cups. The infants were then explicitly introduced
to the colour of the cups, and fed alternately from each cup with the contrasting flavours of
apple sauce. After this colour-flavour pairing session, each infant was invited to choose one or
the other cup by arm-reaching to the pair of cups presented at a distance. A significant proportion
of the infants selected the cup associated with the sweet stimulus, a choice that could only rely
on visual cues in the conditions used. Thus, both of these experiments provide evidence
that infants aged 4–7 months are prone to associate arbitrary odours/flavours with co-occurring
visual cues. Indeed, very few pairing trials were needed to acquire the contingency (two 30-second
odour–visual trials in Fernandez and Bahrick’s study and three taste–visual trials in Reardon
and Bushnell’s study). The opposite matching tasks in both studies (odour to visual in Fernandez
and Bahrick, visual to flavour in Reardon and Bushnell) suggest the possibility of symmetric
binding processes. In addition, presenting only the visual cue in the choice task, the Reardon and
Bushnell study indicates that the infants very easily monitored colour as predicting an absent
flavour.
The occurrence of intersensory contingency learning involving biologically-relevant stimuli
further confirms the early associative readiness of olfaction, but with important qualifica-
tions. So far, such associations have been analyzed mostly in the context of ingestion and
social interaction so that one element in the intersensory process is, or concerns, adaptive
psychobiological responses. The domain of feeding is indeed especially suitable to investigate
the integration of chemosensory events with either beneficial or harmful interoceptive
consequences.
In the first case, children exposed to distinct novel flavours in drinks that differ in carbohydrate-
based energetic content express subsequent preference for the flavour of the more energetic drink
over the less energetic drink (Birch et al. 1990). In a replication, where sweet taste was decoupled
from post-ingestive sensation, a similar linkage between novel flavours and monitored interocep-
tive consequences based on post-ingestive nutritional effects were obtained for foods differing in
fat content in children aged 2–5 years (Johnson et al. 1991). So far, the exact nature and locus of
food-related interoceptive cues that associatively bind with flavours remain unclear (Yeomans
2006), and their discussion is beyond the scope of the present chapter.
46 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

The affectively-opposite linkage between flavours and post-ingestive consequences has been
established in conditioned aversion. Children (aged 2–16 years) who are being treated with toxic
chemotherapy exhibit radical changes in the chemosensory appreciation of any associated food
(Bernstein 1978). When given an unusually flavoured ice-cream before drug administration,
the participants exposed to a toxic drug inducing gastrointestinal malaise rejected the ice-cream
2–4 weeks later, as opposed to control children receiving the drug unpaired with the ice-cream or
the ice-cream unpaired with a toxic drug. This aversion persisted for at least 4.5 months after
the initial flavour–nausea pairing. It is notable that positive flavour–interoception associations
mentioned above needed several pairing trials to establish (e.g. eight trials in Johnson et al. 1991),
while Bernstein’s negative flavour–interoception association was specified in a single pairing trial
and remembered over long periods of time. The crossmodal incorporation of chemosensory
stimuli involving interoceptive cues thus appears differentiated in terms of adaptive outcome.
In real-life conditions, these early interactions between negative interoceptive cues and flavour
percepts are highly prevalent and leave persistent memories that inhibit the intake of similarly
flavoured foods over the course of a lifetime (Garb and Stunkart 1974; Logue et al. 1981). But
even stimuli that do not provoke interoceptive malaise but evoke disgust or fear, for example
visual stimuli (e.g. a cute kitten versus the open mouth of a bat), can be enough to durably influence
the hedonic meaning attached to odorants (Hvastja and Zanuttini 1989).
Another essential setting for the establishment of arbitrary relations between multisensory
events involving olfaction is the social environment. Indeed, caretakers and other conspecifics
provide recurrent occasions for an infant to acquire a complex set of cues that characterizes their
appearance (face, eyes, hands, dynamic behaviour) at the same time as their vocal/verbal, tactile,
vestibular, and olfactory features. These perceptual properties of people are idiosyncratically
arranged in time, space, multisensory complexity on different occasions, and dispositions to
interact. Much of the research on infants’ and children’s multisensory perception of these percep-
tual and dispositional properties of people has been into responsiveness to auditory and visual
cues and their relationship (cf. Lewkowicz and Lickliter 1994, and the chapters therein), but there
has been little research on the developmental mechanisms by which olfaction contributes to social
perception (beyond the established fact that odours are part of the cues that mediate person iden-
tification; e.g. Ferdenzi et al. 2010; Mallet and Schaal 1998; Olsson et al. 2006; Weisfeld et al.
2003). Again, olfaction is assumed to operate in human multisensory social cognition in two
ways. Firstly, odours gain meaning as a result of interaction with the multisensory reinforcing
base constituted by the mother and other people. Secondly, by virtue of their reinforcing proper-
ties, odours can precipitate the learning of social stimuli in other sensory modalities.
Human newborns display subtle abilities to recognize significant individuals (mother) or
classes of individuals (lactating women), as inferred from their differential attraction towards
their odour. The reliability of such early recognition abilities is directly linked with the recurrence
of exposure to conspecifics (Macfarlane 1975). Such early socially-acquired olfactory memories
can persist into infancy for months or years. For example, following exposure to an arbitrary
odorant while suckling during the first postnatal days engenders memories that are traceable at
the age of 7 months, and up to 21 months (Delaunay-El Allam et al. 2010). Thus odour cues
acquired in the multisensory context of the mother’s body can be transferred into competent
responses in domains in which the multisensory assortment of the initially-learned sensory cues
is radically dissimilar, such as when interacting with inanimate objects or toys (Delaunay-El
Allam et al. 2010; Mennella and Beauchamp 1998). This also applies to the food domain, where
familiarization to a given flavour in utero or in lacto influences the subsequent appreciation of the
same flavour despite blatant departure of the actual multisensory context (a non-milk food in a
cup or a spoon) from the acquisition context (breast feeding) (Mennella et al. 2001). In this way,
INTERSENSORY BINDING OF OLFACTION IN INFANCY AND EARLY CHILDHOOD 47

stimuli unprecedented in the context of food (e.g. texture, temperature) may gain attention-
evoking properties and affective equivalence with the chemosensory stimulus experienced
earlier.
Olfaction has a manifest role in an infant’s building of multisensory social representations,
although we do not yet fully understand this role. There are multiple ways to assess the influence
of odours in early social cognition. One is to observe the effects of adulterating the previously
encoded odour features associated with a given conspecific, via olfactory masking or suppression.
This approach has been successful in showing the prominence of odours in the interaction with
the mother in non-human altricial infants (Rosenblatt 1983; Blass 1990), as well as in human
newborns (Kroner 1882; Preyer 1885; Doucet et al. 2007). In puppies, kittens, or rat pups, the
perturbation of the maternal (or nest) odour tends to induce restlessness and distress responses
as if the mother (or nest) were not present or not recognized. Similarly, in human neonates,
spreading intense alien odorants on the nipple causes aversion and crying (Kroner 1882; Preyer
1885); simply removing the natural odour of the breast markedly reduces the infants’ responses
that indicate their wanting to grasp the nipple (Doucet et al. 2007).
Little evidence exists of such effects in older human infants, but some interesting data are at
hand in non-human primates. Harlow, in his early maternal deprivation and surrogate-rearing
experiments, deconstructed the multisensory array of cues that female rhesus monkeys convey to
infants. Following this research, Harlow emphasized the importance of comfort contact (Harlow
and Harlow 1965; see also Gallace and Spence 2010), but never seemed to recognize the impor-
tance of the confounded smell infants spread to, and experience on, a cuddly surrogate. The effec-
tive role of olfactory cues in the representation of the mother was noted subsequently in infant
squirrel monkeys (Saimiri). When mothers were sprayed with artificial odorants, their offspring
did not display the typical visual preference for her against a control female (Redican and Kaplan
1978). Thus, in Saimiri infants aged 1–5 months, the olfactory ‘disfiguration’ of the mother was
not compensated for by her visual identity and behaviour, and accordingly might be considered
to have altered the multisensory representation of the mother. A previous experiment had shown
that Saimiri infants rely more heavily on odour properties than on (static) visual properties of
the (anaesthetized) mother (Kaplan et al. 1977). Similarly surrogate-reared Saimiri infants (aged
1–3 months) exhibit a clear recognition of and preference for their own body odour impregnated
in the cloth covering the surrogate, regardless of its visual aspect (colour; Kaplan and Russell
1973). Using the surrogate-rearing paradigm, Kaplan et al. (1977) further explored the infants’
ability to associate a given odour and colour in two groups exposed to two conditions of colour–
odour pairing (green-floral, GF, and black-clove, BC) over the first 6 months of life. Much to the
researchers’ surprise, both groups differed in the salience assigned to the odour or the colour in
choice tests between surrogates contrasting in odour, colour, or both: While the GF group was
more consistent and precocious in choosing the rearing odour regardless of the colour, the BC
group responded conversely—these infants based their choice much earlier on the rearing colour,
and did not care about the odour. As both colours were pilot-tested to elicit equal attraction, this
difference resided in the odours that indeed elicited differential sniffing behaviours. The clove
odorant appeared a posteriori to either induce avoidance or to have pharmacological effects,
which led the Saimiri infants exposed to them to rely on colour in their selective response.
This set of studies is interesting, first because it illustrates that odorants are not easy-to-
manipulate: their multiple impacts (olfactory, but also trigeminal, or even pharmacological) and
dose-related qualitative variations can strongly affect outcomes. Second, despite evidence for an
apparent dominance of olfaction in Saimiri infants, Kaplan et al.’s (1977) study serendipitously
indicates that Saimiri infants concurrently monitor the visual and olfactory properties of
conspecifics. Furthermore, visual cues can compensate when olfactory cues fail in some way.
48 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

Finally, these studies raise important questions about underlying mechanisms responsible for the
effects of odours attached to social relations: Do they have a psychobiological impact by them-
selves in acting unimodally on the pathways that control arousal and distress responses? Or do
they act through higher cognitive mediation, gating the multisensory perceptual gestalt of the
mother or of the self that regulate affective responses? Or, finally, do both types of perceptual loop
come into play in the control of behaviour at different ages or developmental stages? In older
infants (Schaal et al. 1980), children (Ferdenzi et al. 2008), and adults (McBurney et al. 2006;
Shoup et al. 2008) it is clear that the odour of significant others is sought for its calming effects.
But even then, despite easy introspective assessment (in both children and adults), it is not
yet clear what the underlying perceptual/affective processes governing these behaviours are.
Nevertheless, providing the olfactory essence of the mother (or parent) is a common and appar-
ently effective practice, which can be used to manage an infant or child’s affective upheavals
caused by separation or distress. This ‘transitional object’ practice, whereby an odour is (for a
certain time) substituted for the physical presence of a significant other, may be a productive
context in which to empirically explore the emergence and development of multisensory social
processes involving vision, touch, and olfaction, and their affective and cognitive mediation and
consequences.
Thus during early development, odour percepts appear to become integral parts of object or
person representations, as suggested by the fact that altering only the olfactory facet of objects/
persons appears to degrade the recognition of such objects/persons as being familiar. The proc-
esses underlying such multisensory integration of odours are certainly variable at different ages
(e.g. Pirogovsky et al. 2009). They may be facilitated in early developmental stages, when odour
pairing is mandatory during presumed sensitive periods or when chemosensation may have
greater salience relative to vision (Lewkowicz 1988). Suggestive results indicate that when the
odour is novel, no matter whether it is pleasant or unpleasant, the object that carries it appears to
be treated indiscriminately with regard to the object’s visual–auditory–tactile novelty (Schmidt
and Beauchamp 1989, 1990; in 9-month-olds). In contrast, when the object is presumably novel
in terms of visual–auditory–tactile features, and when the odour is familiar, a different outcome
becomes apparent. Mennella and Beauchamp (1998; in 6–13-month-olds) and Delaunay-El
Allam et al. (2010; in 5–23-month-olds) noted that infants sequentially presented with identical
toys differing in odour, one novel and one to which they had been previously exposed during
breast-feeding, explored the object with the familiar odour more. Finally, when neither the odour
nor the object are familiar, 7–15-month-old infants prefer to interact with the unscented rather
than with the scented version of the object (Durand et al. 2008). Interestingly, though, in this lat-
ter object exploration experiment, the scented and unscented objects did not appear to be sig-
nificantly differentiated immediately, but only after several minutes of manipulation and
mouthing. Thus, infants may need some exposure to an object before attending and reacting to
its odour. An alternative possibility is that the odour is perceived immediately, but that the reac-
tions to it are postponed by competing processes, such as the dominance of other sensory systems
mobilized by actions on the object. Thus when interacting with objects, infants’ attentional
resources may be captured to first process the properties that are most immediately meaningful
in specifying the objects’ physical nature and potential affordances. Odours, supposed to have less
predictive value, may thus be treated secondarily. Although odour cues always co-occur with
other object cues, they cannot be as directly ‘observed’ as visual, tactile, or sound cues by infants
and may be less relevant to learning about object function. Therefore, infants may be so engrossed
with the visual and tactual properties of objects that their attention to their olfactory properties
may appear, at first, overcome. However, making odorants more salient by manipulating their
intensity can reverse this effect. When asked to rank by preference four bottles containing four
INTERSENSORY BINDING OF OLFACTION IN INFANCY AND EARLY CHILDHOOD 49

differently coloured flowers associated with four different scents, 3–5-year-old children relied on
colour when the scents where delivered at low intensities, but they relied on odour information at
higher odour concentrations (Fabes and Filsinger 1986).
In sum, the sensory system(s) that is (are) prevalent in the control of attitudes, decisions and
actions might change within the course of a behaviour sequence underlying object exploration.
This corroborates Ruff et al.’s (1992) proposal that children’s exploratory behaviour may be
organized as a succession of habituations in the different modalities involved to the different cues
emitted by an object. Thus, as suggested by Turkewitz (1994), an actogenetic sensory dominance,
i.e. the relative salience of sensory cues that unfolds during the realization of an action, may be
dissociated from the more established notion of ontogenetic sensory dominance.

2.5.2 Odours as background cues to multisensory events


Another set of studies has investigated whether odours diffused as background cues (i.e. as stimuli
that are not directly relevant to the learned stimuli or contingence between them) can be moni-
tored as implicit cues to multisensory events. Numerous studies have demonstrated that adult
humans encode contextual odours, sometimes outside of awareness or of any explicit focus of
attention on them. For example, ambient odours can be encoded as cues to reinforcing events or
outcomes: When paired with a stressful task, they negatively affect subsequent cognitive perform-
ance in adults (Kirk-Smith et al. 1983; Zucco et al. 2009). Such integration of an undetected
contextual odour can even control actual behaviour. For example, when tested in a room suffused
with a citrus odorant that is evocative of cleaning, participants are more prone to respond to
cleaning-related words, to report having the intention to clean when at home, and to exhibit
effective cleaning gestures while eating (Holland et al. 2005). Finally, the presentation of odour
and visual cues engages the tracking of functional links in space, time or common affordances:
When asked to rate whether a set of odours matched with pictures of everyday settings, adults
assign a better fit between an undetected odour (e.g. coffee odour) and a picture that contains a
visual cue linked with that odour (e.g. a coffee cup; Degel and Köster 1999). These effects of back-
ground odours on cognitive orientation or actual behaviour in adult humans have often been
obtained under conditions of unawareness that an odour is being delivered.
There is some evidence that such encoding of unattended background odours also occurs in
early development. One experimental model, the mobile conjugate reinforcement paradigm, has
been developed by Rovee-Collier and her colleagues (e.g. Rovee-Collier and Cuevas 2009), and
has lately been applied to issues concerning the significance of the olfactory background in infant
learning and memory. This paradigm consists in teaching infants to kick a foot in order to move
a mobile suspended overhead and in testing at various subsequent time points the retention of the
action–outcome contingency. They first demonstrated that visual/auditory contextual informa-
tion available during the encoding phase facilitates later recall in 3-month-olds. Long-term reten-
tion or recall degrades considerably along a period of 5–7 days, and it is completely annihilated
when the visual/auditory properties of the encoding context are modified (Butler and Rovee-
Collier 1989; Fagen et al. 1997). In summary, as the vividness of the infant’s memory for the
learned association decreases, the more the context becomes important as a reminder of the con-
tingent response. This applies to the visual or auditory properties of the learning context, but the
olfactory background of learning has also been assessed in terms of its role in infants’ later mem-
ory performance.
When 3-month-olds learned the effect of their foot kicking activity on the movements of the
mobile in the presence of an odour during two 15-minute training sessions, and were tested for
retention 1, 3, and 5 days thereafter, their responses depended on the olfactory context and on the
nature of the odorant. Those re-exposed to the same odour remembered the contingency between
50 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

their kicking and the induced mobile movement well. By contrast, those exposed to no odour
(control) exhibited only partial recall, while those exposed to a novel odour displayed no signs of
recall (Rubin et al. 1998; Schroers et al. 2007). Thus, firstly, infants detect background odours
and appear to rely on them in their retrieval of the learned kicking–object-mobility conjugation.
The differential effect of the three odour conditions on retrieval performance indicates the con-
tribution level of contextual olfactory cues to the representation of a situation dominated by
vision, action, and proprioception: The matched odour facilitates recall of the original context,
the no odour condition was followed by degraded recall, but the non-matched odour was clearly
detrimental to recall. Second, the persistence of the odour cue to the kicking–object mobility
contingence differed as a function of the odour quality. Whereas arbitrary floral/woody odours
(Lily of the Valley or Douglas fir) were not recalled five days after learning, fruity/nutty odours
(cherry and coconut odours) were. This was interpreted as resulting from the meaning of stimuli
for the infants as cues for food. However, one may doubt that 3-month-olds (whose feeding
status is unreported) can assign unprecedented odorant qualities to a functional category of
‘edible’ items, and the effect may rather be sought in some intrinsic properties of the compounds
(namely trigeminal or otherwise potentially aversive features related to their intensity or novelty).
Third, based on these and prior studies, Fagen and his associates (Rubin et al. 1998; Schroers et al.
2007) made the interesting suggestion that, at least in young infants, odour cues might not func-
tion in the same way as visual/auditory cues. These latter cues were interpreted as occasion setters,
i.e. signalling the occurrence of the learned contingency, whereas odour cues may become part of
a compound percept aggregating the mobile and the odour context. When the associated odour
is lacking, the mobile is no longer recognized as such. Thus, according to such an interpretation,
odour + visual + proprioceptive properties of the learning situation may become amalgamated
into a single multisensory representation, which may be acquired holistically by 3-month-old
infants.
This hypothetical process of blending inputs from different sensory modalities into a novel,
emergent representation reminds one of the issues raised above concerning the formation
of multisensory representations of conspecifics (or of the self): are contexts or people perceptually
constructed by the continuously updated additive encoding and storing of information from
different sensory modalities? Do these cues then become, to a certain extent, substitutable for
each other in eliciting attention, recognition, and attraction? The mother’s odour by itself can
thus separately act on an infants’ behavioural state, at least at a certain age, for a certain amount
of time, and in particular situations, in the much same way as the whole mother herself can.
Altering only her odour properties can disrupt the infant’s recognition of her and also induce
avoidance responses until other cues take control. Whether, when, and how odours can function
as cues to representations of people as whole gestalts in infants and children is an interesting area
open to future investigation.
In older children, as noted above in foetuses or newborns, background odours easily become
fused with the multisensory perceptual events that compose an emotionally arousing situation.
For example, five-year-olds were asked to resolve a maze task in the presence of a subliminal
odour for 5 min. But the maze-solving task was impossible, hence producing a feeling of failure
and stress as inferred from the actual behaviour of the participants while undergoing the
task (Epple and Herz 1999). After a distracting interlude, the children were taken to another room
for the completion of an ‘odd-one-out’ test. This room was suffused with either the same odour,
another odour, or no odour at all. The children exposed to the odour previously paired
with frustration obtained lower cross-out scores than those belonging to the no-odour or to the
different-odour groups. In another study (Chu 2008), children aged 11–13 years were selected
according to their academic achievements and only underachieving subjects were enrolled.
INTERSENSORY BINDING OF OLFACTION IN INFANCY AND EARLY CHILDHOOD 51

These children were first introduced into a room suffused with a given scent, where they had
to complete a cognitive task, the difficulty of which was exaggerated to produce a feeling of
unexpected success in association with the odour. Two days later, when these children performed
as quickly as possible ‘odd-one-out’ and ‘same-as’ tests, those re-exposed to the same scent exhib-
ited better performance than those exposed to a different one.
In both of the above studies, the contingency between odour and emotional experience led the
children to mentally label the odour as either negative or positive. What is being associated with
the background smell is unclear. Epple and Herz (1999) suggest a mechanism of emotional
conditioning, whereby a negative (positive) experience induces an emotional reaction that incor-
porates the co-occurring odour; subsequent presentation of that same odour then evokes a simi-
lar emotional reaction and ensuing negative (positive) effects on performance. Chu (2008)
proposes, alternatively, that the intervention of higher-level evaluative processes such as an
increase in self-confidence or self-esteem plays a role. However, one could explain the findings at
a more elemental level as, in the above experiments, odour stimuli were certainly associated with
a set of sensory cues that were differentially attended as a function of the manipulated affect.
Distinct affective experiences can be discriminated on the basis of the contrasting activation
patterns of somatic and visceral effectors (e.g. facial muscles, heart rate, blood pressure, respira-
tion rate, skin temperature, sweating, gut motility, endocrine release, etc.; see e.g. Levenson et al.
1990; Stockhorst et al. 1999). These patterns engender contrasted interoceptive states or feelings
that may function as ‘somatic markers’ (Damasio 1993). Such somatic markers of bodily state
may become paired with co-occurring exteroceptive cues (namely an odour), generating a multi-
sensory image of situations where the sentient self (Craig 2009) takes in the perception of external
events. Ultimately, the sentient self integrates interoceptive representations with environmental,
hedonic, motivational, social, and cognitive activities to produce a global emotional moment
(Craig 2009, p. 67). Such inclusive emotional experience may explain why children’s responses to
given odours can subtly differ as a function of their association with events concerning significant
others. For instance, in families where parents consume alcohol to reduce dysphoria, children
dislike alcohol-related odours more than children whose parents usually drink for convivial
entertainment (in five-year-olds: Mennella and Garcia 2000; in 5–8-year-olds: Mennella and
Forestell 2008). Similarly, children whose mothers smoke to alleviate their stress dislike the odour
of tobacco smoke more than children of mothers who report smoking for other reasons (Forestell
and Mennella 2005). Thus, seeing and empathically feeling beloved others’ emotional distress or
affliction in the presence of an odour changes the hedonic value of the odour. Therefore, our
odour perceptions are not only shaped in association with multisensory perceptions originating
in our own body, but also in reaction to someone else’s emotional state or behaviour. This might
be linked to the fact that seeing pain in others activates the same representations and overlapping
brain structures as when one is directly exposed to physical or social pain (Eisenberg 2006; Singer
et al. 2004).
To sum up, rare experiments examining odours in multisensory processes in infancy and
childhood clearly show that odours are readily integrated into multisensory percepts.
Chemosensory percepts appear to form unitary perceptual events with inputs from other sensory
modalities, regardless of whether these originate from external stimuli or from internal sensations
or emotional states. This perceptual unification is well evidenced in naturalistic social contexts
where, once ‘glued’ to a multisensory representation, odours by themselves provide efficient
cues to evoke in vivid multisensory detail the whole scene in which they were encoded. This view
of fusion of odours into unified multisensory percepts is also valid for food objects. Not only
are olfaction and taste engaged in synesthesic processes due to their unavoidable contingency
in eating (cf. Section 2.2), but they become combined with the other sensory cues in foods.
52 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

For example, this perceptual blending is so tight that odour/taste attributes of a drink are not
easily separated from its colour properties before nine years of age in an identification task
where flavour and colour were manipulated (Oram et al. 1995; see also Chapter 3 by Spence).
Another point of the current state of research on children is that when a learning contingency is
established between an odour and an affective state, this odour can subsequently reactivate the
corresponding mood and influence (positively or negatively) cognitive performance.

2.6 Conclusions
In this review, we have attempted to understand how olfaction interacts with other sensory systems
in early development, and at the same time to survey findings that can shed light on how intersen-
sory interactions involving olfaction can contribute to perceptual development. It is now clear that
smell is far from dormant in the early orchestration of action in the multisensory environment of
young organisms. In newborns, olfactory stimuli can control arousal states and accordingly tutor
attention in the other sensory modalities. By these regulatory functions, olfaction contributes early
on in life to open the brain to the multisensory stream of information.
Several experiments show that arbitrary odours are easily bound with arbitrary stimuli or
sensations in other sensory modalities, suggesting that similar encoding processes may operate
for multisensory interactions going on in natural and species-typical situations. Thus, for exam-
ple, a mother’s odour cues, in the same way as the sound of her voice (Sai et al. 2005), may pro-
mote attentiveness to the sensory information pertaining to her face and body, and hence may
facilitate identity learning. More generally, odours may operate as sensory tags for other stimulus
attributes of objects, persons or contexts, including the subject’s own internal states. This tagging
function of olfaction is supported by persistent memory processes (Engen 1991). These odour
tags can then operate in two inclusive domains of adaptive responsiveness. First, in tracking con-
tinuity in the otherwise constantly changing multisensory environment, providing the young
organism with partially continuous or overlapping sensory cues in different niches. Second, such
odour tags may contribute to the assimilation of perceptual discontinuities, in that their prior
association with familiar contexts confers on them a dose of reassuring or incentive value that
enhances the sustained intake of information in novel situations. These odour tags may derive
from (postnatal or prenatal) learning processes or from predisposed processes that do apparently
develop independently of experience. Accordingly, olfaction is to be considered as a key sense in
the multisensory organization of adaptive responses in early developmental transitions.
At all developmental stages considered above, odour stimuli were shown to become part of
interoceptive experience related to emotional challenge, physical pain, or malaise. Once paired,
often after a single contingency opportunity, these odour or flavour cues become predictive of
similar states. In other words, the chemosensory cues become linked with the cues related with
the bodily or mental state of the organism. This ubiquitous phenomenon provides a useful
paradigm for developmental investigations of intersensory processes involving olfaction.
It invites us to assess whether the separate presentation of the conditioned odour later has the
potency to evoke similar response patterns controlled by the autonomic nervous system and to
retrieve the associated group of non-olfactory reminiscences, as suggested by studies on the long-
term consequences of elation or traumatic life events (e.g. Hinton et al. 2004; Vermetten and
Bremner 2003). How far can the odour stimulus function as a metonym (Van Toller and Kendall-
Reed 1995) of the original object or context to which it has been paired? Specifically, how far
and under what conditions can the individual odour of another person give rise to expectations
about her or even be taken as the person herself? Along which multisensory developmental
pathways and time-courses will olfaction be integrated when it is compensatorily over-invested,
REFERENCES 53

as in blindness (e.g. Cuevas et al. 2009; Wakefield et al. 2004), or more or less disinvested,
as in situations where it is uncoupled from the multisensory context of parental nurturance
(e.g. in case of mother–infant separation) or of eating (e.g. in case of prolonged early tube-feeding
or enteral nutrition by gastrostomy, e.g. Harding et al. 2010)?
Finally, a wealth of studies has now established that olfaction functions unimodally in labora-
tory tasks as well as in various real-life adaptive challenges that individuals have to face through
development. Now considering the exponential increase of psychological and neurobiological
investigations that integrate sensory systems into unified perceptual processes and intercon-
nected brain structures (e.g. Calvert et al. 2004), the time is ripe to more systematically assess the
development of olfaction in the context of co-occurring inputs from the other senses. Olfactory
perceptions will reveal their full developmental significance only when we consider co-encoded
stimulations, feelings and knowledge raised by complementary sensory entries.

Acknowledgements
The authors thank Drs. Roger Lécuyer, André Holley, and Alix Seigneuric, and the editors of
the present book for their significant comments on a previous draft of the manuscript. We also
express our thanks to André Holley for the line drawing of Fig. 2.1. Finally, the authors are also
especially thankful to Giovanna Lux-Jesse for her insightful requests for clarifications and linguis-
tic skills. During the writing of this chapter, the authors were funded by the Centre National de la
Recherche Scientifique (CNRS), Paris; the Université de Bourgogne, Dijon, and the Conseil
Régional de Bourgogne, Dijon.

References
Abate, P., Pepino, M.Y., Dominguez, H.D., Spear, N.E. and Molina, J.C. (2000). Fetal associative learning
mediated through maternal alcohol intoxication. Alcoholism: Clinical and Experimental Research, 24,
39–47.
Aglioti, S.M., and Pazzaglia, M. (2010). Sounds and scents in (social) action. Trends in Cognitive Science, 15,
47–55.
Alberts, J.R. (1981). Ontogeny of olfaction: reciprocal roles of sensation and behavior in the development
of perception. In Development of Perception: psychobiological perspectives, Vol. 1 (eds. R.N. Aslin, J.R.
Alberts, and M.R. Petersen), pp. 321–57. Academic Press, New York.
Alberts, J.R. (1987). Early learning and ontogenetic adaptation. In Perinatal development. A psychobiological
perspective (eds. N.A. Krasnegor, E.M. Blass, M.A. Hofer, and W.P. Smotherman), pp. 12–38.
Academic Press, Orlando, FL.
Anseloni, V.C., Ren, K., Dubner, R., and Ennis, M. (2005). A brainstem substrate for analgesia elicited by
intraoral sucrose. Neuroscience, 133, 231–43.
Arditi, H., Feldman, R., and Eidelman, A.I. (2006). Effects of human contact and vagal regulation on pain
reactivity and visual attention in newborns. Developmental Psychobiology, 48, 561–73.
Astic, L., and Saucier, D. (1982). Ontogenesis of the functional activity of rat olfactory bulb:
autoradiographic study with the 2-deoxyglucose method. Developmental Brain Research, 2, 1243–56.
Auvray, M., and Spence, C. (2008). The multisensory perception of flavor. Consciousness and Cognition, 17,
1016–31.
Badalian, S.S., Chao, C.R., Fox, H.E., and Timor-Tritsch, I.E. (1993). Fetal breathing-related nasal fluid flow
velocity in uncomplicated pregnancies. American Journal of Obstetrics and Gynecology, 169, 563–67.
Bahrick, L.E., and Lickliter, R. (2002). Intersensory redundancy guides early perceptual and cognitive
development. Advances in Child Development and Behavior, 30, 153–87.
Batsell, W.R., Brown, A.S., Ansfield, M.E., and Paschall, G.Y. (2002). ‘You will eat all of that!’: a
retrospective analysis of forced consumption episodes. Appetite, 38, 211–19.
54 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

Beauchamp, G.K., and Mennella, J.A. (1998). Sensory periods in the development of human flavor
perception and preferences. Annales Nestlé, pp. 19–31. Raven Press, New York.
Beauchamp, G.K., Cowart, B.J. and Moran, M. (1986). Developmental changes in salt acceptability in
human infants. Developmental Psychobiology, 19, 17–25.
Beidler, L.M. (1961). Taste receptor stimulation. Progress in Biophysics and Biophysical Chemistry, 12, 107–32.
Bende, M., and Nordin, S. (1997). Perceptual learning in olfaction: professional wine tasters versus
controls. Physiology and Behavior, 62, 1065–70.
Bensafi, M., Rouby, C., Farget, V., Bertrand, B., Vigouroux, M., and Holley, A. (2002). Autonomic nervous
system responses to odours: the role of pleasantness and arousal. Chemical Senses, 27, 703–9.
Bensafi, M., Brown, W.M., Khan, R., Levenson, B., and Sobel, N. (2004) Sniffing human sex-steroid derived
compounds modulates mood, memory and autonomic nervous system function in specific behavioral
contexts. Behavioural Brain Research, 152, 11–22.
Bensafi, M., Pouliot, S., and Sobel, N. (2005). Odorant-specific patterns of sniffing during imagery
distinguish ‘bad’ and ‘good’ olfactory imagers. Chemical Senses, 30, 521–29.
Bernstein, I.L. (1978). Learned taste aversions in children receiving chemotherapy. Science, 200, 1302–13.
Bernstein, I.L. (1991). Flavor aversion. In Smell and taste in health and disease (eds. T.V. Getchell, R.L.
Doty, L.M. Bartoshuk, and J.B. Snow), pp. 417–28. Raven Press, New York.
Birch, L.L., McPhee, L., Steinberg, L., and Sullivan, S. (1990). Conditioned flavor preferences in young
children. Physiology and Behavior, 47, 501–507.
Blass, E.M. (1990). Suckling: determinants, changes, mechanisms, and lasting impressions. Developmental
Psychology, 26, 520–33.
Boisson, C. (1997). La dénomination des odeurs: variations et régularités linguistiques [The naming of
odours: variation and linguistic regularities]. Intellectica, 24, 29–49.
Bolhuis, J.J. (1996). Development of perceptual mechanisms in birds: predispositions and imprinting. In
Neuroethological studies of cognitive and perceptual processes (eds. C.F. Mossand, and S.J. Shettleworth),
pp. 158–84. Westview Press, Boulder, CO.
Booth, D. (1990). Learned roles of taste in eating motivation. In Taste, experience, and feeding (eds. E.D.
Capaldi and T.L. Powley), pp. 179–94. American Psychological Association, Washington, DC.
Brake, S.C. (1981). Suckling infant rats learn a preference for a novel olfactory stimulus paired with milk
delivery. Science, 211, 506–508.
Bremner, A.J., Holmes, N.P., and Spence, C. (2008). Infants lost in (peripersonal) space? Trends in
Cognitive Sciences, 12, 298–305.
Brunjes, P.C., and Alberts, J.R. (1979). Olfactory stimulation induces filial preferences for huddling in rat
pups. Journal of Comparative Physiology and Psychology, 93, 548–55.
Butler, J. and Rovee-Collier, C. (1989). Contextual gating of memory retrieval. Developmental
Psychobiology, 22, 533–52.
Cain, W.S. (1977). Differential sensitivity for smell: ‘noise’ at the nose. Science, 195, 796–98.
Cain, W.S., and Johnson F., Jr. (1978). Lability of odor pleasantness: influence of mere exposure.
Perception, 7, 459–65.
Calvert, G., Spence, C., and Stein, B.E. (2004). Handbook of multisensory processing. MIT Press, Cambridge, MA.
Carmichael, S.T., Clugnet, M.C., and Price J.L. (1994) Central olfactory connections in the Macaque
monkey. Journal of Comparative Neurology, 346, 403–34.
Cernoch, J.M. and Porter, R.H. (1985). Recognition of maternal axillary odors by infants. Child
Development, 56, 1593–98.
Chu, S. (2008). Olfactory conditioning of positive performance in humans. Chemical Senses, 33, 65–71.
Chu, S., and Downes, J.J. (2000). Long live Proust: the odour-cued autobiographical memory bump.
Cognition, 75, 41–50.
Chu, S., and Downes, J.J. (2002). Proust nose best: odours are better cues of autobiographical memory.
Memory and Cognition, 30, 511–518.
REFERENCES 55

Coureaud, G., Moncomble, A.S., Montigny, D., Dewas, M., Perrier, G., and Schaal, B. (2006). A pheromone
that rapidly promotes learning in the newborn. Current Biology, 16, 1956–61.
Craig, A.D. (2009). How do you feel—now? The anterior insula and human awareness. Nature Reviews
Neuroscience, 10, 59–70.
Cuevas, I., Plaza, P., Rombaux, P., De Volder, A.G., and Renier L. (2009). Odour discrimination and
identification are improved in early blindness. Neuropsychologia, 47, 3079–83.
Damasio, A.R. (1993). Descartes’ error: emotion, reason and the human brain. Putnam, New York.
de Wijk, R.A., and Cain, W.S. (1994). Odor identification by name and by edibility: life-span development
and safety. Human Factors, 36, 182–87.
Degel, J., and Köster, E.P. (1999). Odours: implicit memory and performance effects. Chemical Senses, 24,
317–32.
Delaunay-El Allam, M., Marlier, L., and Schaal, B. (2006). Learning at the breast: preference formation for
an artificial scent and its attraction against the odor of maternal milk. Infant Behavior and Development,
29, 308–21.
Delaunay-El Allam, M., Soussignan, R., Patris, B., and Schaal, B. (2010). Long lasting memory for an odor
acquired at the mother’s breast. Developmental Science, 13, 849–63.
Delplanque, S., Grandjean, D., Chrea, C., et al. (2008). Emotional processing of odors: evidence for non
linear relation between pleasantness and familiarity evaluations. Chemical Senses, 33, 469–79.
Doty, R.L. (2003). Handbook of olfaction and gustation, 2nd edn Marcel Dekker, New York.
Doty, R.L., and Cometto-Muniz J.E. (2003). Trigeminal sensation. In Handbook of olfaction and gustation,
2nd Edn (ed. R.L. Doty), pp. 981–99. Marcel Dekker, New York.
Doucet, S., Soussignan, R., Sagot, P., and Schaal, B. (2007). The ‘smellscape’ of the human mother’s breast:
effects of odour masking and selective unmasking on neonatal arousal, oral and visual responses.
Developmental Psychobiology, 49, 129–38.
Doucet S., Soussignan R., Sagot P., Schaal B. (2009) The secretion of areolar (Montgomery's) Glands from
lactating women elicits selective, unconditional responses in neonates. PLoS One, 4(10), e7579.
doi:10.1371/journal.pone.0007579.
Durand, K., Baudon, G., Freydefont, L., and Schaal, B. (2008). Odorisation of a novel object can
influence infant’s exploratory behavior in unexpected ways. Infant Behavior and Development, 31,
629–36.
Edmond, K.M., Kirkwood, B.R., Amenga-Etego, S., Owusu-Agyei, S., and Hurt, L.S. (2007a). Effect of early
feeding practices on infection-specific neonatal mortality. An investigation of the causal links with
observational data from rural Ghana. American Journal of Clinical Nutrition, 86, 1126–31.
Edmond, K.M., Zandoh, C., Quigley, M.A., Amenga-Etego, S., Owusu-Agyei, S., and Kirkwood, B.R.
(2007b). Delayed breastfeeding initiation increases risk of neonatal mortality. Pediatrics, 117, 380–86.
Eisenberg, N.I. (2006). Identifying the neural correlates underlying social pain: implications for
developmental processes. Human Development, 49, 273–93.
Engen,T. (1987). Remembering odors and their names. American Scientist, 75, 497–503.
Engen, T. (1991). Odor sensations and memory. Praeger Press, New York.
Engen, T., and Ross, B.M. (1973). Long-term memory odors with and without verbal descriptions. Journal
of Experimental Psychology, 100, 221–27.
Engen, T., Kuisma, J.E., and Eimas P.D. (1973). Short-term memory for odors. Journal of Experimental
Psychology, 99, 222–25.
Epple, G., and Herz, R. (1999). Ambient odors associated to failure influence cognitive performance in
children. Developmental Psychobiology, 35, 103–107.
Faas, A.E., Sponton, E.D., Moya, P.R., and Molina, J.C. (2000). Differential responsiveness to alcohol odor
in human neonates. Effects of maternal consumption during gestation. Alcohol, 22, 7–17.
Fabes, R.A., and Filsinger, E.E. (1986). Olfaction and young children’s preferences: a comparison of odor
and visual cues. Perception and Psychophysics, 40, 171–76.
56 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

Fagen, J., Prigot, J., Carroll, M., Pioli, L., Stein, A., and Franco, A. (1997). Auditory context and memory
retrieval in young infants. Child Development, 68, 1057–66.
Ferdenzi, C., Coureaud, G., Camos, V., and Schaal, B. (2008). Human awareness and uses of odor cues in
everyday life: results from a questionnaire study in children. International Journal of Behavioral
Development, 32, 417–26.
Ferdenzi, C., Schaal, B., and Roberts, C.S. (2010). Family scents: developmental changes in body odor
perception of kin? Journal of Chemical Ecology, 36, 847–54.
Fernandez, M., and Bahrick, L. (1994). Infants’ sensitivity to arbitrary object-odour pairings. Infant
Behavior and Development, 17, 471–74.
Forestell, C.A., and Mennella, J.A. (2005). Children’s hedonic judgment of cigarette smoke odor: effect of
parental smoking and maternal mood. Psychology of Addiction and Behavior, 19, 423–32.
Gallace, A., and Spence, C. (2010). The science of interpersonal touch: an overview. Neuroscience and
Biobehavioral Reviews, 34, 246–59.
Ganshrow, J.R. and Mennella J.A. (2003). The ontogeny of human flavor perception. In Handbook of
olfaction and gustation, 2nd edn (ed. R.L. Doty), pp. 823–46. Marcel Dekker, New York.
Garb, J.L., and Stunkart, A.J. (1974). Taste aversion in man. American Journal of Psychiatry, 131,
1204–1207.
Gibson, E.J. (1969). Principles of perceptual learning and development. Academic Press, New York.
Goldman, W.P., and Seamon, J.G. (1992). Very long-term memory for odors: retention of odors-name
associations. American Journal of Psychology, 105, 549–63.
Gottfried, J.A. (2006). Smell: central nervous processing. Advances in Oto-Rhino-Laryngology, 63, 44–69.
Gottfried, J.A., and Dolan, R.J. (2003). The nose smells what the eye sees: crossmodal visual Facilitation of
human olfactory perception. Neuron, 39, 375–86.
Gottlieb, G. (1971). Ontogenesis of sensory function in birds and mammals. In The biopsychology of
development (eds. E. Tobach, L. Aronson, E. Shaw), pp. 67–128. Academic Press, New York.
Goubet, N., Rattaz, C., Pierrat, V., Bullinger, A., and Lequien, P. (2003). Olfactory experience mediates
response to pain in preterm newborns. Development Psychobiology, 42, 171–80.
Goubet, N., Strasbaugh, K., and Chesney, J. (2007). Familiarity breeds content? Soothing effect of a familiar
odor on full-term ne.wborns. Journal of Developmental and Behavioral Pediatrics, 28, 189–94.
Haller R., Rummel, C., Henneberg, S., Pollmer, U., and Köster, E.P. (1999). The effect of early experience
with vanillin on food preference later in life. Chemical Senses, 24, 465–67.
Harding, C., Faiman, A., and Wright J. (2010). Evaluation of an intensive desensitisation, oral tolerance
therapy and hunger provocation program for children who have had prolonged periods of tube feeds.
International Journal of Evidence-Based Healthcare, 8, 268–76.
Harlow, H.F., and Harlow, M.K. (1965). The affectional systems. In Behavior of nonhuman primates:
Modern research trends, Vol. 2 (eds. A.M. Schrier, H.F. Harlow and F. Stollnitz), pp. 287–334. Academic
Press, New York.
Heilmann, S., and Hummel, T. (2004). A new method for comparing orthonasal and retronasal olfaction.
Behavioral Neuroscience, 118, 412–19.
Hepper, P.G. (1988a). Adaptive foetal learning: prenatal exposure to garlic affects postnatal preferences.
Animal Behaviour, 36, 935–36.
Hepper, P.G. (1988b). Fetal ‘soap’ addiction. The Lancet, June 11, 1(8598), 1347–1348.
Hepper, P.G. (1991a). Transient hypoxic episodes: a mechanism to support associative fetal learning,
Animal Behavior, 41, 477–80.
Hepper, P.G. (1991b). An examination of fetal learning before and after birth. Irish Journal of Psychology,
12, 95–107.
Hepper, P.G. (1993). In utero release from a single transient hypoxic episode: a positive reinforcer?
Physiology and Behavior, 53, 309–11.
Hepper, P.G. (1995). Human fetal ‘olfactory’ learning. International Journal of Prenatal and Perinatal
Psychology and Medicine, 7, 147–51.
REFERENCES 57

Hermans, D., and Baeyens, F. (2002). Acquisition and activation of odor hedonics in everyday situations:
conditioning and priming studies. In Olfaction, taste and cognition (eds. C. Rouby, B. Schaal, D.
Dubois, D. Gervais, R. and A. Holley), pp. 119–39. Cambridge University Press, New York.
Herz, R.S. (2004). A naturalistic analysis of autobiographical memories triggered by olfactory, visual and
auditory stimuli. Chemical Senses, 29, 217–24.
Herz, R.S., and Engen, T. (1990). Odor memory: review and analysis. Psychonomic Bulletin and Review, 3,
300–313.
Hinton, M.D., Pich, M.S., Dara Chean, B.A., et al. (2004). Olfactory-triggered panic attacks among
Cambodian refugees attending a psychiatric clinic. General Hospital Psychiatry, 26, 390–97.
Hogan, J.A. (1998). Cause and function in the development of behavior systems. In Handbook of behavioral
neurobiology, Developmental psychobiology and behavioral ecology, Vol. 9 (ed. E.M. Blass), pp. 63–106.
Plenum Press, New York.
Holland, R. W., Hendriks, M., and Aarts, H.A.G. (2005). Smells like clean spirit: nonconscious effects of
scent on cognition and behavior. Psychological Science, 16, 689–93.
Hvastja and Zanuttini (1989). Odour memory and odour hedonics in children. Perception, 18, 391–96.
Johnson, B.A., and Leon, M. (2002). Modular representations of odorants in the glomerular layer of the rat
olfactory bulb and the effects of stimulus concentration. Journal of Comparative Neurology, 422, 496–509.
Johnston, C.C., Filion, F., Snider, L., et al. (2002). Routine sucrose analgesia during the first week of life in
neonates younger than 31 weeks’ postconceptional age. Pediatrics, 110, 523–28.
Johnston, C.C., Filion, F., and Nuyt, A.M. (2007). Recorded maternal voice for preterm neonates
undergoing heel lance. Advances in Neonatal Care, 7, 258–66.
Johnson, S.L., McPhee, L., and Birch, L.L. (1991). Conditioned preferences: young children prefer flavors
associated with high dietary fat. Physiology and Behavior, 50, 1245–51.
Kaplan, J.N., and Russell, M. (1973). Olfactory recognition in the infant squirrel monkey. Developmental
Psychobiology, 7, 15–19.
Kaplan, J.N., Cubicciotti, D.D., and Redican, W.K. (1977). Olfactory and visual differentiation of
synthetically scented surrogates by infant squirrel monkeys. Developmental Psychobiology, 12, 1–10.
Kirk-Smith, M.D., Van Toller, C. and Dodd,G.H. (1983). Unconscious odour conditioning in human
subjects. Biological Psychology, 17, 221–31.
Kobal, G., Hummel, T., and Van Toller, S. (1992). Differences in human chemosensory evoked-potentials
to olfactory and somatosensory chemical stimuli presented to the left and right nostrils. Chemical
Senses, 17, 233–44.
Kobayakawa, K., Kobayakawa, R., Matsumoto, H., et al. (2007). Innate versus learned odor processing in
the mouse olfactory bulb. Nature, 450, 503–510.
Korner, A.F., and Thoman, E.B. (1970). Visual alertness in neonates as evoked by maternal care. Journal of
Experimental Child Psychology, 10, 67–78.
Köster, E.P., Degel, J. and Piper, D. (2002). Proactive and retroactive interference in implicit odor memory.
Chemical Senses, 27, 191–207.
Kroner, T. (1882). Über die Sinnesempfindungen des Neugeborenen [On the sensations of the newborn].
Bresslauer Ärztliche Zeitschrift, 4, 37–58.
Lawless, H., and Engen, T. (1977). Associations to odors: interference, mnemonics, and verbal labeling.
Journal of Experimental Psychology, 3, 52–59.
Lecanuet, J.P., and Schaal, B. (1996). Fetal sensory competences. European Journal of Obstetrics,
Gynaecology, and Reproductive Biology, 68, 1–23.
Leon, M., Coopersmith, R., Lee, S., Sullivan, R.M., Wilson, D.M., and Woo, C.C. (1987). Neural and
behavioral plasticity induced by early olfactory learning. In Perinatal development, A psychobiological
perspective (eds. N.A. Krasnegor, E.M. Blass, M.A. Hoferand, and W.P. Smotherman), pp. 145–67.
Academic Press, Orlando, FL.
Levenson, R.W., Ekman, P., and Friesen, W.V. (1990). Voluntary facial action generates emotion-specific
effects on autonomic nervous system function in humans. Behavioral Neuroscience, 117, 1125–37.
58 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

Lewkowicz, D.J. (1988). Sensory dominance in infants: 2. Ten-month-old infants’ response to auditory-
visual compounds. Developmental Psychology, 24, 172–82.
Lewkowicz, D.J. (1991). Development of intersensory functions in human infancy: auditory/visual
interactions. In Newborn Attention: Biological Constraints and the Influence of Experience (eds.
M.J. Weiss and P.R. Zelazo), pp. 308–38. Ablex Publishing Corp, Norwood, NJ.
Lewkowicz, D.J., and Lickliter, R. (1994). The development of intersensory perception: comparative
perspectives. Lawrence Erlbaum Associates, Hillsdale.
Lewkowicz, D.J., and Turkewitz, G. (1980). Cross-modal equivalence in early infancy: auditory-visual
intensity matching. Developmental Psychology, 16, 597–607.
Li, W., Luxenberg, E., Parrish, T., and Gottfried, J.A. (2006). Learning to smell the roses: experience-
dependent neural plasticity in human piriform and orbitofrontal cortices. Neuron, 52, 1097–1108.
Li, W., Moallem, I., Paller, K.A., and Gottfried, J.A. (2007). Subliminal smells can guide social preferences.
Psychological Science, 18, 1044–49.
Li, W., Howard, J.D., Parrish, T., and Gottfried, J.A. (2008). Aversive learning enhances perceptual and
cortical discrimination of indiscriminable odor cues. Science, 319, 1842–45.
Logue, A.W., Ophir, I., and Strauss, K.E. (1981). The acquisition of taste aversions in humans. Behavior
Research and Therapy, 19, 319–33.
Lundstrom, J.N., and Olsson, M.J. (2005). Subthreshold amounts of social odorant affect mood, but not
behavior, in heterosexual women when tested by a male, but not a female, experimenter. Biological
Psychology, 70, 197–204.
Macfarlane, A.J. (1975). Olfaction in the development of social preferences in the human neonate. Ciba
Foundation Symposia, 33, 103–117.
Mallet, P., and Schaal, B. (1998). Rating and recognition of peer’s personal odours in nine-year-old
children: an exploratory study. Journal of General Psychology, 125, 47–64.
Maone, T.R., Mattes, R.D., Bernbaum, J.C., and Beauchamp, G.K. (1990). A new method for delivering a
taste without fluids to preterm and term infants. Developmental Psychobiology, 23, 179–81.
McBurney, D.H., Shoup, M.L., and Stretter, S.A. (2006). Olfactory comfort: smelling a partner’s clothing
during periods of separation. Journal of Applied Social Psychology, 36, 2325–35.
Mennella, J.A., and Beauchamp, G.K. (1998). Infants’ exploration of scented toys: effects of prior
experiences. Chemical Senses, 23, 11–17.
Mennella, J.A., and Forestell, C.A. (2008). Children’s hedonic responses to the odors of alcoholic beverages:
a window to emotions. Alcohol, 42, 249–60.
Mennella, J.A., and Garcia, P.L. (2000). Children’s hedonic response to the smell of alcohol: effects of
parental drinking habits. Alcohol: Clinical and Experimental Research, 24, 1167–71.
Mennella, J.A., Johnson, A., and Beauchamp, G.K. (1995) Garlic ingestion by pregnant women alters the
odor of amniotic fluid. Chemical Senses, 20, 207–209.
Mennella, J.A., Jagnow, C.P., and Beauchamp, G.K. (2001). Pre- and post-natal flavor learning by human
infants. Pediatrics, 107, 1–6.
Michael, G.A., Jacquot, L., and Brand, G. (2003). Ambient odors modulate visual attentional capture.
Neuroscience Letters, 352, 221–25.
Möller, P., Wulff, C., and Köster, E.P. (2004). Do age differences in odour memory depend on differences
in verbal memory? Neuroreport, 15, 915–917.
Mouélé, M. (1977). L’apprentissage des odeurs chez les Waanzi: notes de recherché [Learnin odours in the
Waanzi : reserach notes]. In L’odorat chez l’enfant : perspectives croisées [Smell in children: crossed
perspectives] (ed. B. Schaal), pp. 209–22. Presses Universitaires de France, Paris.
Murphy, C., and Cain, W.S. (1980). Lability of odor pleasantness: influence vs interaction. Physiology and
Behavior, 24, 601–605.
Olsson, S.B., Barnard, J., and Turri, L. (2006). Olfaction and identification of unrelated individuals. Journal
of Chemical Ecology, 32, 1635–45.
REFERENCES 59

Oram, N., Laing, D.G., Hutchinson, I., et al. (1995). The influence of flavor and color on drink
identification by children and adults. Developmental Psychobiology, 28, 239–46.
Patrick, J., Campbell, K., Carmichael, L., Natale, R., and Richardson, B. (1980). Patterns of human fetal
breathing activity during the last 10 weeks of pregnancy. Obstetrics and Gynecology, 56, 24–28.
Patrick, J., Campbell, K., Carmichael, L., Natale, R., and Richardson, B. (1982). Patterns of gross fetal body
movements over 24-hour observation intervals during the last 10 weeks of pregnancy. American
Journal of Obstetrics and Gynecology, 142, 363–71.
Pedersen, P.A., and Blass, E.M. (1982). Prenatal and postnatal determinants of the 1st suckling episode in
albino rats. Developmental Psychobiology, 15, 349–55.
Pihet, S., Schaal, B., Bullinger, A., and Mellier, D. (1996). An investigation of olfactory responsiveness in
premature newborns. Infant Behavior and Development, ICIS Issue: 676.
Pihet, S., Mellier, D., Bullinger, A., and Schaal, B. (1997). Réponses comportementales aux odeurs chez le
nouveau-né prématuré: étude préliminaire (Behavioural responses to odours in preterm infants : a
preliminary study). In L’odorat chez l’enfant: perspectives croisées (ed. B. Schaal), pp. 33–46. Presses
Universitaires de France, Collection Enfance, Paris.
Pirogovsky, E. Murphy, C., and Gilbert, P.E. (2009). Developmental differences in memory for cross-modal
associations. Developmental Science, 12, 1054–59.
Pomonis, J.D., Jewett, D.C., Kotz, J.E., et al. (2000). Sucrose consumption increases naloxone-induced
c-Fos immunoreactivity in limbic system. American Journal of Physiology, 364, R712–R719.
Poncelet, J., Rinck, F., Bourgeat, F., et al. (2010). The effect of early experience on odor perception in
humans: psychological and physiological correlates. Behavioural Brain Research, 208, 458–65.
Prechtl, H.F.R. (1958). The directed head turning response and allied movements of the human baby.
Behaviour, 13, 212–42.
Preyer, W. (1885). Die Seele des Kindes. [The soul of the child] (French translation). Editions Alcan, Paris.
Price, J.L. (1990). Olfactory system. In The Human Nervous System, G. Paxinos (ed.), pp. 979–1001.
Academic Press, San Diego.
Rabin, M.D. (1988). Experience facilitates olfactory quality discrimination. Perception and Psychophysics,
44, 532–40.
Rattaz, C., Goubet, N., and Bullinger, A. (2005). The calming effect of a familiar odor on full-term
newborns. Journal of Developmental and Behavioral Pediatrics, 26, 86–92.
Raybould, H.E. (1998). Does your gut taste? Sensory transduction in the gastrointestinal tract. News in
Physiological Science, 13, 275–80.
Reardon, P., and Bushnell, E.W. (1988). Infants’ sensitivity to arbitrary pairings of color and taste. Infant
Behavior and Development, 11, 245–50.
Redican, W.K., and Kaplan, J.N. (1978). Effects of synthetic odors on filial attachment in infant squirrel
monkeys. Physiology and Behavior, 20, 79–85.
Rieser, J., Yonas, A., and Wilkner, K. (1976). Radial localization of odors by human newborns. Child
Development, 47, 856–59.
Robinson, S.R., and Mendèz-Gallardo, V. (2010). Amniotic fluid as an extended milieu intérieur. In
Handbook of developmental science, behavior, and genetics (Eds. K.E. Hood, C.T. Halpern, G. Greenberg,
and R.M. Lerner), pp. 234–84. Blackwell Publishing Ltd, Oxford, UK.
Rolls, E.T. (2005). Taste, olfactory, and food texture processing in the brain, and the control of food intake.
Physiology and Behavior, 85, 45–56.
Ronca, A.E., Abel, R.A., and Alberts, J.R. (1996). Perinatal stimulation and adaptation of the neonate. Acta
Paediatrica, Suppl. 416, 8–15.
Rosenblatt, J.S. (1983). Olfaction mediates developmental transitions in the altricial newborn of selected
species of mammals. Development Psychobiology, 16, 347–75.
Rosenstein, D., and Oster, H. (1990). Differential facial responses to four basic tastes in newborns. Child
Development, 59, 1555–68.
60 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

Rovee-Collier, C., and Cuevas, K. (2009). Multiple memory systems are unnecessary to account for infant
memory development: an ecological model. Developmental Psychology, 45, 160–74.
Rozin, P. (1982). ‘Taste-smell confusions’ and the duality of the olfactory sense. Perception and
Psychophysics, 31, 397–401.
Rubin, G.B., Fagen, J.W., and Carroll, M.H. (1998). Olfactory context and memory retrieval in 3-month-
old infants. Infant Behavior and Development, 21, 641–58.
Rudy, J.W., and Cheatle, M.D. (1977). Odor-aversion learning in neonatal rats. Science, 198, 845–46.
Ruff, H.A., Saltarelli, L.M., Capozzoli, M., and Dubiner, K. (1992). The differentiation of activity in infants’
exploration of objects. Developmental Psychology, 28, 851–61.
Saî, F. Z. (2005). The role of the mother’s voice in developing mother’s face preference: evidence for
intermodal perception at birth. Infant and Child Development, 14, 29–50.
Schaal, B. (1988). Olfaction in infants and children: developmental and functional perspectives. Chemical
Senses, 13, 145–90.
Schaal, B. (2005). From amnion to colostrum to milk: odor bridging in early developmental transitions. In
Prenatal development of postnatal functions (eds. B. Hopkins and S.P. Johnson), pp. 51–102. Praeger,
London.
Schaal, B. (2006). The development of flavor perception from infancy to adulthood. In Flavour in food
(ed. A. Voilley), pp. 401–36. Woodhead Publishing, Cambridge.
Schaal, B. (2012). Emerging chemosensory preferences. Another playground for the innate-acquired
dichotomy in human cognition. In Olfactory cognition (eds. G.H. Zucco, R. Herz, and B. Schaal),
pp. 237–68. Benjamin Publishing, Amsterdam.
Schaal, B., and Marlier, L. (1998). Maternal and paternal perception of individual odor signatures in
human amniotic fluid—potential role in early bonding? Biology of the Neonate, 74, 274–80.
Schaal, B., and Orgeur, P. (1992). Olfaction in utero: can the rodent model be generalized? Quarterly
Journal of Experimental Psychology. B. Comparative and Physiological Psychology, 44B, 245–78.
Schaal, B., Montagner, H., Hertling, E., Bolzoni, D., Moyse, R., and Quichon, R. (1980). Olfactory
stimulations in mother–infant relations. Reproduction, Nutrition, and Development, 20, 843–58.
Schaal, B., Orgeur, P., and Arnould, C. (1995a). Olfactory preferences in newborn lambs: possible influence
of prenatal experience. Behaviour, 132, 351–65.
Schaal, B., Orgeur, P., and Rognon, C. (1995b). Odor sensing in the human fetus: anatomical, functional
and chemo-ecological bases. In Prenatal Development, A Psychobiological Perspective (eds. J.P. Lecanuet,
N.A. Krasnegor, W.A. Fifer and W. Smotherman), pp. 205–37. Lawrence Erlbaum Associates,
Hillsdale, NJ.
Schaal, B., Marlier, L., and Soussignan, R. (1995c). Neonatal responsiveness to the odour of amniotic fluid.
Biology of the Neonate, 67, 397–406.
Schaal, B., Marlier, L., and Soussignan, R. (1998). Olfactory function in the human fetus: evidence from
selective neonatal responsiveness to the odor of amniotic fluid. Behavioral Neuroscience, 112, 1438–49.
Schaal, B., Marlier, L., and Soussignan, R. (2000). Human foetuses learn odours from their pregnant
mother’s diet. Chemical Senses, 25, 729–37.
Schaal, B., Coureaud, G., Langlois, D., Giniès, C., Sémon, E., and Perrier, G. (2003). Chemical and
behavioural characterization of the mammary pheromone of the rabbit. Nature, 424, 68–72.
Schaal, B., Hummel T., and Soussignan, R. (2004). Olfaction in the fetal and premature infant: functional
status and clinical implications. Clinics in Perinatology, 31, 261–85.
Schiffman, S.S. (1974). Physicochemical correlates of olfactory quality. Science, 185, 112–17.
Schleidt, M., and Genzel, C. (1990). The significance of mother’s perfume for infants in the first weeks of
their life. Ethology and Sociobiology, 11, 145–54.
Schmidt, H. (1990). Adult-like hedonic responses to odors in 9-month-old infants. Chemical Senses, 15, 634.
Schmidt, H., and Beauchamp, G.K. (1989). Sex differences in responsiveness to odors in 9-month-old
infants. Chemical Senses, 14, 744.
REFERENCES 61

Schneirla, T.C. (1965). Aspects of stimulation and organization in approach/withdrawal processes


underlying vertebrate behavioural development. In Advances in the Study of Behaviour, Vol.1 (eds.
D.S. Lehrman, R.A. Hinde and E. Shaw), pp. 1–74. Academic Press, New York.
Schroers, M., Prigot, J., and Fagen, J. (2007). The effect of a salient odor context on memory retrieval in
young infants. Infant Behavior and Development, 30, 685–89.
Sclafani, A. (2007). Sweet taste signaling in the gut. Proceedings of the National Academy of Sciences of the
USA, 104, 14887–88.
Seigneuric, A., Durand, K., Jiang, T., Baudouin, J.Y., and Schaal, B. (2011). The nose tells it to the eyes:
crossmodal associations between olfaction and vision. Perception, 39, 1541–1554.
Shoup, M.L., Streeter, S.A., and McBurney, D.H. (2008). Olfactory comfort and attachment within
relationships. Journal of Applied Social Psychology, 38, 2954–63.
Simon, S.A., de Araujo, I.E., Gutierrez, R., and Nicolelis, A.L. (2006). The neural mechanisms of gustation:
a distributed processing code. Nature Reviews Neuroscience, 7, 890–901.
Singer, T., Seymour, B., O’Doherty, J., Kaube, H., Dolan, R.J., and Frith, C.D. (2004). Empathy for pain
involves the affective but not sensory components of pain. Science, 303, 1157–62.
Smotherman, W.P. (1982). Odor aversion learning by the rat fetus. Physiology and Behavior, 29, 769–71.
Smotherman, W.P., and Robinson, S.R. (1987). Psychobiology of fetal experience in the rat. In Perinatal
development: A psychobiological perspective (eds. N.E. Krasnegor, E.M. Blass, M.A. Hofer, and
W.P. Smotherman), pp. 39–60. Academic Press, Orlando, FL.
Smotherman, W.P., and Robinson, S.R. (1988). The uterus as environment: the ecology of fetal experience.
In Handbook of behavioral neurobiology: Developmental psychobiology and behavioral ecology, Vol. 9
(ed. E.M. Blass), pp. 149–96. Plenum Press, New York.
Smotherman, W.P., and Robinson, S.R. (1995). Tracing developmental trajectories into the prenatal
period. In Fetal development: a psychobiological perspective (eds. J.P. Lecanuet, W.P. Fifer, N.A.
Krasnegor and W.P. Smotherman), pp. 15–32. Lawrence Erlbaim Associates, Hillsdale, NJ.
Soussignan, R., Schaal, B., Marlier, L., and Jiang, T. (1997). Facial and autonomic responses to biological
and artificial olfactory stimuli in human neonates: re-examining early hedonic discrimination of odors.
Physiology and Behavior, 62, 745–58.
Spear, N.E., and Molina, J.C. (1987). The role of sensory modality in the ontogeny of stimulus selection.
In Perinatal development: a psychobiological perspective (eds. N. Krasnegor, E.M. Blass, M.A. Hofer and
W.P. Smotherman), pp. 83–110. Academic Press, Orlando, FL.
Spector, F., and Maurer, D. (2009). Synesthesia: a new approach to understanding the development of
perception. Developmental Psychology, 45, 175–89.
Steiner, J. (1979). Human facial expressions in response to taste and smell stimulations. In Advances in child
development, Vol. 13 (eds. L.P. Lipsitt, H.W. Reese), pp. 257–95. Academic Press, New York.
Steiner, J.E., Glaser, D., Hawilo, M.E., and Berridge, K. (2001). Comparative evidence of hedonic impact:
affective reactions to taste by human infants and other primates. Neuroscience and Biobehavioral
Reviews, 25, 53–74.
Stevenson, R.J., and Boakes, R.A. (2004). Sweet and sour smells: learned synaesthesia between the senses of
taste and smell. In Handbook of multisensory processing (eds. G. Calvert, C. Spence, and B.E. Stein),
pp. 69–83. MIT Press, Cambridge, MA.
Stevenson, R. J., and Tomiczek, C. (2007). Olfactory-induced synesthesias: a review and model.
Psychological Bulletin, 133, 294–309.
Stickrod, G., Kimble, D.P., and Smotherman, W.P. (1982). In utero taste odor aversion conditioning of the
rat. Physiology and Behavior, 28, 5–7.
Stirnimann, F. (1936). Versuche über Geschmack und Geruch am ersten Lebenstag. [Experiments on taste
and smell on the first day of life], Jahrbuch der Kindereilkunde, 146, 211–27.
Stockhorst, U., Gritzmann, E., Klopp, K., et al. (1999). Classical conditioning of insulin effects in healthy
humans. Psychosomatic Medicine, 61, 424–35.
62 THE ROLE OF OLFACTION IN HUMAN MULTISENSORY DEVELOPMENT

Sullivan, R.M., and Toubas, P. (1998). Clinical usefulness of maternal odor in newborns: soothing and
feeding preparatory responses. Biology of the Neonate, 74, 402–408.
Sullivan, R.M., Taborsky, S.B., Mendoza, R. et al. (1991). Olfactory classical conditioning in neonates.
Pediatrics, 87, 511–517.
Tatzer, E., Schubert, M.T., Timischl, W., and Simbruner, G. (1985). Discrimination of taste and preference
for sweet in premature babies. Early Human Development, 12, 23–30.
Teerling, A., Köster, E.P., and van Nispen, V. (1994). Early childhood experiences and future preferences.
11th Congress of the European Chemoreception Research Organization. Blois, France.
Tees, R.C. (1994). Early stimulation history, the cortex, and intersensory functioning in infrahumans: space
and time. In The development of intersensory perception: comparative perspectives (eds. D.J. Lewkowicz
and R. Lickliter), pp. 107–31. Lawrence Erlbaum Associates, Hillsdale, NJ.
Turkewitz, G. (1979). The study of infancy. Canadian Journal of Psychology, 33, 408–412.
Turkewitz, G.A. (1994). Sources of order for intersensory functioning. In The development of intersensory
perception: comparative perspectives (eds. D.J. Lewkowicz and R. Lickliter), pp. 3–17. Lawrence Erlbaum
Associates, Hillsdale, NJ.
Turkewitz, G., and Kenny, P.A. (1985). The role of developmental limitations of sensory input on sensory/
perceptual organization. Journal of Developmental and Behavioral Pediatrics, 15, 357–68.
Turkewitz, G.A., and Mellon, R.C. (1989). Dynamic organization of intersensory function. Canadian
Journal of Psychology, 43, 286–301.
van Toller, S., and Kendal-Reed, M. (1995). A possible protocognitive role for odour in human infant
development. Brain and Cognition, 29, 275–93.
Varendi, H., and Porter, R.H. (2001). Breast odour as the only maternal stimulus elicits crawling towards
the odour source. Acta Paediatrica, 90, 372–75.
Varendi, H., Porter, R.H., and Winberg, J. (2002). The effect of labor on olfactory exposure learning within
the first postnatal hour. Behavioral Neuroscience, 116, 206–211.
Verhagen, J.V., and Engelen, L. (2006). The neurocognitive bases of human multimodal food perception:
sensory integration. Neuroscience and Biobehavioral Reviews, 30, 613–50.
Vermetten, E., and Bremner, J.D. (2003). Olfaction as a traumatic reminder in posttraumatic stress
disorders: case report and review. Journal of Clinical Psychiatry, 64, 202–207.
Wakefield, C.E., Homewood, J., and Taylor, A.J. (2004). Cognitive compensations for blindness in children:
an investigation using odour naming. Perception 33, 429–42.
Walker-Andrews, A.S. (1994). Taxonomy for intermodal relations. In The development of intersensory
perception: comparative perspectives (eds. D.J. Lewkowicz and R. Lickliter), pp.39–56. Lawrence
Erlbaum Associates, Hillsdale, NJ.
Wang, H., Wysocki, C.J., and Gold, G. (1993). Induction of olfactory receptor sensitivity in mice. Science,
260, 998–1000.
Weisfeld, G.E., Czilly T., Phillips K.A., Gall J.A., and Lichtman, C.M. (2003). Possible olfaction-based
mechanisms in human kin recognition and inbreeding avoidance. Journal of Experimental Child
Psychology, 85, 279–95.
Willander, J., and Larsson, M. (2007). Olfaction and emotion: the case of autobiographical memory.
Memory and Cognition, 35, 1659–63.
Wyart, C., Webster, W.W., Chen, J.H., et al. (2007). Smelling a single component of male sweat alters levels
of cortisol in women. Journal of Neuroscience, 27, 1261–65.
Yeomans, M.R. (2006). The role of learning in development of food preferences. In The psychology of food
choice (ed. R. Shepherd and M. Raats), pp. 93–112. CABI, Wallingford, UK.
Yeshurun, Y., and Sobel, N. (2010). An odor is not worth a thousand words: from multidimensional odors
to unidimensional odor objects. Annual Review of Psychology, 61, 219–41.
Zucco, G. M., Paolini, M., and Schaal, B. (2009). Unconscious odour conditioning 25 years later: Revisiting
and extending ‘Kirk-Smith, Van Toller and Dodd’. Learning and Motivation, 40, 364–75.
Chapter 3

The development and decline of


multisensory flavour perception
Assessing the role of visual (colour)
cues on the perception of taste and flavour
Charles Spence

3.1 Introduction
Flavour perception is one of the most multisensory of our everyday experiences (Spence 2010b,
2012; Stillman 2002), involving as it does not only the taste and smell of a food or drink item, but
also its texture, the sound it makes, and even what it looks like (though see below).1 The pain
associated with eating certain foods (such as, for example, chilli) also contributes to the pleasure
of many foods. Usually, all of these unisensory cues are seamlessly integrated into our perception
of a particular flavour (located subjectively) in the mouth (Spence 2012; Stevenson 2009). Flavour
perception is, however, also one of the least well understood of our multisensory experiences.
This is especially true from a developmental perspective, where the majority of textbooks and
review papers tend not even to discuss the development of flavour perception (e.g. see Lewkowicz
and Lickliter 1994; Pick and Pick 1970). While the evidence is currently fairly sparse, and in many
cases inconsistent, I would argue that research on the topic of multisensory flavour perception is
nevertheless very important—both in terms of understanding why it is that young children do
not like certain foods (such as vegetables) and what can be done to improve the quality of the food
eaten by those at the other end of the age spectrum who may be suffering from a loss of olfactory
and, to a lesser extent, gustatory sensitivity (e.g. Schiffman 1997). Furthermore, the growing
global obesity epidemic has led to a recent increase in interest in multisensory flavour perception
(see Mennella and Beauchamp 2010, for a review).
Flavour perception is a difficult area to study, in part because researchers cannot agree on
a definition (see Auvray and Spence 2008; Spence et al. 2010; Stevenson and Tomacziek 2007).
Part of the problem here is that there is a great deal of uncertainty over whether or not flavour
should be conceptualized as a separate sensory modality (e.g. McBurney 1986; Stevenson 2009).
That said, the last few years have seen a growing number of cognitive neuroscientists successfully

1 Although the words ‘taste’ and ‘flavour’ are used interchangeably in everyday English, food scientists
typically give each term a very specific, and distinct, meaning. In particular, the word ‘taste’ is used to
describe only those sensations primarily associated with the stimulation of the taste-buds, namely sweet-
ness, sourness, bitterness, saltiness, and the savoury taste of umami. By contrast, the word ‘flavour’ is used
to refer to the experiences resulting from the simultaneous stimulation of the taste buds and the olfactory
receptors in the nasal epithelium. In order to avoid any confusion, this is also how the two terms will be
used in the present article.
64 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

applying their understanding of the mechanisms underlying multisensory integration borrowed


from investigations of audiovisual or visuotactile integration to the study of flavour perception
(e.g. Auvray and Spence 2008; Spence 2010b; Verhagen and Engelen 2006). Given that we now
understand more about the multisensory perception of flavour in adults, I would argue that we
are in a better position than ever before to examine how the senses converge to influence flavour
perception developmentally. Below, I review what is currently known about the development and
decline of multisensory flavour perception across the human lifespan (the focus will primarily be
on the role that visual cues play in modulating taste and flavour perception). One caveat at the
outset is that there is not as yet a great deal of evidence relevant to this question, at least not when
compared to other areas of developmental perception (see, for example, the other chapters in this
volume). What is more, many of the studies that have been published to date have generated
results that are either seemingly mutually inconsistent or else have provided only relatively weak
empirical evidence for the claims being made by the authors.

3.2 Which senses contribute to flavour perception?


Our enjoyment of food and drink comes not only from the unified oral sensation of taste and
smell (both orthonasal and retronasal),2 but also from the sound it makes, not to mention what
it looks like. The oral-somatosensory qualities of foods are also very important: texture, tempera-
ture, and even pain, as in the case of eating chilli peppers (see Green 2002), all contribute to the
overall multisensory flavour experience (or gestalt; Spence 2010b; Verhagen and Engelen 2006).
A number of reviews of multisensory flavour perception in adults have been published over
the last few years (e.g. see Spence 2012; Stevenson 2009; Verhagen and Engelen 2006, for some
representative examples). Therefore, given the developmental theme of this volume, I will not
discuss the adult data in any great detail here. I do, however, want to highlight the important
distinction between taste and flavour: the ‘basic’ tastes, which can be detected by receptors on the
human tongue (intriguingly, there also appear to be gustatory receptors in the gastrointestinal
tract; see Egan and Margolskee 2008), consist of sweet, sour, bitter, salty, umami, and metallic
(see Erikson 2008; Spence 2010b). By contrast, flavour perception involves the stimulation of
retronasal olfaction, gustation (i.e. taste), and on occasion oral irritation (transduced by the
trigeminal nerve). It is the combination of odours and tastes that gives rise to the perception of
fruit flavours, meaty flavours, etc. (see Spence et al. 2010). To put the relative contribution of
these two senses into some perspective, it is frequently stated that as much as 80% of our percep-
tion of flavour comes from the information provided by the nose (rather than from the tongue;
e.g. see Martin 2004; Murphy et al. 1977; although note that it is unclear whether this figure
should be taken to refer to the perception of intensity or to the identification of flavour).
According to The International Standards Organization (ISO 5492, 1992), flavour is a ‘complex
combination of the olfactory, gustatory and trigeminal sensations perceived during tasting.
The flavour may be influenced by tactile, thermal, painful and/or kinaesthetic effects.’ (see
Delwiche 2004, p. 137). Visual and auditory cues may modify a food’s flavour, but according to

2 Researchers now believe that there are two relatively distinct olfactory sensory systems (see Chapter 2 by
Schaal and Durand). One system (which is older in phylogenetic terms), associated with the inhalation of
external odours, is known as orthonasal olfaction. The other (newer system involving the posterior nares) is
associated with the detection of the olfactory stimuli emanating from the food we eat, as odours are peri-
odically forced out of the nasal cavity when we chew or swallow food, and is known as retronasal olfaction.
It is an interesting, although as yet unanswered, question as to whether orthonasal and retronasal olfaction,
in addition to their different phylogenetic origins, also have different developmental trajectories.
WHICH SENSES CONTRIBUTE TO FLAVOUR PERCEPTION? 65

the ISO definition at least, they are not intrinsic to it. However, many other researchers disagree
with what they see as an overly restrictive definition and have argued that all five of the major
senses can and do contribute to the multisensory perception of flavour (e.g. see Auvray and
Spence 2008; Stevenson 2009). To make matters all the more complicated, visual cues, such as a
food’s colour, may modify the perception of a food’s flavour by influencing the gustatory qualities
of the food, by influencing the olfactory attributes of the food (as perceived orthonasally and/or
retronasally; Koza et al. 2005), by influencing the oral-somatosensory qualities of the food,
and/or by influencing the overall multisensory flavour percept (or gestalt; see Fig. 3.1). As yet, it
is not altogether clear at which stage(s) vision interacts with the other senses. Furthermore, vari-
ous top-down factors play a profoundly important role in modulating our responses to foods as
well (see Rozin and Fallon 1987; Yeomans et al. 2008).
Here, I would like to argue, as I have done elsewhere (see Spence et al. 2010), that the ISO
definition is overly restrictive, and that audition should be included in the definition of flavour,
whereas (contrary to the claims of a number of contemporary neuroscientists: Stevenson 2009;
Verhagen and Engelen 2006; see also Auvray and Spence 2008) vision should not, and hence its
influence should be considered as crossmodal. (By crossmodal, I mean that one sense influences
another without the two sensory inputs necessarily being integrated into a unified perceptual
whole, or gestalt.) The reason, I would like to argue, why audition should be included is that both
spatial and temporal coincidence play a critical role in terms of what we hear, influencing our

Olfaction Gustation
Flavour

Oral-
somatosensation
Fig. 3.1 This figure highlights the multiple ways in which visual cues might influence flavour
perception. Visual cues (such as the colour of a beverage) may exert a crossmodal influence on
olfaction, gustation, and/or on oral-somatosensation. Such crossmodal effects, should they exist,
might then have a carry-over effect on the experienced multisensory flavour percept once the various
unisensory cues have been integrated. Alternatively, however, visual information might influence
flavour perception only once the olfactory, gustatory, and/or oral-somatosensory cues have been
integrated into a multisensory flavour percept. Unfortunately, as yet, there is no clear answer with
regard to the way(s) in which vision exerts its effect on multisensory flavour perception. (Reproduced
from Charles Spence, Does Food Color Influence Taste and Flavor Perception in Humans?,
Chemosensory Perception, 3 (1), pp. 68-84, © 2010, Springer Science + Business Media.)
66 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

perception of a food or drink’s flavour. Spatiotemporal coincidence also plays a critical role in
modulating any oral-somatosensory contribution to flavour perception. By contrast, the effect of
vision usually occurs despite the fact that visual cues are experienced in a different location and
time from the other flavour cues (e.g. in the mouth). The reason why vision should probably not
be included in the definition of flavour is that it may exert its effect on flavour perception by
setting-up an expectation (just like a label or verbal description) about the likely identity/intensity
of what we are about to taste/consume. True, those expectations can modulate the experienced
taste and flavour of the food in the mouth, but the rules of spatial and temporal correspondence
are not as strict as is typical in other examples of multisensory integration.
In this review, I will focus on the effect of visual cues on taste and flavour perception. First,
however, I briefly review the evidence concerning our early flavour learning experiences.

3.3 Prenatal and perinatal flavour learning: gustation and


olfaction
In terms of development, the senses of taste and smell (technically referred to as gustation and
olfaction, respectively) begin to develop after touch, which starts about 7 weeks after fertilization.
The foetus starts to breathe, inhaling and exhaling amniotic fluid around 9–10 weeks after con-
ception, and has a functioning olfactory epithelium by 11 weeks (Doty 1992). Specialized taste
cells appear around the seventh or eighth week of gestation in the human foetus, with structurally
mature taste buds emerging after 13–15 weeks (Bradley and Mistretta 1975; Bradley and Stern
1967; Cowart 1981; Mennella and Beauchamp 1994). During the later stages of gestation, the
foetus takes in considerable amounts of amniotic fluid, inhaling more than twice the volume it
swallows (Ganchrow and Mennella 2003). The development of the senses of taste and smell there-
fore occurs well before the development of fully functional auditory (about 6–7 months) or visual
receptors (see Gottlieb 1971).
While certain of our responses to basic tastes are present at birth, the majority (but by no means
all) of our responses to odours are learnt (Khan et al. 2007; see also Chapter 2 by Schaal and
Durand). We are all born liking sweet- and disliking sour-tasting foodstuffs, while being indiffer-
ent to bitter- and salty-tasting solutions (e.g. Birch 1999; Desor et al. 1973, 1975). Our liking for
salt appears to emerge after approximately 4–6 months (Desor et al. 1975), while a liking for bitter
substances emerges much later in life (see Mennella and Beauchamp 1994, for a review). We are
born liking certain of the flavours/smells of the foods that our mothers happen to have consumed
during pregnancy (e.g. Abate et al. 2008; Schaal et al. 2000; see also DeSnoo 1937; Ganchrow 2003).
It turns out that flavours from a mother’s diet are transmitted to the amniotic fluid and breast milk
and thus swallowed by the foetus and neonate during and after pregnancy (Blake 2004). In turn,
newborns tend to find the odour of their mother’s amniotic fluid attractive (e.g. Varendi et al.
1996). Given that our earliest flavour learning takes place in the womb and at our mother’s breast
(Galef and Sherry 1973; Hausner et al. 2008; Mennella 1995; Mennella and Beauchamp 1994;
Mennella et al. 2001), crossmodal olfactory–gustatory flavour learning presumably starts before
functional vision has come online (i.e. with the opening of a baby’s eyes at, or after, birth; see
Chapter 2 by Schaal and Durand, for further discussion of early flavour learning).

3.4 The later development of multisensory flavour perception:


gustation and olfaction
Olfactory–gustatory flavour learning continues well into adulthood. The latest research from
Stevenson and his colleagues in Australia have demonstrated that novel food odours (i.e. odours
that do not themselves elicit any taste percept when initially presented in isolation) can come to
THE LATER DEVELOPMENT OF MULTISENSORY FLAVOUR PERCEPTION 67

take on specific taste qualities for adults (e.g. Stevenson and Boakes 2004). So, for example, in the
West, adding the tasteless odour of strawberry or vanilla, say, to a drink will make it taste sweeter
(see also Frank and Byram 1988). Lavin and Lawless (1998) have demonstrated that both adults
(18–31 years old) and children (5–14 years old) show this crossmodal enhancement effect, rating
low-fat milk drinks as tasting sweeter when tasteless vanilla odor is added than when it is absent.
This form of crossmodal associative learning takes place very rapidly. Within a few trials of a
novel odourant being paired with a particular tastant (such as pairing the odour of water chestnut
with a sweet taste for a Western European participant), the odour comes to take on the properties
of the tastant (e.g. see Stevenson et al. 1995, 1998). In further experiments, Stevenson and his
colleagues have gone on to demonstrate that a given novel odorant can actually be associated with
a variety of different tastants. So, for example, it turns out that it is just as easy to pair the aroma
of water chestnut with a bitter taste, should a participant first experience it (i.e. the aroma)
together with a bitter taste (see Stevenson 2012, for a review). In the context of the present review,
it would be intriguing to determine whether there is any fall-off in this ability to learn novel taste–
odour associations in old-age (cf. Nusbaum 1999), given the absence of evidence on this question
at present.
Taste and flavour preferences change across the lifespan: for example, children between 9 and
15 years of age appear to like sweet (e.g. sugar), salty (sodium chloride), and extremely sour tastes
more than adults (see Desor et al. 1975; Liem and Mennella 2003), but often avoid anything
remotely resembling a vegetable (Blake 2004; Horne et al. 2004, 2009). In part, this may reflect an
aversion to bitter tastes, a sensible evolutionary strategy for a young child given that bitter-tasting
foods are often poisonous in nature (Bartoshuk and Duffy 2005; Glendinning 1994). There is,
though, also some evidence to suggest that young children may be more sensitive to bitter tastes
than adults (see Cowart 1981; Harris 2008; Mennella and Beauchamp 2010, for reviews). In fact,
as adults, our liking for bitter foods emerges in many cases as the result of social conditioning
and/or the pairing of the unpleasant taste (e.g. of caffeine) with sugar (many people start drinking
sweetened coffee; cf. Zellner et al. 1983) or with the pleasant physiological consequences of other
pharmacologically-active bitter substances, such as the caffeine in coffee or the ethanol in alco-
holic beverages (see Blake 2004; Mennella and Beauchamp 2010).
Recent research from Houston-Price et al. (2009) has demonstrated that merely (‘visually’)
exposing 21–24-month-old toddlers to pictures of fruit and vegetables can influence their subse-
quent willingness to taste those fruits and vegetables (though see also Birch et al. 1987). When
such results are put together with earlier findings showing that adults like unfamiliar fruit juices
more, the more that they have tried (or been exposed to) those flavours previously (Pliner 1982),
the suggestion that emerges is that mere exposure effects (both in the womb, see above, and after
birth) also help to explain many of the changes in the liking for various tastes/flavours that occur
over the course of human development (see Capretta et al. 1975; Harris 2008).
The available evidence suggests that the different sensory attributes of a food (such as its aroma,
flavour, colour, texture, shape, and/or temperature) may play different roles in people’s food
preferences at different ages. So, for example, research comparing the preferences of children
(mean age of 11 years) with those of young adults (with a mean age of 20 years) has revealed that,
if anything, sweetness (e.g. in a soft drink) is more important to children than to adults, whereas
adults tend to rate visual appearance and odour as being more important sensory attributes than
do younger children (Tuorila-Ollikainen et al. 1984). Colour preferences in foodstuffs may also
differ as a function of age: for example, while Lowenberg (1934) reported that preschool children
preferred orange and yellow, Walsh et al. (1990) observed that both 5- and 9-year-old children
preferred red, green orange, and yellow candies in that order (see also Marshall et al. 2006).
However, given the paucity of research in this area, and the large temporal separation between the
studies just reported, further (possibly longitudinal) research will clearly be needed before any
68 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

firm conclusions can be drawn with regards to changes in preferred food colours over the course
of development. During stages of maximal growth, humans ought, if anything, to express an
increased liking for carbohydrates (e.g. sweet-tasting foods; Drewnowski 2000; Mennella and
Beauchamp 2010). Consequently, given the correlation in nature between ripeness and sweetness
(e.g. in fruits; Maga 1974), one might expect that red foods ought to be particularly appealing to
children during these periods of maximal growth (when their energy requirements are at their
peak).
While there has been a recent growth of interest in studying the nature of any oral-somatosen-
sory contributions to flavour perception (e.g. Bult et al. 2007), this research has so far primarily
only been conducted in adults (see Spence 2012, for a review; though see Blossfeld et al. 2007, for
a solitary study examining texture perception in 12-month-old infants). Similarly, the resurgence
of interest in auditory contributions to food texture and flavour perception over the last few years
has been restricted to studies conducted on adults (normally college-age students; see Spence and
Shankar 2010; Zampini and Spence 2004, 2010). Hence, there is not yet really a developmental
story to tell concerning any changes in the role of oral-somatosensory or auditory cues to multi-
sensory flavour perception across the life-span. Given that the majority of developmental studies
of multisensory flavour perception have tended to focus on assessing the (changing) crossmodal
influence of visual cues on taste and flavour perception, it is on these studies that we will focus in
the sections below.

3.5 How might the influence of vision in flavour perception be


expected to change over the course of development?
At the outset, two main contrasting predictions can be put forward in terms of how the influence
of visual (specifically colour) cues might be expected to change over the course of early human
development:
1. According to the most commonly expressed view in the literature, we are all born with dis-
tinct sensory systems, and we learn to integrate the cues (or information) provided by each
sense over the course of development as a result of our experience of crossmodal correlations
between the patterns of stimulation presented in the different modalities in the environment
(see Lewkowicz and Lickliter 1994). According to this account (e.g. Christiansen 1985; Lavin
and Lawless 1998), the influence of the colour of a food or drink on our perception of flavour
ought to increase over the first few years of life as we come to learn the correlations that exist
in nature (or, for that matter, in the supermarket; see Shankar et al. 2010a) between colour and
flavour (Wheatley 1973). So, for example, we might come to learn that in many fruits there
is an association between redness (e.g. ripeness) and sweetness (Maga 1974). Similarly, in the
case of many processed foods and drinks in the supermarket, there tends to be a crossmodal
association between the intensity of the colour and the intensity of the taste/flavour. Conse-
quently, over time, certain colours may be expected to come to signify (or lead to the expecta-
tion) that they will be followed by certain tastes/flavours (this is sometimes described as ‘visual
flavour’).3 Such learning presumably starts very early. Indeed, the available evidence suggests
that by 4 months of age, female infants have already started learning crossmodal associations

3 However, the source of the crossmodal association is not always easy to figure out. Gilbert et al. (1996), for
example, have highlighted the existence of certain reliable colour–odour associations present in adults
where it is currently much harder to fathom where people might have come across these associations in
nature (see also Schifferstein and Tanudjaja 2004; Spector 2009).
HOW MIGHT THE INFLUENCE OF VISION IN FLAVOUR PERCEPTION CHANGE? 69

between odours and colours/shapes (Fernandez and Bahrick 1994; see also Hurlbert and Ling
2007; Reardon and Bushnell 1988; Spence 2011).
2. However, according to an alternative view of the development of multisensory perception, we
are all born confusing our senses (the ‘blooming buzzing confusion’ mentioned by William
James, 1890), and we learn through experience (and possibly as a result of parcellation; see
Maurer 1997; Maurer and Mondloch 2005) to distinguish between the attributes that rightfully
belong to each of the senses over the course of early development (see Lewkowicz and Lickliter
1994). According to the latter view, one might expect the influence of visual cues on multisen-
sory flavour perception to decline over the course of development, as individuals become
increasingly competent at individuating their sensory experiences. The development of the
ability to direct one’s attention/cognitive resources to the inputs associated with a particular
sense may also aid this process of individuating sensory inputs (i.e. pulling apart the multisen-
sory flavour gestalt), and, once again, likely improves over the course of development. It would
seem plausible that younger children might find it harder to focus on the flavour of food and
hence might be more easily distracted than adults by any highly salient changes in the colour of
food. Indeed, there is some evidence that the highly-developed ability to focus one’s attention
solely on the taste (e.g. sweetness) of a foodstuff (such as a wine), and not be influenced by the
aroma—that is, the ability to treat flavours analytically—is something that can only be acquired
(typically in adulthood) as a result of extensive training (e.g. Prescott et al. 2004).
Of course, predictions regarding the changing role of vision in multisensory flavour perception
are also going to be complicated by the fact that acuity in each of the senses develops/declines at
different rates/times (see below; cf. Zampini et al. 2008). As we will see below, the weight of the
(admittedly limited) empirical evidence appears to support the view that the crossmodal effect of
visual cues on flavour perception (specifically flavour identification) declines over the course of
early development (i.e. up to adulthood). With regards to what happens at the other end of the
age spectrum (i.e. in old age), the extant evidence (although limited) currently supports the view
that the impact of visual (colour) cues of multisensory flavour perception increases slightly.
Before we come to evaluate the developmental data with regards to vision’s influence on mul-
tisensory flavour perception, however, it is worth noting that while there is good evidence that
visual cues (specifically relating to colour) play an important role in the perception of flavour
identity in adults (e.g. see DuBose et al. 1980; Stillman 1993; Zampini et al. 2007, 2008, for empir-
ical evidence), the literature with regard to the effects of colour intensity on taste and flavour
intensity is much more mixed (e.g. Lavin and Lawless 1998). Indeed, the literature on adults has
not yet delivered a particularly clear story in terms of the effects of changes in colour identity (i.e.
hue) or colour intensity (i.e. saturation) on either taste or flavour perception (see Spence et al.
2010, for a review). Thus, taken as a whole, the literature on adults currently supports the view
that colour has a much more reliable crossmodal effect on flavour judgments than on taste judg-
ments (see Spence et al. 2010, for a review).4 Below, we will see that the same appears to hold true
in the developmental data. (Remember here that flavour judgments include such attributes as
fruitiness, spiciness, etc. that involve the contribution of both olfaction and taste, whereas taste
judgments refer just to basic tastes: sweetness, sourness, bitterness, saltiness etc.).

4 Note here that the strongest crossmodal effects appear to be on qualitative judgments (e.g. of flavour
identity) rather than on quantitative judgments (e.g. of taste or flavour intensity). While it is possible that
colour might also have an effect on qualitative judgments of taste identity (e.g. sweet versus sour), no one
has conducted (or at least published) such a study to date (see Spence et al. 2010).
70 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

3.6 Developmental changes in the crossmodal influence of


visual cues on flavour perception
Oram et al. (1995) conducted one of the only studies to have looked at the influence of visual cues
on multisensory flavour perception that specifically tested several different age groups of children
using exactly the same experimental method. In their study, over 300 visitors to a university open
day in Australia were presented with a tray of four drinks whose flavour they had to try and dis-
criminate. In total, 16 drinks were prepared for use in this study, resulting from the crossing of
four possible flavours (chocolate, orange, pineapple, and strawberry) and four possible colours
(brown, orange, yellow, and red). Four of the possible colour-flavour combinations in Oram
et al.’s study were congruent (as determined by the experimenters) while the remaining twelve
were deemed to be incongruent (though see Shankar et al. 2010a on the problematic notion of
congruency in this area of research). Each colour and flavour was represented once on the drinks
tray presented to each participant. The participants were not given any information about the
colours of the drinks and whether or not they might be meaningfully related to the flavours of the
drinks. After tasting each of the drinks, the participant had to try and discriminate whether it had
a chocolate, orange, pineapple, or strawberry flavour. The four choices were written on a card
next to the participant. Additionally, an actual chocolate bar, an orange, a pineapple, and a carton
of strawberries were each also placed next to the appropriate card.
The results of Oram et al.’s (1995) study (see Fig. 3.2) highlighted a clear developmental trend
toward an increased ability to correctly report (i.e. discriminate) the actual flavour of the drinks,

100

90 Colour-associated
Flavour-associated
80
Not associated with flavour or colour
Percentage option selected

70

60

50

40

30

20

10

0
2 to 7 8 and 9 10 and 11 12 to 18 Adult
Age group (in years)
Fig. 3.2 Graph highlighting the percentage of trials in which the participants’ flavour discrimination
response matched the colour of the drink, the actual flavour of the drink, or matched neither the
colour or flavour of the drink as a function of the age of the participants in Oram et al.’s (1995)
study. (Adapted from Nicholas Oram, David G. Laing, Ian Hutchinson, Joanne Owen, Grenville Rose,
Melanie Freeman, and Graeme Newell, The influence of flavor and color on drink identification by
children and adults, Developmental Psychobiology, 28 (4), pp. 239–49, (c) 1995, John Wiley and
Sons, with permission.)
DEVELOPMENTAL CHANGES IN THE CROSSMODAL INFLUENCE OF VISUAL CUES ON FLAVOUR PERCEPTION 71

regardless of their colour. That is, the crossmodal modulation of flavour perception by vision
apparently decreases in a fairly orderly manner with increasing age. Although Fig. 3.2 collapses the
data across all 16 of the possible colour–flavour combinations tested in the study, similar results were
apparently observed for each colour and flavour when they were examined individually. The only
noticeable exceptions to this generalization was that the participants were somewhat more likely to
respond on the basis of flavour for the chocolate-flavoured drink, and more likely to respond on the
basis of colour (i.e. responding ‘strawberry’) when a drink was coloured red (consistent with previous
research showing that red appears to be a particularly powerful colour in terms of modulating flavour
perception; see Spence et al. 2010). That said, more than 80% of the participants in each age group
identified the flavour of the drinks correctly when they were coloured congruently.
Oram et al. (1995) suggested that the most likely explanation for the developmental trend high-
lighted by their data was that, with increasing age, children become better able to focus their
attention on the flavour of food and drink items. Hence their judgments become less and less
influenced by any expectations that they may have regarding the likely flavour of the drink that
happens to be based on its colour. Note here that younger children have sometimes been shown
to be more influenced in their judgments of stimuli by the background within which that stimu-
lus happens to be presented than adults (see, for example, Moskowitz 1985). Oram et al. preferred
the quantitative change account of sensory dominance to the alternative possibility that there may
be an age-dependent qualitative change (or switch) in the reliance on specific sensory cues (visual
versus flavour-based) that children may exhibit. (Of course, it is worth bearing in mind here,
given the uncertainty that surrounds the definition of flavour in adults that it might not be
surprising if children’s understanding of the term were to change with age too.)
None of the participants in Oram et al.’s (1995) study was informed that the colours of the
drinks might be misleading with regard to their actual flavours. Hence, age-dependent changes in
the effects of task demands on participants’ performance cannot be ruled out as a potential factor
influencing Oram et al.’s results (cf. Zampini et al. 2007 for similar concerns with much of the
adult data in this area). That is, younger children may simply be more likely to assume (in the
context of an experimental setting) that the colour is likely to provide a meaningful indicator of a
drink’s flavour (that, or perhaps, children might simply include colour in their definition of fla-
vour at a younger age). By contrast, as adults we may all be more wary of the possibility of trickery
in the context of ‘a food experiment’. It could, however, be argued that the very fact that (for
whatever reason) children are more strongly influenced by colour than are adults when judging
flavour identity is, in itself, an interesting observation. Perhaps then the argument here can best
be framed in terms of there being uncertainty (on the basis of the study presented) about whether
this developmental change in the influence of vision should be thought of as reflecting an auto-
matic crossmodal effect on multisensory integration, or instead some more voluntary attentional
strategy that changes as a function of age.
Finally, when thinking about the results of Oram et al.’s (1995) study, it is worth noting that
even as adults, we tend to be particularly bad at identifying odours (e.g. Cain 1979; see Zellner
et al. 1991, for a review). It is therefore a shame that Oram and colleagues did not collect any data
concerning the baseline olfactory discrimination/identification abilities across the various age
groups that they tested. Without such data, it becomes difficult to rule out the possibility that any
developmental changes that they observed might, in part, simply have reflected the consequences
of age-related changes in olfactory and/or gustatory perception, rather than any age-related
changes in multisensory integration/perception per se (see Doty et al. 1984; Ganchrow and
Mennella 2003; see also Bryant 1974).
It is interesting to compare the slow and gradual change in sensory dominance observed
in Oram et al.’s (1995) study with the rather more sudden changes seen in Gori et al.’s (2008)
recent study of visual–haptic multisensory integration. There, children younger than 8 years of
72 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

age were found to use visual (rather than haptic) cues in order to judge the orientation of an
object (and haptic rather than visual cues in order to judge an object’s size). This total dominance
of one sense over the other had switched to a response strategy based on statistically optimal
multisensory integration (i.e. weighting each estimate according to its reliability according to
maximum likelihood estimation) by the time that children reached 8–10 years of age. The changes
in visual dominance observed by Oram et al. would appear to have been taking place much more
slowly.
More generally, though, one might ask whether the maximum likelihood estimation account
(or Bayesian decision theory) could be used to explain the developmental changes in multisen-
sory flavour perception data. One problem with applying this approach to flavour identification
is that most work on Bayesian decision theory has to date focused on situations in which people
have to make quantitative judgments (e.g. of relative size or position) rather than qualitative judg-
ments (such as what shape or speech sound is it). While it may be possible to model certain kinds
of qualitative (or categorical) judgments in terms of Bayesian decision theory (cf. Helbig and
Ernst 2008, pp. 13–14), it is, at present, by no means clear that it will be possible to do so for
categorical judgments such as those involved in flavour identification that are more difficult to
transcribe onto any kind of meaningful continuum.
It is also important to note here that the influence of visual cues on multisensory flavour
perception does not necessarily obey either the spatial or temporal rules (Shankar et al. 2010b):
that is, the sight of a drink on the table can still influence a person’s perception of flavour in
their mouth, despite the fact that the location of the cues is different. Similarly, colour cues are
normally available some time before the flavour of the food is actually experienced in the mouth.
This has led some researchers to argue that visual cues may be better conceptualized as influenc-
ing flavour perception by means of expectancy effects (Cardello 1994; Hutchings 2003; Spence
et al. 2010) rather than by multisensory integration based on the spatial and temporal rules
derived from single-cell neurophysiology (e.g. see Stein and Meredith 1993; see also Spence 2012;
Stevenson et al. 2000).
While a mechanistic explanation of expectancy effects in multisensory flavour perception is still
lacking (see Cardello 1994; Spence et al. 2010), one currently appealing way of thinking about
the integration/influence of colour on flavour is in terms of Bayesian priors (cf. Ernst and Bülthoff
2004; Shankar et al. 2010c; Spence 2011). That is, most likely through experience, we may
build up Bayesian priors concerning the fact that certain food colours normally co-occur with
certain flavours. Of course, it could be argued that Bayesian priors need not be learned, but could
perhaps reflect some bias in the way in which the brain happens to represent different kinds
of information neutrally (see Scholl 2005; Spence 2011). Only further developmental research
will allow us to distinguish between these various possibilities, although, at present, it is probably
safe to say that the majority of researchers favour the experience-based learning account. Red,
for example, often co-occurs with sweetness, while the majority of green fruits are sour (Kostyla
1978; Maga 1974). Given the commercial opportunities associated with being able to model
and predict multisensory flavour perception, it seems likely that within a few years researchers
will have extended Bayesian decision theory to try and account for the contribution of visual
cues to the perception of flavour. Neuroimaging research may, of course, also help researchers
to understand the neural mechanisms underlying the influence of visual cues on multisensory
flavour perception (Österbauer et al. 2005; Skrandies and Reuther 2008; see also De Araujo et al.
2003).
One other influential study to have looked for developmental changes in terms of vision’s influ-
ence on multisensory flavour perception was reported by Lavin and Lawless (1998). They conducted
an experiment in which the influence of colour intensity on ratings of sweetness intensity (i.e. on
CHANGES IN THE INFLUENCE OF COLOUR ON TASTE/FLAVOUR IN ADULTHOOD 73

taste rather than flavour judgments) in North American children and adults was investigated. The
participants were given two pairs of strawberry-flavoured beverages to compare and to rate in
terms of their sweetness (using a nine-point scale). One pair consisted of light- and dark-red
drinks while the other pair consisted of light- versus dark-green drinks. All of the drinks actually
had the same physical sweetness, varying only in terms of their colour. Lavin and Lawless tested
three groups of children (5–7 years, 8–10 years, and 11–14 years) and a group of adults.
The results showed that the adults rated the dark-red and light-green samples as being sweeter
than the light-red and dark-green samples, respectively. By contrast, colour intensity did not
have a significant effect on the responses of the younger age groups (although, if anything, the
11–14 year olds showed a trend in the opposite direction to that of the adults). In contrast to
Oram et al.’s (1995) results, then, Lavin and Lawless’s (1998) results demonstrate that changes in
the level of food coloring appear to have more of an effect on sweetness judgments in adults than
in children. Meanwhile, in another North American study, Alley and Alley (1998) demonstrated
no effect of colour (red, blue, yellow, green, or colourless) on the perceived sweetness (rated on a
ten-point scale) of sugar solutions served in either liquid or solid (i.e. gelatin) form to a group of
11–13 year olds.
However, it is perhaps worth pointing out at this point that judging the degree of sweetness,
as in the studies of Lavin and Lawless (1998), and Alley and Alley (1998), is not the same thing as
trying to identify (or discriminate) the flavour, as in Oram et al.’s (1995) study. What is more,
children are more likely to be able to perform relatively easy categorical judgments (i.e. identify-
ing the flavour of a beverage), than quantitative intensity judgments (i.e. judging how sweet a
drink happens to be). These possibilities raise the suggestion that the developmental story with
regards to colour’s changing influence on flavour perception may not be a simple one. Perhaps,
just like for the literature on adults (see Spence et al. 2010), colour may exert a qualitatively
different effect on flavour (or taste) identification versus on taste/flavour intensity judgments (see
also Koza et al. 2005). At present, the strongest evidence regards colour’s influence (in particular,
the hue of the colour) on flavour identification judgments and the decline of this crossmodal
influence over early development. I would argue that further research is really needed here before
one can draw any firm conclusions regarding the existence of developmental changes in the effect
of colour intensity (or saturation) changes on the perception of taste/flavour intensity.

3.7 Changes in the influence of colour on taste/flavour in


adulthood: the role of expertise
Barring accident, the sensitivities of the human senses do not change much over the course of
adulthood (e.g. between the ages of 18 and 50 years). There is some gradual decline, but the more
severe drops in sensory acuity have yet to occur (see Section 3.8). Hence, the only significant
changes (or development) that one is likely to see during this period relate to the altered sensory
perceptions of those who acquire an expertise in a particular domain of flavour perception. Of
such experts, the most widely studied have been wine tasters (e.g. see Lehrer 2009; Parr et al. 2002;
Spence 2010a). Researchers have, for example, demonstrated that when the students on a univer-
sity degree course in oenology in Bordeaux (i.e. in some sense experts) were given a glass of white
wine that had been artificially coloured red, they could be fooled into smelling the aromas that
they normally (and had previously) associated with a red wine (Morrot et al. 2001).
According to the suggestion put forward earlier, the presence of colour in food influences
flavour perception by means of the expectations that those colours set up in the mind of the
observer. What is more, the stronger the expectations, the stronger the crossmodal influence
of colour on flavour identification is likely to be (e.g. Hutchings 2003; Shankar et al. 2010b, c;
74 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

Shankar et al. 2010c).5 If one accepts the logic of this argument then one would expect that wine
experts ought to be more strongly influenced by inappropriate coloration than less experienced
drinkers (see Spence 2010a). That, indeed, is what has now been demonstrated. So, for example,
in an early study, Pangborn et al. (1963) gave expert and non-expert wine drinkers a set of dry
white wines that had been coloured pink, yellow, brown, red, or purple to simulate a rosé, or
blush, wine, Sauternes, sherry, claret, and Burgundy wine, respectively. The experts judged the
pink wine as tasting sweeter than when no colouring had been added, while the non-experts’
sweetness judgments were unaffected by the addition of colour to the drinks.
Parr et al. (2003) conducted a follow-up to Morrot et al.’s (2001) study in New Zealand, but this
time they tested both experts (including professional wine tasters and wine makers) and ‘social’
drinkers. They demonstrated that the experts’ descriptions of the aroma of a Chardonnay when it
was coloured red were more accurate when it was served in an opaque glass than when it was
served in a clear glass. This colour-induced biasing of their olfactory flavour judgments occurred
despite the fact that the experts had been explicitly instructed to rate each wine irrespective of its
colour (thus suggesting that this crossmodal effect of vision is not under cognitive control; cf.
Stillman 1993; Zampini et al. 2007). When the same experiment was conducted in social drinkers,
however, it turned out that they were so bad at reliably identifying the aromas present in the wine
that it was difficult to discern any pattern in the data when an inappropriate wine colour was
added. Nevertheless, taken together, the evidence that has been published to date is consistent
with the view that expert wine drinkers differ from social wine drinkers (i.e. non-experts) in the
degree to which visual (colour) cues influence their orthonasal perception of flavour (Parr et al.
2003) and their perception of sweetness (Pangborn et al. 1963).
That said, not all food/flavour experts exhibit the same increased responsiveness to visual
colour cues when evaluating flavour or taste. For example, Shankar et al. (2010d) recently reported
that flavour experts (those working on a descriptive panel at an international flavour house,
and who all had more than three years of experience flavor profiling food and drink products)
exhibited an equivalent amount of visual capture over their orthonasal olfactory flavour judg-
ments as non-experts (i.e. normal people). All of the participants who were selected to take part
in the main study were shown, in pre-testing, to expect a purple-coloured drink to taste of grape
and an orange-coloured drink to taste of orange. These colours were found to bias both groups
of participants’ judgments on the critical experimental trials when the cranberry- or blueberry-
flavoured drinks were coloured purple and when the grapefruit- and lemon-flavoured drinks
were coloured orange. Thus, the conclusion from the research that has been published to date
on flavour experts would appear to be that while some experts (specifically those with an expertise
in wine—the same may also go for tea and coffee experts) show an enhanced susceptibility to
the crossmodal influence of colour on their judgments of food and drink items within their area
of expertise (Pangborn et al. 1963; Parr et al. 2003), this pattern of results does not necessarily
extend to other groups of flavour experts (Shankar et al. 2010d; see also Lelièvre et al. 2009;
Teerling 1992).

5 According to this account, it should not matter much whether the expectation happens to be set-up by the
colour of a food or drink item, or simply by the name of the colour itself (if, say, the participant happened
to be blindfolded). While this has never been tested for the perception of flavour, the results of a study by
Davis (1981) demonstrated that simple colour word cues can be as effective as colour patches in modulat-
ing a participant’s odour identification responses. That said, it could also be argued that the assumption of
unity will be stronger when the colour comes from the food or drink itself, rather from labeling/verbal
description, packaging colour etc. (see Shankar et al. 2010b).
THE DECLINE OF MULTISENSORY FLAVOUR PERCEPTION 75

At present, it is not clear what explains these differences in the modulatory effect of visual
(colour) cues on the taste and flavour judgments of different groups of flavour experts. Several
possibilities spring to mind, including whether or not the participants in these various studies
were aware that the colour of a foodstuff that they were evaluating may have been misleading
(cf. Stillman 1993; Zampini et al. 2007, 2008). That is, experts may be more influenced by the
colour of a food or drink item if they believe that it is informative with regard to the taste, aroma,
and/or flavour, while at the same time being better able to discard the information before their
eyes (and adopt an analytic approach to tasting; Prescott et al. 2004) if they have reason to believe
that the colour may be misleading. It is perhaps also worth noting that while there is typically a
meaningful relationship (or correlation) between the colour of a ‘natural’ product such as wine
(and presumably also coffee and tea) and its taste/aroma/flavour properties (see Spence 2010a),
the same is not necessarily true of the (synthetic and/or processed) coloured foodstuffs that the
flavour experts studied by Shankar et al. (2010d) would normally have to evaluate (where the
relationship between colour and flavour is often manipulated artificially to obtain a particular
commercially-desirable outcome).
The one other way in which the influence of colour on flavour perception might be expected to
change during adulthood relates to cohort effects in terms of exposure to different foods (cf. Blake
2004; Lavin and Lawless 1998). In fact, some of the latest research from the Crossmodal Research
Laboratory here in Oxford has suggested that exposure to different products in the marketplace
(and hence presumably in one’s diet) can influence which flavours different individuals expect
different coloured foods and drinks to taste of (see Shankar et al. 2010a). The participants in
Shankar et al.’s study were simply asked to look at a set of six coloured liquids presented in trans-
parent plastic drinking cups (see Fig. 3.3) and to report what flavour they would expect a drink of
that colour to have. That is, the flavour expectations of the participants were based solely on the
colour of the beverage that they saw. The results showed that the bright clear blue-coloured drink
(see Fig. 3.3B) was associated with raspberry flavour in a group of young British participants but
with the flavour of mint in a group of young Taiwanese participants. While such ‘arbitrary’ cross-
modal associations might at first seem perplexing, Shankar and her colleagues suggested that the
young adult British participants in their study may have picked up the blue–raspberry association
from fruit drinks (such as Cool Blue Gatorade which has a raspberry taste), whereas in the absence
of such products in the marketplace, the Taiwanese participants may have associated the
blue colour with the minty taste of mouthwash instead. While such differences in colour–flavour
associations were demonstrated cross-culturally in Shankar et al.’s study, similar trends are likely
to be present as a function of age within a culture too, given the different patterns of consump-
tion. What is clear is that certain flavours are likely to be more familiar to those of more advanced
years (cf. Blake 2004). Think here only of Palma violets or the heavy scent of patchouli that
are indelibly linked to the 1960s for many of those of a certain age. Angel Delight™, introduced
into the UK marketplace in the 1960s, and incredibly popular for a couple of decades thereafter,
also established a link (for those who grow up at the time) between a particular shade of pinkish-
red and a synthetic strawberry flavour (one that was unusual in containing a hint of pineapple in
the flavour).

3.8 The decline of multisensory flavour perception


All of the senses decline with age. However, that said, we have prostheses (glasses and hearing
aids) to correct for any loss of visual or auditory acuity. The situation is not so fortunate when it
comes to the other senses. Indeed, there is currently nothing that can be done to recover taste,
smell, and touch function once they have been lost. What is more, more and more people are now
76 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

Fig. 3.3 The six coloured drinks used in Shankar et al.’s (2010a) study, in which the crossmodal
flavour expectations associated with particular beverage colours was assessed cross-culturally in
two groups of young adult participants, one from the UK and the other from Taiwan. Reprinted
from Consciousness and Cognition, 19 (1), Maya U. Shankar, Carmel A. Levitan, and Charles Spence,
Grape expectations: The role of cognitive influences in color–flavor interactions, pp. 380–90,
Copyright (2010), with permission from Elsevier (Reproduced in colour in the colour plate section).
THE DECLINE OF MULTISENSORY FLAVOUR PERCEPTION 77

living to a more advanced age, in particular, to an age where the loss of gustatory and olfactory
sensitivity is starting to have a markedly detrimental effect on their health and well-being (see
Doty et al. 1984; Schiffman 1997). This is especially problematic given that eating and drinking
constitute some of the most treasured pleasures for those in their later years. As Anton Brillat-
Savarin (1835) put it in his classic volume, The Philosopher in the Kitchen: ‘The pleasures of the
table, belong to all times and all ages, to every country and to every day; they go hand in hand with
all our other pleasures, outlast them, and remain to console us for their loss’. But what of the
empirical data?
A little over half a century ago, Cooper et al. (1959) reported there to be little change in people’s
taste sensitivity up to 50 years of age, but that after people reached their mid-50s, there was a
sharp decline in sensitivity for the four basic tastes (see Schiffman 1977). Schiffman and Pasternak
(1979) demonstrated that the elderly (72–78 years of age) found it harder to discriminate between
food odours than younger participants (19–25 years of age in this study). In a more recent review
of the literature on the decline of smell and taste in the elderly, Schiffman (1997) reported that the
evidence supported the claim that the decline of taste and smell really starts in earnest once people
reach around 60 years of age, and becomes more severe in those who have reached 70 years or
more (see also Cowart 1981; Doty et al. 1984; Mojet et al. 2003). To put this decline into some
kind of perspective, research shows that thresholds for many tastes and odours can be as much as
12 times higher in the elderly as compared to younger people, especially if the participant happens
to be on medication as the majority of elderly people apparently are (Schiffman 1997; Schiffman
and Warwick 1989). What is more, a number of studies have suggested that olfactory sensitivity
appears to decline more severely than gustatory sensitivity (Cowart 1989; see also Stevens et al.
1984). In addition to any sensory decline, there is also evidence that older participants may strug-
gle with the cognitive demands of chemosensory tasks (e.g. as when participants are required to
remember a taste or flavour and compare it to a subsequently presented stimulus; see Cowart
1981, 1989).
Given that (as pointed out earlier) as much as 80% of flavour perception may come from the
information provided by the nose (rather than from the tongue; see Martin 2004; Murphy et al.
1977), one would expect this decline in olfactory sensitivity to have a particularly severe effect
on multisensory flavour perception. Furthermore, given the much more severe decline of olfac-
tory as compared to gustatory abilities with increasing age, one might also predict that in old
age colour would increasingly come to influence flavour judgments as compared to (the relatively
less impaired) taste judgments. Several researchers have suggested that people’s perception of
food aroma and flavour intensity will increasingly be influenced by food colour as the chemical
senses start their inevitable decline (e.g. Christensen 1985; Clydesdale 1994). However, the evi-
dence that has been published on this topic to date is rather mixed, with several studies actually
failing to find any significant difference between adults and the elderly on the extent to which
colour influences flavour.
So, for example, Chan and Kane-Martinelli (1997) conducted one oft-cited study that exam-
ined the effect of the addition of colour on perceived flavour intensity or acceptability ratings of
chicken bouillon and chocolate pudding in both young and older adults (20–35 years old and
60–90 years old, respectively). Three levels of colour were added: no colour added, standard (the
commercially available colour), and high colour (twice the standard) for each foodstuff. Each
participant tasted and evaluated each of three samples of one food using a series of visual analog
scales. The results suggested that the young adults’ judgments were affected more by the actual
level of food colouring added than were the older adults, although, as one might expect, there was
far more variation in response in the older group (hence making it harder to pick up significant
effects in this group). The younger group’s judgment of the overall flavour intensity of the chicken
78 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

bouillon was affected by the amount of colouring added. The younger group also showed signifi-
cant effects of amount of colour added when rating the acceptability of appearance of both the
chicken bouillon and the chocolate pudding.
Similarly, Christensen (1985) failed to find any evidence that elderly participants were affected
more by visual cues than were younger participants. She compared a group of young adults
(21–40 years of age) with a group of elderly participants (65–85 years of age) on their perception
of processed cheese and grape-flavoured jelly. Each participant in the study evaluated one of the
two foods which was presented at one of three flavour intensities and one of three colour intensity
levels (low, medium, or high) giving rise to nine possible variations of each food (where the
medium flavouring and colour levels were considered as normal for these particular products).
The participants were then presented with 52 pairs of food samples and asked to discriminate
which sample had a more intense flavour or aroma (this involved the participants ‘tasting’
or sniffing the products, respectively). The participants also had to give a certainty judgment
concerning their response. The difficulty of the flavour discrimination task was manipulated by
pairing a high intensity flavour with either a low or medium intensity sample (making for easy
versus hard tasks, respectively). The colour of the two samples could either be the same, the col-
our difference could reinforce the flavour difference (i.e. the higher intensity colour matched with
the more intense flavour), or else it could be incongruent (i.e. where the less strongly flavoured
food had the more intense flavour).
Christensen (1985) observed no significant differences between performance of the older
and younger group when evaluating the grape jelly. However, it is worth noting that in the major-
ity of conditions, the elderly group performed less accurately than the young group (i.e. regardless
of the specific colour condition) when judging flavour. The lack of a significant difference between
the groups in this case then likely simply reflects a lack of statistical power given the small sample
sizes used (there were only 12 participants in each group). What is more, both groups of partici-
pants were at ceiling performance in most of the conditions in the aroma judgment task, hence
making it difficult to draw any firm conclusions from their data. When evaluating the cheese
sample, performance was at ceiling in the flavour-intensity judgment task, and the elderly par-
ticipants performed numerically somewhat better than the younger participants in the aroma-
intensity judgment task. For both foods, the participants performed less accurately in the
incongruently coloured condition than in the condition where the colours of the two samples
were matched (thus showing that the addition of colour did have some impact on flavour percep-
tion). As Christensen herself notes, however, participants may by and large have learned to ignore
the colour of the foods in the context of the experimental setting. Given these problems, it
is therefore difficult to draw any firm conclusions from Christensen’s results about whether or
not the relative importance of visual cues changes with age.
Elsewhere, Philipsen et al. (1995) reported a study in which they compared a young adult
population (18–22 years of age) to a group of older participants (60–75 years of age) when rating
various attributes (e.g. sweetness, flavour intensity, flavour quality, flavour identification etc.) of
15 samples of an artificially-flavoured cherry beverage varying in sucrose, flavour, and colour.
Interestingly, variations in colour intensity did not have a significant effect on sweetness ratings
in either group, but did impact on flavour intensity ratings in the older group, if not in the
younger group. Changes in the colour of the drinks also had a significant effect on flavour quality
and overall acceptability ratings in both age groups. Philipsen et al.’s results therefore support the
claim that older participants are more influenced by visual (colour) cues, likely because of their
reduced sensitivity to olfactory and gustatory flavor cues (see also Tepper 1993). Here, however,
one might also think that this increased reliance on vision could relate to older individuals
being, in some sense, more ‘expert’ than younger tasters (that is, they have certainly had far more
CONCLUSIONS 79

experience in terms of picking up the correlations that exist in nature between tastes/flavours
and colours; remember here also that wine experts tend to be swayed more by visual cues than less
experienced wine drinkers; Pangborn et al. 1963; Parr et al. 2003; Spence 2010a).
In summary, there is some evidence that older individuals may rely more heavily on colour
cues when perceiving and evaluating foods than do younger adults (e.g. Philipsen et al. 1995;
Tepper 1993). That said, it should be noted that the data are noisy and somewhat inconsistent,
with no difference between younger and older adults being reported by researchers using certain
tasks (see Chan and Kane-Martinelli 1997; Christensen 1985). It would therefore be ideal if
someone were to repeat Oram et al.’s (1995) flavour discrimination study (discussed earlier)
with older adults. I would predict that such a study, because it involves flavour identification (or
discrimination) rather than taste/flavour intensity ratings, would give rise to more profound
age-related changes in the modulatory role of visual cues (remember that, in adults, flavour-
discrimination responses have proven to be more affected by colour cues than have taste or
flavour-intensity responses; Spence et al. 2010 ). What is more, the relative simplicity of
the experimental design utilized by Oram et al. (involving the presentation of just four drinks to
each participant) is also less likely to give rise to a situation in which participants learn to ignore
the visual cues in the specific context of the experiment situation (cf. Christensen 1985; Spence
et al. 2010).

3.9 Conclusions
Despite the fact that flavour perception constitutes one of the most multisensory of our everyday
experiences (e.g. Auvray and Spence 2008; Spence 2010b; Stillman 2002), developmental research-
ers have seemingly not been overly interested in investigating any developmental changes affect-
ing the relative contribution of each of the senses to multisensory flavour perception. Indeed,
Piaget himself apparently never gave the development of flavour perception much thought (at
least not in print). William James (1890) only got as far as smell, but failed to mention taste in his
oft-cited quote: ‘The baby, assailed by eye, ear, nose, skin and entrails at once, feels it all as one
great blooming buzzing confusion’. Perhaps though this latter omission can be explained by
James’s further comment elsewhere in The Principles of Psychology that ‘Taste, smell, as well as
hunger, thirst, nausea and other so-called “common” sensations need not be touched on . . . as
almost nothing of psychological interest is known concerning them’. In fact, the situation had not
changed much by the time of Pick and Pick’s (1970) influential review of sensory and perceptual
development. More recently, Lewkowicz and Lickliter’s (1994) edited volume on infant develop-
ment also contained nothing on the development of taste, smell, and/or the flavour senses.
This lacuna in the developmental literature is all the more surprising given the fundamental
importance of food and food acquisition to brain development and survival. As Young (1968, p. 21)
puts it so eloquently: ‘No animal can live without food. Let us then pursue the corollary of this:
namely, food is about the most important influence in determining the organization of the brain
and the behavior that the brain organization dictates’. What is more, as Mennella and Beauchamp
(1994, p. 25) point out: ‘Anyone who has observed infants for any period of time can testify to the
intense activity occurring in and around their mouths – a primary site for learning in the first few
months of life. During feeding, or while mouthing objects such as their hands and toys, infants learn
to discriminate the varying features of their new world.’ In their latest review, Mennella and
Beauchamp (2010, p. 204) go on to highlight the fact that: ‘one of the most important decisions that
an animal makes’ is ‘whether to reject a foreign substance or take it into the body’.
Given the alarming rise in childhood obesity in recent years, much of the currently interest in
the developmental aspects of multisensory flavour perception relates to the establishment and
80 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

modification of infant preferences for (and acceptance of) particular classes of typically ‘healthier/
healthful’ foodstuff (see Harris 2008 and Mennella and Beauchamp 2010 for reviews). While the
situation is somewhat better at the other end of the age spectrum, research with elderly popula-
tions has nevertheless primarily been driven by concerns over the consequences of sensory decline
for healthy eating, rather than necessarily because of any particular curiosity about how multisen-
sory integration changes over the latter stages of the lifespan (see Schiffman 1997). Given the
paucity of empirical data concerning age-related changes in the multisensory integration of uni-
sensory flavour signals at either end of the age spectrum, this chapter has focused primarily on
those (slightly more common) studies that have investigated the relative contribution of visual
cues (specifically colour cues) to multisensory flavour perception.
In summary, the results, although somewhat messy, tend to support the claim that visual cues
have a greater influence on multisensory flavour perception in childhood and (to a lesser extent)
in old age than during adulthood. Just as for the adult literature, the clearest age-related changes
concern vision’s influence on flavour identification (Oram et al. 1995), whereas the data on fla-
vour or taste intensity are far more mixed (Alley and Alley 1998; Chan and Kane-Martinelli 1997;
Lavin and Lawless 1998; though see also Léon et al. 1999; Philipsen et al. 1995; Tepper 1993). That
said, the reasons behind this change in visual dominance may be somewhat different in the two
cases. It seems plausible that in children, the increased role of vision in flavour identification may
result from a strategy of relying (or tendency to rely) on a single source of sensory information
rather than integrating all of the available sensory cues. Indeed, as Ganchrow and Mennella (2003,
p. 839) note, it is also unclear when exactly during the course of infancy taste, retronasal olfaction,
and oral-irritation fuse into a single sensory gestalt, that of flavour (cf. Spence 2012; Verhagen
and Engelen 2006).
At the opposite end of the age spectrum, taste and smell (the two most important senses for
flavour perception) start their inevitable decline, with the more severe drops in sensitivity occur-
ring during the sixth/seventh decades of life. As a result this loss of gustatory and olfactory acuity,
one might have expected that visual cues would come to play a much more dominant role in
influencing multisensory flavour perception in old age (cf. Christensen 1985). However, that said,
the extent of the increase in visual dominance over flavour perception in old age is perhaps not as
great as one might have predicted given the profound drops in olfactory and gustatory sensitivity
that have now been observed in numerous studies (and by comparison with the developmental
changes in visual dominance one sees at the other end of the age spectrum; though see Mojet et al.
2003). How, then, should the relatively modest changes in vision’s influence over flavour percep-
tion in old age be explained? Well, one possibility is that there may perhaps be some form of
compensatory behaviour in terms of enhanced multisensory integration in older participants (cf.
Chapter 11 by Laurienti and Hugenschmidt; Laurienti et al. 2006). Although there is little evi-
dence directly supporting it yet, the alternative possibility that there might be a more general
breakdown in multisensory integration in old age should also be borne in mind (see Nusbaum
1999, on this possibility). One might also consider the possibility that older individuals can be
thought of as being in some sense more ‘expert’ than younger people. Perhaps any expertise
attributable to age may then increase an individual’s analytic tasting abilities (i.e. their ability to
focus on the inputs from just a single sense; cf. Prescott et al. 2004). Such cognitive factors may
help older people to focus their attention on gustatory and/or olfactory inputs, hence potentially
reducing vision’s influence over flavour perception in this age group. On the other hand, exper-
tise (at least in the food and drink sector) is also associated with people having stronger expecta-
tions about the consequences of changes in the hue and/or intensity (or saturation) of a colour for
the perception of flavour and taste.
REFERENCES 81

Given the recent success of Bayesian decision theory in accounting for and predicting the
patterns of sensory dominance in the conflict situation seen in a range of different situations
(Ernst and Bülthoff 2004), it will be interesting in the coming years to see whether it can also be
used to account for visual influences on multisensory flavour and taste perception (see Shankar
et al. 2010c), and its change over the lifespan (cf. Gori et al. 2008). Of course, if one adopts the
view that vision’s role in modulating flavour perception occurs primarily through the setting-up
of expectations then it might turn out to be more appropriate to model vision’s influence on fla-
vour in terms of a coupling prior in Bayesian decision theory (cf. Shams and Beierholm 2010;
Spence 2011).
As noted earlier, the majority of age-related studies of multisensory flavour perception have
tended to focus on the (changing) influence of visual cues on taste and flavour perception, and
hence it is on these studies that I have focused in this review. Nevertheless, it seems at least plau-
sible that what is now known about the changing contribution of visual cues to multisensory
flavour perception over the lifespan may provide a model for thinking about how other multisen-
sory interactions, such as the influence of auditory or oral-somatosensory cues to flavour percep-
tion (Blossfeld et al. 2007; Bult et al. 2007; Zampini and Spence 2004; see Spence and Shankar
2010; Zampini and Spence 2010, for reviews) may also develop. However, in the absence of any
empirical evidence on this question thus far, confirmation of such claims will clearly need to
await future research. Furthermore, it must also be born in mind that unlike audition and oral-
somatosensation, vision is not integral to many researchers’ definition of flavour (see Spence et al.
2010). Hence, it could also be argued that vision’s influence on flavour perception, in terms of
setting-up expectations, is different in kind from the multisensory integration that results in the
binding of olfactory, gustatory, oral-somatosensory, and auditory cues into unified flavour
gestalts. (Here, the prediction is that a verbal description, such as hearing the phrase ‘this is a red
drink’, might have just as much of an influence on a blindfolded individual’s flavour responses,
as actually colouring a drink red has on the perception of a person who can see the drink.) If the
latter view turns out to be correct, then vision’s influence of flavour perception may end up
having a rather different developmental trajectory than that of audition or oral somatosensation.
Given such uncertainty, more research, both theoretical and empirical, is clearly needed in order
for scientists to make progress in understanding the development and decline of multisensory
flavour perception.

References
Abate, P. Pueta, M., Spear, N.E., and Molina, J.C. (2008). Fetal learning about ethanol and later ethanol
responsiveness: evidence against ‘safe’ amounts of prenatal exposure. Experimental Biology and
Medicine, 233, 139–54.
Alley, R.L., and Alley, T.R. (1998). The influence of physical state and color on perceived sweetness. Journal
of Psychology: Interdisciplinary and Applied, 132, 561–68.
Auvray, M., and Spence, C. (2008). The multisensory perception of flavor. Consciousness and Cognition, 17,
1016–31.
Bartoshuk, L.M., and Duffy, V.B. (2005). Chemical senses: taste and smell. In The taste culture reader:
experiencing food and drink (ed. C. Korsmeyer), pp. 25–33. Berg, Oxford.
Birch, L.L. (1999). Development of food preferences. Annual Review of Nutrition, 19, 41–62.
Birch, L.L., McPhee, L., Shoba, B.C., Pirok, E., and Steinberg, L. (1987). What kind of exposure reduces
children’s food neophobia? Looking vs. tasting. Appetite, 9, 171–78.
Blake, A.A. (2004). Flavour perception and the learning of food preferences. In Flavor perception (ed. A.J.
Taylor and D.D. Roberts), pp. 172–202. Blackwell, Oxford.
82 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

Blossfeld, I., Collins, A., Kiely, M., and Delahunty, C. (2007). Texture preferences of 12-month-old infants
and the role of early experience. Food Quality and Preference, 18, 396–404.
Bradley, R.M., and Mistretta, C.M. (1975). Fetal sensory receptors. Physiology Reviews, 55, 352–82.
Bradley, R.M., and Stern, I.B. (1967). The development of the human taste bud during the foetal period.
Journal of Anatomy, 101, 743–52.
Brillat-Savarin, J.A. (1835). Physiologie du goût [The philosopher in the kitchen / The physiology of taste].
J.P. Meline: Bruxelles. (translated by A. Lalauze (1884), A handbook of gastronomy. Nimmo and Bain,
London.
Bryant, P. (1974). Perception and understanding in young children: an experimental approach. Methuen, London.
Bult, J.H.F., de Wijk, R.A., and Hummel, T. (2007). Investigations on multimodal sensory integration:
texture, taste, and ortho- and retronasal olfactory stimuli in concert. Neuroscience Letters, 411, 6–10.
Cain, W.S. (1979). To know with the nose: keys to odor identification. Science, 203, 467–70.
Capretta, P.J., Petersik, J.T., and Stewart, D.J. (1975). Acceptance of novel flavors is increased after early
experience of diverse tastes. Nature, 254, 689–91.
Cardello, A.V. (1994). Consumer expectations and their role in food acceptance. In Measurement of food
preferences (ed. H.J.H. MacFie and D.M.H. Thomson), pp. 253–97. Blackie Academic and Professional,
London.
Chan, M.M., and Kane-Martinelli, C. (1997). The effect of color on perceived flavor intensity and acceptance
of foods by young adults and elderly adults. Journal of the American Dietetic Association, 97, 657–959.
Christensen, C. (1985). Effect of color on judgments of food aroma and food intensity in young and elderly
adults. Perception, 14, 755–62.
Clydesdale, F.M. (1994). Changes in color and flavor and their effect on sensory perception in the elderly.
Nutrition Reviews, 52 (8 Pt 2), S19–20.
Cooper, R.M., Bilash, I., and Zubeck, J.P. (1959). The effect of age on taste sensitivity. Journal of
Gerontology, 14, 56–58.
Cowart, B.J. (1981). Development of taste perception in humans: sensitivity and preference throughout the
life span. Psychological Bulletin, 90, 43–73.
Cowart, B.J. (1989). Relationships between taste and smell across the adult life span. Annals of the New York
Academy of Sciences, 561, 39–55.
Davis, R.G. (1981). The role of nonolfactory context cues in odor identification. Perception and
Psychophysics, 30, 83–89.
De Araujo, I.E.T., Rolls, E.T., Kringelbach, M.L., McGlone, F., and Phillips, N. (2003). Taste-olfactory
convergence, and the representation of the pleasantness of flavour, in the human brain. European
Journal of Neuroscience, 18, 2059–68.
Delwiche, J. (2004). The impact of perceptual interactions on perceived flavor. Food Quality and Preference,
15, 137–46.
DeSnoo, K. (1937). Das trinkende kind im uterus [The drinking baby in the uterus]. Monatschrift für
Geburtshilfe und Gynäkologie, 105, 88–97.
Desor, J.A., Maller, O., and Turner, R.E. (1973). Taste in acceptance of sugars by human infants. Journal of
Comparative and Physiological Psychology, 84, 496–501.
Desor, J.A., Greene, L.S., and Maller, O. (1975). Preferences for sweet and salty in 9- to 15-year-old and
adult humans. Science, 190, 686.
Desor, J.A., Maller, O., and Andrews, K. (1975). Ingestive responses of human newborns to salty, sour, and
bitter stimuli. Journal of Comparative and Physiological Psychology, 89, 966–70.
Doty, R.L. (1992). Olfactory function in neonates. In The human sense of smell (ed. D.G. Laing, R.L. Doty,
and W. Breipohl), pp. 155–65. Springer-Verlag, Heidelberg.
Doty, R.L., Shaman, P., Applebaum, S.L., Giberson, R., Siksorski, L., and Rosenberg, L. (1984). Smell
identification ability: changes with age. Science, 226, 1441–43.
Drewnowski, A. (2000). Sensory control of energy density at different life stages. Proceedings of the Nutrition
Society, 59, 239–44.
REFERENCES 83

DuBose, C.N., Cardello, A.V., and Maller O. (1980). Effects of colorants and flavorants on identification,
perceived flavor intensity, and hedonic quality of fruit-flavored beverages and cake. Journal of Food
Science, 45, 1393–99, 1415.
Egan, J.M., and Margolskee, R.F. (2008). Taste cells of the gut and gastrointestinal chemosensation.
Molecular Intervention, 8, 78–81.
Erikson, R.P. (2008). A study of the science of taste: on the origins and influence of the core ideas.
Behavioral and Brain Sciences, 31, 59–105.
Ernst, M.O., and Bülthoff, H.H. (2004). Merging the senses into a robust percept. Trends in Cognitive
Sciences, 8, 162–69.
Fernandez, M., and Bahrick, L.E. (1994). Infants’ sensitivity to arbitrary object-odor pairings. Infant
Behavior and Development, 17, 471–74.
Frank, R.A., and Byram, J. (1988). Taste-smell interactions are tastant and odorant dependent. Chemical
Senses, 13, 445–55.
Galef, B.G., and Sherry, D.F. (1973). Mother’s milk: a medium for transmission of cues reflecting the flavor
of mother’s diet. Journal of Comparative and Physiological Psychology, 83, 374–78.
Ganchrow, J.R., and Mennella, J.A. (2003). The ontogeny of human flavour perception. In Handbook of
olfaction and gustation (ed. R.L. Doty), pp. 823–46. Marcel Dekker, New York.
Gilbert, A.N., Martin, R., and Kemp, S.E. (1996). Cross-modal correspondence between vision and
olfaction: the color of smells. American Journal of Psychology, 109, 335–51.
Glendinning, J.I. (1994). Is the bitter rejection response always adaptive? Physiology and Behavior, 56,
1217–27.
Gori, M., Del Viva, M., Sandini, G., and Burr, D.C. (2008). Young children do not integrate visual and
haptic information. Current Biology, 18, 694–98.
Gottlieb, G. (1971). Ontogenesis of sensory function in birds and mammals. In The biopsychology of
development (ed. E. Tobach, L.R., Aronson, and E.F. Shaw), pp. 67–128. Academic Press, New York.
Green, B.G. (2002). Studying taste as a cutaneous sense. Food Quality and Preference, 14, 99–109.
Harris, G. (2008). Development of taste and food preferences in children. Current Opinion in Clinical
Nutrition and Metabolic Care, 11, 315–319.
Hausner, H., Bredie, W.L.P., Mølgaard, C., Petersen, M.A., and Møller, P. (2008). Differential transfer of
dietary flavour compounds into human breast milk. Physiology and Behavior, 95, 118–24.
Helbig, H.B., and Ernst, M.O. (2008). Visual-haptic cue weighting is independent of modality-specific
attention. Journal of Vision, 8, 1–16.
Hepper, P. (1995). Human fetal ‘olfactory’ learning. International Journal of Prenatal and Perinatal
Psychology, 2, 147–51.
Horne, P.J., Hardman, C.A., Lowe, C.F., Tapper, K., Le Noury, J., Madden, P., Patel, P., and Doody, M.
(2009). Increasing parental provision and children’s consumption of lunchbox fruit and vegetables in
Ireland: the Food Dudes intervention. European Journal of Clinical Nutrition, 63, 613–618.
Horne, P.J., Tapper, K., Lowe, C.F., Hardman, C. A., Jackson, M.C., and Woolner, J. (2004). Increasing
children’s fruit and vegetable consumption: a peer-modelling and rewards-based intervention.
European Journal of Clinical Nutrition, 58, 1–12.
Houston-Price, C., Butler, L., and Shiba, P. (2009). Visual exposure impacts on toddlers’ willingness to
taste fruit and vegetables. Appetite, 53, 450–53.
Hurlbert, A.C., and Ling, Y. (2007). Biological components of sex differences in color preference. Current
Biology, 17, R623–25.
Hutchings, J.B. (2003). Expectations and the food industry: the impact of color and appearance. Plenum
Publishers, New York.
ISO (1992). Standard 5492: Terms relating to sensory analysis. International Organization for Standardization
James, W. (1890). Principles of psychology, Vol. 1. Henry Holt, New York.
Khan, R.M., Luk, C.H., Flinker, A., et al. (2007). Predicting odor pleasantness from odorant structure:
pleasantness as a reflection of the physical world. Journal of Neuroscience, 27, 10015–23.
84 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

Kostyla, A.S. (1978). The psychophysical relationships between color and flavor of some fruit flavored beverages.
Ph.D. thesis. University of Massachusetts, Amherst, Massachusetts.
Koza, B.J., Cilmi, A., Dolese, M., and Zellner, D.A. (2005). Color enhances orthonasal olfactory intensity
and reduces retronasal olfactory intensity. Chemical Senses, 30, 643–49.
Laurienti, P.J., Burdette, J.H., Maldjian, J.A., and Wallace, M.T. (2006). Enhanced multisensory integration
in older adults. Neurobiology of Aging, 27, 1155–63.
Lavin, J., and Lawless, H. (1998). Effects of color and odor on judgments of sweetness among children and
adults. Food Quality and Preference, 9, 283–89.
Lehrer, A. (2009). Wine and conversation (2nd Ed.). Oxford University Press, Oxford.
Lelièvre, M., Chollet, S., Abdi, H., and Valentin, D. (2009). Beer-trained and untrained assessors rely more
on vision than on taste when they categorize beers. Chemosensory Perception, 2, 143–53.
Léon, F., Couronne, T., Marcuz, M., and Köster, E. (1999). Measuring food liking in children: a
comparison of non-verbal methods. Food Quality and Preference, 10, 93–100.
Lewkowicz, D.J., and Lickliter, R. (Eds.). (1994). The development of intersensory perception: comparative
perspectives. Lawrence Erlbaum Associates, Hillsdale, NJ.
Liem, D.G., and Mennella, J.A. (2003). Heightened sour preferences during childhood. Chemical Senses, 28,
173–80.
Lowenberg, M.E. (1934). Food for the young child. Collegiate Press, Ames, IO.
Maga, J.A. (1974). Influence of color on taste thresholds. Chemical Senses and Flavor, 1, 115–119.
Marshall, D., Stuart, M., and Bell, R. (2006). Examining the relationship between product package colour
and product selection in preschoolers. Food Quality and Preference , 17, 615–21.
Martin, G.N. (2004). A neuroanatomy of flavour. Petits Propos Culinaires, 76, 58–82.
Maurer, D. (1997). Neonatal synaesthesia: implications for the processing of speech and faces.
In Synaesthesia: classic and contemporary readings (ed. S. Baron-Cohen, and J.E. Harrison), pp. 224–42.
Blackwell, Oxford.
Maurer, D., and Mondloch, C.J. (2005). Neonatal synaesthesia: a reevaluation. In Synaesthesia: perspectives
from cognitive neuroscience (ed. L.C. Robertson and N. Sagiv), pp. 193–213. Oxford University Press,
Oxford.
McBurney, D.H. (1986). Taste, smell, and flavor terminology: taking the confusion out of fusion.
In Clinical measurement of taste and smell (ed. H.L. Meiselman and R.S. Rivkin), pp. 117–25.
Macmillan, New York.
Mennella, J.A. (1995). Mother’s milk: a medium for early flavor experiences. Journal of Human Lactation,
11, 39–45.
Mennella, J.A., and Beauchamp, G.K. (1994). Early flavor experiences: when do they start? Nutrition Today,
29, 25–31.
Mennella, J.A., and Beauchamp, G.K. (2010). The role of early life experiences in flavor perception and
delight. In Obesity prevention: the role of brain and society on individual behavior (ed. L. Dubé,
A. Bechara, A. Dagher, A. Drewnowski, J. Lebel, P. James, R.Y. Yada, and M.-C. LaFlamme-Sanders),
pp. 203–217. Elsevier, Amsterdam.
Mennella, J.A., Jagnow, C.P., and Beauchamp, G.K. (2001). Prenatal and postnatal flavor learning by
human infants. Pediatrics, 107, e88, 1–6.
Mojet, J., Heidema, J., and Christ-Hazelhof, E. (2003). Taste perception with age: Generic or specific losses
in supra-threshold intensities of five taste qualities? Chemical Senses, 28, 397–413.
Morrot, G., Brochet, F., and Dubourdieu, D. (2001). The color of odors. Brain and Language, 79, 309–20.
Moskowitz, H.R. (1985). Product testing with children. In New directions for product testing and sensory
analysis of foods (ed. H.R. Moskowitz), pp. 147–64. Food and Nutrition Press, Westport, CT.
Murphy, C., Cain, W.S., and Bartoshuk, L.M. (1977). Mutual action of taste and olfaction. Sensory
Processes, 1, 204–211.
Nusbaum, N.J. (1999). Aging and sensory senescence. Southern Medical Journal, 92, 267–75.
REFERENCES 85

Oram, N., Laing, D.G., Hutchinson, I., et al. (1995). The influence of flavor and color on drink
identification by children and adults. Developmental Psychobiology, 28, 239–46.
Österbauer, R.A., Matthews, P.M., Jenkinson, M., Beckmann, C.F., Hansen, P.C., and Calvert, G.A. (2005).
Color of scents: chromatic stimuli modulate odor responses in the human brain. Journal of
Neurophysiology, 93, 3434–41.
Pangborn, R.M., Berg, H.W. and Hansen, B. (1963). The influence of color on discrimination of sweetness
in dry table-wine. American Journal of Psychology, 76, 492–95.
Parr, W.V., Heatherbell, D., and White, K.G. (2002). Demystifying wine expertise: Olfactory threshold,
perceptual skill and semantic memory in expert and novice wine judges. Chemical Senses, 27, 747–55.
Parr, W.V., White, K.G., and Heatherbell, D. (2003). The nose knows: influence of colour on perception of
wine aroma. Journal of Wine Research, 14, 79–101.
Philipsen, D.H., Clydesdale, F.M., Griffin, R.W., and Stern, P. (1995). Consumer age affects response to
sensory characteristics of a cherry flavored beverage. Journal of Food Science, 60, 364–68.
Pick, H.L., and Pick, A.D. (1970). Sensory and perceptual development. In Carmichael’s manual of child
psychology (Vol. 1) (ed. P. H. Mussen). Wiley, New York.
Pliner, P. (1982). The effects of mere exposure on liking for edible substances. Appetite, 3, 283–90.
Prescott, J., Johnstone, V., and Francis, J. (2004). Odor-taste interactions: effects of attentional strategies
during exposure. Chemical Senses, 29, 331–40.
Reardon, P., and Bushnell, E.W. (1988). Infants’ sensitivity to arbitrary pairings of color and taste. Infant
Behavior and Development, 11, 245–50.
Rozin, P., and Fallon, A.E. (1987). A perspective on disgust. Psychological Review, 94, 23–41.
Schaal, B., Marlier, L., and Soussignan, R. (2000). Human foetuses learn odours from their pregnant
mother’s diet. Chemical Senses, 25, 729–37.
Schifferstein, H.N.J., and Tanudjaja, I. (2004). Visualizing fragrances through colors: the mediating role of
emotions. Perception, 33, 1249–66.
Schiffman, S.S. (1977). Food recognition in the elderly. Journal of Gerontology, 32, 586–92.
Schiffman, S.S. (1997). Taste and smell losses in normal aging and disease. Journal of the American Medical
Association, 278, 1357–62.
Schiffman, S., and Pasternak, M. (1979). Decreased discrimination of food odors in the elderly. Journal of
Gerontology, 34, 73–79.
Schiffman, S.S., and Warwick, Z.S. (1989). Use of flavor-amplified foods to improve nutritional status in
elderly persons. Annals of the New York Academy of Sciences, 561, 267–76.
Scholl, B.J. (2005). Innateness and (Bayesian) visual perception: reconciling nativism and development. In
The innate mind: structure and contents (ed. P. Carruthers, S. Laurence, and S. Stich), pp. 34–52.
Oxford University Press, Oxford.
Shams, L., and Beierholm, U.R. (2010). Causal inference in perception. Trends in Cognitive Sciences, 14, 425–32.
Shankar, M.U., Levitan, C.A., Prescott, J., and Spence, C. (2009). The influence of color and label
information on flavor perception. Chemosensory Perception, 2, 53–58.
Shankar, M.U., Levitan, C., and Spence, C. (2010a). Grape expectations: the role of cognitive influences in
color-flavor interactions. Consciousness and Cognition, 19, 380–90.
Shankar, M., Simons, C., Levitan, C., Shiv, B., McClure, S., and Spence, C. (2010b). An expectations-based
approach to explaining the crossmodal influence of color on odor identification: the influence of
temporal and spatial factors. Journal of Sensory Studies, 25, 791–803.
Shankar, M., Simons, C., Shiv, B., Levitan, C., McClure, S., and Spence, C. (2010c). An expectations-based
approach to explaining the influence of color on odor identification: the influence of degree of
discrepancy. Attention, Perception, and Psychophysics, 72, 1981–93.
Shankar, M., Simons, C., Shiv, B., McClure, S., and Spence, C. (2010d). An expectation-based approach to
explaining the crossmodal influence of color on odor identification: the influence of expertise.
Chemosensory Perception, 3, 167–73.
86 THE DEVELOPMENT AND DECLINE OF MULTISENSORY FLAVOUR PERCEPTION

Skrandies, W., and Reuther, N. (2008). Match and mismatch of taste, odor, and color is reflected by
electrical activity in the human brain. Journal of Psychophysiology, 22, 175–84.
Spector, F. (2009). The colour and shapes of the world: testing predictions from synesthesia about the development
of sensory associations. Unpublished PhD thesis. McMaster University, Hamilton, Ontario, Canada.
Spence, C. (2010a). The color of wine—Part 1. The World of Fine Wine, 28, 122–29.
Spence, C. (2010b). The multisensory perception of flavour. The Psychologist, 23, 720–23.
Spence, C. (2011). Crossmodal correspondences: a tutorial review. Attention, Perception, and Psychophysics,
73, 971–95.
Spence, C. (2012). Multisensory integration and the psychophysics of flavour perception. In Food oral
processing—fundamentals of eating and sensory perception (ed. J. Chen and L. Engelen) pp. 203–219.
Blackwell Publishing, Oxford.
Spence, C., and Shankar, M.U. (2010). The influence of auditory cues on the perception of, and responses
to, food and drink. Journal of Sensory Studies, 25, 406–30.
Spence, C., Levitan, C., Shankar, M.U., and Zampini, M. (2010). Does food color influence taste and flavor
perception in humans? Chemosensory Perception, 3, 68–84.
Stein, B.E., and Meredith, M.A. (1993). The merging of the senses. MIT Press, Cambridge, MA.
Stevens, J.C., Bartoshuk, L.M., and Cain, W.S. (1984). Chemical senses and aging: taste versus smell.
Chemical Senses, 9, 167–79
Stevenson, R.J. (2009). The psychology of flavour. Oxford University Press, Oxford.
Stevenson, R.J. (2012). Multisensory interactions in flavor perception. In The new handbook of multisensory
processes (ed. B.E. Stein) pp. 283–99. MIT Press, Cambridge, MA.
Stevenson, R.J., and Boakes, R.A. (2004). Sweet and sour smells: learned synaesthesia between the senses of
taste and smell. In The handbook of multisensory processing (ed. G.A. Calvert, C. Spence, and B.E.
Stein), pp. 69–83. MIT Press, Cambridge, MA.
Stevenson, R.J., and Tomiczek, C. (2007). Olfactory-induced synesthesias: a review and model. Psychological
Bulletin, 133, 294–309.
Stevenson, R.J., Prescott, J., and Boakes, R.A. (1995). The acquisition of taste properties by odors. Learning
and Motivation, 26, 433–55.
Stevenson, R.J., Boakes, R.A., and Prescott, J. (1998). Changes in odor sweetness resulting from implicit
learning of a simultaneous odor-sweetness association: an example of learned synaesthesia. Learning
and Motivation, 29, 113–32.
Stevenson, R.J., Boakes, R.A., and Wilson, J.P. (2000). Counter-conditioning following human odor-taste
and color-taste learning. Learning and Motivation, 31, 114–27.
Stillman, J. (1993). Color influences flavor identification in fruit-flavored beverages. Journal of Food Science,
58, 810–812.
Stillman, J.A. (2002). Gustation: intersensory experience par excellence. Perception, 31, 1491–1500.
Teerling, A. (1992). The colour of taste. Chemical Senses, 17, 886.
Tepper, B.J. (1993). Effects of a slight color variation on consumer acceptance of orange juice. Journal of
Sensory Studies, 8, 145–54.
Tuorila-Ollikainen, H., Mahlamäki-Kultanen, S., and Kurkela, R. (1984). Relative importance of color, fruity
flavor and sweetness in the overall liking of soft drinks. Journal of Food Science, 49, 1598–1600, 1603.
Varendi, H., Porter, R.H., and Winberg, J. (1996). Attractiveness for amniotic fluid odor: evidence for
prenatal learning? Acta Paediatria, 85, 1223–27.
Verhagen, J.V., and Engelen, L. (2006). The neurocognitive bases of human multimodal food perception:
sensory integration. Neuroscience and Biobehavioral Reviews, 30, 613–50.
Walsh, L.M., Toma, R.B., Tuveson, R.V., and Sondhi, L. (1990). Color preference and food choice among
children. Journal of Psychology, 124, 645–53.
Wheatley, J. (1973). Putting colour into marketing. Marketing, 67, 24–29.
REFERENCES 87

Yeomans, M., Chambers, L., Blumenthal, H., and Blake, A. (2008). The role of expectancy in sensory and
hedonic evaluation: the case of smoked salmon ice-cream. Food Quality and Preference, 19, 565–73.
Young, J.Z. (1968). Influence of the mouth on the evolution of the brain. In Biology of the mouth: a
symposium presented at the Washington meeting of the American Association for the Advancement of
Science, 29–30 December 1966 (ed. P. Person), pp. 21–35. American Association for the Advancement
of Science, Washington, D.C.
Zampini, M., and Spence, C. (2004). The role of auditory cues in modulating the perceived crispness and
staleness of potato chips. Journal of Sensory Science, 19, 347–63.
Zampini, M., and Spence, C. (2010). Assessing the role of sound in the perception of food and drink.
Chemosensory Perception, 3, 57–67.
Zampini, M., Sanabria, D., Phillips, N., and Spence, C. (2007). The multisensory perception of flavor:
assessing the influence of color cues on flavor discrimination responses. Food Quality and Preference,
18, 975–84.
Zampini, M., Wantling, E., Phillips, N., and Spence, C. (2008). Multisensory flavor perception: assessing
the influence of fruit acids and color cues on the perception of fruit-flavored beverages. Food Quality
and Preference, 19, 335–43.
Zellner, D.A., Rozin, P., Aron, M., and Kulish, C. (1983). Conditioned enhancement of humans’ liking for
flavour by pairing with sweetness. Learning and Motivation, 14, 338–50.
Zellner, D.A., Bartoli, A.M., and Eckard, R. (1991). Influence of color on odor identification and liking
ratings. American Journal of Psychology, 104, 547–61.
Chapter 4

Crossmodal interactions in the


human newborn
New answers to Molyneux’s question
Arlette Streri

4.1 Introduction
The environment is characterized by inputs to more than one sensory system: each modality
provides original and unique information about an event. For example, only the visual system can
encode the colour of an event, whereas the haptic sense is needed to perceive the hardness, weight,
and temperature of an object. Only the ears (and to some extent the somatosensory system) are
sensitive to sound, whereas the taste and smell of something can only be coded by the chemical
senses, taste and olfaction. As adults, we integrate the multiple inputs arriving through these
sense organs into unified functional representations because the effective control of actions in the
environment depends on the inputs from a single event being linked and possibly integrated.
Multisensory perception has often been shown to be more precise than unimodal perception (e.g.
Ernst and Banks 2002), bestowing functional advantages such as economy of learning (Hatwell
2003) or intersensory substitution and thus crossmodal plasticity in people born blind or deaf
(Cohen et al. 1997; Röder and Rösler 2004; see also Chapter 13 by Röder). This spontaneous,
effortless integration is especially striking, given the wealth of research in this area that has pro-
vided evidence that integration depends on a number of different complex processes (see Calvert
et al. 2004; Spence and Driver 2004). The organism has to combine information from different
senses to enhance or complete perception, but this combination also poses many challenges for
the nervous system, due to the substantial differences between each sensory system. In the same
way, several decades of study have revealed various functional complexities and developmental
changes in intersensory functioning during the first year after birth (see Lewkowicz and Lickliter
1994) and in adult learning tasks (Shams and Seitz 2008).
The varieties of stimulation (distal versus proximal) and, still more, the varieties of receptors, make
the problem of integrating information across the senses very complex and challenging. This chapter
focuses first on the intersensory interactions observed in human newborns involving the chemical
senses, audition, and vision. Then the relations between the visual and the tactile modalities in new-
borns are examined in detail. These specific visual–tactile relations shed light on an old philosophical
question: Molyneux’s famous question of 7 July 1688 concerning the ‘visual-haptic mapping and the
origin of crossmodal identity’ (Held 2009). They therefore constitute the focus of this chapter.

4.2 The diversity of interactions between the senses


in human newborns
Adults possess a variety of perceptual mechanisms that enable them to deal with multisensory
inputs. Such performance may be due to a long period of learning to combine various inputs (this
THE DIVERSITY OF INTERACTIONS BETWEEN THE SENSES IN HUMAN NEWBORNS 89

is known as the empiricist hypothesis). Newborns enter the world largely naïve, so how can they
make sense of the wealth of stable or moving things, events, and peoples, that they encounter
through audition, vision, touch, and olfaction? Plausibly, they should possess some means that
allow them to have a coherent, if incomplete, representation of these objects, events, and people
(Slater and Kirby 1998). E.J. Gibson (1969) proposed that spatial dimensions as well as temporal
dimensions are amodal in nature, i.e. they are available to all sensory modalities right from birth.
Amodal perception can occur whenever two or more senses provide equivalent information. It is
quite likely that the ability to detect amodal relations is innately given to the infant. For instance,
Wertheimer (1961) reported consistent eye movements in the direction of an auditory stimulus
positioned close to either the left or right ear of a 10-minute-old infant. He interpreted these
ipsilateral eye movements as providing evidence for an innate mechanism subserving the integra-
tion of visual and auditory information. After several failures to replicate this first observation
with brief sounds (Butterworth and Castillo 1976; McGurk et al. 1977), Crassini and Broerse
(1980) finally determined the adequate parameters for the auditory stimulus in order to trigger a
newborn’s eye movements towards the sound source. They observed eye orientations toward a
sound when its duration was sufficiently long, and when the auditory information had a complex
spectral composition (such as speech). In the same way, von Hofsten (1982) provided evidence
that newborns orient their arm and hand to reach for a gently moving object while they look at it.
Thus, newborns’ behaviour is directed to a common auditory, tactile, and visual space (see
Chapter 5 by Bremner et al. for discussion of developments in multisensory representations of
space beyond the first months).
Some preferential orientation towards a stimulus can stem from the prenatal life of the new-
born. Chemosensory interactions provide a good illustration of multisensory interactions because
several senses are involved during the foetal period. Flavour, the most salient feature of foods and
beverages, is defined as the combination of at least three anatomically distinct chemical senses:
taste, smell, and chemosensory irritation (Beauchamp and Mennella 2009). Gustatory stimuli are
detected by receptor cells located in the tongue and palate. The smell component is composed of
volatile compounds detected by receptors in the upper regions of the nose. Chemosensory irrita-
tion is detected by receptors in the skin all over the head, but food is detected by receptors that are
particularly located in the mouth and nose. In adults, these chemical senses (taste, smell and
chemical irritation) work contingently to determine food choices (see Beauchamp and Mennella
2009). In newborns, volatile components of flavour, detected by the olfactory system, are strongly
influenced by early exposure in utero. Chemical molecules soluble in the amniotic fluid soak in
continuously through the nose, lips, and tongue of the foetus. The foetus can detect and store the
unique chemosensory information available in the prenatal environment. At birth, when exposed
to paired-choice tests contrasting the odours of familiar or non-familiar amniotic fluids, infants
orient preferentially and selectively to the odour of familiar amniotic fluid (Schaal et al. 1998; see
also Chapter 2 by Schaal and Durand). Thus from the volatile and non-soluble fragrant informa-
tion alone, the newborn is able to recognize the composite chemical fluid information learned
in utero. Odour cues from lactating women affect the newborn’s behaviours in multiple ways. The
odour of the lactating breast reduces arousal in active newborns and increases it in somnolent
newborns. Such cues elicit positive head- (and nose-) turning and increase oral activity, etc. (cf.
Chapter 2 by Schaal and Durand; Doucet et al. 2007). Auditory stimulation can also modify and
influence the infant’s state of arousal and preference for visual stimuli. Lewkowicz and Turkewitz
(1981) have demonstrated that the newborns exposed first to light spots of different intensities
preferred looking at the light of intermediate intensity. In contrast, the newborns who were first
exposed to a sound (white noise) and then to various light spots preferred the light of lowest
intensity (see also Gardner et al. 1986; Garner and Karmel 1983; Turkewitz et al. 1984 for similar
results). Lewkowicz and Turkewitz (1981) concluded from these results that newborns attend to
90 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

quantitative variations in stimulation. They also concluded that newborns ignore qualitative
attributes of stimulation in favour of quantitative ones.
However, the interactions between auditory and visual information can occur and are
perceived at different levels. Perceiving the equivalent nature of the visible and audible aspects
of an event testifies to an ability to integrate the multimodal properties of temporal events into
unified experiences. Lewkowicz (2000; see also Chapter 7 by Lewkowicz) has proposed that
the four basic features of multisensory temporal experience—temporal synchrony, duration,
rate, and rhythm—emerge in a sequential and hierarchical fashion during the first year after
birth. From birth, synchrony appears to be the fundamental dimension for the perception
of intersensory unity. Many multisensory events give both amodal and arbitrary auditory–visual
information. Infants’ learning about auditory–visual intersensory relations, both amodal
and arbitrary, has been investigated in detail in studies by Bahrick ( 1987 , 1988 ; Bahrick
and Pickens 1994; see also Chapter 8 by Bahrick and Lickliter). Morrongiello et al. (1998) have
demonstrated that newborns can associate objects and sounds on the basis of the combined
cues of collocation and synchrony. They are also capable of learning arbitrary auditory–visual
associations (e.g. between an oriented coloured line and a syllable), but only in the condition
where the visual and auditory information were presented synchronously (Slater et al. 1997,
1999). All these results suggest that, thanks to the temporal synchrony available in the informa-
tion, newborns already have the perceptual mechanisms for later learning the meaning of lexical
words.
Right from birth, infants see many faces speaking to them. The synchrony of voice and mouth
provides amodal information whereas the pairing of the face and the sound of the voice is
arbitrary. Using familiarization and preferential looking times with alternated presentations of
familiar and new stimuli (Spelke 1976), Coulon et al. (2011) recently showed that unfamiliar mov-
ing and talking faces in videos are salient and can enhance face recognition (see also, Guellai et al.
2011; Guellai and Streri, 2011). Moreover, the face of a previously unfamiliar woman was recog-
nized more efficiently when seen talking than when silent. This result supports the idea that audio-
visual integration is fundamental to efficient face processing and learning immediately after birth.
If temporal synchrony is a fundamental dimension to link visual and auditory information
about an event or object, it is not always necessary when the newborns have to abstract informa-
tion not directly available in the layout, such as large numerosities. Izard et al. (2009) recently
revealed that newborn infants spontaneously associate slowly and smoothly moving visual spatial
arrays of 4–18 objects with rapid auditory sequences of events on the basis of number (see
Fig. 4.1). In these experiments, infants were familiarized with sequences of either 4 or 12 sounds
(6 or 18 sounds) accompanied by visual arrays of either 4 or 12 objects (6 or 18 objects). In all the
familiarisation conditions, newborn infants looked longer at the visual image with the matching
number of objects. Despite the absence of synchrony between the sounds and objects, newborns
responded to abstract numerical quantities across different modalities and formats (i.e. sequential
versus simultaneous).
In short, newborns have to understand a complex environment providing a variety of changing
inputs. They have the intermodal mechanisms necessary to perceive this environment in an organ-
ized manner, allowing for an immediate adaptation to facilitate their chances of survival. All these
studies shed light on the different means by which newborns integrate these multiple inputs. These
mechanisms can involve cognitive processes at different levels. For instance, turning the eyes towards
a sound source does not require the same competencies (or have the same ecological value) as
orienting toward or detecting the mother’s milk. The modifications of a newborn’s arousal due to
a milk odour or to an auditory sound make fewer cognitive demands than integration of the arbi-
trary auditory–visual inputs aided by temporal and spatial synchrony. Finally, abstracting and
THE DIVERSITY OF INTERACTIONS BETWEEN THE SENSES IN HUMAN NEWBORNS 91

(A) Familiarisation (2 mins)

. . . “tu-tu-tu-tu-tu-tu-tu-tu-tu-tu-tu-tu-” ... “ra-ra-ra-ra-ra-ra-ra-ra-ra-ra-ra-ra-” . . .

or

. . . “tuuuuu-tuuuuu-tuuuuu-tuuuuu” ... “raaaaa-raaaaa-raaaaa-raaaaa” . . .

Test (4 trials)

(B)

40 s
*** ***
p<0.0001
p<0.1

30
***
Looking time

20

10

0
4 vs. 12 6 vs. 18 4 vs. 8

Congruent number Incongruent number


Fig. 4.1 (A) Auditory and visual displays for an experiment on number representation in newborn
infants, and (B) infants’ looking times to the visual arrays that corresponded or differed in number
from the accompanying auditory sequences (after Izard et al. 2009).

discriminating large numerosities across vision and audition without the aid of synchrony are
abilities that are also present in newborns. In other words, newborns have at their disposal various
fundamental competencies to develop more complex abilities. It is plausible that these competen-
cies may be insufficient or vague or incomplete, and that learning will be necessary for a further
extension of knowledge.
92 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

4.3 Crossmodal relations between touch and vision


In recent decades there have been considerable advances in our understanding of the matura-
tional development of vision in the first year after birth. Newborns’ neurological and behavioural
visual systems are very poor, with weak acuity and contrast sensitivity (Allen et al. 1996), poor
fixation ability, and uncoordinated saccadic and other ocular movements (Ricci et al. 2008).
Nevertheless, a series of studies has demonstrated the successful perception of both faces (see
Pascalis and Slater 2003) and objects (Slater 1998). Between the ages of 6 and 9 months, visual
acuity reaches near-mature levels. In contrast, few studies have focused on the tactile system in
this period of development. A recent study using magneto-encephalography (MEG) has provided
evidence concerning the cortical maturation of tactile processing in human subjects from birth
onward (Pihko et al. 2009). Maturation of short-latency cortical responses to tactile stimulation
is largely complete two years after birth. A contrast worth noting here is that while the somes-
thetic system, the first to function in utero, develops slowly even after birth, the visual system, the
last to really begin to work, develops extremely quickly after birth (Granier-Deferre et al. 2004;
Lecanuet et al. 1995).
In the first part of the following section, the complexities of the tactile system in adults and
its development in infancy are described, invoking some important methodological considera-
tions along the way. In the second part, experimental evidence concerning interconnections
between the tactile and visual modalities in newborns is presented in an attempt to evaluate pos-
sible explanations for both the successes and the failures of these interconnections. These findings
will be brought to bear on current theoretical views of the development of relations between
modalities.

4.3.1 Characteristics of the tactile system in adults


The mouth and the hands are the best organs for perceiving and knowing the properties of the
environment using the tactile modality. In many mammals, the mouth and the vibrissal system1
are the most sensitive body parts and the main tools of perceptual exploration. For non-human
primates, the hands are also used for feeding, running, stroking, active manipulation, etc.,
allowing animals to vary their activity in situationally appropriate ways (Tomasello 1998). This
specialization culminates in human beings, whose hands and arms have become a principal tool
of investigation, manipulation and transformation of the world.
It is widely recognized that one characteristic of touch stems from the fact that the hands
are both perceptual systems, able to explore the environment, and motor organs, able to modify
it (Hatwell et al. 2003; Loomis and Lederman 1986). These two skills, perceptual and motor, give
to the haptic system its unique and original property among the senses. Although it is necessary
to distinguish between ‘action for perception’ (exploratory action) and ‘perception for action’
(perception subserving or preparing action; Lederman and Klatzky 1996), perception and action
are closely linked in haptic functioning in all cases.
Knowing an object by touch depends on contact with surfaces and on the quality of proximal
reception. The tactual perceptual field is limited to the zone of contact with an object and has the
exact dimensions of the surface of the skin in contact with the stimulus. However, to obtain a
good representation of the whole object and identify it, voluntary movements must be made in
order to compensate for the smallness of the tactile perceptual field. The kinesthetic perceptions
resulting from these movements are necessarily linked to the purely cutaneous perceptions
generated by skin contact, and they form a whole called ‘haptic’ (or tactilo-kinesthetic, or active

1 Vibrissae (or whiskers) are hairs usually employed for tactile sensation by mammals.
CROSSMODAL RELATIONS BETWEEN TOUCH AND VISION 93

touch) perception. As a result, object perception in the haptic mode is highly sequential. This
property increases the load on working memory and requires a mental reconstruction and syn-
thesis in order to obtain a unified representation of a given object (Revesz 1950).
In adults, the hands alone are able to gather information about the different properties
of objects, but these properties do not have the same salience under haptic exploration with or
without vision (Klatzky et al. 1987). In a free-sorting task, participants had to sort by similarity
objects with different properties according three instructions or conditions: without vision, with
visual imagery, or with visual inspection of the objects. Hardness and texture were highly salient
in the haptic-alone group. For the haptic-plus-imagery group, shape was especially salient. The
haptic-plus-visual group showed salience to be well distributed over all the dimensions. Haptic
identification of different properties of objects is possible, each being detected by means of
specific exploratory procedures (EPs). EPs are voluntary movements for detecting a particular
property of object. For example, enclosure is a necessary procedure to detect global shape; contour
following allows a good detection of precise shape, while lateral motion on the object surface is
sufficient to detect texture. However, not of all these exploratory procedures are exhibited by
infants. Therefore, it is difficult to know which properties the infants can detect. Perhaps they
detect only one or two properties when they are holding an object in their hands but they are
unable to detect all the properties of an object with precision because they do not yet have the
specific EPs that have been described in adults.

4.3.2 Characteristics of the tactile system in infants


In recent years, there has been an increased interest in the manual perception of objects in human
newborns. Earlier, newborns had typically been described as displaying mainly reflex reactions
and clumsy arm movements. Newborns’ hands, in particular, have often been described as closed,
or exhibiting either grasping or avoiding reflexes, which are inappropriate behaviours for holding
an object and gathering and processing information about it (Twitchell 1965). As a consequence,
while a wealth of research studying infants and very young children has been devoted to the
description and development of manual action skills (displacement and manipulation of
objects)—skills which, as mentioned above, reflect the haptic system’s distinctive coupling of
motor and perceptual abilities—relatively few studies have focused on the use of the hands
for perceptual information-gathering. Another reason for the lack of emphasis on manual explo-
ration and perception in young infants is that they spontaneously display marked and varied
behaviours with their mouth, but not with their hands. Numerous studies of the neonatal imita-
tion of tongue protrusion or mouth opening (Field et al. 1982; Fontaine 1984; Maratos 1973;
Meltzoff and Moore 1977; Vinter 1986) and coordination of hand–mouth actions (Butterworth
and Hopkins 1988) reveal the early-developing perceptual and motor abilities of young babies.
Thus, the mouth has been considered by researchers as an exploratory tool and an important part
of the haptic system, playing a major role for the infant when gathering information about
objects. Because of these factors, studies on the specifically manual abilities of newborns have
unfortunately been rather neglected, almost to the point of being forgotten altogether. Thus,
numerous researchers have held that manual exploration is very limited or absent in infancy
(Gibson and Walker 1984; Meltzoff and Borton 1979; Rochat 1989). It is very likely that, from
birth, mouthing and oral exploration provide fundamental means for infants to interact with
objects. However, the hands often aid in these interactions.
While perceiving with the hands is intensively practiced by the young infant, at about
5–6 months of age (see Streri 1993), infants start to maintain a sitting position. They transport a
held object to their mouths or their eyes to bring it into view, and often, if they see an object in
their peripersonal space, i.e. near them, they attempt to catch or grasp it. As a consequence, the
two modalities interact in the course of exploration. In this simultaneous bimodal exploration,
94 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

infants have to judge as equivalent items of information gathered simultaneously by the haptic
and visual systems in order to obtain a complete and high-quality representation of the object.
A series of studies has offered a better understanding of the efficiency of each modality (visual,
oral, manual) in babies’ play activities (Gottfried et al. 1978; Ruff 1984; Ruff et al. 1992; Jovanovic
et al. 2008). However, in multisensory exploration, it is difficult to isolate and evaluate the effects
of the manual system’s activity and its consequences for other sensory systems. Our research has
focused mainly on the role of the hands as perceptual systems considered in isolation (i.e. without
visual inspection) and their interaction with vision in crossmodal tasks.

4.3.3 Methodological considerations when assessing tactile


perception in infants
In newborns and young infants, grasping and avoiding reflexes were once regarded as the domi-
nant behaviours mediating interactions with the environment. At first glance, the grasping reflex
in neonates seems to be similar to the adult’s enclosure procedure. However, according to Katz
(1925) and Roland and Mortensen (1987), newborns’ tendency to strongly close their fingers on
an object or an adult’s finger makes it impossible for them to perceive the fine details of an object,
or even recognize an object with a single grasp. In adults, tactile information about object shape
is sampled sequentially by several fingertips sweeping over the object’s surface in different direc-
tions with different velocities. Under such conditions, a newborn’s grasping may be insufficient
to detect fine features. However, a series of several grasps might be adequate to perceive global
shape, rough texture, etc. and give an involuntary EP to detect the global shape.
For this reason, the studies have been performed using habituation with an infant-control proce-
dure, as opposed to fixed-duration familiarization. The familiarization procedure was used in Rose’s
(1994) well-known work in 6- to 24-month-olds. The procedure makes it possible to determine the
minimum duration of tactile familiarization required to obtain visual recognition in crossmodal
transfer tasks.2 The babies feel the object with their hand, without visual control, for a fixed duration
in one trial alone and then see the familiar object and the novel object. One disadvantage of this
method is that it does not necessarily allow the infant to explore the object with more than one grasp,
and thus extract a more full representation of its shape. The habituation procedure, usually used in
the visual modality, was applied for the first time in the haptic modality with 5-month-old infants
(Streri and Pêcheux 1986a), and can easily be adapted to newborns (see Fig. 4.2).
The haptic habituation/dishabituation procedure presents several advantages:
◆ The decrease in holding times seems to reveal the infants’ abilities to perceive and memorize
the shape and recognize it. Moreover, it is now well established that habituation and memory

2 The haptic habituation phase included a series of trials in which the infants received a small object in one
hand. A trial began when the infant held the object and ended when the infant dropped it or after a maxi-
mum duration defined by the experimenter. This procedure was repeated several times. As a consequence,
the habituation process entailed several grasps of different durations. Usually, the minimum duration for a
simple grasp was 1 sec of holding and the maximum was 60 sec. The inter-trial intervals were short (between
4 and 10 sec). Trials were continued until the habituation criterion was met. The infant was judged to have
been habituated when the duration of holding on any two consecutive trials, from the third onwards,
totalled a third (or a quarter according to the infant’s age) or less of the total duration of the first two trials.
The total holding time was taken as an indicator of familiarization duration, exclusive of inter-trial interval.
The mean number of trials to reach habituation was between four and twelve, and often varied according
to the complexity of shapes. Then, in the dishabituation phase, a novel object was put in the infant’s hand.
If an increase in holding time of the novel object was observed, one can infer that the baby was reacting to
novelty, and thus had noticed the difference between the novel and the familiar objects.
CROSSMODAL RELATIONS BETWEEN TOUCH AND VISION 95

50

40

30
Right hand

20

10
Mean duration of holding (seconds)

0
-

50

40

30
Left hand

20

10

0
N-4 N-3 N-2 N-1 N+1 N+2 T1 T2
Haptic habituation PC trials Haptic test
Fig. 4.2 Habituation and discrimination of object shapes by newborn infants in both the left and
the right hand (after Streri et al. 2000). The superimposed shapes indicate the stimuli which were
presented to newborns haptically (the shape which newborns were habituated to was
counterbalanced between participants). ‘PC trials’ are post-criterion trials in which newborns
response to the habituated stimulus were measured. Adapted from Arlette Streri, Myriam Lhote,
and Sophie Dutilleul, Haptic perception in newborns, Developmental Psychobiology, 3 (3),
pp. 319–327, © 2000, John Wiley and Sons, with permission.
processes are closely linked, revealing a form of mental representation of stimuli (cf. Pascalis
and De Haan 2003; Rovee-Collier and Barr 2001).
◆ It gives the infant more than one haptic sample.
◆ This procedure, controlled by the infant, effectively reveals the early perceptual capacities of
young babies (cf. Streri 1993).
96 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

◆ Using the same procedure for both visual and haptic tasks makes it possible to compare
performance in these two modalities at the same age. If performance is identical in both
modalities, one can conclude that the processes for taking in information are comparable and
the exchange of information between the senses is possible.

4.3.4 Experimental evidence of perceptual manual


abilities in newborns
Streri et al. (2000) demonstrated that newborns were able to detect differences in the contours
of two small objects (a smoothly curved cylinder versus a sharply angled prism) with both the right
and left hands. Habituation and test phases were performed with the same hand without visual
inspection. After habituation with one of the two objects placed in the right or left hand, confirmed
by two more trials (this is known as the partial-lag design, see Berthental et al. 1983), a novelty
reaction was observed when a new object (the prism or the cylinder) was put in the same hand. This
experiment provided the first evidence of habituation and reaction to novelty observed with the
left as well as the right hand in human newborns (cf. Fig. 4.2). Recently, manual habituation and
discrimination of shapes in preterm infants from 33 to 34 weeks post-conceptional age have
been demonstrated in the same conditions as in full-term newborns, regardless of hand side (Lejeune
et al. 2010). Pre-term and full-term newborns are able to discriminate between curvilinear and
rectilinear contours in small objects. As a consequence, it may be concluded that haptic memory in
its manual mode is also present from birth. Within the first 6 months after birth, the young baby is
also capable of reasonably good performance in the detection of other shape distinctions.
Nevertheless, this behaviour does not show that the baby has a clear representation of what
is being held in the hand. They do not explore the objects actively in their hands. Some squeeze-
releases on the objects are observed. Because young infants are unable to perform the integration
and synthesis of information in working memory required under haptic exploration, shape per-
ception is probably partial or limited to the detection of features such as points, curves, and edges.
The information gathered is provided from enclosure of the object, which seems to be an effective
exploratory procedure for these limited purposes.

4.4 The links between the haptic and visual modalities:


an old–new philosophical debate
If the visual and haptic systems characterize perceptual objects according to some common
property or feature, and if information for that property flows between the two systems, the pre-
requisites to reveal links between modalities in crossmodal transfer tasks are present. The ques-
tion of whether there are links between sensory modalities in newborns is crucial because although
both haptic and auditory information is available in the fetal period, visual information is not. We
may thus ask whether human newborns can perceive shape equivalence between vision and
touch.
For several centuries, the question of crossmodal integration between the senses has been
addressed by philosophical answers to Molyneux’s famous question: will a person born blind
who recovers sight as an adult immediately be able to distinguish visually between a cube and a
sphere? (Bruno and Mandelbaum 2010; Gallagher 2006; Proust 1997). But Diderot (1749–1792),
the first philosopher to compare the blind person to a neonate, contended that there is no doubt
‘that vision must be very imperfect in an infant that opens his/her eyes for the first time, or in a
blind person just after his/her operation’. Diderot was right when he said that newborn’s vision is
very imperfect. However, many experiments have since provided evidence that neonates perceive
THE LINKS BETWEEN THE HAPTIC AND VISUAL MODALITIES 97

several elementary properties of the world, such as colour, shape, movement, objects, and faces
(cf. Kellman and Arterberry 1998; Pascalis and Slater 2003). Moreover, Molyneux’s question
precisely describes the crossmodal task from hands to eyes in infancy. Although newborns are not
directly comparable to congenitally blind individuals (see Gallagher 2006, for a discussion), if
they are able to form a perceptual representation of the shape of objects from the hand and to
recognize this shape visually, this would suggest an affirmative answer to Molyneux’s question.
This would mean that a link between the hands and the eyes exists before infants have had the
opportunity to learn from the pairings of visual and tactual experiences.

4.4.1 Methodological considerations when assessing


crossmodal transfer in infants
Crossmodal transfer tasks have been used to study Molyneux’s question. They involve two
successive phases: a habituation phase in one modality, followed by a recognition phase in a
second modality. In the ‘tactile-to-visual modality’ task, newborns undergo tactile habituation
to an object that is placed in their right hand. Then, in the second phase, the familiar and
the novel objects are visually presented in alternation during four trials in a counterbalanced
order between participants. The ‘tactile-to-visual modality’ task is very similar to the situation
described in Molyneux’s question. It makes it possible to uncover not only links between
the senses but also, in older babies, the capability of using the hands to go beyond directly
perceived information about the environment to abstract some information, as in the case
of perception of partly occluded objects presented in the visual mode (Kellman and
Spelke 1983)—adapted to the haptic modality (Streri and Spelke 1988, 1989)—or in the case of
the perception of number of held objects (Féron et al. 2006). The opposite transfer is also possi-
ble. In the ‘visual-to-tactile modality’ task, newborns are first visually habituated to an object and
then, in the second phase, the familiar and the novel objects are tactually presented to their right
hands in alternation during four trials. In both tasks a reaction to novelty is expected. Its occur-
rence means that the newborns recognize the familiar object and explore the novel object for a
longer time.
This paradigm, widely used in infancy studies, involves several cognitive resources. In the first
phase, the baby has to collect a piece of information on an object in one modality, memorize this
information, and then, in the second phase, choose which object is the familiar one in another
modality.

4.4.2 Experimental evidence of crossmodal transfer of information


Several lines of investigation have approached but not resolved Molyneux’s question. Crossmodal
transfer of texture (smooth versus granular) and substance (hard versus soft) from the oral
modality to vision has been shown in 1-month-old infants (Gibson and Walker 1984; Meltzoff
and Borton 1979). Two-month-old infants were able to visually recognize the shape of an object
that they had previously manipulated with their right hand (Streri 1987). However, several
questions regarding the origins of intermodal transfer remain. First, research on the visual
abilities of newborns indicates a clear possibility that babies might have learned links between the
modalities over 1 or 2 months. Second, data on oral-to-visual transfer of texture were sceptically
received, and the results have not always been reproduced (cf. Maurer et al. 1999). Third, mouth-
ing behaviour involves different exploratory procedures from handling, which is more complex
and varied. These discrepancies suggest that information processing by the two modes of explora-
tion might not develop in a similar manner or have the same function. Finally, according to
Klatzky and Lederman’s (1993) classification, texture and substance are material properties and
98 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

not geometric properties. Molyneux’s question, in contrast, concerns the perception of object
shape. Shape is a geometric property under Klatsky and Lederman's (1993) classification, and
thus it would require different exploratory procedures to those available orally.
Streri and Gentaz (2003 ) conducted an experiment on crossmodal recognition of shape
from the right hand to the eyes in human newborns. They used an intersensory paired-preference
procedure that included two phases: a haptic familiarization phase in which newborns were given
an object to explore manually without seeing it, followed by a visual test phase in which infants
were shown the familiar object paired with a novel one for 60 seconds. The participants consisted
of 24 newborns (mean age: 62 hours). Tactile objects were a small cylinder (10 mm in diameter)
and a small prism (10-mm triangle base). Because the vision of newborns is immature and their
visual acuity is weak, visual objects were the same 3D shapes, but much larger (45-mm triangle
base and 100 mm in length for the prism and 30 mm in diameter and 100 mm in length for the
cylinder). An experimental group (12 newborns) received the two phases successively (haptic
then visual) whereas a baseline group (12 newborns) received only the visual test phase with the
same objects as the experimental group but without the haptic familiarization phase.
The comparison of looking times between the two groups allowed us to provide evidence of
crossmodal recognition of shape from hand to eyes in the experimental group. The results revealed
that the newborns in the experimental group looked at the novel object for longer than the famil-
iar one. In contrast, the newborns in the baseline group looked equally at both objects. Moreover,
in the experimental group, infants made more gaze shifts toward the novel object than the famil-
iar object. In the baseline group this was not the case. Thus, this recognition in the experimental
group stems from the haptic habituation phase. These results suggest that newborns recognized
the familiar object through a visual comparison process as well as a comparison between the haptic
and visual modes. Moreover, the discrepancy between the size of visual and tactile objects was
apparently not relevant for crossmodal recognition. Shape alone seems to have been considered
by newborns.
In conclusion, newborns are able to transfer shape information from touch to vision before
they have have the opportunity to learn the pairing of visual and tactile experiences. These results
challenge the empiricist philosophical view, as well as modern connectionist models (Elman et al.
1996; McClelland et al. 1986; see also Chapter 15 by Mareschal et al.) arguing that sensory modal-
ities cannot communicate in newborns. The results reveal an early-developing ability, largely
independent of experience, to detect abstract, amodal higher-order properties of objects. This
ability may be a necessary prerequisite to the development of knowledge in infancy. At birth,
various perceptions of objects are unified to make the world stable. How should these results be
interpreted?
Recently, Held (2009) proposed another interpretation of these results. After several attempts
to obtain an affirmative answer to Molyneux’s question (cf. Jeannerod 1975; Pascual-Leone and
Hamilton 2001), Held’s experiments on congenitally blind adults whose vision was restored
have revealed that they learned very quickly to map a felt object with a seen object by a process
called ‘capture’. This capture consists, for adults, of seeing their hands in normal activities. It is
sufficient for recalibrating the two senses and serves as basis for the crossmodal matching abilities.
In a second experiment (Streri and Gentaz 2004), newborns were 54 hours old. Referring to
this experiment, Held proposed that newborns had the possibility, within these 54 hours, to see
their hand and as a consequence learn to map the felt shape with the seen object through
‘capture’. A third, more recent experiment by Sann and Streri (2007), nevertheless cast doubt
on this interpretation. In Sann and Streri’s experiment, the youngest newborn was 12 hours old.
It seems less likely that during this short period, the ‘capture’ and recalibration effects could have
occurred.
THE LINKS BETWEEN THE HAPTIC AND VISUAL MODALITIES 99

The contrast between the early-emerging perceptual capacities of newborn infants, and the
slower-to-adapt capacities of blind adults whose sight is restored, suggests that the fundamental
question concerning the origins of perceptual knowledge may be better addressed through studies
of infants than through studies of adults. Although both newborn infants and newly operated
blind adults are similarly devoid of visual experience, they differ greatly in other ways. In particu-
lar, blind adults have a lifetime of experience using touch, audition, and other senses to guide
their actions. They also have a nervous system that is fully mature. For both reasons, adults may
adapt to newly incoming visual information, but they are likely to achieve this much more slowly
than infants.
Other explanations have been proposed in the literature to explain these early perceptual
abilities:
1 Synaesthesia hypothesis: Maurer (1993; for a more recent version see Chapter 10 by Maurer et al.;
Spector and Maurer 2009) suggested that newborns cannot yet distinguish among the various
senses ‘ . . .The newborn’s senses are not well differentiated but are instead intermingled in a synes-
thetic confusion . . . Energy from the different senses, including the proprioceptive sense of his
or her own movement, is largely if not wholly undifferentiated: the newborn perceives changes
over space and time in the quantity of energy, not the sense through which it arose’ (Maurer 1993,
pp. 111–112). On this view, early crossmodal transfer is based on a lack of distinction between
senses rather than on a crossmodal recognition of objects and the concept of synaesthesia
would be more appropriate for understanding the relations between sensory modalities. In
the adult field, synaesthesia has a different meaning. It has been described as a ‘mixing’ of the
senses (Rich et al. 2004) or as occurring in a person who experiences sensations in one modal-
ity when a second modality is stimulated, even though no physical stimulation is presented in
the first modality (Marks 1975; Ramachandran and Hubbard 2001). In other words, synaes-
thesia is a clearly defined sensation which is perceived in addition to the percepts of objects
that are physically present and induce the synesthetic sensation. If the most common type of
intramodal synaesthesia is coloured numbers, coloured hearing is a frequent form of intermodal
synaesthesia wherein speech sounds or music produce colored visual images as well as auditory
perceptions.
2. Active intermodal matching hypothesis: Meltzoff (1993) proposed the active intermodal
matching hypothesis (AIM), according to which, 1-month-old infants match what they see
and what they explore orally (with a preference for familiar texture) in experiments on cross-
modal transfer (Meltzoff and Borton 1979). On this view, young infants use a supramodal code
that allows them to unify information derived from different sensory modalities in a common
framework. The same mechanism is invoked to explain neonatal imitations, such as tongue
protrusion or mouth opening. According to Meltzoff’s hypothesis, imitation taps perception,
crossmodal coordination, and motor control. Early imitation is mediated by AIM; to perform
it young infants use the crossmodal equivalence between the dynamics of the visual stimulus
and their own motor signals.
3. Abstract amodal information hypothesis: E.J. Gibson (1969) suggested that infants are able to
abstract amodal or common information. Abstraction of common information is based on
the equivalence between modalities with regard to amodal properties such as location, texture,
movement, shape, and duration. Since Gibson’s suggestion, the concept of amodal perception
has often been used to describe the link between modalities. According to this view, a common
percept, independent from any one sense, is created.
In the remainder of this section I will bring the findings observed in Streri and Gentaz’s experi-
ments to bear on the three hypotheses discussed above. Questions we have addressed, and which
100 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

are relevant to distinguishing these hypotheses, concern whether crossmodal transfer is a general
property of the newborn human, or whether it is specific to certain parts of the body. We have
also examined whether crossmodal transfer occurs only in a single direction or not. In concrete
terms, the issues we have addressed are:
◆ Considering that habituation and discrimination have been demonstrated in both the right
and left hands, is crossmodal transfer realized regardless of which hand is stimulated?
◆ If newborns cannot distinguish between senses then, in crossmodal transfer tasks, whatever
modality is stimulated first, recognition should be found in the second modality. The same
argument can be taken up for testing the concept of amodal perception: a bidirectional transfer
should be evidenced.
◆ If the AIM hypothesis is correct, then a preference for the familiar object should be shown in
the recognition test phase—infants matching the felt (or seen) object with the seen (or felt)
object. This is because the AIM proposes that infants should actively attempt to match the
senses. One could argue thus, that active matching would result in a familiarity preference
following crossmodal transfer.
Streri and Gentaz (2004) tested for crossmodal transfer from the left hand to vision, as well as
transfer from the right hand to vision but with another methodology. In the visual test phase,
newborns looked at only one object per trial and the objects were presented successively and not
simultaneously as in previous experiments (Streri and Gentaz 2003). Twenty-four newborns
received the tactile habituation phase with the left hand, 24 other newborns received it with the
right hand, and 12 newborns received no tactile habituation phase. In the visual test phase,
newborns in the experimental groups looked at the familiar object and at the novel object over
four sequential trials presented in alternation (one object per trial). In the control group, new-
borns looked at the cylinder and the prism over four trials in the same manner as the experimen-
tal groups. Thus, all newborns looked twice at both the prism and the cylinder. The order of
presentation of the objects was counterbalanced across participants.
The results were as follows: crossmodal recognition was again evidenced from the right hand:
newborns looked longer at the novel object. No visual recognition was evidenced from infants in
the left-hand group. The haptic-habituation phase did not influence the visual-test phase. In the
control group, newborns looked equally at the two objects. The interpretation of laterality effects
at birth is very difficult (Streri and Gentaz 2004). Some authors have suggested that they might be
determined by foetal position (Michel and Goodwin 1979). Two-thirds of infants are born in a
left vertex presentation, which is probably also their position in the womb 3–4 weeks prior to
birth. It has been suggested that, due to the fetal position, the left arm is stuck to the mother’s
back and consequently receives a lesser flow of arterial blood than the right arm. Other studies in
connected fields have also observed asymmetries favouring the right side before and after birth
(Streri 1993). Newborns present a spontaneous preference for lying with their head turned
toward the right while supine (Coryell and Michel 1978; Gesell 1938; Turkewitz et al. 1965).
Laterality effects in crossmodal transfer from hand to eyes might have the same origin.
Sann and Streri (2007, Exps. 1 and 2) tested transfer from eyes to hand and from hand to eyes
in order to ascertain whether a complete primitive ‘unity of the senses’ would be demonstrated.
After haptic habituation to an object (cylinder or prism), the infants saw the familiar and the
novel shape in alternation. After visual habituation with either the cylinder or the prism, the
familiar and the novel shape were put in the infant’s right hand. The tactile objects were presented
sequentially in an alternating manner. Following haptic habituation, visual recognition was again
THE LINKS BETWEEN THE HAPTIC AND VISUAL MODALITIES 101

observed, whereas no haptic recognition was found following visual habituation. These results
have implications for the three models described earlier.
1. Our data do not support an account that relies on a confusion of the senses as suggested by
Maurer (1993). Haptic inputs are transferred to the visual modality, but the reverse is not true.
That is, the senses are not confused. Some research suggests that synaesthetic correspondences
between sensory modalities, namely between the auditory and visual modalities, exist in chil-
dren (Mondloch and Maurer 2004; although see Spence 2011) and 3- to 4-month-old infants
(Walker et al. 2010), but currently there is no evidence that newborns’ perception is synaes-
thetic. Our results cannot stem from synaesthesia for several reasons. Some of these correspon-
dences can be attributed to intensity matching, but shape is a structural property that cannot
be defined in terms of ‘intensity of stimulation.’ According to Ramachandran and Hubbard
(2001), synaesthesia is a sensory effect rather than a cognitive one based on memory associa-
tions. Our crossmodal transfer tasks between touch and vision with successive presentations
require several cognitive processes such as memory, comparison between senses, and choice
between the objects. Such processes are not involved in an intersensory paired-preference pro-
cedure, as used in Mondloch and Maurer’s (2004) experiments, where the visual and auditory
stimulation were presented simultaneously. Moreover, in visual and auditory transfer tasks,
the senses are both distal, while the tactile sense deals with proximal stimuli. As a consequence,
several forms of intersensory integration are possible at birth, because intersensory integration
is not a unitary process (see Lewkowicz 2000, for a review).
2. Intermodal matching is also ruled out because the transfer is not reciprocal. A felt object is
matched with a seen object, but a seen object is not matched with a felt object. A common
framework may exist, but the matching process is not reciprocal with regard to object proper-
ties, at least in the case of crossmodal transfer of shape information. Moreover, in our experi-
ments, newborns looked more at the novel object than the familiar object, unlike in Meltzoff
and Borton’s (1979) experiments.
3. Detection of amodal information seems to be the best-supported hypothesis because habitua-
tion and dishabituation are observed in both modalities. Newborns detect invariants whatever
the modality that receives information. But this hypothesis must be made more precise. How
exactly is amodal information perceptually linked, given that transfer is not reciprocal?

4.4.3 Similarity of processes versus similarity of representations


Our results suggest that similarity of processes does not necessarily lead to similarity of represen-
tations. Recall that newborns are able to habituate and discriminate between various features in
vision and touch. They detect the same invariants in the course of habituation and react to the
novelty in the phase test when they perceive a novel feature. However, according to Slater et al.
(1991), information processed by vision is global and complete from birth. Although the ‘global’
character of early visual perception is debated (Cohen and Younger 1984), vision, with its large
perceptual field and numerous receptors, is relatively ‘high-bandwidth’ and parallel. Information
gathering by touch, in contrast, is relatively ‘low-bandwidth’ and sequential: even though, thanks
to the grasp, babies are capable of taking in information about the contours of the object, this
information is limited at any one moment to the stimulation of a small patch of skin receptors.
Moreover, successively gathered information may not be synthesized. This discrepancy in the
manner of information-gathering could be the cause of non-reciprocal transfer between vision
and touch. A seen object can be processed as a structured whole, but the newborn cannot manually
102 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

recognize this totality from a felt feature. In contrast, an infant can visually recognize a feature of
an object that he or she has previously felt. Amodal perception would be obtained from the sim-
plest processed information; that is, information gathered by the haptic modality. The amodal
percept is only partial because, as discussed above, the newborn is not able to establish a complete
percept of the object using the hand alone. Consequently, crossmodal transfer of shape informa-
tion between touch and vision is not a general property of the organism, since it is found only in
one direction, from touch to vision, and only from the right hand, not from the left.
An important question is how haptic input could be translated into a visual format given that
the sensory impressions are so different? To date, in adults, there is substantial neuroimaging
evidence showing that vision and touch are intimately connected, even though different views
are proposed (see Amedi et al. 2005; Sathian 2005 for reviews). Cerebral cortical areas, specifically
the lateral occipital complex (LOC), previously considered as exclusively visual areas become
activated during haptic perception of shape (Lacey et al. 2007). Perhaps this mechanism is already
present in newborns’ brains.

4.4.4 Shape versus texture


Shape and texture are two amodal properties processed by both vision and touch. Studying these
two properties could allow us to test the hypothesis of amodal perception in newborns anew,
as well as shedding light on the processes involved in gathering information by both sensory
modalities. However, shape is best processed by vision, whereas texture is thought to be best
detected by touch (see Bushnell and Boudreau 1998; Klatzky et al. 1987). According to Guest and
Spence (2003), texture is ‘more ecologically suited’ to touch than vision. In many studies con-
cerning shape (a macrogeometric property), transfer from haptics to vision has been found to be
easier than transfer from vision to haptics in both children and adults (Connolly and Jones 1970;
Jones and Connoly 1970; Juurmaa and Lehtinen-Railo 1988; Newham and MacKenzie 1993;
cf. Hatwell 1994). In contrast, when the transfer concerns texture (a microgeometric property),
for which touch is as efficient as (if not better than) vision, this asymmetry does not appear.
Neuroimaging data obtained in human adults has suggested that there exists a functional
separation in the cortical processing of micro- and macrogeometric cues (Roland et al. 1988). In
this study, adults had to discriminate the length, shape, and roughness of objects with their right
hand. Discrimination of object roughness activated the lateral parietal opercular cortex signifi-
cantly more than did length or shape discrimination. Shape and length discrimination activated
the anterior part of the intraparietal sulcus more than did roughness discrimination. More
recently, Merabet et al. (2004) confirmed the existence of a functional separation and suggested
that the occipital (visual) cortex is functionally involved in tactile tasks requiring fine spatial
judgments in normally sighted individuals. More specifically, a transient disruption of visual
cortical areas using repetitive transcranial magnetic stimulation did not hinder texture judgments
but impaired the subject’s ability to judge the distance spacing between raised dot patterns.
Conversely, transient disruption of somatosensory cortex impaired texture judgments, while
interdot distance judgments remained intact. In short, shape and texture properties need differ-
ent exploratory procedures to be detected, and take place in two different pathways in adult
brains. In our behavioural research, we attempted to determine whether shape and texture are
processed similarly in human newborns or if the different processing modes develop with age
through learning and maturation.
To the best of our knowledge, no study has examined whether crossmodal transfer between
vision and touch occurs in newborns for a property other than shape—for example texture,
although some results about texture processing have been presented. Molina and Jouen (1998)
demonstrated that newborns are able to modulate their manual activity with objects that vary
THE LINKS BETWEEN THE HAPTIC AND VISUAL MODALITIES 103

only according to their texture. Their study was based on the recording of the pressure activity
level exerted on objects. To measure haptic perception, they used hand pressure frequency (HPF),
which consists in the recording of successive contacts between the skin and an object. The results
of this study showed that a continuous high hand pressure was exhibited when newborns held
smooth objects, and discrete continuous low pressure when they held granular objects. Elsewhere,
Molina and Jouen (2001) conducted an experiment to investigate intersensory functioning
between vision and touch for texture information in neonates. Using HPF, they compared man-
ual activity recorded on objects with variable texture in the presence or absence of visual informa-
tion. The visual object had either the same (matching condition) or different (mismatching
condition) texture information as the held object. In the matching condition, holding times
increased but HPF remained unchanged. In the mismatched condition, holding times remained
unchanged, but HPF systematically changed over the test period. Taken together, these results
revealed that manual activity is modulated by vision. This experiment revealed an ability to detect
a common texture across vision and touch in newborns. Nevertheless, these simultaneous match-
ing tasks do not involve memory; they do not tell us anything about the infants’ abilities to trans-
fer information between vision and touch in successive phases.
Sann and Streri 2007 undertook a comparison between shape and texture in bidirectional cross-
modal transfer tasks. They sought to reveal how information is gathered and processed by the vis-
ual and tactile modalities and, as a consequence, to shed light on the perceptual mechanisms of
newborns. If the perceptual mechanisms implicated in gathering information on object properties
are equivalent in both modalities at birth, we would expect to observe a reverse crossmodal trans-
fer. By contrast, if the perceptual mechanisms differ in the two modalities, a non-reversible transfer
should be found. Thirty-two newborns participated in two experiments (16 in crossmodal transfer
from vision to touch, and 16 in the reverse transfer). The material was one smooth cylinder versus
one granular cylinder (a cylinder with pearls stuck on it). The results revealed crossmodal recogni-
tion of texture between modalities in both directions. These findings suggest that for the property
of texture, exchanges between the sensory modalities are bidirectional. Why then is a reverse cross-
modal transfer observed for texture and not for shape (see Fig. 4.3)?
All of our findings suggest that newborns are not able to obtain a good representation of object
shape while they are manipulating it. Two experimental arguments run in favour of this hypoth-
esis: inter-manual transfer of shape versus texture in newborns and bidirectional transfer of shape
in 2-month-olds, under certain conditions.
The study of inter-manual transfer of information does not involve the inherent difficulties in
the decoding process of crossmodal tasks. The inter-manual transfer of tactile information can
be defined as the ability to recognize that an object experienced in one hand is the same or
equivalent to an object experienced in the opposite hand. In adults, neuroimaging studies have
revealed that the transfer of tactile information from one hand to the other rests on the func-
tional integrity of the corpus callosum (Fabri et al. 2001, 2005). The corpus callosum is the
major neural pathway that connects homologous or equivalent regions of the two hemispheres
in mammals and plays a central role in interhemispheric integration and information transfer
(Bloom and Hynd 2005). Although its development begins during fetal life, the corpus callosum
is not fully formed at birth and matures very slowly (Cernacek and Podivinski 1971). However,
neuroimaging data obtained using diffusion tensor imaging (DTI) indicates that, despite uncom-
pleted myelination, the main fiber bundles are already in place at birth (Dubois et al. 2006).
Thus the organization and maturation of white matter bundles seem to be present in infants
from the beginning of life. As a consequence, questions arise as to whether, at birth, the
two hemispheres function independently, and whether inter-manual transfer of information is
possible in the newborn infant.
104 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

Unidirectionality of crossmodal transfer of shape

Vision Touch

Bidirectionality of crossmodal transfer of texture

Vision Touch

Fig. 4.3 (top) Visual and tactual objects used in the Streri and Gentaz (2003, 2004) and Sann and
Streri (2007) experiments on crossmodal transfer of shape. (below) Visual and tactual objects used
in the Sann and Streri (2007) experiments on crossmodal transfer of texture. The arrows indicate
the sense of transfer.

Numerous behavioural studies have shown that human neonates are able to tactually discrimi-
nate between two objects varying in texture (smooth/granular) or rigidity (soft/hard) by modu-
lating their manual activity (e.g. Molina and Jouen 1998 ; Rochat 1987 ). We reported that
newborns are able to process and encode some information about shape and discriminate two
different shapes (cylinder/prism) with either the right or the left hand (Streri et al. 2000). The
latter result revealed that neonates possess haptic memory. It should be possible then to observe
an inter-manual transfer at birth.
Two experiments, using a habituation/reaction-to-novelty procedure, were performed to assess
human neonates’ ability to process and exchange information about texture or shape between
their hands, without visual control (Sann and Streri 2008). Forty-eight newborn infants (24 per
experiment) received haptic habituation either with their right or left hand, followed by a haptic
discrimination test in the opposite hand. In the test phase, the novel and the familiar object were
each presented twice, in alternation. In both conditions, a reaction to the novel object in the
opposite hand was expected. The results revealed two patterns of behaviour, depending on the
object property to be processed. After tactile habituation to a texture in one hand, newborns held
the novel texture longer in the other hand than the familiar one. In contrast, after tactile habitu-
ation to a shape in one hand, the familiar shape was held longer in the opposite hand than the
novel one. These findings suggest that inter-manual transfer is possible at birth despite the rela-
tive immaturity of the corpus callosum. Although, in both cases, an intermanual transfer occurred,
CONCLUSIONS 105

the discrepancies in results between object properties (preference for the novel texture and pref-
erence for the familiar shape) reveal once again newborns’ difficulties in obtaining a sufficient
representation of shape in haptic mode, as well as the possibility of a different haptic processing
of texture and shape in the newborn brain. This view is supported by a developmental inter-
manual transfer study revealing recognition of the felt object and a preference for the novel shape
at the test phase beginning only at 6 months of age (Streri et al. 2008).
The difference may be related to the manner of gathering information involved in each case.
Previous studies have shown that adults perceive texture most precisely when they are permitted
to move their hands in a lateral gliding motion across the surface of stimulus objects (Lederman
and Klatzky 1987, 1990). Newborns do not display this exploratory procedure. The grasping
reflex is a rough exploratory procedure, but it seems to be sufficient to differentiate between
granular and smooth objects. Visual scanning of the surface is also sufficient to differentiate
a granular from a smooth object. As a consequence, our results suggest that the manner of
gathering and exchanging information about texture across modalities is equivalent in touch
and vision from birth. However, this exchange fails if the object is flat and not volumetric,
because newborns do not exhibit the exploratory procedure of ‘lateral motion’ needed to detect
the different textures of flat objects (Sann and Streri 2008). Grasping is an inefficient ‘enclosure
procedure’ to detect differences between textured flat objects. Texture is a material or microgeo-
metric property better processed by touch than by vision (see Hatwell et al. 2003). Our results
reveal that newborns process this property using the hands as well as the eyes. Shape, a geometri-
cal or macrogeometric property, is better processed by vision than by touch. Our results confirm
this as well.
The second argument is built on previous research performed in 2-month-old infants and
using a bidirectional crossmodal shape transfer task (Streri 1987). The findings revealed that
2-month-old infants visually recognize an object they have previously held, but do not manifest
tactile recognition of an already-seen object. A plausible explanation of these results on crossmo-
dal transfer is that, as in newborns, the levels of representation reached by each modality are not
sufficiently equivalent to exchange information between sensory modalities. This hypothesis
seems to be validated by the fact that, if a 2-month-old baby is presented with degraded visual
stimulation (a bidimensional sketch of an 3D object) in which volumetric and textural aspects are
missing, leading to a blurred percept, tactile recognition is possible, which is not the case with a
visual volumetric object (Streri and Molina 1993). Although the methodology is quite different,
our results are supported by Ernst and Banks’s (2002) model of human adults’ performance on
an intersensory integration task. In this study, adults compared the size of objects between touch
and vision. Visual capture was observed when the visual stimulus was clear (‘noise-free’), and
haptic capture was observed when the visual stimulus was noisy. In other words, the sensory
modality that dominates performance is the one with the lowest level of variance. A similar result
was subsequently observed in children, with integration between systems not appearing before
8 years of age (Gori et al. 2008).

4.5 Conclusions
We recognize, understand, and interact with objects through both vision and touch. In infancy,
despite the various discrepancies between the haptic and visual modalities, such as asynchrony in
the maturation and development of the different senses, distal versus proximal inputs, or parallel
processing in vision versus sequential processing in the haptic modality, from birth onward both
systems detect regularities and irregularities when they are in contact with different objects.
Conceivably, these two sensory systems may encode object properties such as shape and texture
106 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

in similar ways. This ability may facilitate crossmodal communication. Our investigations in
crossmodal transfer tasks have revealed some links between the haptic and visual modes at birth.
Newborns are able to visually recognize a held object (Streri and Gentaz 2003). This is the new-
born’s answer both to Molyneux’s question and support of Diderot’s position. This neonatal
ability is independent of learning or the influence of the environment.
The concept of ‘amodal perception’ (Gibson 1969), understood as the creation of a new percept
independent of the sensory modality, fits with our first results on the intermodal transfer from
touch to vision in infancy. However, by means of bidirectional crossmodal transfer tasks, Streri
and colleagues have provided evidence about the perceptual mechanisms present at birth that
constrain or limit the exchange of information between the sensory modalities. Newborns visu-
ally recognize the shape of a felt object, but are unable to recognize the shape of a seen object with
their hands (Sann and Streri 2007). The link is obtained from the simplest information gathered.
Moreover, it is observed only with the newborn’s right hand and not with the left (Streri and
Gentaz 2004). A third striking result is that crossmodal transfer depends on object properties, as
it is bidirectional with texture but not with shape (Sann and Streri 2007)—though this finding
holds if, and only if, the felt textured object is volumetric, and not if it is flat (Sann and Streri
2008). For shape, just as for texture, the newborn’s exploratory procedures are limited to the
grasping reflex, which makes effective exploration of object properties impossible. All these find-
ings suggest that at birth, the links between the senses are specific to individual modalities and are
not yet or entirely a general property of the brain.
A plausible explanation for these failures to obtain an amodal perception could be due to the
situation itself. Recall that crossmodal transfer tasks require several cognitive processes, such as
encoding, memorization, decoding, the comparison of objects, etc. This combination of resources
may be too challenging for the newborn. In simultaneous bimodal situation of haptic and visual
exploration of an object, which is possible starting only at 5 months of age, infants may obtain
a better representation of perceived objects. In infancy, the hands are used as instruments for
transporting objects to the eyes or mouth, and the acquisition of this new ability comes to the
detriment of the perceptual function of the hands.
Several studies have also revealed that over the course of development the links between the
haptic and the visual modes are fragile, often not bidirectional, and representation of objects is
never complete: this holds not only in infancy (Rose and Orlian 1991; Sann and Streri 2007; Streri
and Pêcheux 1986b), but also in children (Gori et al. 2008) and adults (Kawashima et al. 2002).
For example, in a behavioural and PET study on human adults, Kawashima et al. found that the
human brain mechanisms underlying crossmodal discrimination of object size have two different
pathways depending on the temporal order in which the stimuli are presented. Crossmodal infor-
mation transfer was found to be less accurate with visual-to-tactile (VT) transfer compared with
tactile to visual (TV) transfer. In addition, more brain areas were activated during VT than during
TV. Crossmodal transfer of information is rarely reversible, and is generally asymmetrical even
when it is bidirectional.
All of these results contrast with other studies that have provided evidence for amodal percep-
tion in infancy. For example, Féron, Streri, and Gentaz (2005) have demonstrated that 5-month-
old infants are able to use their hands, without visual reinforcement, to discriminate a set of two
objects from a set of three in a crossmodal transfer task. This result means that discrimination of
number is amodal, since it is revealed also in the visual and auditory modes (Izard et al. 2009;
Kobayashi et al. 2005; Starkey et al. 1990). Four-month-old infants are able, thanks to bimanual
haptic exploration, to complete their information about a large partially felt object, and to track
its unity in the same ways as in the visual mode (Kellman and Spelke 1983; Streri and Spelke
1988). How should one interpret these results?
REFERENCES 107

In short, all these studies on the links between the haptic and visual modalities suggest two
levels of amodal perception or amodal representation. In the conditions where the exchanges
concern object properties, directly perceived by means of the senses, amodal perception is never
complete, or is limited to the less efficient modality, i.e. the haptic modality. But if the infants
have to abstract some not-directly-available properties from their environment to better under-
stand it, then amodal perception and representation occur, and operate independently from the
individual sensory modalities.

Acknowledgements
These studies were financed by Paris Descartes University, the CNRS, and a grant from the
Institut Universitaire de France. The author thanks Andrew Bremner, Paul Reeves, Elizabeth
Spelke, and Charles Spence for their remarks, stylistic corrections and suggestions on earlier ver-
sions of this manuscript.

References
Allen, D., Tyler, C.W., and Norcia, A.M. (1996). Development of grating acuity and contrast sensitivity in
the central and peripheral visual field of the human infant. Vision Research, 13, 1945–53.
Amedi, A., Kriegstein, K. von, Atteveldt, N.M. van, Beauchamp, M.S., and Naumer, M.J. (2005). Functional
imaging of human crossmodal identification and object recognition. Experimental Brain Research, 166,
559–71.
Bahrick, L.E. (1987). Infants’ intermodal perception of two levels of temporal structure in natural events.
Infant Behavior and Development, 10, 387–416.
Bahrick, L.E. (1988). Intermodal learning in infancy: learning on the basis of two kinds of invariant
relations in audible and visible events. Child Development, 59, 197–209.
Bahrick, L.E., and Pickens, J.N. (1994). Amodal relations: the basis for intermodal perception and learning
in infancy. In The development of intersensory perception: comparative perspectives (eds. D.J. Lewkowicz,
and R. Lickliter), pp. 205–33. Lawrence Arlbaum Associates, Hillsdale, NJ.
Beauchamp, G.K., and Mennela, J.A. (2009). Early flavor learning and its impact on later feeding behavior.
Journal of Pediatric Gastroenterology and Nutrition, 48, 25–30.
Berthental, B., Haith, M., and Campos, J. (1983). The partial-lag design: a method for controlling
spontaneous regression in the infant-control habituation paradigm. Infant Behavior and Development,
6, 331–38.
Bloom, J.S., and Hynd, G.W. (2005). The role of the corpus callosum in interhemispheric transfer of
information: excitation or inhibition? Neuropsychology Review, 15, 59–71.
Bruno, M., and Mandelbaum, E. (2010). Locke’s answer to Molyneux’s thought experiment. History of
Philosophy Quarterly, 27, 165–80.
Bushnell, E.W., and Boudreau, J.P. (1998). Exploring and exploiting objects with the hands during infancy.
In The psychobiology of the hand (ed. K.J. Connolly), pp. 144–61. Mac Keith Press, London.
Butterworth, G.E., and Castillo, M. (1976). Coordination of auditory and visual space in newborn human
infants. Perception, 5, 155–60.
Butterworth, G.E., and Hopkins, B. (1988). Hand-mouth coordination in the newborn human infant.
British Journal of Developmental Psychology, 6, 303–314.
Calvert, G., Spence, C., and Stein, B.E. (eds.) (2004). The handbook of multisensory processes. MIT Press,
Cambridge, MA.
Cernacek, K.J., and Podivinski, F. (1971). Ontogenesis of handedness and somatosensory cortical response.
Neuropsychologia, 9, 219–32.
Cohen, L., Celnik, P., Pascual-Leone, A., et al. (1997). Functional relevance of crossmodal plasticity in blind
humans. Nature, 389, 180–83.
108 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

Cohen, L.B., and Younger, B.A. (1984). Infant perception of angular relations. Infant BehaviorBehaviour
and Development, 7, 37–47.
Connolly, K., and Jones, B. (1970). A developmental study of afferent-reafferent integration. British Journal
of Psychology, 61, 259–66.
Coryell, J.F., and Michel, G.F. (1978). How supine postural preferences of infants can contribute toward
the development of handedness. Infant Behavior and Development, 1, 245–57.
Coulon, M., Guellai, B., and Streri, A. (2011). Recognition of unfamiliar talking faces at birth. International
Journal of Behavioral Development, 35, 282–87.
Crassini, B., and Broerse, J. (1980). Auditory-visual integration in neonates: a signal detection analysis.
Journal of Experimental Child Psychology, 29, 144–55.
Diderot, D. (1749/1972). La lettre sur les aveugles à l’usage de ceux qui voient (The letter on blind people to
sighted people). Garnier-Flammarion, Paris.
Doucet, S., Soussignan, R., Sagot, P., and Schaal, B. (2007). The ‘smellscape’ of mother’s breast: effects of
odor, masking and selective unmasking on neonatal arousal, oral and visual responses. Developmental
Psychobiology, 49, 129–38.
Dubois, J., Hertz-Pannier, L., Dehaene-Lambertz, G., Cointepas, Y., and Le Bihan, D. (2006). Assessment of
the early organization and maturation of infants’ cerebral white matter fiber bundles: a feasibility study
using quantitative diffusion tensor imaging and tractography. NeuroImage, 30, 1121–32.
Elman, J.L., Bates, E., Johnson, M.H., Karmiloff-Smith, A., Parisi, D., and Plunkett, K. (1996). Rethinking
innateness: a connectionist perspective on development. MIT Press, Cambridge, MA.
Ernst, M.O., and Banks, M.S. (2002). Humans integrate visual and haptic information in a statistically
optimal fashion. Nature, 415, 429–33.
Fabri, M., Polonara, G., Del Pesce, M., Quattrini, A., Salvolini, U., and Manzoni, T. (2001). Posterior corpus
callosum and interhemispheric transfer of somatosensory information: an fRMI and neuropsychological
study of partially callosotomized patient. Journal of Cognitive Neuroscience, 13, 1071–79.
Fabri, M., Del Pesce, M., Paggi, A., et al. (2005). Contribution of posterior corpus callosum to the
interhemispheric transfer of tactile information. Cognitive Brain Research, 24, 73–80.
Féron, J., Gentaz, E., and Streri, A. (2006). Evidence of amodal representation of small numbers across
visuo-tactile modalities in 5-month-old infants. Cognitive Development, 21, 81–92.
Field, T.M., Woodson, R., Greenberg, R., and Cohen, D. (1982). Discrimination and imitation of facial
expressions by neonates. Science, 218, 179–81.
Fontaine, R. (1984). Imitative skills between birth and six months. Infant Behavior and Development, 7, 323–33.
Gallagher, S. (2006). Neurons and neonates: reflections on the Molyneux problem. In How the body shapes
the mind (ed. S. Gallagher), pp. 45–58. Oxford University Press, Oxford.
Garner, J.M., and Karmel, B.Z. (1983). Attention and arousal in preterm and full-term neonates, in Infant
born at risk: physiological, perceptual, and cognitive processes (eds. T. Field, and A. Sostek), pp. 69–98.
Grune and Stratton, New York.
Garner, J.M., Lewkowicz, D.J., Rose, S.A., and Karmel, B.Z. (1986). Effects of visual and auditory stimulation
on subsequent visual preferences in neonates. International Journal of Behavioral Development, 9, 251–63.
Gesell, A. (1938). The tonic neck reflex in the human infant. Journal of Pediatrics, 13, 455–64.
Gibson, E.J. (1969). Principles of perceptual learning and development. Academic Press, New York.
Gibson, E.J., and Walker, A. (1984). Development of knowledge of visual-tactual affordances of substance.
Child Development, 55, 453–60.
Gogate, L.J., and Bahrick, L.E. (1988). Intersensory redundancy facilitates learning of arbitrary relation
between vowel sounds and objects in seven-month-old-infants. Journal of Experimental Child
Psychology, 69, 133–49.
Gori, M., Del Viva, M., Sandini, G., and Burr, D.C. (2008). Young children do not integrate visual and
haptic form information. Current Biology, 18, 694–98.
Gottfried, A.W., Rose, S.A., and Bridger, W.H. (1978). Effects of visual, haptic and manipulatory
experiences on infants’ visual recognition memory of objects. Developmental Psychology, 14, 305–312.
REFERENCES 109

Granier-Deferre, C., Schaal, B., and DeCasper, A.J. (2004). Les prémices foetales de la cognition (the fœtal
beginnings of cognition). In Le développement du nourrisson (ed. R. Lecuyer), pp. 101–38. Dunod, Paris.
Guellai, B., Coulon, M., and Streri, A. (2011). The role of motion and speech in face recognition at birth.
Visual Cognition, 19:9, 1212–33.
Guellai, B., Streri, A. (2011). Cues for early social skills: direct gaze modulates newborns’recognition of
talking faces. Plos One, doi:10.1371/journal.pone.0018610.
Guest, S., and Spence, C. (2003). Tactile dominance in speeded discrimination of textures. Experimental
Brain Research, 150, 201–207.
Hatwell, Y. (1994). Transferts intermodaux et intégration intermodale (crossmodal transfers and
intersensory integration). In Traité de psychologie expérimentale, Vol. 1 (eds. M. Richelle, J. Requin, and
M. Robert) pp. 543–84. PUF, Paris.
Hatwell, Y. (2003). Intermodal coordinations in children and adults. In Touching for knowing: cognitive
psychology of tactile manual perception (eds. Y. Hatwell, A. Streri, and E. Gentaz), pp. 191–206. Johns
Benjamins Publishing Company, Amsterdam.
Hatwell, Y., Streri, A., and Gentaz, E. (2003) Touching for knowing: cognitive psychology of tactile manual
perception. Johns Benjamins Publishing Company, Amsterdam.
Held, R. (2009). Visual-haptic mapping and the origin of crossmodal identity. Optometry and Vision
Science, 86, 595–98.
Izard, V., Sann, C., Spelke, E.S., and Streri, A. (2009). Newborn infants perceive abstract numbers.
Proceedings of the National Academy of Sciences USA, 106, 10382–85.
Jeannerod, M. (1975). Deficit visuel persistant chez les aveugles-nés opérés. Données cliniques et
expérimentales (Persistent visual deficit in congenital blind people after surgery: clinical and
experimental data). L’Année Psychologique, 75, 169–95.
Jones, B., and Connolly, K. (1970). Memory effects in crossmodal matching. British Journal of Psychology,
61, 267–70.
Jovanovic, B., Duemmler, T., and Schwarzer, G. (2008). Infant development of configural object processing
in visual and visual-haptic contexts. Acta Psychologica, 129, 376–86.
Juurmaa, J., and Lehtinen-Railo, S. (1988). Visual experience and access to spatial knowledge. Journal of
Visual Impairment and Blindness, 88, 157–70.
Katz, D. (1925/1989). The world of touch (translated by L.E. Krueger 1989). Lawrence Erlbaum Associates,
Hillsdale, NJ.
Kawashima, R., Watanabe, J., Kato, T., et al. (2002). Direction of crossmodal information transfer affects
human brain activation: a PET study. European Journal of Neuroscience, 16, 137–44.
Kellman, P.J., and Arterberry, M.E. (1998). The cradle of knowledge. MIT Press, Cambridge, MA.
Klatzky, R.L., and Lederman, S.J. (1993). Toward a computational model of constraint-driven exploration
and haptic object identification. Perception, 22, 597–621.
Kellman, P., and Spelke, E.S. (1983). Perception of partly occluded objects in infancy. Cognitive Psychology,
15, 483–524.
Klatzky, R.L., Lederman, S.J., and Reed, C. (1987). There’s more to touch than meets the eye: the salience of
object attributes for haptic with and without vision. Journal of Experimental Psychology: General, 116,
356–69.
Kobayashi, T., Hiraki, K., and Hasegawa, T. (2005) Auditory-visual intermodal matching of small
numerosities in 6-month-old infants. Developmental Science, 8, 409–419.
Lacey, S., Campbell, C., and Sathian, K. (2007). Vision and touch: multiple or multisensory representations
of objects? Perception, 36, 1513–21.
Lecanuet, J.-P., Granier-Deferre, C., and Busnel, M.-C. (1995). Human fetal auditory perception. In Fetal
development: a psychobiological perspective (eds. J.-P. Lecanuet, W.P. Fifer, N.A. Krasnegor, and W.P.
Smotherman), pp. 239–62. Lawrence Arlbaum Associates, Hillsdale, NJ.
Ledermann, S.J., and Klatzky, R.L. (1987). Hand movements: a window into haptic object recognition.
Cognitive Psychology, 19, 342–68.
110 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

Lederman, S.J., and Klatzky, R.L. (1990). Haptic classification of common objects: knowledge-driven
exploration. Cognitive Psychology, 22, 421–59.
Lederman, S.J., and Klatzky, R.L. (1996). Haptic object identification II. Purposive exploration. In Somesthesis
and the neurobiology of the somatosensory cortex (ed. O. Franzen), pp. 153–61. Birkhauser Verlag, Basel.
Lejeune, F., Audeoud, F., Marcus, L., et al. (2010). The manual perception of shapes in preterm human
infants from 33 to 34 + 6 post-conceptional age. PLoS ONE, 5(2): e9108.
Lewkowicz, D.J. (2000). The development of intersensory temporal perception: an epigenetic systems/
limitations view. Psychological Bulletin, 126, 281–308.
Lewkowicz, D.J., and Lickliter, R. (1994). The development of intersensory perception. Comparative
perspectives. Lawrence Erlbaum Associates, Hillsdale, NJ.
Lewkowicz, D.J., and Turkewitz, G. (1981). Intersensory interaction in newborns: modification of visual
preferences following exposure to sound. Child Development, 52, 827–32.
Loomis, J.M., and Lederman, S.J. (1986). Tactual perception. In Handbook of perception and human performance.
Vol. II: Cognitive processes and performance (ed. K.R. Boff), pp. 1–41. John Wiley and Sons, New-York.
Maratos, O. (1973). The origin and development of imitation in the first six months of life. PhD thesis.
University of Geneva, Switzerland.
Marks, L.E. (1975). On coloured-hearing synaesthesia: crossmodal translations of sensory dimensions.
Psychological Bulletin, 82, 303–31.
Maurer, D. (1993). Neonatal synaesthesia: implications for the processing of speech and faces. In
Developmental neurocognition: speech and face processing in the first year of life (eds. B. de Boysson-
Bardies, and S. de Schönen), pp. 109–24. Kluwer Academic Publishers, Netherlands.
Maurer, D., Stager, C., and Mondloch, C. (1999). Crossmodal transfer of shape is difficult to demonstrate
in one-month-olds. Child Development, 70, 1047–57.
McClelland, J., Rumelhart, J., and the PDP research group. (1986). Parallel distributed processing:
explanations in the microstructure of cognition, Vol. 2. MIT Press, Cambridge, MA.
McGurk, H., Turnure, C., and Creighton, S.J. (1977). Auditory-visual coordination in neonates. Child
Development, 48, 138–43.
Meltzoff, A.N. (1993). Molyneux’ s babies: crossmodal perception, imitation and the mind of the preverbal
infant. In Spatial representation: problems in philosophy and psychology (eds. N. Eilan, R. McCarthy, and
B. Brewer), pp. 219–35. Oxford University Press, Oxford.
Meltzoff, A.N., and Borton, R.W. (1979). Intermodal matching by human neonates. Nature, 282, 403–404.
Meltzoff, A.N., and Moore, M.K. (1977). Imitation of facial and manual gestures by human neonates,
Science, 198, 75–78.
Merabet, L., Thut, G., Murray, B., Andrews, J., Hsiao, S., and Pascual-Leone, A. (2004). Feeling by sight or
seeing by touch? Neuron, 42, 173–79.
Michel, G.F., and Goodwin, R. (1979). Intrauterine birth position predicts newborn supine head
preferences. Infant and Development, 2, 29–38.
Molina, M., and Jouen, F. (1998). Modulation of the palmar grasp behaviour in neonates according to
texture property. Infant and Development, 21, 659–66.
Molina, M., and Jouen, F. (2001). Modulation of manual activity by vision in human newborns.
Developmental Psychobiology, 38, 123–32.
Mondloch, C.J., and Maurer, D. (2004). Do small white balls squeak? Pitch-object correspondences in
young children. Cognitive, Affective and al Neuroscience, 4, 133–36.
Morrongiello, B.A., Fenwick, K.D., and Chance, G. (1998). Crossmodal learning in newborn infants:
inferences about properties of auditory–visual events. Infant and Development, 21, 543–54.
Newham, C., and McKenzie, B.E. (1993). Crossmodal transfer of sequential and haptic information by
clumsy children. Perception, 22, 1061–73.
Pascalis, O., and de Haan, M. (2003). Recognition memory and novelty preference: what model? In Progress
in infancy research, Vol. 3 (eds. H. Hayne, and J. Fagen), pp. 95–119. Lawrence Erlbaum Associates,
Mahwah, NJ.
REFERENCES 111

Pascalis, O., and Slater, A. (2003). The development of face processing in infancy and early childhood: current
perspectives. Nova Science Publishers, Inc., New York.
Pascual-Leone, A., and Hamilton, R. (2001). The metamodal organization of the brain. In Progress in Brain
Research, Vol. 134 (eds. C. Casanova, and M. Petito), pp. 427–45. Elsevier Science B.V., Amsterdam.
Pihko, E., Nevalainen, P., Stephen, J., Okada, Y., and Lauronen, L. (2009). Maturation of somatosensory
cortical processing from birth to adulthood revealed by magnetoencephalography. Clinical
Neurophysiology, 120, 1552–61.
Proust, J. (1997). Perception et Intermodalité (Perception and intersensory relations). Presses Universitaires
de France, Paris.
Ramachandran, V.S., and Hubbard, E.M. (2001). Synaesthesia—a window into perception, thought and
language. Journal of Consciousness Studies, 8, 3–34.
Revesz, G. (1950). Psychology and art of the blind. Longmans Green, London.
Ricci, D., Cesarini, L., Groppo, M., et al. (2008). Early assessment of visual function in full term newborns.
Early Human Development, 84, 107–113.
Rich, A.N., Bradshaw, J.L., and Mattingley, J.B. (2004). A systematic, large-scale study of synaesthesia:
implications for the role of early experience in lexical-colour associations. Cognition, 98, 53–84.
Rochat, P. (1987). Mouthing and grasping in neonates: evidence for the early detection of what hard or soft
substances afford for action. Infant Behavior and Development, 10, 435–49.
Rochat, P. (1989). Object manipulation and exploration in 2- to 5-month-old infants. Developmental
Psychology, 25, 871–84.
Röder, B., and Rösler, F. (2004). Compensatory plasticity as a consequence of sensory loss. In Handbook of
multisensory processing (eds. G. Calvert, C. Spence, and B.E. Stein), pp. 719–47. MIT Press, Cambridge, MA.
Roland, P.E., and Mortensen, E. (1987). Somatosensory detection of microgeometry, macrogeometry and
kinaesthesia in man. Brain Research Review, 12, 1–42.
Roland, P.E., O’Sullivan, B., and Kawashima, R. (1998). Shape and roughness activate different somatosensory
areas in the human brain. Proceedings of the National Academy of Sciences USA, 95, 3295–300.
Rose, S.A. (1994). From hand to eye: findings and issues in infant crossmodal transfer. In The development
of intersensory perception: comparative perspectives (eds. D.J. Lewkowicz, and R. Lickliter), pp. 265–84.
Lawrence Erlbaum Associates, Hillsdale, NJ.
Rose, S.A., and Orlian, E.K. (1991). Asymmetries in infant crossmodal transfer. Child Development,
62, 706–18.
Rovee-Collier, C., and Barr, R. (2001). Infant learning and memory. In Handbook of infant development
(eds. G. Bremner and A. Fogel), pp. 139–68. Blackwell Publishing, Malden, MA.
Ruff, H.A. (1984). Infants’ manipulative exploration of objects: effects of age and object characteristics.
Developmental Psychology, 20, 9–20.
Ruff, H.A., Saltarelli, L.M., Cappozoli, M., and Dubiner, K. (1992). The differentiation of activity in infants’
exploration of objects. Developmental Psychology, 28, 851–61.
Sann, C., and Streri, A. (2007). Perception of object shape and texture in human newborns: evidence from
crossmodal transfer tasks. Developmental Science, 10, 398–409.
Sann, C., and Streri, A. (2008). The limits of newborn’s grasping to detect texture in a crossmodal transfer
task. Infant Behavior and Development, 31, 523–31.
Sathian, K. (2005). Visual cortical activity during tactile perception in the sighted and the visually deprived.
Development Psychobiology, 46, 279–86.
Schaal, B., Marlier, L., and Soussignan, R. (1998). Olfactory function in the human fetus: evidence from
selective neonatal responsiveness to the odor of amniotic fluid. Behavioral Neuroscience, 112, 1438–49.
Shams, L., and Seitz, A.R. (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12, 411–417.
Slater, A. (1998). Perceptual development: visual, auditory and speech perception in infancy. Psychology Press,
Hove.
Slater, A., and Kirby, R. (1998). Innate and learned perceptual abilities in the newborn infant. Experimental
Brain Research, 123, 90–94.
112 CROSSMODAL INTERACTIONS IN THE HUMAN NEWBORN

Slater, A., Brown, E., and Badenoch, M. (1997). Intermodal perception at birth: newborn infants’ memory
for arbitrary auditory–visual pairings. Early Development and Parenting, 6, 99–104.
Slater, A., Mattoch, A., Brown, E., and Bremner, J.G. (1991). Form perception at birth: Cohen and Younger
(1984) revisited. Journal of Experimental Child Psychology, 51, 395–406.
Slater, A., Quinn, P.C., Brown, E., and Hayes R. (1999). Intermodal perception at birth: intersensory redundancy
guides newborn infants’ learning of arbitrary auditory–visual pairings. Developmental Science, 2, 333–38.
Spector, F., and Maurer, D. (2009). Synaesthesia: a new approach to understanding the development of
perception. Developmental Psychology, 45, 175–89.
Spelke, E.S. (1976). Infants’ intermodal perception of events. Cognitive Psychology, 8, 553–60.
Spence, C. (2011). Crossmodal correspondences: a tutorial review. Attention, Perception, and Psychophysics,
73, 971–95.
Spence, C., and Driver, J. (2004). Crossmodal space and crossmodal attention. Oxford University Press, Oxford.
Starkey, P., Spelke, E.S., and Gelman, R. (1990). Numerical abstraction by human infants. Cognition, 36, 97–127.
Streri, A. (1987). Tactile discrimination of shape and intermodal transfer in 2- to 3- month-old infants.
British Journal of Developmental Psychology, 5, 213–20.
Streri, A. (1993). Seeing, reaching, touching: the relations between vision and touch in infancy. Halverster
Wheatsheaf, London.
Streri, A., and Gentaz, E. (2003). Crossmodal recognition of shape from hand to eyes in human newborns.
Somatosensory and Motor Research, 20, 11–16.
Streri, A., and Gentaz, E. (2004). Crossmodal recognition of shape from hand to eyes and handedness in
human newborns. Neuropsychologia, 42, 1365–69.
Streri, A., and Molina, M. (1993). Visual-tactual and tactual-visual transfer between objects and pictures in
2-month-old infants. Perception, 22, 1299–318.
Streri, A., and Pêcheux, M.G. (1986a). Tactual habituation and discrimination of form in infancy: a
comparison with vision. Child Development, 57, 100–104.
Streri, A., and Pêcheux, M.G. (1986b). Vision to touch and touch to vision transfer of form in 5-month-old
infants. British Journal of Developmental Psychology, 4, 161–67.
Streri, A., and Spelke, E.S. (1988). Haptic perception of objects. Cognitive Psychology, 20, 1–23.
Streri, A., and Spelke, E.S. (1989). Effects of motion and figural goodness on haptic object perception in
infancy. Child Development, 60, 1111–25.
Streri, A., Lhote, M., and Dutilleul, S. (2000). Haptic perception in newborns. Developmental Science, 3, 319–27.
Streri, A., Lemoine, C., and Devouche, E. (2008). Development of inter-manual transfer information in
infancy. Developmental Psychobiology. 50, 70–76.
Tomasello, M. (1998). Uniquely primate, uniquely human. Developmental Science, 1, 1–16.
Turkewitz, G., Gardner, J., and Lewkowicz, D.J. (1984). Sensory/perceptual functioning during early
infancy: the implications of a quantitative basis for responding. In Behavioral evolution and integrative
levels (eds. G. Greenberg and E. Tobach), pp. 167–95. Lawrence Erlbaum Associates, Hillsdale, NJ.
Turkewitz, G., Gordon, E.W., and Birch, H.G. (1965). Head turning in the human neonate: spontaneous
pattern. Journal of Genetic Psychology, 107, 143–58.
Twitchell, T.E. (1965). The automatic grasping responses of infants. Neuropsychologia, 3, 247–59.
Vinter, A. (1986). The role of movement in eliciting early imitations. Child Development. 57, 66–71.
von Hofsten, C. (1982). Eye-hand coordination in the newborn. Developmental Psychology, 18, 450–61.
Walker, P., Bremner, J.G., Mason, U., et al. (2010). Preverbal infants’sensitivity to synaesthetic
crossmodality correspondences. Psychological Science, 21, 21–25.
Wertheimer, M. (1961). Psychomotor coordination of auditory and visual space at birth. Science, 134, 1692.
Chapter 5

The development of multisensory


representations of the body and
of the space around the body
Andrew J. Bremner, Nicholas P. Holmes,
and Charles Spence

5.1 Introduction
Developing mature multisensory representations of the spatial disposition of our body and limbs
is vital if we are to move around the environment in a physically competent manner. As adults,
we efficiently integrate information arriving from multiple sensory modalities pertaining to
the spatial layout of the body and limbs. Indeed, multisensory body representations underpin
a fundamental aspect of all our mental processes, by providing ‘embodied’ representations
that give a point of reference to objects in the external world (Bermudez et al. 1995; Gallagher
2005; Varela et al. 1991). Such embodied representations provide the basis for action in both the
nearby (peripersonal) and distant (extrapersonal) environments (see Previc 1998; Rizzolatti et al.
1997). Despite the fundamental role of such representations, there is clear scope for significant
development in them; any early ability to represent the layout of one’s body would need to be
continually re-tuned throughout development in order to cope with physical changes in the sizes,
disposition, and movement capabilities of the limbs (see A.J. Bremner et al. 2008a; Gori et al.
2010; King 2004).
A casual observation of young infants’ behaviour with respect to their own bodies indicates
significant limitations in their perception and understanding of their limbs. As suggested by the
following anecdotal observation of a 4-month-old infant, a naïve appreciation of the relation
between tactile and visual spatial frames of reference could lead to a number of problems in
controlling and executing appropriate actions:
Sometimes the hand would be stared at steadily, perhaps with growing intensity, until interest reached
such a pitch that a grasping movement followed as if the infant tried by an automatic action of the
motor hand to grasp the visual hand, and it was switched out of the centre of vision and lost as if it had
magically vanished. (Hall 1898; p. 351).

Even with children well into their second year of life, DeLoache et al. (2004) have observed
striking ‘scale errors’ indicating that, despite being able to arrange their limbs competently
with respect to the actions they select (see also Glover 2004), children misunderstand or fail to take
into account the scale relationship between their body and an object when selecting appropriate
actions.
In this chapter, we discuss the research that has investigated the development, through infancy
and early childhood, of multisensory representations of the body and of the space around
114 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

the body. We will limit our focus to the emergence of representations that subserve action in rela-
tion to the nearby (peripersonal) environment (for a discussion of representations subserving
action with respect to the more distant environment—in particular, balance and locomotion—
see Chapter 6 by Nardini and Cowie).

5.2 The multisensory nature of body representations


The sensory information that provides the most obvious contribution to body representations is
that arising from the senses that are only stimulated by the body or by objects touching it. Two
inputs that have particular importance in this respect are those of touch (i.e. cutaneous somato-
sensory inputs) and proprioception (i.e. deep or non-cutaneous somatosensory inputs). However,
this is not to say that it is this sensory information alone which provides embodied experience. On
the contrary, behavioural and physiological evidence from adult humans and monkeys points to
the conclusion that the neural systems subserving representations of the body and the embodied
environment (including peripersonal space) integrate information from the receptors relating to
the body and its surface (touch and proprioception) with that arriving from the receptors that
provide information about the near and distant environment (i.e. vision and audition). For
instance, as early as Brodmann’s area 5 in the higher-order somatosensory cortex, tactile and
proprioceptive information about the arm and hand is integrated with visual inputs, generating
multisensory representations of body-part location (Graziano et al. 2000). Similarly, parts of the
ventral and dorsal premotor cortex in adult monkeys and humans integrate tactile, propriocep-
tive, visual, and even auditory inputs in the representation of limbs, of limb position and of
nearby objects in body-part-centred representations of space (Ehrsson et al. 2004; Graziano et al.
1997; Holmes and Spence 2004; Makin et al. 2008; Maravita et al. 2003). Interestingly, research
also suggests that auditory-tactile integration may be of particular relevance to representations of
the space behind the head (e.g. Kitagawa et al. 2005).

5.3 The development of visual–proprioceptive and visual–tactile


correspondence: early body representations?
The question of whether infants and young children are able to perceive spatial correspondences
between sensory inputs relating to the distant environment (in particular, visual inputs) and
direct inputs concerning the body is one that has fascinated philosophers and psychologists since
the time of Molyneaux and Locke (Locke 1690; see also Chapter 4 by Streri). The first empirical
approaches attempted to discern whether children and later on infants could recognise an object
in one sensory modality, which had previously been presented only in another (‘crossmodal
transfer tasks’). Birch and Lefford (1963), the pioneers of this technique in children, observed that
the accuracy of children’s crossmodal matching of stimuli between vision and touch increases
from the ages of 5 to 11 years of age. However, their conclusion that the crossmodal integration
of these senses undergoes extended development across childhood has since received substantial
criticism. In particular, Bryant and his colleagues (Bryant 1974; Bryant et al. 1972; Bryant and Raz
1975; Hulme et al. 1983) refuted Birch and Lefford’s (1963) finding of age-related developments
in crossmodal matching, by noting that it could be explained by corresponding developments in
unimodal perceptual matching.
Bryant et al. (1972) argued, contrary to Birch and Lefford’s (1963) account, that the crossmodal
integration of touch and vision is in fact present in early infancy, and backed this up with evidence
that 6–12-month-old infants are able to identify shapes visually, which they had only previously
experienced through touch. Rose and her colleagues confirmed in a series of studies that
THE DEVELOPMENT OF VISUAL–PROPRIOCEPTIVE AND VISUAL–TACTILE CORRESPONDENCE 115

bi-directional crossmodal transfer between touch and vision is available by 6 months of age (Rose
et al. 1981a), but they also demonstrated that between 6 and 12 months infants become progres-
sively more efficient at encoding information into memory, allowing them to achieve crossmodal
recognition in a more robust way (Gottfried et al. 1977, 1978; Rose et al. 1981b; for a review see
Rose 1994). More recently, Streri and colleagues have showed that an ability to perceive commo-
nalities between touch and vision is present at birth (although see also Maurer et al. 1999; Meltzoff
and Borton 1979 for controversy surrounding the early emergence of crossmodal transfer),
by demonstrating that newborn infants are capable of some visual–tactile crossmodal transfer
of shape recognition (see Chapter 4 by Streri; also Sann and Streri 2007; Streri and Gentaz 2003,
2004).
The early development of spatial correspondence between vision and proprioception has
also been investigated in infants using a crossmodal matching method, which measures infants’
preference for looking toward visual information that is either congruent or incongruent with
proprioception. Following the development of this method by Bahrick and Watson (1985), a
series of studies conducted by several different research groups have examined whether infants
from as young as 3 months of age were able to recognize crossmodal visual–proprioceptive (VP)
correspondences arising from the movement of their own limbs (e.g. Bahrick and Watson 1985;
Rochat and Morgan 1995; Schmuckler and Fairhall 2001; Schmuckler and Jewell 2007). In these
studies, infants were presented with a visual display showing their own legs moving in synchrony
with their own movements, and compared this with the visual presentation of another child’s leg
movements. The preferences infants showed for the asynchronous display across a number of
different conditions of stimulus presentation provided strong evidence that young infants are
indeed able to detect VP spatiotemporal correspondences.
Thus, the evidence seems fairly conclusive that, soon after birth, humans are able to register
correspondences across the direct and distance receptors (at least for visual–tactile (VT) and VP
correspondences). However, in considering the development of multisensory representations of
the body and limbs, it is important to examine whether these VT and VP spatial correspondences
are perceived with respect to parts of the body (e.g. head-, eye-, body-, limb-centred spatial coor-
dinates). The crossmodal matching and crossmodal transfer studies we have discussed in this
section suffer from a limitation in this respect because although they permit investigation of
infants’ ability to perceive a multisensory spatial correspondence (either between touch and
vision or between proprioception and vision), they do not provide an indication of the spatial
frame of reference in which this correspondence is encoded. Given young infants’ skill with rep-
resenting external (allocentric) spatial frames of reference (A.J. Bremner et al. 2007; Kaufman and
Needham 1999), it is quite plausible that crossmodal transfer is achieved with respect to a frame
of reference unrelated to a representation of the body or indeed any intrinsic spatial framework
such as the eye or the hand (A.J. Bremner et al. 2008a).
In their studies of VP matching (using the crossmodal matching paradigm described above),
Rochat and Morgan (1995), and Schmuckler (1996) attempted to determine the underlying
frame of reference governing the infants’ behaviour. They did this via some manipulations of the
left–right and up–down spatial matches between the visual and proprioceptive stimuli. The
authors of these two papers found, independently, that infants only appear to detect VP corre-
spondence when the visual inputs move in temporal synchrony with the limb, and when they
correspond spatially in the left–right dimension of movement (Rochat and Morgan 1995;
Schmuckler 1996). The infants did not demonstrate any differential looking with respect to
changes in crossmodal correspondence in the up–down (vertical) dimension of movement.
Arguing that the left–right dimension of movement is specific to an egocentric frame of reference
(cf. Driver 1999), Rochat (1998) concluded that the infants’ crossmodal matching is based on a
116 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

representation of movements of their own body coded relative to an intrinsic frame of reference.
Rochat has further asserted that these multisensory egocentric representations are made possible
by an innate human capacity to represent one’s own body (Rochat 1998, 2010).
However, there are a number of problems with concluding from these data that young infants
perceive visual and proprioceptive cues with respect to a body-, or limb-centred spatial frame of
reference. In these VP matching studies, the visual cues concerning the body were presented on a
video display outside bodily and peripersonal space (i.e. beyond reach, in extrapersonal space),
and so it is difficult to rule out the possibility that infants were responding on the basis of some
other correspondence between these sensory inputs, such as their correlated movements
(cf. Parise, Spence, and Ernst, 2012). Indeed, it is important to highlight that infants’ detection of
VP correspondence in this paradigm need not necessarily be registered with respect to limb- or
body-centred spatial frames of reference. For instance, VP contiguity occurs when the move-
ments of an infant’s arm cause objects in his or her environment to move (such as a blanket or a
teddy bear). This kind of experience may lead young infants to expect VP spatial correspondence,
without any recourse to a multisensory spatial representation of their own body. Thus, it is pos-
sible that the findings from these studies may tell us more about infants’ perceptions of contin-
gencies between their own movements and the movements of objects or people in their
extrapersonal environment than about representations of their own bodies.1 Recall that the
4-month-old in Hall’s (1898) anecdote mentioned earlier treated their visual hand as if it were an
extrapersonal object.
Indeed, the limitations in the matches that the infants made suggest that this extrapersonal
interpretation of VP correspondence might be the most appropriate. Recall that while infants
detected left–right VP congruency, they failed to notice such crossmodal correspondence in the
up–down (vertical) dimension (Rochat 1998; Schmuckler 1996; see also Snijders et al. 2007).
There seems to be little reason to expect such a problem with up–down VP correspondence with
reference to body- or limb-centred frames of reference because infants will have had crossmodal
experience of their arms moving with respect to both horizontal and vertical axes. Conversely,
experience with VP correspondence arising from the movements of extrapersonal objects is likely
to be much more restricted to the left–right dimension, for the simple reason that the arrange-
ments of objects and surfaces with respect to gravity in our environment allows for less variation
within the vertical plane (at least up until infants start picking up and dropping objects towards
the end of the first year). Thus, young infants’ abilities to detect left–right, but not up–down spa-
tial correspondence between their proprioceptively perceived leg movements and their visually
perceived movements on the video screen may well be based on representations of, and arise from
the experience of, crossmodal correspondences between the proprioceptive body and visual
extrapersonal objects.
Finally, even if young infants do register VP commonalities according to a representation of
their own bodies (as argued by Rochat 1998), we have little information regarding the spatial
precision of such representations. It is unclear to what extent the crossmodal perceptual abilities
of young infants could provide the basis for useful sensorimotor coordination in the immediate
personal and peripersonal environment. It is our assertion (see later, and also A.J. Bremner et al.
2008a) that, in order to glean unambiguous information concerning the development of spatial
representations of the body, spatial-orienting paradigms are needed. Spatial-orienting responses
have the advantage of requiring coordination between the stimulus location and the intrinsic

1 Indeed, this prroblem is also present in a more recent study on infants’ detection of visual-tactile syn-
chrony of stimulation to the legs (Zmyj et al. 2011).
TWO MECHANISMS OF MULTISENSORY INTEGRATION 117

spatial framework within which the response is made. We will shortly move on to a description of
the findings of a study of infants’ orienting responses to tactile stimuli presented to their hands.
However, first we will consider the computational demands of orienting toward the body or
stimuli in the nearby (peripersonal) environment. As will become clear, the task of orienting to a
stimulus with respect to the body is by no means a simple one.

5.4 Computational challenges of forming multisensory


representations of the body
The existence of many separate multisensory representations of space and the body, across indi-
viduals, brain areas, and species, belies the computational complexity of constructing such repre-
sentations. Not only do the senses convey information about the environment in different neural
codes and reference frames, but the relationship between sensory modalities frequently changes,
for example with changes in body posture, such as when the eyes move in their sockets or the
arms move. Take, for instance, the task of retrieving a nearby seen object: first, the brain must
represent the object’s location visually and then it must translate this retinocentric location into
the limb-centred coordinates necessary for initiating a reaching movement. In other words, the
brain calculates the location of the retrieving limb relative to the object. The necessary multisen-
sory information specifying the layout of our bodies with respect to the world is typically pro-
vided by touch, proprioception, vision, and occasionally audition. The neural representations of
limb position and nearby stimuli dynamically remap the ways in which information is integrated
across the senses in response to different circumstances (e.g. changes in posture, or the use of
tools; Groh and Sparks 1996a,b; Graziano et al. 2004; Spence et al. 2008). This dynamic integra-
tion of multisensory spatial cues occurs automatically in human adults (e.g. Kennett et al. 2002),
but we cannot necessarily infer that the same is the case for earlier stages in development.
However, the challenge to multisensory integration posed by changes in the relationships
between the limbs and body is even greater when considered across development. Firstly, the
number and variety of postural changes that a child can readily make increase substantially, par-
ticularly in the first years of life (Bayley 1969; Carlier et al. 2006; Morange and Bloch 1996; Provine
and Westerman 1979; Van Hof et al. 2002). Indeed, such developmental changes in posture
are known to have a significant impact on infants’ and children’s abilities to act on and navigate
around their environment (see Chapter 6 by Nardini and Cowie; see also work by Adolph, e.g.
Adolph 2008). In addition to changes in the postural variations available to children, the spatial
distribution of the limbs and the body also vary profoundly right up to adulthood. Figure 5.1
shows how the relative sizes, shapes and distributions of the limbs, body, and head change sig-
nificantly across development. These changes in body and proportions necessitate continuous
adaptation of sensorimotor integration across early life (Adolph and Avolio 2000 and see Chapter
6 by Nardini and Cowie).

5.5 Two mechanisms of multisensory integration underlying


peripersonal spatial representations and their development
As has been seen, when we orient toward locations in peripersonal space, and indeed when we
orient toward locations on the body, our brains must integrate and align the spatial frames of
reference used by vision, audition, and by the body senses. In a recent review of the literature on
peripersonal spatial representations (A.J. Bremner et al. 2008a), we highlighted two mechanisms
of multisensory integration that adult humans and primates (typically) use in order to achieve
unified, consistent representations of the body and peripersonal space. In A.J. Bremner et al.
118 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

(a)

Birth 2 years 5 years 15 years Adult

(b)
70
Sitting height as a percentage of stature

65

Boys
60 Girls

55

50
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Age (years)
Fig. 5.1 The distributions of limbs and the body change continually across infancy and childhood.
(a) An approximation (after Gilbert 2000) of body proportions in a human girl at birth, 2 years, 5 years,
15 years, and as an adult. Note that the head’s size relative to the body and limbs decreases, and
the length of the limbs relative to body stature increase significantly across development. (b) Changes
in sitting height as a percentage of stature across 0–17 years of age. Decreases in this statistic are
directly proportional to increases in leg length as a percentage of stature (With kind permission
from Springer Science+Business Media: Manual of Physical Status and Performance in Childhood:
Volume One Parts A and B; Physical Status, 1983, Roche, Alex F., Malina, Robert M.).

(2008a), we labelled these as visual spatial reliance, and postural remapping. Due to informative
conversations with colleagues we present this framework in a modified form. We have relabelled
visual spatial reliance here as canonical multisensory body representations.

5.5.1 Canonical multisensory body representations


The available research shows that we rely heavily on statistical information about the sensory
stimulation we receive in order to make more precise estimates about and actions on objects and
TWO MECHANISMS OF MULTISENSORY INTEGRATION 119

our spatial environment (e.g. Ernst and Bülthoff 2004; Kersten et al. 2004; Körding and Wolpert
2004). For instance, a number of studies have demonstrated that our reliance on information
from a given sensory modality depends on the variability of information in that modality in the
context of a particular task (Alais and Burr 2004; Ernst and Banks 2002; Ernst and Bülthoff 2004;
Van Beers et al. 2002). Statistical priors can also play a role in telling us how our body is arrayed:
by relying on the fact that the limbs usually occupy particular locations with respect to the body
(i.e. that there is a high prior probability associated with seeing feeling or even hearing a limb in
a particular location), one can approximate limb position on the basis of such priors (e.g. when
one feels a tactile sensation on the left hand the object which has produced that will more often
than not be in the left visual field).
This approach predicts that the influence of canonical body representations should be observ-
able when participants are asked to undertake tasks in which their limbs are placed into positions
that are at odds with the canonical representation of the body derived from its typical layout. One
particularly striking example of this can be seen in those tasks in which adults make temporal
order judgments (TOJs) concerning tactile stimuli presented first on one hand and then on the
other, in quick succession; performance is much less accurate in an unusual crossed-hands pos-
ture than in the more typical (canonical) uncrossed-hands posture (Schicke and Röder 2006;
Shore et al. 2002; Yamamoto and Kitazawa 2001). Interestingly, these crossed-hands deficits do
not occur, or occur much less strongly, in congenitally blind participants, indicating that, in this
case at least, visual input has an important role in setting up such canonical representations of the
body (Röder et al. 2004; see also Chapter 13 by Röder, and Pagel et al. 2009).
But, from what information might such a canonical representation of the body be derived? As
we described above, Röder et al. (2004) demonstrated that visual experience appears to mediate the
development of canonical representations of the body’s layout. Indeed, vision often provides the
most reliable information about where the hands are in space (cf. Alais and Burr 2004; Van Beers
et al. 2002). It is possible to observe our reliance on vision when vision becomes misleading in the
context of striking bodily illusions, such as the ‘rubber hand’ and ‘mirror’ illusions (Botvinick and
Cohen 1998; Holmes et al. 2004). However, vision is not always the most reliable sensory input
about a given stimulus or in the context of a given task. For instance, Van Beers et al. (2002) have
demonstrated that proprioceptive and visual information about the hand’s location are weighted
relatively more strongly in the contexts in which they are most reliable: proprioception is more
reliable than vision when determining location depth away from the body, whereas vision is more
reliable than proprioception in the left–right dimension (azimuth). Adults thus place more weight
on the most reliable modality. Indeed, research suggests that we integrate the senses optimally
by weighting them in accordance with their task-dependent reliabilities (Ernst and Banks 2002).
We thus suggest that canonical representations of the body are multisensory.

5.5.2 Postural remapping


Of course, canonical multisensory body representations should lead to problems when the limbs
are not in their usual positions. By taking account of postural changes (either passively through
visual and proprioceptive cues, or actively through ‘efferent copies’ of the movement plans used
to change posture), the spatial correspondence between external stimuli and the limbs can be
‘remapped’. Research with adult humans has shown that the multisensory interactions that direct
attention take account of postural changes such as when a hand crosses the body midline (Azañón
and Soto-Faraco 2008; Spence et al. 2008). Additionally, brain areas identified as sites of multi-
sensory integration have been implicated in processes of postural remapping: neurons that remap
sensory correspondences across changes in posture have been reported in the monkey superior
colliculus (SC; Groh and Sparks, 1996a,b) and ventral premotor cortex (PMv; Graziano 1999;
Graziano et al. 1997).
120 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

Such remapping mechanisms may operate very early in the neural processing between stimulus
and response, at least in adults. Azañón and Soto-Faraco (2008) used an exogenous cuing experi-
ment to investigate the remapping of tactile stimulus location when the hands were crossed
over the body midline. They examined whether adults’ speeded responses to a visual stimulus on
the hand (a light) were influenced by the spatial proximity of a tactile cue that appeared prior to

(a) Saccades at 0 ms latency

(b) Saccades at 600–1000 ms latency

Fig. 5.2 Saccades made by an adult human to tactile stimuli on their right hand in a crossed-hands
posture (with the right hand in the left visual field). (a) Saccades are made to the tactile stimulus without
any intervening delay. (b) Saccades are made to the tactile stimulus with a delay of 600–1000 ms
between stimulus and response. Gradations indicate 10° of visual angle. Note that in (a), several saccades
begin by heading in the direction in which the tactile stimulus would normally lie and then a later
corrective process takes account of current hand posture, and shifts the saccade direction. (Reproduced
from Journal of Neurophysiology, 75 (1), J. M. Groh and D. L. Sparks, Saccades to somatosensory targets.
I. behavioral characteristics, pp. 412-427 © 1996 , The American Physiological Society with permission.)
CANONICAL MULTISENSORY BODY REPRESENTATIONS 121

the visual target. They found that exogenous crossmodal spatial cuing effects to visual
stimuli appearing within the first 100 ms following a tactile stimulus were strongest in an
anatomical frame of reference—that is, they were strongest when the visual target was in the
position where the tactile cue would typically occur if the hands were uncrossed. 100 ms
after stimulus onset, cuing effects were remapped into an external frame of reference,
taking account of the unusual crossed-hands posture. A similar time-locked process of postural
remapping can be observed in adults’ saccades to tactile stimuli. If saccades are made immediately
following a tactile stimulus applied to a hand crossed over the midline, a proportion of saccades
are initially directed toward the visual hemispace where the tactile stimulus would normally be.
However, if saccadic orienting responses to tactile stimuli are delayed by 600–1000 ms, then
they are directed correctly to the actual location of the stimulus in space, even when the tactile
stimulus is in the visual hemifield opposite that to where the hand would normally be (Groh and
Sparks 1996c; see Fig. 5.2). Thus, it would seem as though an integrative mechanism that is sensi-
tive to posture is required in order to make correct gaze-orienting responses to the hands
when they are placed in atypical locations. Interestingly, it appears that adults are only
conscious of such tactile sensations once they have been remapped (Azañón and Soto-Faraco
2008; Kitazawa 2002).

5.6 Canonical multisensory body representations and


postural remapping in infants’ orienting responses
to tactile stimuli on the hands
A.J. Bremner et al. (2008b) recently traced the development of these mechanisms of multisensory
integration in 6.5- and 10-month-old infants, by measuring their spontaneous manual orienting
responses to vibrotactile sensations presented to their hands when placed both in the uncrossed-,
and crossed-hands postures (see Fig. 5.3a). The 6.5-month-old infants demonstrated a bias to
respond to the side of the body where the hand would typically rest (i.e. to the left of space for a
vibration in the left hand), regardless of the posture (uncrossed or crossed) of the hands. This
indicates a reliance on the typical location of the tactile stimulus in visual space. Later, at
10 months of age, a greater proportion of manual responses were made appropriately in both
postures, suggesting the development of an ability to take account of posture in remapping
correspondences between visual and tactile stimulation.
These developmental findings converge with the results of neuroscientific and behavioural
research in suggesting that representations of peripersonal space arise from two distinct mecha-
nisms of sensory integration, which follow separate developmental trajectories. The first mecha-
nism, canonical multisensory body representations, integrates bodily and visual sensory information
but relies substantially on the probable location of the hand, derived primarily from prior multi-
sensory experience about the canonical layout of the body. This mechanism is present early in the
first 6 months of life. The second mechanism, postural remapping, updates these multisensory
spatial correspondences by dynamically incorporating information about the current posture of
the hand and body. This mechanism develops after 6.5 months of age. We are not suggesting that
the early mechanism of canonical body representations is wholly replaced by that of postural
remapping, but that they continue to work together, as has been observed in adults (see above,
and Fig. 5.3b).
Thus we have put forward a dual-mechanism framework for understanding the development
during the first years of life of representations of the body and limbs and the stimuli that impinge
upon them. However, as we stated at the beginning of this chapter, the functional purpose of
such body representations is to provide a basis for action. Whilst orienting responses can them-
selves be considered as exploratory actions with respect to the body and nearby environment,
122 (a) An infant receiving tactile stimulation to the right hand

Uncrossed-hands Crossed-hands

(b) Spatial information available for orienting

hand hand

Tactile location Locations of hand Typical location of R Posture of R hand


on body (R hand) targets in visual field hand in visual field (proprioceptive & visual)

(c) 6.5-month-olds’ responses (/5) (d) 10-month-olds’ responses (/10)


Number of responses

5 5
Contralateral
4 Ipsilateral 4
3 3
2 2
1 1
0 0
Uncrossed- Crossed- Uncrossed- Crossed-
hands hands hands hands
Fig. 5.3 Orienting to touch in familiar and unfamiliar postures. In the uncrossed-hands posture both
the visual information about the hand (circle) and a tactile stimulus on that hand (zig-zag pattern) arrive
at the contralateral hemisphere (tactile stimuli were always delivered when the infants were fixating
centrally). But with crossed hands, these signals initially arrive in opposite hemispheres (a). (b) shows
the sources of information available to be integrated into a representation of stimulus location. Our
framework suggests that all sources of information are available to 10-month-olds, and all but current
postural information is available to 6.5-month-olds. (c) 6.5-month-old infants’ manual responses to
tactile stimuli (d) 10-month-old infants’ manual responses to tactile stimuli. The infants’ first responses
on each trial were coded (from video-recordings) in terms of their direction in visual space with respect
to the hemisphere receiving the tactile signal. Thus, contralateral responses are appropriate in the
uncrossed-hands posture and ipsilateral responses in the crossed-hands posture. The 6.5-month-olds’
manual responses (n = 13; c) showed an overall contralateral bias, as predicted by a hypothesized
reliance on the typical layout of their body relative to vision. The 10-month-olds (n = 14; c) were able
to respond manually in the appropriate direction in either posture, suggesting, in agreement with the
proposed framework, that this age group are able to use information about current posture to remap
their orienting responses (Figure adapted from Bremner et al. 2008a). Asterisks represent significant
comparisons. Solid arrows represent a strong contribution of a particular source of information to
behaviour. Dotted arrows represent a weak contribution of that particular source of information.
Error bars show standard error of the mean. Reprinted from Trends in Cognitive Sciences, 12 (8),
Andrew J. Bremner, Nicholas P. Holmes, and Charles Spence, Infants lost in (peripersonal) space?,
pp. 298–305, Copyright (2008), with permission from Elsevier.
PERIPERSONAL SPATIAL REPRESENTATIONS UNDERLYING ACTION IN EARLY INFANCY 123

it is important to investigate whether our framework is also applicable to overt actions within
peripersonal space.

5.7 Peripersonal spatial representations underlying


action in early infancy
Of the measurable behaviours in early infancy, perhaps the most relevant ways to observe
the development of action in peripersonal space are reaches and grasps made toward nearby
objects. A key question raised by the framework outlined here concerns whether infants’ reaching
movements at any given stage of development are based on multisensory systems that take account
of current limb posture (postural remapping), or whether instead successful reaches are
based on canonical representations of the limbs in their familiar locations derived from multi-
sensory experience (canonical multisensory body representations; also see Körding and Wolpert
2004).
While newborn infants do not often manually contact objects, their reaches are more often
directed toward an object if they are looking at it (Von Hofsten 1982, 2004). Newborns have also
been shown to change the position of their hand, and initiate decelerations of movement in order
to bring it into sight under the illumination of a spotlight that alternated between two locations
near their body (Van der Meer 1997). Thus, at birth, there is at least some spatial integration
between the information coming from nearby visible objects, and that coming from the body
parts with which responses are made.
An important question is whether early reaching is guided by visual feedback concerning the
relative locations of hand and object. Newborns demonstrate a deceleration of arm movements
in anticipation of their hand’s appearance in a spotlight (Van der Meer 1997), and this is sugges-
tive of the coordination of visual, proprioceptive, and kinaesthetic information (purely visual
guidance cannot explain the anticipatory adjustments, since the hand was invisible when outside
of the spotlight). However, it is difficult to determine whether this indicates early crossmodal
spatial correspondence between proprioceptive and visual space, or rather operant conditioning
of particular arm movements, contingent upon the reward of seeing one’s own hand.
The coordination of proprioceptive and visual space in the guidance of reaching has been
investigated more fully by comparing infants’ early successful reaches for distal targets in the light
against those in the dark (i.e. toward sounding or glowing targets without visual cues to the loca-
tion of their hand; Clifton et al. 1993, 1994; Robin et al. 1996). These studies have shown that
successful reaching in the dark develops at the same age as in the light, indicating that the first
reaches (at around 3–4 months of age) can be based on proprioceptive guidance of hand position
toward a sighted visual target.
Given that infants’ first successful reaches toward visual targets can occur without any visual
input about limb position, these actions are clearly generated by peripersonal spatial representa-
tions that integrate visual and proprioceptive cues to the target and the hand, respectively.
Nonetheless, it remains possible that their reaches in the dark are not guided by current proprio-
ceptive information, but rather by a multisensory representation of limb position that is strongly
weighted toward the canonical location that the limb would normally occupy. Because studies of
infants’ reaching in the dark (Clifton et al. 1993, 1994; Robin et al. 1996) have not systematically
varied limb posture prior to reaching, it is difficult to disentangle these interpretations (cf. Holmes
et al. 2004). However, within the framework put forward here (and by A.J. Bremner et al. 2008a),
the predictions are that if posture were to be varied, young infants’ early reaches would be error-
prone, but that in the second 6 months of life they will become better able to take account of the
current position of the limbs in order to reach accurately from a variety of starting postures.
124 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

From 4 months of age, an infant’s reaching movements gradually become more ‘goal-directed’
in nature. Grasps which anticipate the orientation of an object begin to emerge at around 5
months (Lockman et al. 1984; Von Hofsten and Fazel-Zandy 1984). By 8 months of age, re-
orienting of the hand in anticipation of the orientation of a visual target also occurs independ-
ently of vision of the hand (McCarty et al. 2001), indicating that postural guidance is achieved
proprioceptively at this age. Grasps that anticipate the size of an object are first observed from
9 months of age (Von Hofsten and Rönnqvist 1988).
Improvements in the ability to use postural information to maintain spatial alignment between
different sensory inputs arising from peripersonal space can also explain the later development of
an infant’s ability to produce more fine-grained (‘goal-directed’) postural adjustments (especially
those made without sight of the hand (McCarty et al. 2001). These behaviours clearly require
postural calibration, and feed-forward prediction in actions made toward objects.

5.8 Neural construction of bodily and peripersonal spatial


representations in the first year of life
We have argued that two mechanisms of multisensory integration underlying peripersonal space
(canonical multisensory body representations and postural remapping) develop independently
during the first year of life. The sensory interactions subserving the early canonical multisensory
body representations mechanism could be governed both by subcortical (e.g. SC or putamen)
and cortical loci for multisensory integration. The strongest evidence for neural systems underly-
ing the dynamic updating of peripersonal space across changes in posture (postural remapping)
has been obtained from single-unit recordings made in the macaque PMv by Graziano and col-
leagues (Graziano et al. 1997; Graziano 1999). Thus, the more protracted development of mecha-
nisms subserving postural remapping could be explained by a developmental shift from
sub-cortical to cortical processing of multisensory stimuli in early infancy (cf. Wallace and Stein
1997). However, a number of factors speak against cortical maturation as the sole explanation for
these developments.
First, there have been a number of demonstrations of the effect of experience on multisensory
integration. In one such study (Nissen et al. 1951), a newborn chimpanzee’s multisensory and
motor experience with his own hands and feet was severely restricted during the first 30 months
of life by fixing restricting cylinders over these limbs. This chimpanzee later demonstrated almost
no ability to learn a conditioned crossmodal orienting response between two tactile cued loca-
tions on the index finger of either hand. Consistent with this finding, neurophysiological evidence
has demonstrated that multisensory neurons in the SC of dark-reared cats fail to demonstrate
the distinct responses to multisensory and unimodal stimuli seen in normally reared animals
(Wallace et al. 2004).
More recently, Röder et al. (2004; see also Chapter 13 by Röder) have shown that early visual
experience may play a key role in establishing how tactile stimuli are related to visual spatial coor-
dinates, and the typical (visual) posture of the limbs. Indeed, visual experience has also been
shown to play a key role in the developmental integration of auditory and tactile spatial represen-
tations (Collignon et al. 2009). As we shall describe later, there are a number of indications that
changes in patterns of sensory weighting in spatial tasks involving the body may continue well
beyond infancy and into late childhood (Gori et al. 2008; Nardini et al. 2008; Pagel et al. 2009;
Renshaw 1930; Smothergill 1973; Warren and Pick 1970).
Second, the more protracted development of postural remapping in infancy may depend
largely on changes in the kinds of active experience that infants have of their environment. The
developments in postural remapping observed between 6.5 and 10 months of age coincide with
DEVELOPMENTAL CHANGES IN BODY REPRESENTATIONS BEYOND INFANCY 125

the emergence (at about 6 months) of spontaneous reaching toward and across the body midline
for visually-presented objects (Morange and Bloch 1996; Provine and Westerman 1979; Van Hof
et al. 2002). The multisensory experience associated with this behaviour is well-suited for driving
the development of mechanisms of postural remapping.
Roles for experience in the development of representations of the body and peripersonal space
are consistent with ‘interactive specialization’ frameworks for neural systems development
(Johnson 2011; Mareschal et al. 2007) in that some degree of specialization of earlier developed
brain regions (such as the SC) for multisensory orienting responses may lay down the behavioural
foundations required for experientially driven development of more specialized networks under-
lying representations of the body and peripersonal space. The provision of a default canonical
representation of the body underpinned by patterns of relative weighting of the senses may pro-
vide a basis upon which (later developing) experience-dependent dynamic networks can be effi-
ciently deployed, when changes in the posture of the body make this necessary for successful
orienting. This is not to say that brain networks underlying a default canonical representation
would be unaffected by experience. Changes in the body across development would require such
networks to be flexible, and indeed the evidence suggests that sensory experience is necessary for
their normal development (cf. Röder et al. 2004). Rather, it seems more reasonable to suggest that
the general function of such networks in establishing a unitary (if vague) default representation
of the body may be well specified prior to birth.2

5.9 Developmental changes in body representations


beyond infancy
As explained above, it is likely that the neural mechanisms underlying representations of the body
and peripersonal space continue to undergo development beyond infancy. The abilities of young
infants to represent the layout of the body would need to be tuned and re-tuned throughout
development in order to cope with physical changes in the disposition, sizes, and movements of
the limbs that continue even beyond adolescence (see Fig. 5.1).
Although we know of no published studies that have directly examined the development
of visual influences on the spatial localization of the limbs beyond infancy, some classic and more
recent studies have indicated that the multisensory processes involved in representations of the
limbs may change substantially in early childhood. For instance, a number of researchers have
asserted that across early childhood to adolescence, children come to rely more on vision (Renshaw
and Wherry 1931; Renshaw et al. 1930; Warren and Pick 1970). Evidence from spatial orienting
tasks has been used to support this view. For instance, Renshaw (1930) asked children and adults
to localize punctate tactile stimuli on their right arm and hand. The task involved pointing, while
blindfolded, to the stimulated locations using the left hand. Interestingly, pre-adolescent children
performed better than adults on this task, and Renshaw thus suggested that adults rely much more
on vision for directing proprioceptive responses with respect to an external frame of reference.
A similar argument has been put forward more recently by Pagel et al. (2009), who examined
the spatial coordinate systems children use when attributing tactile stimuli to their hands. Using
a tactile TOJ task, they demonstrated developments in the ability to detect the temporal sequence

2 While we consider it likely that such a representation of the canonical layout of the body would be formed
via prenatal multisensory experience, we do not rule out the possibility that such a representation could
exist independently of sensory experience, as has been argued in the case of the newborn’s representations
of the visual appearance of the human face (see Johnson et al. 1991; Morton and Johnson 1991).
126 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

of tactile stimuli presented across the hands in a familiar uncrossed posture. However, these
developmental improvements with uncrossed hands were not matched when the children per-
formed the same task with their hands crossed. Pagel et al. explained this crossed-hands deficit in
the older children (the same deficit has previously been documented in adults: Shore et al. 2002;
Yamamoto and Kitazawa 2001) by suggesting that they adopt an extrapersonal frame of reference
for locating tactile stimuli on the hands: when the hands are crossed, extrapersonal and anatomi-
cal frames of reference conflict, explaining the difficulty with this posture. Given that this crossed-
hands deficit is not observed in congenitally blind adults (Röder et al. 2004), it seems likely that
the developmental changes observed by Pagel et al. (2009) in encoding tactile stimuli are due to
visual experience.
While these findings suggest that spatial representations of the body and limbs become increas-
ingly visual in nature, this need not necessarily be the case. Certainly, it does not inevitably follow
that multisensory spatial representations of the body undergo the same developments as spatial
representations of stimuli impinging on the body (the tactile stimuli used in Pagel et al.’s 2009
study can to some extent be considered as extrapersonal). A number of authors have suggested
that adults may perceive stimuli on the body with respect to different spatial frames of reference
(internal and/or external) depending on the task (Martin 1995; Spence et al. 2001), and it is quite
plausible that such internal and external frames of reference emerge according to different devel-
opmental time courses (cf. A.J. Bremner et al. 2008b). Thus, it may be that even very young chil-
dren optimally integrate the senses (vision and proprioception) when representing the position of
their body and limbs, but undergo development in the ways that they integrate multisensory
spatial information regarding objects with respect to the body.

5.10 Using illusions to explore the development of visual spatial


reliance in young children’s limb representations
In order to investigate developmental changes in visual influences on limb position during
early childhood, we recently conducted an experiment in which we utilized Holmes et al.’s (2004)
‘mirror illusion’ task as a means of comparing the extent of visual influence on limb position
as measured by subsequent reaching behaviours in 5- to 7-year-old children (A.J. Bremner
et al. submitted). In this illusion, participants view one of their hands on both the left and right
of their midline (via a mirror placed at the midline facing one arm and obscuring the other; see
Fig. 5.4). When the position of the hidden right hand (perceived non-visually) is put into spatial
conflict with the illusory visual image, participants’ perceptions of the location of the hidden
hand and their subsequent reaching movements are typically biased by the illusory visual infor-
mation about their hand position (Holmes et al. 2004; Holmes and Spence 2005; Snijders et al.
2007). We measured this visual bias in our developmental groups by examining the extent
to which children’s reaches were affected by illusory visual cues concerning the location of
the hand.
The appeal of investigating multisensory weighting of limb position with this visual bodily
illusion does not simply lie in the fact that it provides an elegant measure of visual spatial reliance
in young children, but also because this particular paradigm places little demand on executive
resources: bear in mind here the fact that memory limitations are known to limit children’s
performance on a range of tasks, even into adolescence (Anderson 2003; Hitch 2002). Because in
this task children do not have to maintain sensory information in mind before making a response,
it represents an ideal way of investigating sensory weighting in a relatively pure way. Finally, it is
easy to motivate children to complete large numbers of trials in illusory tasks like this one because
the perceptual phenomenon being measured captures participants’ interest.
(a) = starting
locations 127
= target

17cm 12 7 0 cm 12
cm cm cm

+ –
(b)
Visual capture of reaching (mm/mm of visual conflict)

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
56–64 mths 65–73 mths 74–82 mths 83–91 mths Adults
Age group
Fig. 5.4 (a) The mirror apparatus from the point of view of the experimenter. The scale below the
diagram indicates distance to the participant’s right with respect to the mirror, measure in
centimetres. Participants viewed their left hand on both the left and right of their midline (by means
of a mirror placed at the midline facing the left arm, and obscuring the right arm). The left hand
was placed at 12 cm from the mirror. The participant’s hidden right hand was either congruent
with the visual image (12 cm right of the mirror, with respect to the participant), or was put into
spatial conflict in the azimuthal dimension with the illusory visual image (at 7 cm or 17 cm to the
right of the mirror). Participants reached toward the target (12 cm to the right of the mirror, and in
front of the starting positions—indicated by the visible arrow above it) and the lateral terminal
errors were measured. Errors to the participant’s right (left) with respect to the target were scored
128 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

Fig. 5.4 (Cont.) as positive (negative). Participants’ use of illusory visual information is indicated by
reaching errors under conditions of crossmodal conflict (i.e. when visual information about the
location of the hand conflicts with veridical information arriving from proprioception). A reliance on
visual information is thus indicated by negative errors (away from the target) at 7 cm, errors around
zero at 12 cm, and positive errors at 17 cm. (b) A comparison of visual influence in the mirror
illusion across age groups (56–64 months, n = 12; 65–73 months, n = 10; 74–82 months, n = 13;
83–91 months, n = 10; adults, n = 15). The measure of visual influence given here is a difference
score, which is compared between the gradients of reach error against starting position in Mirror
and No Mirror conditions (error gradient in the Mirror condition − error gradient in the No Mirror
condition). In a comparison of visual influence on reaching across the four age groups of children,
a main effect of age group was observed (p < 0.05). The error bars represent the SE of the means,
and asterisks indicate statistically significant post-hoc comparisons (p d 0.05).

We tested four age-groups of children on the task. As Fig. 5.4 shows, all of the age-groups tested
showed evidence of being influenced by the illusory visual information. Thus, from age 5 years (at
the latest), children, like adults, demonstrate some reliance on vision for locating their hands in
the azimuthal plane (Holmes et al. 2004; Van Beers et al. 2002). Despite this continuous role of
vision, our data also indicate some developmental changes in its influence on behaviour through
early childhood. In order to compare the magnitude of visual influence in the mirror illusion
across age-groups, we obtained measures of the degree of visual influence (see Fig. 5.4) by com-
paring reaching errors in the ‘Mirror’ and ‘No Mirror’ conditions. This age-group comparison
showed a significant non-monotonic development of visual weighting in limb position as
we observed a sharp increase in visual influence between the 56–64-month-old (approx. 5 years)
age group and the 65–73-month-old (approx. 5¾ years) age group followed by a subsequent
reduction in visual influence between 65–73 months (5¾ years) and 74–82 months (6½ years).
In the next section, we attempt to explain this finding in the context of the relevant literature.

5.11 Functions and developmental processes underlying


changes in visual spatial reliance beyond infancy
One view concerning the functional purpose of an increased dominance of vision in orienting
tasks (as observed in the findings reported above, and by Renshaw 1930) was put forward by both
Renshaw (1930) and Warren and Pick (1970) who argued that children come to be dominated by
visual cues because they derive from the most spatially accurate sensory organ. A more recent
refinement of this argument has been proposed in response to the ‘optimal integration’ account
of multisensory integration in adults (e.g. Ernst and Banks 2002). Remember that this account
proposes that all relevant senses contribute to spatial representations, but that the contributions
of the individual senses are weighted in proportion to their relative task-specific reliabilities, thus
yielding statistically optimal responses. Both Gori et al. (2008), and Nardini and colleagues
(Chapter 6 by Nardini and Cowie; Nardini et al. 2008) have proposed that multisensory develop-
ment converges in middle to late childhood onto an optimal sensory weighting. The evidence for
this claim rests on recent demonstrations in which children develop from a state of either relying
entirely on one sensory modality (which sensory modality this is depends on the task; Gori et al.
2008), or alternating between reliance on one modality or the other (Nardini et al. 2008), to a
situation in which they weight the senses in a statistically optimal fashion.
This account can provide a framework for understanding developmental shifts in middle to late
childhood toward an increased visual influence observed across a number of spatial-orienting
DEVELOPMENTS IN VISUAL SPATIAL RELIANCE BEYOND INFANCY 129

behaviours (Pagel et al. 2009; Renshaw 1930, Renshaw et al. 1930; Smothergill 1973; Warren and
Pick 1970). In addition, however, the optimal integration account can also explain developmental
decreases in visual influence as being due to convergence upon an optimal weighting from a prior
state in which vision was weighted disproportionately to its reliability. We speculate that this
account can also explain the non-monotonic developmental changes in visual influence on reach-
ing observed in the mirror illusion (reported in Section 5.10), as follows. The initial shift toward
a greater influence of vision at 65–73 months of age could be due to convergence upon a multi-
sensory weighting in which vision is weighted more in the context of locating the hand in the
azimuthal dimension, just as it is in adults (see Van Beers et al. 2002). We also suggest that the
later shift away from visual influence in the 74–82-month-old group represents a developmental
process of optimization in which children reclaim some degree of reliance on the less reliable
modality (in this case proprioception) following an overemphasis on vision at an earlier stage
(65–73 months).
Thus, we argue that developmental change in visual spatial reliance in limb representations in
early childhood represents a process of convergence upon an optimal weighting of the senses. But
what developmental processes underlie this extended process of development? Warren and Pick
(1970), following on from Birch and Lefford (1967), posited that increased visual reliance is made
possible by a progressive general linking of the senses across childhood. However, this account
has difficulty explaining evidence indicating that much younger children (infants in their
first year of life) are sensitive to multisensory correspondences, both through their visual and
auditory senses (e.g. for correspondences between vision and audition, see Lewkowicz 2000, and
Chapter 7 by Lewkowicz) and their bodily senses (see above).
Thus, it seems more likely that the changes in visual influence observed in young children,
described above, occur as part of a developmental process of multisensory fine-tuning (rather
than a registering of multisensory correspondence per se) in which the specific weightings of the
senses are modified in order to improve the efficiency of sensorimotor performance. Gori et al.
(2008) propose an interesting account for why the development of optimal multisensory integra-
tion may take so long; they suggest that the optimal weighting of the senses is delayed so as to
provide a point of reference against which to calibrate the senses in multisensory tasks. Because
the body and limbs continue to grow over the course of childhood and adolescence, this recalibra-
tion is constantly required, and so, according to Gori et al. (2008), optimal weighting does
not take place until the body has stopped growing substantially (see also King, 2004 for a similar
argument in the ferret). It remains unclear however, how such a process of delay in or inhibition
of integration might come about.
Some clues to the developmental processes involved in changes in visual spatial reliance in body
representations can be gleaned from different patterns of development observed across different
crossmodal tasks. For instance, whereas Gori et al. (2008), and Nardini et al. (2008) identify
developmental shifts in crossmodal weighting between 5 and 8–10 years of age, and 7–8 years and
adulthood, respectively, A.J. Bremner et al.’s (submitted) findings suggest that children settle on
a weighting similar to that used by adults by 6½ years of age. There are several possible explana-
tions for this discrepancy. For one, it is likely that the different kinds of multisensory integration
(e.g. different sensory modalities and weightings) required in these tasks would have different
developmental time-courses. Such discrepancies are certainly evident in the emergence of multi-
sensory integration in infancy (e.g. Lewkowicz 2002). Another explanation we consider particu-
larly likely rests not on the kinds of multisensory integration required but on the different
executive demands required by our task and others used with young children. For example, the
tasks used by both Gori et al. and Nardini et al. both involve short-term maintenance of cues prior
to a response being made. In contrast, the mirror illusion task reported in this chapter requires
130 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

children to integrate proprioception and vision concurrently and make an immediate reaching
response. An important question for future research will therefore be to address whether the
development of optimal multisensory weighting of body representations in early childhood is
constrained by the amelioration of executive limitations, which continues across childhood and
into early adolescence (Anderson 2003; Hitch 2002) and likely affects the maintenance and
manipulation of multisensory inputs (Gallace and Spence 2009).

5.12 Summary
A significant challenge for development is to form detailed, accurate multisensory representa-
tions of the body and peripersonal space. The task of integrating the senses to form a unified
percept of the body and its relation to the environment is not simple. Movement of the sense
organs with respect to one another when the body changes posture means that the developing
infant must learn dynamically to alter the way in which he or she integrates the senses (in particu-
lar, the spatial relationships between visual, auditory, and bodily receptors). Here, we have
described a framework that argues for the independent development (at least over the first year of
life) of two integrative mechanisms that give rise to multisensory representations of peripersonal
space: canonical multisensory body representations and postural remapping. We have argued, on
the basis of evidence from tactile spatial orienting behaviours in infants that a mechanism of
canonical multisensory body representations provides an approximate default form of multisen-
sory integration, upon which more dynamic systems of integration can later be built. The later
development during infancy of more dynamic integrative systems may arise in response to
changes in the demands of multisensory and sensorimotor interactions in peripersonal space,
commensurate with the emergence of certain kinds of postural changes related to exploratory
behaviours.
Beyond infancy, and into early childhood, there are developmental changes in the relative
weightings given to vision and proprioception in locating an arm in the azimuthal plane with
respect to the body. We identified continued developmental changes in the weightings given to
vision, characterized by an increase in visual weighting at 65–73 months, followed by a later shift
away from visual weighting at 74–82 months and beyond into adulthood. Such developmental
changes likely play a role in the fine tuning of both canonical multisensory body representations
and postural remapping such that children converge on increasingly optimal weightings of
current and prior sensory inputs regarding limb position.
Multisensory spatial representations of the body not only underpin our ability to act on our
nearby environment, but also provide an embodied point of reference to the external
world (Bermudez et al. 1995; Gallagher 2005; Varela et al. 1991). As such, an understanding of
the development of these representations has important implications for children’s developing
conceptions of the world they inhabit. Indeed, the surprising prevalence of sensorimotor difficul-
ties of one kind or another, across a wide range of developmental disorders, also hints that
body representations may have an important role in atypical development (see Chapter 12 by
Hill et al.). Developmental research is only just beginning to scratch the surface of the rich set
of multisensory interactions that underlie representations of the body and its surroundings.
Despite the emphasis on tactile–proprioceptive–visual interactions in this chapter, it is important
to note that other sensory modalities may also play important roles in the development of body
representations (see e.g. Kitagawa and Spence 2005). In the final section of this chapter we will
discuss how the developmental course of embodied representations may help explain some
of the puzzling dissociations infants and even children demonstrate during their early cognitive
development.
REFERENCES 131

5.13 Epilogue: lessons for early cognitive development


Despite a relatively limited repertoire of sensorimotor spatial behaviours in early infancy, research
using measures of the time spent looking at arrays of objects has shown that, within a few months
of birth, human infants are able to form sophisticated spatial representations of objects in their
environment (e.g. Slater 1995) and their interactions (Baillargeon 2004; Scheier et al. 2003; Spelke
et al. 1992). By 3–4 months of age, infants form the perceptual categories of ‘above’ and ‘below’
(Quinn, 1994), encode the spatial distance between objects (Newcombe et al. 2005), and can
represent the continued existence and extent of a hidden object in a particular spatial location
(Baillargeon 2004). Strikingly, young infants can recognise the locations of objects and features in
relation to external frames of reference across changes in their orientation (A.J. Bremner et al.
2007; Kaufman and Needham 1999).
However, while looking-duration measures have tended to indicate precocious abilities to
represent objects in the extrapersonal environment, spatial-orienting tasks that require infants
to locate objects in the environment relative to themselves (with manual responses, or visual-
orienting behaviour) have provided a mixed picture of early spatial abilities. Typically, before
8 months of age, infants do not even attempt to uncover hidden objects within reach (Piaget
1954). When orienting to targets, young infants seem to code their responses with respect to their
own body and ignore changes in the position of the target or their own body (Acredolo 1978;
J.G. Bremner 1978). When either is moved before the orienting response occurs, young infants
make ‘egocentric’ errors, and it is only in their second year that they correctly update their
responses.
One way of resolving these conflicting findings is to consider the kinds of spatial representation
required in these two different types of task. In looking-duration tasks, it is possible to identify
changes in the location of objects by reference to purely visual, extrapersonal, spatial coordinates.
By contrast, spatial-orienting tasks require infants to represent the location of objects identified
via the ‘distance’ receptors (typically vision) and, in order to translate this target location into the
intrinsic body-centred coordinates required to orient toward it, they require multisensory embod-
ied representations of the target. The extended development of such representations as discussed
in this chapter may help explain a well-known paradox of cognitive development in the first
year of life: that infants’ early competence as demonstrated in looking-duration measures
(e.g. Baillargeon 2004; Spelke et al. 1992) is not matched by their ability to act manually on that
information until much later and, in many cases, beyond infancy (e.g. J.G. Bremner 2000; Hood
et al. 2000; Mareschal 2000; Munakata et al. 1997).

Acknowledgements
AJB is supported by European Research Council Grant No. 241242 (European Commission
Framework Programme 7). We would also like to thank some of our colleagues for helpful discus-
sions that have fed into the development of this chapter; particularly JJ Begum, Dorothy Cowie,
Fran Knight, David Lewkowicz, and Silvia Rigato.

References
Acredolo, L.P. (1978). Development of spatial orientation in infancy. Developmental Psychology, 14, 224–34.
Adolph, K.E. (2008). Learning to move. Current Directions in Psychological Science, 17, 213–18.
Adolph, K.E., and Avolio, A.M. (2000). Walking infants adapt locomotion to changing body dimensions.
Journal of Experimental Psychology: Human Perception and Performance, 26, 1148–66.
Alais, D., and Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration.
Current Biology, 14, 257–62.
132 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

Anderson, P. (2003). Assessment and development of executive function (EF) during childhood. Child
Neuropsychology (Neuropsychology, Development and Cognition: Section C), 8, 71–82.
Azañón, E., and Soto-Faraco, S. (2008). Changing reference frames during the encoding of tactile events.
Current Biology, 18, 1044–49.
Bahrick, L.E., and Watson, J.S. (1985). Detection of intermodal proprioceptive-visual contingency as a
potential basis of self-perception in infancy. Developmental Psychology, 21, 963–73.
Baillargeon, R. (2004). Infants’ reasoning about hidden objects: evidence for event-general and event-
specific expectations. Developmental Science, 7, 391–414.
Bayley, N. (1969). Manual for the Bayley scales of infant development. Psychological Corp., New York.
Bermudez, J.L., Marcel, A.J., and Eilan, N. (1995). The body and the self. MIT Press, Cambridge, MA.
Birch, H.G., and Lefford, A. (1963). Intersensory development in children. Monographs of the Society for
Research in Child Development, 28, 1–48.
Birch, H.G., and Lefford, A. (1967). Visual differentiation, intersensory integration, and voluntary motor
control. Monographs of the Society for Research in Child Development, 32, 1–87.
Botvinick, M., and Cohen, J. (1998). Rubber hands ‘feel’ touch that eyes see. Nature, 391, 756.
Bremner, A.J., Bryant, P.E., Mareschal, D., and Volein, Á. (2007). Recognition of complex object-centred
spatial configurations in early infancy. Visual Cognition, 15, 1–31.
Bremner, A.J., Holmes, N.P., and Spence, C. (2008a). Infants lost in (peripersonal) space? Trends in
Cognitive Sciences, 12, 298–305.
Bremner, A.J., Mareschal, D., Fox, S., and Spence, C. (2008b). Spatial localization of touch in the first year
of life: early influence of a visual code, and the development of remapping across changes in limb
position. Journal of Experimental Psychology: General, 137, 149–62.
Bremner, A.J., Hill, E.L., Pratt, M., Rigato, S., and Spence, C. (submitted). Bodily illusions in young
children: developmental change in visual and proprioceptive contributions to perceived hand
position.
Bremner, J.G. (1978). Egocentric versus allocentric coding in nine-month-old infants: factors influencing
the choice of code. Developmental Psychology, 14, 346–55.
Bremner, J.G. (2000). Developmental relationships between perception and action in infancy. Infant
Behavior and Development, 23, 567–82.
Bryant, P.E. (1974). Perception and understanding in young children. Methuen, London, UK.
Bryant, P.E., and Raz, I. (1975). Visual and tactual perception of shape by young children. Developmental
Psychology, 11, 525–26.
Bryant, P.E., Jones, P., Claxton, V., and Perkins, G.M. (1972). Recognition of shapes across modalities by
infants. Nature, 240, 303–304.
Carlier, M., Doyen, A.-L., and Lamard, C. (2006). Midline crossing: Developmental trend from 3 to 10
years of age in a preferential card-reaching task. Brain and Cognition, 61, 255–61.
Clifton, R.K., Muir, D.W., Ashmead, D.H., and Clarkson, M.G. (1993). Is visually guided reaching in early
infancy a myth? Child Development, 64, 1099–1110.
Clifton, R.K., Rochat, P., Robin, D.J., and Berthier, N.E. (1994). Multimodal perception in the control of
infant reaching. Journal of Experimental Psychology: human Perception and Performance, 20, 876–86.
Collignon, O., Charbonneau, G., Lassonde, M., and Lepore, F. (2009). Early visual deprivation alters
multisensory processing in peripersonal space. Neuropsychologia, 47, 3236–43.
DeLoache, J.S., Uttal, D.H., and Rosengren, K.S. (2004). Scale errors offer evidence for a perception-action
dissociation early in life. Science, 304, 1027–29.
Driver, J. (1999). Egocentric and object-based visual neglect. In The hippocampal and parietal foundations of
spatial cognition (eds. N. Burgess, K.J. Jeffrey, and J. O’Keefe), pp. 67–89. Oxford University Press, Oxford.
Ehrsson, H.H., Spence, C., and Passingham, R.E. (2004). ‘That’s my hand!’ Activity in the premotor cortex
reflects feeling of ownership of a limb. Science, 305, 875–77.
Ernst, M.O., and Banks, M.S. (2002). Humans integrate visual and haptic information in a statistically
optimal fashion. Nature, 415, 429–33.
REFERENCES 133

Ernst, M.O., and Bülthoff, H.H. (2004). Merging the senses into a robust percept. Trends in Cognitive
Sciences, 8, 162–69.
Gallace, A., and Spence, C. (2009). The cognitive limitations and neural correlates of tactile memory.
Psychological Bulletin, 135, 380–406.
Gallagher, S. (2005). How the body shapes the mind. Oxford University Press, Oxford.
Gilbert, F.S. (2000). Developmental biology, 6th Ed. Sinauer Associates, Inc., Sunderland, MA.
Glover, S. (2004). What causes scale errors in children? Trends in Cognitive Sciences, 8, 440–42.
Gori, M., Del Viva, M.M., Sandini, G., and Burr, D.C. (2008). Young children do not integrate visual and
haptic form information. Current Biology, 18, 694–98.
Gori, M., Sandini, G., Martinoli, C., and Burr, D. (2010). Poor haptic orientation discrimination in
nonsighted children may reflect disruption of cross-sensory calibration. Current Biology, 3, 223–225.
Gottfried, A.W., Rose, S.A., and Bridger, W.H. (1977). Cross-modal transfer in human infants. Child
Development, 48, 118–23.
Gottfried, A.W., Rose, S.A., and Bridger, W.H. (1978). Effects of visual, haptic, and manipulatory
experiences on infants’ visual recognition memory of objects. Developmental Psychology, 14, 305–12.
Graziano, M.S.A. (1999). Where is my arm? The relative role of vision and proprioception in the neuronal
representation of limb position. Proceedings of the National Academy of Sciences, USA, 96, 10418–21.
Graziano, M.S.A., Hu, X.T., and Gross, C.G. (1997). Visuospatial properties of ventral premotor cortex.
Journal of Neurophysiology, 77, 2268–92.
Graziano, M.S.A., Cooke, D.F., and Taylor, C.S.R. (2000). Coding the location of the arm by sight. Science,
290, 1782–86.
Graziano, M.S.A., Gross, C. G., Taylor, C.S.R., and Moore, T. (2004). A system of multimodal areas in the
primate brain. In Crossmodal space and crossmodal attention (eds. C. Spence and J. Driver), pp. 51–67.
Oxford University Press, Oxford.
Groh, J.M., and Sparks, D.L. (1996a). Saccades to somatosensory targets: 2. Motor convergence in primate
superior colliculus. Journal of Neurophysiology, 75, 428–38.
Groh, J.M., and Sparks, D.L. (1996b). Saccades to somatosensory targets: 3. Eye-position dependent
somatosensory activity in primate superior colliculus. Journal of Neurophysiology, 75, 439–53.
Groh, J.M., and Sparks, D.L. (1996c). Saccades to somatosensory targets: 1. Behavioral characteristics.
Journal of Neurophysiology, 75, 412–27.
Hall, G.S. (1898). Some aspects of the early sense of self. American Journal of Psychology, 9, 351–95.
Hitch, G.J. (2002). Developmental changes in working memory: a multicomponent view. In Lifespan
development of human memory (eds. P. Graf and N. Ohta), pp. 15–37. MIT Press, Cambridge, MA.
Holmes, N.P., and Spence, C. (2004). The body schema and multisensory representation(s) of peripersonal
space. Cognitive Processing, 5, 94–105.
Holmes, N.P., and Spence, C. (2005). Visual bias of unseen hand position with a mirror: spatial and
temporal factors. Experimental Brain Research, 166, 489–97.
Holmes, N.P., Crozier, G., and Spence, C. (2004). When mirrors lie: ‘Visual capture’ of arm position
impairs reaching performance. Cognitive, Affective, and Behavioral Neuroscience, 4, 193–200.
Hood, B., Carey, S., and Prasada, S. (2000). Predicting the outcomes of physical events: two-year-olds fail
to reveal knowledge of solidity and support. Child Development, 71, 1540–54.
Hulme, C., Smart, A., Moran, G., and Raine, A. (1983). Visual kinaesthetic and cross-modal development:
relationships to motor skill development. Perception, 12, 477–83.
Johnson, M.H. (2011). Interactive specialization: a domain-general framework for human functional brain
development? Developmental Cognitive Neuroscience, 1, 7–21.
Johnson, M.H., and De Haan, M. (2011). Developmental cognitive neuroscience, 3rd edn. Blackwell-Wiley, Oxford.
Johnson, M.H., Dziurawiec, S., Ellis, H., and Morton, J. (1991). Newborns’ preferential tracking of face-like
stimuli and its subsequent decline. Cognition, 40, 1–19.
Kaufman, J., and Needham, A. (1999). Objective spatial coding by 6.5-month-old infants in a visual
dishabituation task. Developmental Science, 2, 432–41.
134 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

Kennett, S., Spence, C., and Driver, J. (2002). Visuo-tactile links in covert exogenous spatial attention
remap across changes in unseen hand posture. Perception and Psychophysics, 64, 1083–94.
Kersten, D., Mamassian, P., and Yuille, A. (2004). Object perception as bayesian inference. Annual Review
of Psychology, 55, 271–304.
King, A. (2004). Development of multisensory spatial integration. In Crossmodal space and crossmodal
attention (eds. C. Spence, and J. Driver), pp. 1–24. Oxford University Press, Oxford University Press.
Kitagawa, N., Zampini, M., Spence, C. (2005). Audiotactile interactions in near and far space. Experimental
Brain Research, 166, 528–37.
Kitazawa, S. (2002). Where conscious sensation takes place. Consciousness and Cognition, 3, 475–77.
Körding, K.P., and Wolpert, D. (2004). Bayesian integration in sensorimotor learning. Nature, 427, 244–47.
Lewkowicz, D.J. (2000). The development of intersensory temporal perception: an epigenetic systems/
limitations view. Psychological Bulletin, 126, 281–308.
Lewkowicz, D.J. (2002). Heterogeneity and heterochrony in the development of intersensory perception.
Cognitive Brain Research, 14, 41–63.
Locke, J. (1690). An essay concerning human understanding. Oxford University Press, Oxford, UK.
Lockman, J.J., Ashmead, D.H., and Bushnell, E.W. (1984). The development of anticipatory hand
orientation during infancy. Journal of Experimental Child Psychology, 37, 176–86.
Makin, T.R., Holmes, N.P., and Ehrsson, H.H. (2008). On the other hand: dummy hands and peripersonal
space. Behavioural Brain Research, 191, 1–10.
Maravita, A., Spence, C., and Driver, J. (2003). Multisensory integration and the body schema: close to
hand and within reach. Current Biology, 13, R531–39.
Mareschal, D. (2000) Infant object knowledge: current trends and controversies. Trends in Cognitive
Sciences, 4, 408–416.
Mareschal, D., Johnson, M.H., Sirois, S., Spratling, M., Thomas, M., and Westermann, G. (2007).
Neuroconstructivism, Vol. I: How the brain constructs cognition. Oxford University Press, Oxford.
Martin, M.G.F. (1995). Bodily awareness: a sense of ownership. In The body and the self (eds. J.L. Bermudez,
A. Marcel, and N. Eilan), pp. 267–89. MIT Press, Oxford.
Maurer, D., Stager, C., and Mondloch, C. (1999) Cross-modal transfer of shape is difficult to demonstrate
in 1-month-olds. Child Development, 70, 1047–57.
McCarty, M.E., Clifton, R.K., Ashmead, D.H., Lee, P., and Goubet, N. (2001). How infants use vision for
grasping objects. Child Development, 72, 973–87.
Meltzoff, A.N., and Borton, R.W. (1979). Intermodal matching by human neonates. Nature, 282, 403–404.
Morange, F., and Bloch, H. (1996). Lateralization of the approach movement and the prehension
movement in infants from 4 to 7 months. Early Development and Parenting, 5, 81–92.
Morton, J., and Johnson, M.H. (1991). CONSPEC and CONLERN: a two-process theory of infant face
recognition. Psychological Review, 98, 164–81.
Munakata, Y., McClelland, J.L., Johnson, M.H., and Siegler, R.S. (1997). Rethinking infant knowledge:
toward an adaptive process account of successes and failures in object permanence tasks. Psychological
Review, 104, 686–713.
Nardini, M., Jones, P., Bedford, R., and Braddick, O. (2008). Development of cue integration in human
navigation. Current Biology, 18, 689–93.
Newcombe, N.S., Sluzenski, J., and Huttenlocher, J. (2005). Pre-existing knowledge versus on-line learning:
what do young infants really know about spatial location? Psychological Science, 16, 222–27.
Nissen, H.W., Chow, K.L., and Semmes, J. (1951). Effects of restricted opportunity for tactual, kinesthetic, and
manipulative experience on the behavior of a chimpanzee. American Journal of Psychology, 64, 485–507.
Orlov, T., Makin, T.R., and Zohary, E. (2010). Topographic representation of the human body in the
occipitotemporal cortex. Neuron, 68, 586–600.
Pagel, B., Heed, T., and Röder, B. (2009). Change of reference frame for tactile localization during child
development. Developmental Science, 12, 929–37.
REFERENCES 135

Parise, C.V., Spence, C., and Ernst, M.O. (2012). Multisensory integration: when correlation implies
causation. Current Biology, 22, 46–49.
Piaget, J. (1954). The construction of reality in the child. Routledge and Kegan-Paul, London.
Previc, F.H. (1998). The neuropsychology of 3-D space. Psychological Bulletin, 124, 123–64.
Provine, R.R., and Westerman, J.A. (1979). Crossing the midline: limits of early eye-hand behavior. Child
Development, 50, 437–41.
Quinn, P.C. (1994). The categorisation of above and below spatial relations by young infants. Child
Development, 65, 58–69.
Renshaw, S. (1930). The errors of cutaneous localization and the effect of practice on the localizing
movement in children and adults. Journal of Genetic Psychology, 28, 223–38.
Renshaw, S., and Wherry, R.J. (1931). Studies on cutaneous localization III: the age of onset of ocular
dominance. Journal of Genetic Psychology, 29, 493–96.
Renshaw, S., Wherry, R.J., and Newlin, J.C. (1930). Cutaneous localization in congenitally blind versus
seeing children and adults. Journal of Genetic Psychology, 28, 239–48.
Rizzolatti, G., Fadiga, L., Fogassi, L., and Gallese, V. (1997). The space around us. Science, 277, 190–91.
Robin, D.J., Berthier, N.E., and Clifton, R.K. (1996) Infants’ predictive reaching for moving objects in the
dark. Developmental Psychology, 32, 824–35.
Rochat, P. (1998). Self-perception and action in infancy. Experimental Brain Research, 123, 102–109.
Rochat, P. (2010). The innate sense of the body develops to become a public affair by 2–3 years.
Neuropsychologia, 48, 738–45.
Rochat, P., and Morgan, R. (1995). Spatial determinants in the perception of self-produced leg movements
by 3- to 5-month-old infants. Developmental Psychology, 31, 626–36.
Roche, R.M., and Malina, A.F. (1983). Manual of physical status and performance in childhood, Volume 1,
Physical status. Plenum, New York.
Röder, B., Rösler, F., and Spence, C. (2004). Early vision impairs tactile perception in the blind. Current
Biology, 14, 121–24.
Rose, S.A. (1994). From hand to eye: findings and issues in infant cross-modal transfer. In The development
of intersensory perception: comparative perspectives (eds. D.J. Lewkowicz, and R. Lickliter), pp. 265–84.
Lawrence Erlbaum Associates, Hillsdale, NJ.
Rose, S.A., Gottfried, A.W., and Bridger, W.H. (1981a). Cross-modal transfer and information processing
by the sense of touch in infancy. Developmental Psychology, 17, 90–98.
Rose, S.A., Gottfried, A.W., and Bridger, W.H. (1981b). Cross-modal transfer in 6-month-old infants.
Developmental Psychology, 17, 661–69.
Sann, C., and Streri, A. (2007). Perception of object shape and texture in human newborns: evidence from
cross-modal transfer tasks. Developmental Science, 10, 399–410.
Scheier, C., Lewkowicz, D.J., and Shimojo, S. Sound induces perceptual reorganization of an ambiguous
motion display in human infants. Developmental Science, 6, 233–44.
Schicke, T., and Röder, B. (2006). Spatial remapping of touch: confusion of perceived stimulus order across
hand and foot. Proceedings of the National Academy of Science, USA, 103, 11808–13.
Schmuckler, M. (1996). Visual-proprioceptive intermodal perception in infancy. Infant Behavior and
Development, 19, 221–32.
Schmuckler, M.A., and Fairhall, J.L. (2001). Infants’ visual-proprioceptive intermodal perception of
point-light information. Child Development, 72, 949–62.
Schmuckler, M.A., and Jewell, D.T. (2007). Infants’ visual-proprioceptive intermodal perception with
imperfect contingency information. Developmental Psychobiology, 48, 387–98.
Shore, D.I., Spry, E., and Spence, C. (2002). Confusing the mind by crossing the hands. Cognitive Brain
Research, 14, 153–63.
Slater, A.M. (1995). Visual perception and memory at birth. In Advances in Infancy Research, 9 (eds. C.
Rovee-Collier and L.P. Lipsitt), pp. 107–62. Ablex, Norwood, NJ.
136 DEVELOPMENT OF MULTISENSORY REPRESENTATIONS OF THE BODY

Smothergill, D.W. (1973). Accuracy and variability in the localization of spatial targets at three age levels.
Developmental Psychology, 8, 62–66.
Snijders H.J., Holmes N.P., and Spence, C. (2007). Direction-dependent integration of vision and
proprioception in reaching under the influence of the mirror illusion. Neuropsychologia, 45, 496–505.
Spelke, E.S., Breinlinger, K., Macomber, J., and Jacobson, K. (1992). Origins of knowledge. Psychological
Review, 99, 605–32.
Spence, C., Nicholls, M.E.R., and Driver, J. (2001). The cost of expecting events in the wrong sensory
modality. Perception and Psychophysics, 63, 330–36.
Spence, C., Pavani, F., Maravita, A., and Holmes, N.P. (2008). Multisensory interactions. In Haptic rendering:
foundations, algorithms, and applications (eds. M.C. Lin and M.A. Otaduy), pp. 21–52. AK Peters,
Wellesley, MA.
Streri, A., and Gentaz, E. (2003). Cross-modal recognition of shape from hand to eyes in human newborns.
Somatosensory and Motor Research, 20, 13–18.
Streri, A., and Gentaz, E. (2004). Cross-modal recognition of shape from hand to eyes and handedness in
human newborns. Neuropsychologia, 42, 1365–69.
van Beers, R.J., Wolpert, D.M., and Haggard, P. (2002). When feeling is more important than seeing in
sensorimotor adaptation. Current Biology, 12, 834–37.
van der Meer, A.L. (1997). Keeping the arm in the limelight: advanced visual control of arm movements in
neonates. European Journal of Paediatric Neurology, 1, 103–108.
van Hof, P., van der Kamp, J., and Savelsbergh, G.J.P. (2002). The relation of unimanual and bimanual
reaching to crossing the midline. Child Development, 73, 1353–62.
Varela, F.J., Thompson, E., and Rosch, E. (1991). The embodied mind: cognitive science and human
experience. MIT Press, Cambridge, MA.
von Hofsten, C. (1982). Eye-hand coordination in the newborn. Developmental Psychology, 18, 450–61.
von Hofsten, C. (2004). An action perspective on motor development. Trends in Cognitive Sciences, 8, 266–72.
von Hofsten, C., and Fazel-Zandy, S. (1984). Development of visually guided hand orientation in reaching.
Journal of Experimental Child Psychology, 38, 208–219.
von Hofsten, C., and Rönnqvist, L. (1988). Preparation for grasping an object: a developmental study.
Journal of Experimental Psychology: Human Perception and Performance, 14, 610–21.
Wallace, M.T., and Stein, B.E. (1997). Development of multisensory neurons and multisensory integration
in cat superior colliculus. Journal of Neuroscience, 17, 2429–44.
Wallace, M.T., Perrault, T.J., Jr., Hairston, W.D., and Stein, B.E. (2004). Visual experience is necessary for
the development of multisensory integration. Journal of Neuroscience. 24, 9580–84.
Warren, D.H., and Pick, H.L., Jr. (1970). Intermodality relations in blind and sighted people. Perception
and Psychophysics, 8, 430–32.
Yamamoto, S., and Kitazawa, S. (2001). Reversal of subjective temporal order due to arm crossing. Nature
Neuroscience, 4, 759–65.
Zmyj, N., Jank, J., Schütz-Bosbach, S., and Daum, M.M. (2011). Detection of visual-tactile contingency in
the first year after birth. Cognition, 120, 82–89.
Chapter 6

The development of multisensory


balance, locomotion, orientation, and
navigation
Marko Nardini and Dorothy Cowie

6.1 Introduction
This chapter describes development of the sensory processes underlying the movement of our
bodies in space. In order to move around the world in an adaptive manner infants and children
must overcome a range of sensory and motor challenges. Since balance is a prerequisite for whole-
body movement and locomotion, a primary challenge is using sensory inputs to maintain balance.
The ultimate goal of movement is to reach (or avoid) specific objects or locations in space. Thus
a further prerequisite for adaptive spatial behaviour is the ability to represent the locations of
significant objects and places. Information about such objects, and about one’s own movement
(‘self-motion’) comes from many different senses. A crucial challenge for infants and children (not
to mention adults) is to select or integrate these correctly to perform spatial tasks.
We will first describe the development of balance and locomotion. We will then go on to
describe the development of spatial orientation and navigation. Basic spatial orienting (e.g. turn-
ing the head to localize multisensory stimuli), which does not require balance or locomotion, is
also described in this section. These early-developing orienting behaviours are building blocks for
more sophisticated navigation and spatial recall.
We will describe a number of situations in which spatial behaviour in response to multiple
sensory inputs undergoes marked changes in childhood before reaching the adult state. Bayesian
integration of estimates is a theoretical framework that may accommodate some of these findings.
In the concluding section of this chapter (Section 6.5), we will describe this framework and
highlight its potential for explaining some of the existing developmental findings. However, we
will argue that most of the experimental work needed to evaluate the Bayesian explanation still
remains to be done.

6.2 Development of multisensory balance and locomotion


Human balance mechanisms allow us to maintain a positional equilibrium by coordinating
internal and external forces on the body. Static balance refers to the maintenance of equilibrium
during quiet stance (standing still), whereas dynamic balance refers to the maintenance of equi-
librium during movement, for example during walking. In line with the focus of available/current
research, we will concentrate on static balance. Maintaining balance is vitally important for devel-
oping infants and children because it provides a base on which to build other skills. For example,
being able to sit upright allows for reaching, being able to stand without falling over allows for
walking. These motor skills in turn permit explorations of the surrounding space and objects.
138 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

Useful information for balance and locomotion comes from multiple senses. However, develop-
ment poses a difficult context in which to integrate these multisensory inputs for balance and
locomotion because the child’s sensory and motor capabilities are still developing, while its body
is also changing in shape, size, and mass.
In this review we will separate the sensory inputs used to maintain balance into three functional
groupings: visual inputs, vestibular inputs, and information from muscle and joint mechanore-
ceptors, which we term ‘proprioception’. The key visual information for balance is that which
signals relative motion between an observer and their environment. Such movement produces
characteristic patterns of change (‘optic flow’) in the visual environment (Gibson 1979). For
example, as an observer sways forwards toward an object, the image of that object expands. This
characteristic ‘expansion’ pattern therefore signals that the observer is moving forwards, and a
corrective, backwards sway response may be made. The medial superior temporal area (MST)
plays a prominent role in processing these signals, as do subcortical structures (Billington et al.
2010; Wall and Smith 2008). In the vestibular system, the semicircular canals provide informa-
tion about the rotation of the head whereas the otolith organs signal linear accelerations (Day and
Fitzpatrick 2005). This information is processed in the vestibular nuclei and subsequently in
higher structures including the cerebellum and cortex (see Angelaki et al. 2009). Finally what we
term proprioception includes information arising from muscle spindles sensing muscle stretch
and similar mechanoreceptors in the joints.

6.2.1 Multisensory balance development


Many studies suggest that balance control is strongly coupled to visual information very early in
infancy. Furthermore, this coupling seems to require little experience of standing or even sitting
upright. Children as young as three days old make head movements in response to expanding
optic flow stimuli, and head movement increases linearly as a function of flow velocity (Jouen
et al. 2000). Preferential looking also shows that 2-month-olds can discriminate global radial
expansion patterns from random patterns. Sensitivity to this motion increases during the first
year of life (Brosseau-Lachaine et al. 2008). The earliest responses may have a subcortical basis,
while later responses are likely to recruit cortical visual areas sensitive to global flow patterns, such
as MT (middle temporal cortex) and MST (medial superior temporal cortex) (Wall et al. 2008;
Wattam-Bell et al. 2010).
The ‘swinging room’ technique (Lishman and Lee 1973) has been widely used to investigate the
balance responses of infants and children (Fig. 6.1). Participants stand on a platform inside what
appears to be a small room. The walls and ceiling of the room move back and forth independently

(a) (b)

Fig. 6.1 The swinging room technique. (a) The participant stands in a room with a stationary floor
but moving walls. (b) An optic flow expansion pattern created by walls moving toward the
participant. The same pattern would be produced by the observer swaying forward.
DEVELOPMENT OF MULTISENSORY BALANCE AND LOCOMOTION 139

of the floor, which is fixed. This causes adult participants to sway with the motion of the room.
When the room moves towards the participant, creating an expansion pattern (Fig. 6.1b), the
participant takes this to signal self-motion forwards and corrects their perceived posture by sway-
ing backwards, with the motion of the room. Thus participants became ‘hooked like puppets’ to
the motion of the room. The development of this phenomenon in childhood can be characterized
by two processes: first, a decrease in the gain of the sway response (older children sway less in
proportion to the sway of the room) and second, an increase in the temporal coupling between
room motion and body sway.
Sway responses are present very early: sitting 5-month-olds sway in synchrony with the room
(Bertenthal et al. 1997). Strong sway responses are maintained across the transition from crawling
to walking (Lee and Aronson 1974). Indeed, responses to the moving room are produced to a
greater degree for infants with crawling or locomotor experience than for less experienced infants
of the same age (Higgins et al. 1996). Many new walkers respond so strongly to the stimulus that
they stumble or fall over. Thus, visual inputs provide stronger inputs to balance at this age than
they do in adults. This is still true of 3- to 4-year-old children (Wann et al. 1998), although the
falling responses have disappeared by this age. The extent of visually driven sway decreases rather
sharply between 4 and 6 years (Godoi and Barela 2008), indicating a transition away from visual
dominance of balance responses, and this transition continues through childhood. To under-
stand the exact developmental trajectory of sway gain, the literature would benefit from studies
that used the same paradigm over a very broad age range, including older children and teenagers.
However, available data demonstrate that for children aged 7–14 years, the gain of sway to a
swinging room is still higher than in adults (Sparto et al. 2006; Godoi and Barela 2008).
Alongside the very gradual weakening of responses to visual information, there is a gradual
increase in the temporal coupling of room movement and body sway through childhood.
Developments in coupling can be seen as early as 5–13 months (Bertenthal et al. 1997) and
continue through mid-childhood. Coherence between room and body sway reaches adult levels
by 10 years (Godoi and Barela 2008; Rinaldi et al. 2009). This may indicate improvements in
muscular control, but could also reflect the refinement of visuomotor mechanisms (Godoi and
Barela 2008; Rinaldi et al. 2009). This increased coupling may be a general feature of perceptual
development, since it is also found using a ‘haptic moving room’ (Barela et al. 2003) where the
relevant sensory information is haptic rather than visual.
While the swinging room technique allows for the systematic manipulation of visual input,
work using platform perturbations has investigated the contributions of visual, proprioceptive,
and vestibular inputs within one task (Shumway-Cook and Woollacott 1985; Woollacott et al.
1987). In conditions in which children stand with eyes closed on a platform which rotates under-
foot to dorsiflex the ankle joint, visual signals are not present and vestibular inputs do not initially
signal sway. Only proprioceptive information from the ankle joint indicates that a balance
response is necessary. Children aged 15–31 months do not sway significantly in this situation
(Woollacott et al. 1987). This result accords with those from the swinging room studies in show-
ing that, for very young children, visual and vestibular inputs are much more effective than pro-
prioceptive inputs in driving balance responses. Similar platform rotations do evoke significant
sway responses in children older than 4 years. At this age, where responses to moving visual scenes
are also decreasing, proprioceptive information alone becomes sufficient to induce sway
(Shumway-Cook and Woollacott 1985).
Further conditions in this study measured the relative contributions of vision and propriocep-
tion to balance, asking the children to stand still in four sensory conditions (Fig. 6.2). There was
no sudden platform perturbation, but different conditions removed reliable visual information,
reliable proprioceptive information, neither, or both. Visual information is simply removed by
140 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

(a) (b) (c) (d)

Reliable sources of Vestibular Vestibular Vestibular Vestibular


information present: Visual Visual
Proprioceptive Proprioceptive

Fig. 6.2 Measuring multisensory contributions to balance. While participants attempt to stand still
on a platform, visual information can be removed by closing the eyes. Ankle joint proprioception
can be made unreliable by rotating the platform to maintain the ankle at a fixed angle. This creates
four sensory conditions (a)–(d) during which sway was measured (Reproduced from The growth of
stability: postural control from a development perspective. Shumway-Cook, A., and Woolacott,
M.H., Journal of Motor Behaviour, 17, 131–147 © 1985 Taylor and Francis with permission
(Taylor & Francis, http://www.tandfonline.com).

having children close their eyes. Reliable proprioceptive information is removed by making the
platform ‘sway-referenced’; that is, the participant’s sway is monitored and the platform is rotated
accordingly, so that the ankle joint always remains at 90°. In this situation proprioceptive infor-
mation remains present, but is incongruent with other inputs. Reliable vestibular information
was available in all conditions.
Removing visual information caused greater increases in sway for 4- to 6-year-olds than for
older children or adults. However, removing reliable proprioceptive information caused even
greater increases in sway than removing vision. Removing both visual and proprioceptive infor-
mation caused dangerous amounts of sway, with the majority of 4- to 6-year-olds falling over.
These results suggest that, as for adults, both these sources of information are very important in
maintaining balance for children of this age. However, 4- to 6-year-olds are more destabilized
than adults under conditions of sensory conflict, for example when the proprioception system
signals no movement while the vision and vestibular systems signal movement.
Initial studies found that 7- to 10-year-olds’ balance during platform translations was
not greatly affected by removing vision (Shumway-Cook and Woollacott 1985). In these respects,
7- to 10-year-olds’ performance was very similar to that of adults tested on the same tasks. This
indicated weaker visual, and stronger proprioceptive contributions to balance than at younger ages.
However, recent work has suggested that fully mature balance responses may not occur until 12–15
years (Hirabayashi and Iwasaki 1995; Peterka and Black 1990; Peterson, Christou, and Rosengren
2006). Again, more detailed studies of balance in the late childhood and early teenage years would
be valuable in developing a fuller picture of multisensory balance control towards adulthood.
Together, the swinging room and platform perturbation experiments suggest an early reliance
on visual information. The end of this period and transition away from reliance on vision emerges
at around 5 years. Sway responses also achieve a tighter temporal coupling to sensory inputs dur-
ing mid-childhood. However, the longer developmental trajectories of these processes are rela-
tively under-researched.

6.2.2 Multisensory locomotor development


There are several distinct roles for sensory inputs to locomotion. First, as we have seen,
the processing of visual, proprioceptive, and vestibular inputs is crucial for balance, and
balance in turn underpins locomotion. Indeed, an important achievement for early walkers is
learning to differentiate visual information for walking from visual information for balance.
For example, balance can be challenged during walking by moving the room walls. In this
DEVELOPMENT OF MULTISENSORY BALANCE AND LOCOMOTION 141

situation, experienced walkers are highly influenced by the nature of the walking task
(presence/absence of obstacles) as well as by the visual information present for balance. In
contrast, more experienced walkers react to balance demands equally well in both navigation
conditions (Schmuckler and Gibson 1989).
A further role of sensory inputs in locomotion is allowing the walker to judge properties of the
environment, and the position of the limbs. This may be done in the planning phase of a move-
ment, or ‘online’, during the movement. The development of visual planning in locomotion has
been tested using judgment tasks, where children are verbally questioned about whether they
think an obstacle is passable or not, or where they are asked to choose a preferred path through
an environment. These visually guided ‘passability’ judgments depend on the perceived
skill required to cross an obstacle and the perceived size of the obstacle relative to one’s own
body dimensions (Adolph 1995; Kingsnorth and Schmuckler 2000; Schmuckler 1996). Visual
information is particularly useful in determining obstacle size, and by mid-childhood this body-
referenced size information is very tightly coupled to passability judgments (Heinrichs 1994;
Pufall and Dunbar 1992). These judgments become more refined during early childhood: for
example, older toddlers judge better than younger toddlers whether a barrier can be stepped over
(Schmuckler 1996) or a gap crossed (Zwart et al. 2005).
Haptic information can also contribute to locomotor judgments. For example, Adolph (1995)
compared children’s perceptions of which slopes they could walk on with their actual abilities.
14-month-olds made fewer attempts to descend slopes as they became steeper, but nonetheless
overestimated their abilities. Crucially, these infants explored the slopes before they descended.
The exploration was structured, consisting of a short look at the slope followed, if necessary,
by a long look, then exploring the slope by touch, and then trying alternative methods of descent
such as sliding down (Adolph 1997). Exploration was used to inform locomotor choices, with
most exploration at those slopes that were neither obviously easy to descend nor obviously
impossible, but required careful consideration. Thus haptic information was added to visual
information in locomotor decision-making at this young age. Sensory exploration persists at
older ages. In one task, children of 4.5 years were asked to judge whether they could stand upright
on a ramp (Klevberg and Anderson 2002). Their judgements were compared with their actual
competence. They could explore the ramp using either vision or touch (for touch, they felt the
ramp with a wooden pole). Children overestimated their ability to stand on the slopes. Like
adults, they judged more accurately in the visual exploration condition than in the haptic explo-
ration condition. Thus both visual and haptic information can be used for exploration of the
locomotor environment, though further work is needed to explore the balance of these inputs in
more detail.
Online feedback can help to correct and refine movements that are already underway. In loco-
motion, a wide range of sensory inputs can potentially provide feedback, including visual, prop-
rioceptive, vestibular, and tactile inputs. Using feedback to guide actions may be particularly
important in childhood. This is apparent in a stair-descent task. With vision available during the
whole stepping movement, three-year-olds are skilled at using visual information about stair size
to scale their movements to the size of step they are descending. However, this ability is signifi-
cantly impaired when visual feedback is removed during the step down (Cowie et al. 2010). This
suggests a reliance on visual feedback at 3 years which disappears by 4 years of age. In a more
complicated obstacle-avoidance task that required careful foot placement as well as movement
scaling, degrading vision impaired performance even for 7-year-old children (Berard and Vallis
2006). Thus, some existing data suggests that visual feedback may be crucial for complex aspects
of walking throughout childhood. A recent eye-tracking study with young children (Franchak
and Adolph 2010) confirms that during obstacle navigation children make more visual fixations
142 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

than adults. More work is needed to establish the dynamics of sensory inputs and motor outputs
during locomotion. Furthermore, it is clear that the extent of reliance on visual feedback depends
on task complexity, and one of the challenges for locomotor research is to formulate general
developmental principles regarding the use of sensory information in locomotor tasks.

6.2.3 Multisensory development of balance and


locomotion: conclusions
Although balance and locomotion have here been reviewed separately, in fact these two skills are
highly interdependent. While balance skills allow the child to walk safely, locomotion also refines
the control of balance. Infants who can sit upright respond better to balance perturbations than
those who cannot (Woollacott et al. 1987), and infants with crawling or locomotor experience
sway less than those without (Barela et al. 1999). Locomotor experience can also alter the sensory
control of balance: for example, experienced walkers are more responsive to specific aspects of
sensory stimuli, such as peripheral optic flow (Higgins et al. 1996).
We have reviewed a range of evidence which suggests important transitions from an early reli-
ance on vision for balance and locomotion, to a later pattern where information from across the
visual, proprioceptive, and vestibular systems is responded to in a more even way. What precisely
do we mean by ‘reliance’, and what are the mechanisms underlying such change? One model
of multisensory processing leading to action is that, during development, estimates from one
predominant sense are simply acted upon in disregard of estimates from the other responses.
It is easy to imagine infants’ responses to the visual swinging-room stimulus in terms of this sort
of mechanism. This can be contrasted with a second type of model in which a central integrator
collates and weights information from the independent senses to produce a final multisensory
estimate, which can then be acted upon (Clark and Yuille 1990; Ernst 2005; Körding and Wolpert
2006). In the concluding section of this chapter (Section 6.5), we will discuss models of informa-
tion integration in more detail, argue that they provide a useful framework for future lines of
investigation, and highlight the experimental manipulations that still need to be carried out in
order to evaluate them.

6.3 Development of multisensory orientation and navigation


Balance and locomotion provide a basis for spatial behaviour. To be adaptive, spatial behaviour
must be directed towards (or away from) significant objects and locations in the environment.
In this section we consider the development of the ability to orient and navigate to significant
objects and locations. Information about objects and locations commonly comes from multiple
senses (e.g. vision, audition, touch), which must be combined appropriately. Information about
one’s own movement in space also comes from multiple senses (e.g. vision, audition, propriocep-
tion, and the vestibular system). Many spatial tasks, such as picking up an object or crossing a
road, require an immediate response to environmental stimuli but no lasting record of them.
Other spatial tasks, such as finding the keys or finding the way home, also require memory. In
these tasks, multisensory spatial information must be stored in a form that is useful for later
retrieval.

6.3.1 Multisensory orienting


Orienting the eyes and head towards salient nearby stimuli is a basic spatial behaviour evident
from birth. Newborns orient to visual patterns (especially faces), but also to auditory and tactile
stimuli (Clifton et al. 1981; Fantz 1963; Moreau et al. 1978; Tan and Tan 1999; Wertheimer 1961).
Early visual orienting is driven by the retinotectal pathway to the superior colliculus in the
DEVELOPMENT OF MULTISENSORY ORIENTATION AND NAVIGATION 143

midbrain, which transmits sensory information for eye and head movements (Bronson 1974).
This subcortical system enables orienting to salient single targets, but not fine discrimination
or target selection. Cortical processing of visual stimuli, enabling increasingly fine discrimina-
tions (e.g. of orientation, motion, or binocular disparity), develops in the first months of
life (Atkinson 2002). Selective attention, enabling flexible selection of targets (for example, disen-
gagement from a central target in order to fixate a peripheral target) develops at 3–4 months
(Atkinson et al. 1992) and represents further cortical control over orienting behaviour (Braddick
et al. 1992).
Orienting responses alert infants to potentially interesting or hazardous objects, and enable
them to collect additional sensory information about them. Since the same objects can be sig-
nalled by information from multiple modalities (e.g. vision, audition, touch), orienting responses
need to be driven by multiple sensory inputs. In addition, as multisensory inputs are unlikely to
come from a similar spatial location at the same time purely by chance (see Körding et al. 2007),
such stimuli are likely to represent significant objects in the environment. The superior colliculus
supports multisensory orienting by integrating spatially localized visual, auditory, and somato-
sensory inputs within a common reference frame (see Chapter 11 by Laurienti and Hugenschmidt
and Chapter 14 by Wallace et al.). A proportion of multisensory neurons in the cat superior col-
liculus have ‘superadditive’ properties: they fire to auditory-only and visual-only stimuli localized
at the same point in space, but the firing rate for visual-and-auditory stimuli is greater than for
the sum of the two single stimuli (Meredith and Stein 1983; Meredith and Stein 1986; Wallace
et al. 1996). This property indicates that these neurons have a dedicated role in the processing of
stimuli providing both auditory and visual information at the same time. This multisensory neu-
ral organization is not present in newborn cats or monkeys (Wallace and Stein 1997, 2001) but
develops in a manner dependent on sensory experience (Chapter 14 by Wallace et al.).
Spatial co-location of auditory and visual events is detected by infants at least by 6 months of
age (Lawson 1980). To investigate the early development of multisensory integration for spatial
orienting, Neil et al. (2006) measured the latencies of 1- to 10-month-olds’ head and eye move-
ments towards auditory-only, visual-only, or auditory and visual targets located left or right of the
midline. In theory, having two stimuli available at once can enable a purely ‘statistical facilitation’
of reaction times, since there are two parallel opportunities to respond (Miller 1982; Raab 1962).
Given parallel processing but no interaction between cues, observers responding to whichever cue
is processed first on any given trial will show a predictable decrease in mean reaction time relative
to single cues. This purely ‘statistical’ improvement is described by the ‘race model’: (Miller 1982;
Raab 1962; see also Otto and Mamassian 2010), in which the two cues being processed in parallel
are in a ‘race’ to initiate a response. To show evidence for multisensory interaction, it is therefore
necessary to show speed advantages for two cues versus one that are greater than those predicted
by the race model. Neil and colleagues (2006) found latency gains given two cues rather than one
at all ages; however, it was only at 8 to 10 months that these exceeded the statistical facilitation
predicted by the race model. This indicates that mechanisms for multisensory facilitation
of audiovisual orienting emerge late in the first year of life in humans, consistent with the
experience-dependent postnatal development that has been reported in the superior colliculus in
monkeys and cats (Wallace and Stein 1997, 2001; see Chapter 14 by Wallace et al.).
Interestingly, when participants did not use eye or head movements to localize audiovisual
stimuli, but responded to them by pressing a button as quickly as possible (a ‘detection’ task),
adult-like multisensory improvements in latency were not evident until after 7 years (Barutchu
et al. 2009; Barutchu et al. 2010). It may be that these tasks do not tap into the early-developing
subcortically driven reflexive orienting system, but require cortical evidence integration and
selection for action (Romo et al. 2004). These may develop later. To investigate this possibility,
144 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

the latest research has started to compare manual and eye-movement responses to the same
audiovisual stimuli directly (Nardini et al. 2011).

6.3.2Orienting and reaching while taking own movement


into account
To localize targets that are perceptually available at the time of test, an egocentric coordinate
system is sufficient, and no memory is needed. However, significant objects in the environment
are not continually perceptually available, and may become hidden from view or silent to
the observer. This is particularly the case with mobile observers, who can change their perspective
on spatial layouts from one moment to the next. For infants beginning to crawl and walk
independently, a crucial challenge is to integrate multisensory information into spatial represen-
tations that are useful for relocating objects after a change of position. Infants who are passively
carried must likewise take their movements into account in order to keep track of nearby objects’
locations.1
After movement, a static object initially localized on the observer’s left can take up a new posi-
tion in egocentric space, such as behind or to the right. The original egocentric representation of
the object’s location is then no longer useful for retrieving it. This problem can be overcome in
two ways: by updating objects’ positions in egocentric coordinates while moving (‘spatial updat-
ing’) or by using external landmarks such as the walls of the room to encode and retrieve loca-
tions. For spatial updating, the observer’s own movement (‘self-motion’) needs to be taken into
account. In human adults, both self-motion and landmarks play major roles in spatial orientation
and navigation (for reviews see Burgess 2006; Wang and Spelke 2002). Within the broad category
of ‘landmarks’ key questions concern the extent to which these help to organize space into a geo-
centric ‘cognitive map’, and the extent to which they are used for simpler strategies such as recog-
nition of familiar views (Burgess 2006; Wang and Spelke 2002). For humans, useful landmarks
are overwhelmingly visual.2 Information about self-motion, however, comes from many sensory
sources: from vision, via optic flow (Gibson 1979; see Section 6.2 above), as well as from vestibu-
lar, kinesthetic, and proprioceptive inputs (Howard and Templeton 1966; MacNeilage et al. 2007;
Zupan, et al. 2002).
Until recently, little was known about the neural mechanisms underlying integration of multi-
ple self-motion inputs for spatial behaviour. Important advances have been in showing how
macaque MST neurons integrate visual and vestibular information to compute the animal’s
direction of movement (Gu et al. 2008; Morgan et al. 2008). Self-motion information must fur-
ther be integrated with visual landmarks. A range of cell types in the mammalian medial temporal
lobe (‘head direction cells’, ‘grid cells’, and hippocampal ‘place cells’) encode both self-motion
and landmark information and are likely to be the basis for their integration (for reviews see
Burgess 2008; McNaughton et al. 2006; Moser et al. 2008; Taube 2007).
The earliest situations in which human infants might take their own motion into account
to localize objects are before independent movement, when carried passively. Infants’ spatial
orienting in these situations has been investigated in studies using ‘peekaboo’ tasks. In a study
by Acredolo (1978), 6- to 16- month-olds sat in a parent’s lap at a table inside a square enclosure

1 A related problem is keeping track of own limb positions in order to enable their accurate localization, and
to support accurate reaching towards objects (Chapter 5 by A.J. Bremner et al. and Chapter 13 by Röder
et al.; A.J. Bremner et al. 2008a; 2008b; Pagel et al. 2009).
2 If landmarks do play a role in building up a ‘cognitive map’ of space, an interesting question is how and
whether blind humans do this (Klatzky et al. 1990; Loomis et al. 1993).
DEVELOPMENT OF MULTISENSORY ORIENTATION AND NAVIGATION 145

with windows to the left and right. In a training phase, infants learned that whenever a buzzer
sounded, an experimenter appeared at one of the windows. This was always the same window for
each child. Once the infant had learnt to orient to the correct window when the buzzer sounded,
the testing stage began. The parent and infant moved to the opposite side of the table, a manipu-
lation that swapped the egocentric positions of the ‘left’ and ‘right’ windows. When the buzzer
sounded, the experimenters recorded whether the infant looked to the correct window, showing
that they had correctly processed their change of position. With no distinctive landmarks distin-
guishing the windows, solving the task depended on the use of self-motion information, e.g. from
optic flow and the vestibular system. Infants up to 11 months old failed to use such information
and oriented to the incorrect window.
Other studies have confirmed the poor abilities of infants in the first year to update their
direction of orienting using only self-motion information (Acredolo and Evans 1980; Keating
et al. 1986; McKenzie et al. 1984; Rieser 1979). However, when direct landmarks are added to
distinguish between locations, infants in the first year orient correctly (e.g. at 6 months in Rieser
1979, and at 8–11 months in Acredolo and Evans 1980; Keating et al. 1986, Lew et al. 2000 and
McKenzie et al. 1984). Similarly, when searching for objects hidden in the left or right container
on a table-top, infants in the first year moved to the opposite side of the table tend not to take
their own movement into account unless they are given useful landmarks in the form of colour
cues (J.G. Bremner 1978; J.G. Bremner and Bryant 1977).
These results suggest that in order to maintain the locations of nearby objects following their
own movement, infants in the first year found added visual landmarks significantly more useful
than self-motion information alone. As stated above, self-motion information comes from a
number of sensory inputs. These may mature at different rates, and interactions between them to
guide self-motion perception may also mature unevenly, as is the case with balance (see Section
6.2, above). In ‘peekaboo’ and reaching tasks, infants have optic flow and vestibular cues to
changes of position. Older infants and children carrying out the movement themselves can also
use kinesthetic and motor efference information about the movement of their body. Children
perform better after moving actively than after being carried (Acredolo et al. 1984), and better
when walking around to the opposite viewpoint than when the opposite viewpoint is presented
by rotating the spatial layout (J.G. Bremner 1978; J.G. Bremner and Bryant 1977; Schmuckler and
Tsang 1997 ). This indicates that kinesthetic and motor-efference information is useful.
Performance is also better in the light than in the dark, consistent with use of optic flow
(Schmuckler and Tsang-Tong 2000). There is, however, as yet little direct evidence for whether,
when, and how infants integrate these multiple potential information sources to orient them-
selves in space. J.G. Bremner et al. (2011) separated the contributions of optic flow and vestibular
cues to 6- to 12-month olds’ abilities to orient correctly to a target after a change of position.
Infants were seated in a rotatable cylindrical surround, which allowed manipulation of visual and
vestibular information separately and in conflicting combinations. The results showed a U-shaped
developmental pattern in which 6-month-olds, 12-month-olds, and young adults responded
on the basis of optic flow information, whereas 9-month-olds responded predominantly on the
basis of vestibular information. This indicates that children’s reliance on visual versus vestibular
information for locating targets undergoes marked changes in the first year.

6.3.3 Navigation and spatial recall


The natural extension of early orienting and reaching responses towards nearby objects is to flexible
navigation and spatial recall in extended environments. This depends on observers being able to
encode locations using frames of reference that will be useful for their later retrieval, and to bind
individual objects (‘what’) to their locations (‘where’). As described above, mature navigation
146 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

and recall depend critically both on visual information about landmarks and multisensory infor-
mation about self-motion.
Spatial studies with children of walking age have typically used search tasks in which the child
sees a toy hidden, and must then find it given a specific set of cues or after a specific manipulation.
Evidence for the individual development of landmark-based and self-motion-based spatial recall
has come from tasks that separate these information sources completely. Following spatial tasks
devised for rodents by Cheng and Gallistel (Cheng 1986; Gallistel 1990), Hermer and Spelke
(1994, 1996) used ‘disorientation’ to eliminate self-motion information and so test spatial recall
based only on visual landmarks. In their task, 18- to 24-month-olds saw a toy hidden in one of
four identical containers in the four corners of a rectangular room. Useful landmarks were the
‘geometry’ of the room (i.e. whether the target box had a longer wall on its left or its right), and
in some conditions, different wall colours. If the child was simply turned to face a random direc-
tion and then allowed to search, they could relocate the toy using not only these visual landmarks
but also an egocentric representation. Such a representation might be an initial vector (e.g.
‘on my left’) that had been updated with self-motion to take the child’s recent turn into account.
To eliminate this information source, children were disoriented by repeated turning with eyes
closed before being allowed to search. The resulting searches therefore reflected only the accuracy
of visual landmarks, and not of egocentric representations updated with self-motion.
A large body of research using this technique has demonstrated that children as young as 18 to
24 months can recall locations using only indirect visual landmarks, although their abilities to use
some kinds of indirect landmark (e.g. ‘geometry’, or the shape of the enclosure layout) can be
markedly better than others (Huttenlocher and Lourenco 2007; Learmonth et al. 2002, 2008; Lee
et al. 2006; Lee and Spelke 2008; Nardini et al. 2009; Newcombe et al. 2010; for reviews see Cheng
and Newcombe 2005; Twyman and Newcombe 2010).
The converse manipulation used to test self-motion alone, without use of visual landmarks, is
blindfolding participants. Strictly, this a test for use of non-visual (e.g. vestibular) self-motion
cues only, as removing vision also removes the optic flow cue to self-motion.3 A typical non-
visual ‘self-motion-only’ spatial task is one in which the participant is walked along the two lines
of an ‘L’ and then asked to return directly to the origin, i.e. to complete the triangle (Foo et al.
2005; Loomis et al. 1993). Such a task depends on keeping track of one’s own directions and
angles of movement since leaving the origin. There have been relatively few studies quantifying
development of self-motion-only navigation. Rider and Rieser (1988) found that 2-year-olds
could localize targets after blindfolded walking along routes with two 90° turns, as could 4-year-
olds, who were more accurate. In another study, four-year-olds’ errors, like adults’, increased
with numbers of turns and numbers of targets (Rieser and Rider 1991). While 4-year-olds were
inaccurate compared with adults, they still performed better than chance given up to three turns
and five targets. These results suggest that like the landmark-only based localization of targets,
self-motion-only based localization emerges quite early in development. However, developmen-
tal trajectories for different self-motion inputs (e.g. vestibular versus kinesthetic), or their inte-
gration, have not yet been studied.
The usual case, of course, is one in which both landmarks and self-motion are available. For
example, in order to relocate an object that we have recently put down elsewhere in the room we
can use visual landmarks, as well as a self-motion-updated egocentric estimate of where the object

3 In a study with adults, Riecke et al. (2002) separated the dual roles of vision in providing visual landmarks
and optic flow by having subjects navigate in virtual reality through a field of visual ‘blobs’ that provided
texture for optic flow but could not be used as landmarks.
DEVELOPMENT OF MULTISENSORY ORIENTATION AND NAVIGATION 147

now is relative to us. A key question is how these systems interact to guide spatial behaviour. For
mammals, including humans, landmarks are usually more useful and exert stronger control
over behaviour (Etienne et al. 1996; Foo et al. 2005). Landmark and self-motion systems can be
seen as complementary, in the sense that self-motion can help when landmarks are not available.
On this model, self-motion is a ‘back-up’ system for landmarks. An alternative model is one
in which self-motion and landmarks are integrated to improve accuracy. That is, neither system
provides a fully reliable guide to location, but by integrating them estimates of location can be
made more reliable. This second view is in line with Bayesian models of multisensory and spatial
behaviour (Cheng et al. 2007; Clark and Yuille 1990; Ernst 2005; Körding and Wolpert 2006; see
below and Section 6.5).
A number of studies have examined the development of spatial recall in situations allowing use
of both self-motion and landmarks. In studies by Huttenlocher, Newcombe, and colleagues
(Huttenlocher et al. 1994; Newcombe et al. 1998), children searched for toys after seeing them
buried in a long sandbox. The 1994 study established that 16- to 24-month olds used the edges
and overall shape of the sandbox as visual ‘landmarks’ when retrieving toys while remaining on
the same side of the box. In the 1998 study, 16- to 36-month-olds walked around the box to
retrieve toys from the opposite side. From the age of 22 months, having additional distant land-
marks in the room visible improved the accuracy of retrieval from the opposite side. The role of
these cues may have been to provide distant landmarks situating the box and locations in a wider
reference frame, and to improve spatial updating with self-motion by highlighting the change in
perspective brought about by walking to the opposite side. In either case, these results are consist-
ent with use of both visual landmarks and multisensory self-motion inputs to encode spatial
locations in the first years of life.
Simons and Wang (1998; Wang and Simons 1999) showed that human adults’ recall for spatial
layouts from new viewpoints is more accurate when they walk to the new viewpoint than when
they stay in one place while the layout is rotated. With rotation of the layout, use of visual land-
marks alone allows adult observers to understand the change of perspective and to relocate
objects with a certain level of accuracy. However, this accuracy is enhanced when subjects walk
to the new location, meaning that self-motion information is also available (see also Burgess et al.
2004).
To study the development of coding based on both visual landmarks and self-motion,
Nardini et al. (2006) tested 3- to 6-year olds’ recall for the location of a toy hidden under an
array of cups surrounded by small local landmarks. Vision was always available. However,
on some trials, recall took place from the same viewpoint, while on others recall was from a dif-
ferent viewpoint. Viewpoint changes were produced either by the child walking (so gaining self-
motion information about the viewpoint change) or by rotation of the array (gaining no
self-motion information). At all ages recall was most accurate without any viewpoint change.
With viewpoint changes produced by walking, the youngest children (3-year-olds) were above
chance and quite accurate at relocating objects. However, with viewpoint changes produced by
array rotation, only 5- and 6-year-olds retrieved objects at rates above chance. This indicates that
self-motion information is a major component of early spatial competence, and can dominate
over visual landmarks: without useful self-motion information, and so using only visual land-
marks, 3- and 4-year olds could not correctly process the viewpoint change. Note that when diso-
riented and so unable to use self-motion, children from 18–24 months are competent at using
some kinds of visual landmarks to recall locations (Hermer and Spelke 1994, 1996). The present
case is more complex in that instead of being eliminated as by disorientation, when the array was
rotated self-motion information remained available, but indicated the wrong location. This study
therefore set up a conflict between visual landmarks and self-motion, and found that before
148 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

5 years of age children preferred to rely on self-motion, even when it was inappropriate for
the task.
In a subsequent study, Nardini et al. (2008) used a different method to measure how self-motion
and landmarks interact for spatial recall. The cues were not placed in conflict, as by array rotation
manipulations, but were separated completely in ‘landmark-only’ and ‘self-motion-only’ condi-
tions. This enabled a more formal test for their integration and comparison with a Bayesian
‘ideal observer’ model (Cheng et al. 2007; Clark and Yuille 1990; Ernst 2005; Körding and Wolpert
2006). As in earlier studies, ‘landmark-only’ performance was measured using disorientation,
and ‘self-motion-only’ performance was measured using walking in darkness. However, these
manipulations were now carried out within the same spatial task, and were compared with a
combined ‘self-motion and landmarks’ condition within the same participants.
Four- to eight-year-olds were tested in a dark room with distant glowing landmarks, and a set
of glowing objects on the floor (Fig. 6.3A). The task was to pick up the objects on the floor in
a sequence, and then to return the first object directly to its original place after a short delay. In
‘visual landmarks only’ and ‘(non-visual) self-motion only’ conditions, participants had to return
the object after disorientation and in darkness respectively. A ‘self-motion-plus-landmarks’
condition tested the normal case in which participants were not disoriented and landmarks
remained available. Bayesian integration of cues predicts greater accuracy (reduced variance)
given both cues together than given either single cue alone. In a ‘conflict’ condition the landmarks
were rotated by a small amount before participants responded. They did not detect this conflict,
but it provided information about the degree to which participants followed one or the other of
the two information sources.
Adults were more accurate in their spatial estimates given both self-motion and landmarks, in
line with optimal (ideal observer model) performance. Given two cues rather than one, adults’
searches were on average closer to the target (Fig. 6.3B) and less variable (Fig. 6.3C).4 By contrast,
4- to 5-year-olds and 7- to 8-year-olds did not perform better given two cues together than the
best single cue (Fig. 6.3B, C). Modelling of responses on the ‘conflict’ condition indicated that
while children can use both self-motion and landmarks to navigate, they did not integrate these
but followed one or the other on any trial. Thus, while even very young children use both self-
motion and landmarks for spatial coding, they seem not to integrate these to improve accuracy
until after 8 years of age. Proposed neural substrates for integration of self-motion and landmarks
for spatial tasks are head place cells, direction cells, and grid cells in the hippocampus and medial
temporal lobe (for reviews, see Burgess 2008; McNaughton et al. 2006; Moser et al. 2008; Taube
2007). Late development of spatial information may be reflected in development of these medial
temporal networks. Recent studies with rodents have found these cell types to be functional very
early in life (Langston et al. 2010; Wills et al. 2010), although the development of their abilities to
integrate multisensory information is still to be investigated.
The late development of multisensory integration for reducing uncertainty in spatial tasks
parallels that recently reported for visual–tactile judgments of size and shape (Gori et al. 2008)
and for use of multiple visual depth cues (Nardini et al. 2010). It is possible that while mature
sensory systems are optimized for reducing uncertainty, developing sensory systems are opti-
mized for other goals. One benefit of not integrating multisensory cues could be that keeping cues
separate makes it possible to detect conflicts between them. These conflicts may provide the error
signals needed to learn correspondences between cues, and to recalibrate them while the body is
growing (Gori et al. 2010).

4 The major predicted benefit for integration of cues is a reduction in the variance of responses—see
Section 6.5.
DEVELOPMENT OF MULTISENSORY ORIENTATION AND NAVIGATION 149

B
100 SM (self-motion)
LM (landmarks)
Mean RMSE (cm)

80 SM+LM

60
40
20
0
4–5 yr. 7–8 yr. Adult
Group
C

100 SM (self-motion)
LM (landmarks)
Mean SD (cm)

80 SM+LM

60
40
20
0
4–5 yr. 7–8 yr. Adult
Group
Fig. 6.3 (A) Layout for the Nardini et al. (2008) spatial task. (B) Mean root mean square error
(RMSE) of responses under conditions providing SM (self-motion), LM (landmarks), or SM + LM
(both kinds of information). This measure reflects distances between each search and the correct
location. (C) Standard deviation (SD) of responses. This measure reflects the variability (dispersion)
of searches. Bayesian integration of cues predicts reduced variance (and hence also reduced SD,
which is the square root of the variance), given multiple cues. Reprinted from Current Biology,
18 (9), Marko Nardini, Peter Jones, Rachael Bedford, and Oliver Braddick, Development of Cue
Integration in Human Navigation, pp. 689–693, Copyright (2008), with permission from Elsevier.
Reproduced in colour in the colour plate section.
150 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

6.4 Summary
We have reviewed the development of sensory integration for the skills underpinning human
spatial behaviour: balance, locomotion, spatial orientation, and navigation. Even very young
children are capable of using sensory information in order to control balance and locomotion.
Visual information seems particularly heavily weighted in infancy and early childhood. Around
5–6 years a period of change begins, where visual information is gradually down-weighted in
favour of somatosensory and vestibular inputs. However, on the available evidence, balance and
locomotion are not coupled in an adult-like fashion to either somatosensory or visual information
until adolescence.
Early spatial orienting behaviours show multisensory facilitation late in the first year, consistent
with experience-dependent postnatal development of the superior colliculus. Sensory integration
for more complex orientation and navigation tasks, including localizing objects again after own
movement, shows a much more extended developmental trajectory. Both self-motion informa-
tion and landmarks are used for spatial orientation and navigation from an early age, but they
may not be integrated in an adult-like fashion until after 8 years of age. Neural bases for these
tasks are increasingly becoming understood in animal models, and mathematical models are
increasingly allowing sensory integration during development to be precisely quantified. A major
challenge for the future is to account for developmental change by linking these mathematical
and neural levels of analysis.

6.5 Bayesian models


We have proposed at several points in this chapter that Bayesian models of information integra-
tion can provide a useful framework for understanding developmental balance, locomotion, and
navigation phenomena. Here we briefly describe the models and the prospects for further tests of
their application to multisensory developmental research in the future.
Bayesian models address the problem that individual sensory estimates are inevitably variable
(uncertain) because of random noise. ‘Noise’ is evident in an auditory environment with back-
ground noise, or in a murky visual environment. However, even in ideal environmental condi-
tions, all sensory systems contain some variability (uncertainty) due to their own limited resolution
and noisy transmission of information by neurons. Bayesian models provide a rule by which
the multiple, noisy, sensory estimates we usually have available in parallel can be combined to
provide a single estimate that minimizes noise or uncertainty (Cheng et al. 2007; Clark and Yuille
1990; Ernst 2005; Körding and Wolpert 2006).
The rule for combining estimates to reduce uncertainty is simple: if multiple sensory estimates
are available (e.g. visual and haptic cues to object size) and the noise in each of these is independ-
ent, then taking a weighted average of the estimates can produce a final estimate that is less
variable (more reliable) than the component estimates. The optimal weighting is one in which the
different estimates are weighted in proportion to their reliabilities. A cue that is three times more
variable than another is therefore given three times less weight. Recent multisensory studies with
adults have shown optimal multisensory integration in many situations (e.g. vision and touch,
Ernst and Banks 2002; vision and audition, Alais and Burr 2004; visual and vestibular cues to
direction of heading, Fetsch et al. 2009; reviews, Ernst 2005; Körding and Wolpert 2006).
These findings provide an important new perspective on human multisensory processing
in demonstrating that, in adults at least, multisensory integration is very flexible. Adults are
excellent at dynamically ‘reweighting’ their reliance on different information sources when these
change in reliability. Thus, although vision may usually dominate for a particular task, if visual
information becomes uncertain (e.g. at night or in fog) it will be given less weighting relative to
BAYESIAN MODELS 151

the other available information (Alais and Burr 2004). The conclusion from this and other
studies is that phenomena such as ‘visual dominance’ may be explained simply by the differing
reliabilities of the senses for particular tasks (Ernst 2005 ; Körding and Wolpert 2006 ).
Task-specific explanations for cue weighting: e.g. that vision dominates over audition for spatial
localization (Pick et al. 1969) can then be subsumed by a single, more general explanation: vision
and audition and other sensory inputs are weighted according to their reliabilities (Alais and
Burr 2004).
We have described a range of developmental phenomena in which visual dominance changes
with age. For example, balance is much more influenced by vision in young children than it is in
older children or adults (Godoi and Barela 2008; Shumway-Cook and Woollacott 1985; Wann
et al. 1998; Woollacott et al. 1987). We can state two possible kinds of Bayesian explanation for
this developmental pattern.
Possibility 1: children and adults integrate multisensory information in the same Bayesian/‘optimal’
manner. In this scenario, developmental changes in ‘dominance’ might simply reflect develop-
mental changes in the reliabilities of the underlying estimates. If the development of different
senses were uneven (e.g. if visual information for balance was accurate at an earlier age than
vestibular information) then different ages would show different weightings even though they
were following the same integration rules. A prediction from this account is that adults whose
cue reliabilities were manipulated (e.g. by selectively adding noise) to be the same as children’s
would show the same behaviour as children.
The issue of dynamic sensory reweighting in children’s balance control was first raised by
Forssberg and Nashner (1982), and several studies since have investigated the swinging room
experiment from this perspective, changing the distance between the swinging walls and the
observer to try and detect sensory reweighting as the far walls become a less reliable source of
information about relative motion (Dijkstra et al. 1994; Godoi and Barela 2008). Studies have also
recorded developmental changes in weighting of multisensory information for posture control in
response to differing amplitudes of visual versus somatosensory perturbations (Bair et al. 2007).
However, to be able to test whether patterns of reweighting in either children or adults are in line
with Bayesian predictions, studies will need to measure the sensory reliabilities of visual and other
cues to balance, which remains challenging (see below).
The alternative is Possibility 2: children and adults do not integrate multisensory information in
the same Bayesian/‘optimal’ manner. For example, adults might take a weighted average of esti-
mates in proportion to their reliabilities, while children might do something else. Children could
rely on a single estimate even when other reliable estimates are available, or, they could integrate
estimates using a sub-optimal weighting, or they could be less effective than adults in excluding
estimates that are highly discrepant with others from the integration process.5

5 The simple rule to integrate all available estimates is inappropriate for situations in which one estimate dif-
fers drastically from the others. ‘Robust cue integration’ is an extension to the Bayesian integration frame-
work stating that since large outliers are highly unlikely to be due to normal sensory noise, they should be
presumed to be erroneous and excluded from integration with other estimates (Knill 2007; Körding et al.
2007; Landy et al. 1995). ‘Robust cue integration’ would give observers in the swinging room a basis for
disregarding visual information when it is highly discrepant with vestibular and proprioceptive informa-
tion. The crucial point to note in comparing children and adults is that whether or not estimates can be
detected as ‘highly discrepant’ depends not only on the average difference between them, but also on their
variabilities. Thus, given children and adults subjected to the same experimental manipulation, if adults’
estimates of own posture (via all senses) are much less variable than children’s, then adults might have a
basis for detecting that the visual estimate is discrepant compared with the others, whereas children might
152 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

Most of the studies that have been reviewed cannot separate these two accounts, as changes
with age in relative reliance on different cues are consistent with both. Possibility 1—that children
and adults process information in the same way—provides the simplest account in that it needs
to posit fewer kinds of developmental changes. This account therefore should not be ruled out for
specific tasks until the relevant experimental manipulations have been tested. Interestingly, in the
small number of studies that have examined the development of Bayesian integration explicitly
(Gori et al. 2008; Nardini et al. 2008; Nardini et al. 2010), children were shown not to integrate
estimates in an adult-like way until after 8 years—i.e. Possibility 1 could be rejected. These studies
used visual, manual, and locomotor tasks involving spatial or psychophysical decisions. Whether
the same would be true for more elementary sensory-motor behaviours such as balance is a major
question for future studies. A challenge for developmental studies is the time-consuming nature
of testing Bayesian models of multisensory integration, since they require measurement of
response variability in both single-cue and combined-cue conditions. One way to address this is
use of psychophysical staircase procedures (e.g. Kaernbach 1991; Kontsevich and Tyler 1999) to
obtain more rapid estimates of variability (Nardini et al. 2010). A second challenge is measuring
perceptual variability for elementary sensory-motor behaviours such as balance that lead to pos-
tural changes but do not lend themselves to explicit judgments. For this, new approaches need to
be developed for inferring variability from spontaneous behaviours. This is challenging but pos-
sible, as a recent study has shown by measuring integration of depth cues using only spontaneous
eye movements (Wismeijer et al. 2010).
To sum up, many developmental changes in use of multiple cues for balance, locomotion, and
navigation in childhood could be explained within a Bayesian framework. However, to assess
whether developmental changes correspond either to adult-like integration with a different
inputs in terms of cue reliabilities, or non-adult like integration or lack of integration, more
detailed studies need to be carried out quantifying the variability of the underlying estimates. The
few studies so far that have tested this framework with children suggest that they do not integrate
estimates in an adult like manner until after 8 years, across senses for spatial localization or object
perception (Gori et al. 2008; Nardini et al. 2008), or within vision for three-dimensional shape
discrimination (Nardini et al. 2010).

6.5.1 Other models


We have reviewed development of the sensory processes underlying adaptive spatial behaviour.
While children’s sensory and motor systems are immature, they are also in some senses optimized
for the ecology of their environments. For example, young infants do not need high visual acuity
or accurate spatial localization. Bayesian models stress optimization for accuracy, in terms of the
reduction of sensory uncertainty. However, maximal accuracy is not the most important goal for
many spatial tasks that infants and children face. For developing sensory-motor systems there
may be other goals more important than maximizing accuracy, such as responding rapidly
(Barutchu et al. 2009; Neil et al. 2006) or detecting inter-sensory conflicts (Gori et al. 2010;
Nardini et al. 2010). Therefore many other kinds of model remain to be developed, in which
‘optimality’ is quantified based on different goals or properties. It may be that these models will

not. In this situation, if both age groups were following ‘robust cue integration’ with the same integration
rules, adults would reject vision but children would not. To show that adults and children really differ in
integration behaviour it is necessary to exclude this possibility—for example, by showing that even when
noise is added to adults’ estimates in order to bring them to children’s level, they still behave differently.
REFERENCES 153

provide the best description of how developing perceptual and motor systems are optimized for
spatial behaviour.

Acknowledgements
MN would like to acknowledge support from the James S. McDonnell Foundation 21st Century
Science Scholar in Understanding Human Cognition Program, the National Institute of Health
Research BMRC for Ophthalmology, and United Kingdom Economic and Social Research
Council Grant RES-062–23-0819. DC would like to acknowledge support from the European
Research Council under the European Community’s Seventh Framework Programme (FP7/2007–
2013) / ERC Grant agreement no. 241242.

References
Acredolo, L.P. (1978). Development of spatial orientation in infancy. Developmental Psychology, 14, 224–34.
Acredolo, L.P., and Evans, D. (1980). Developmental changes in the effects of landmarks on infant spatial
behavior. Developmental Psychology, 16, 312–318.
Acredolo, L.P., Adams, A., and Goodwyn, S.W. (1984). The role of self-produced movement and visual
tracking in infant spatial orientation. Journal of Experimental Child Psychology, 38, 312–27.
Adolph, K.E. (1995). A psychophysical assessment of toddlers’ ability to cope with slopes. Journal of
Experimental Psychology: Human Perception and Performance, 21, 734–50.
Adolph, K.E. (1997). Learning in the development of infant locomotion. Monographs of the Society for
Research in Child Development, 62, I–158.
Alais, D., and Burr, D. (2004). The ventriloquist effect results from near optimal crossmodal integration.
Current Biology, 14, 257–62.
Angelaki, D.E., Klier, E.M., and Snyder, L.H. (2009). A vestibular sensation: probabilistic approaches to
spatial perception. Neuron, 64, 448–61.
Atkinson, J. (2002). The Developing Visual Brain. Oxford University Press, Oxford.
Atkinson, J., Hood, B., Wattam-Bell, J., and Braddick, O. (1992). Changes in infants’ ability to switch visual
attention in the first three months of life. Perception, 21, 643–53.
Bair, W.N., Kiemel, T., Jeka, J.J., and Clark, J.E. (2007). Development of multisensory reweighting for
posture control in children. Experimental Brain Research, 183, 435–46.
Barela, J.A., Jeka, J.J., and Clark, J.E. (1999). The use of somatosensory information during the acquisition
of independent upright stance. Infant Behavior and Development, 22, 87–102.
Barela, J.A., Jeka, J.J., and Clark, J.E. (2003). Postural control in children. Coupling to dynamic
somatosensory information. Experimental Brain Research, 150, 434–42.
Barutchu, A., Crewther, D.P., and Crewther, S.G. (2009). The race that precedes coactivation: development
of multisensory facilitation in children. Developmental Science, 12, 464–73.
Barutchu, A., Danaher, J., Crewther, S.G., Innes-Brown, H., Shivdasani, M.N., and Paolini, A.G. (2010).
Audiovisual integration in noise by children and adults. Journal of Experimental Child Psychology,
105, 38–50.
Berard, J.R., and Vallis, L.A. (2006). Characteristics of single and double obstacle avoidance strategies: a
comparison between adults and children. Experimental Brain Research, 175, 21–31.
Bertenthal, B.I., Rose, J.L., and Bai, D.L. (1997). Perception-action coupling in the development of visual
control of posture. Journal of Experimental Psychology: Human Perception and Performance, 23, 1631–43.
Billington, J., Field, D.T., Wilkie, R.M., and Wann, J.P. (2010). An fMRI study of parietal cortex
involvement in the visual guidance of locomotion. Journal of Experimental Psychology: Human
Perception and Performance, 36, 1495–1507.
Braddick, O., Atkinson, J., Hood, B., Harkness, W., Jackson, G., and Vargha-Khadem, F. (1992). Possible
blindsight in infants lacking one cerebral hemisphere. Nature, 360, 461–63.
154 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

Bremner, A.J., Holmes, N.P., and Spence, C. (2008). Infants lost in (peripersonal) space? Trends in
Cognitive Sciences, 12, 298–305.
Bremner, A.J., Mareschal, D., Lloyd-Fox, S., and Spence, C. (2008). Spatial localization of touch in the first
year of life: early influence of a visual spatial code and the development of remapping across changes in
limb position. Journal of Experimental Psychology: General, 137, 149–62.
Bremner, J.G. (1978). Egocentric versus allocentric spatial coding in 9-month-old infants: factors
influencing the choice of code. Developmental Psychology, 14, 346–55.
Bremner, J.G., and Bryant, P. (1977). Place versus response as the basis of spatial errors made by young
infants. Journal of Experimental Child Psychology, 23, 162–71.
Bremner, J.G., Hatton, F., Foster, K.A., and Mason, U. (2011). The contribution of visual and vestibular
information to spatial orientation by 6- to 14-month-old infants and adults. Developmental Science 14,
1033–1045
Bronson, G. (1974). The postnatal growth of visual capacity. Child Development, 45, 873–90.
Brosseau-Lachaine, O., Casanova, C., and Faubert, J. (2008). Infant sensitivity to radial optic flow fields
during the first months of life. Journal of Vision, 8, 1–14.
Burgess, N. (2006). Spatial memory: how egocentric and allocentric combine. Trends in Cognitive Sciences,
10, 551–57.
Burgess, N. (2008). Spatial cognition and the brain. Annals of the New York Academy of Sciences, 1124,
77–97.
Burgess, N., Spiers, H.J., and Paleologou, E. (2004). Orientational manoeuvres in the dark: dissociating
allocentric and egocentric influences on spatial memory. Cognition, 94, 149–66.
Cheng, K. (1986). A purely geometric module in the rat’s spatial representation. Cognition, 23, 149–78.
Cheng, K., and Newcombe, N.S. (2005). Is there a geometric module for spatial orientation? Squaring
theory and evidence. Psychonomic Bulletin and Review, 12, 1–23.
Cheng, K., Shettleworth, S.J., Huttenlocher, J., and Rieser, J.J. (2007). Bayesian integration of spatial
information. Psychological Bulletin, 133, 625–37.
Clark, J.J., and Yuille, A.L. (1990). Data Fusion for Sensory Information Systems. Kluwer Academic, Boston, MA.
Clifton, R.K., Morrongiello, B.A., Kulig, J.W., and Dowd, J.M. (1981). Newborns’ orientation toward
sound: possible implications for cortical development. Child Development, 52, 833–38.
Cowie, D., Atkinson, J., and Braddick, O. (2010). Development of visual control in stepping down.
Experimental Brain Research, 202, 181–88.
Day, B.L., and Fitzpatrick, R.C. (2005). The vestibular system. Current Biology, 15, R583–86.
Dijkstra, T.M., Schoner, G., and Gielen, C.C. (1994). Temporal stability of the action-perception cycle for
postural control in a moving visual environment. Experimental Brain Research, 97, 477–86.
Ernst, M.O. (2005). A Bayesian view on multimodal cue integration. In Human body perception from the
inside out (eds. G. Knoblich, I.M. Thornton, M. Grosjean, and M. Shiffrar), pp. 105–34. Oxford
University Press, New York.
Ernst, M.O., and Banks, M.S. (2002). Humans integrate visual and haptic information in a statistically
optimal fashion. Nature, 415, 429–33.
Etienne, A.S., Maurer, R., and Seguinot, V. (1996). Path integration in mammals and its interaction with
visual landmarks. The Journal of Experimental Biology, 199, 201–209.
Fantz, R.L. (1963). Pattern vision in newborn infants. Science, 140, 296–97.
Fetsch, C.R., Turner, A.H., Deangelis, G.C., and Angelaki, D.E. (2009). Dynamic reweighting of visual and
vestibular cues during self-motion perception. The Journal of Neuroscience, 29, 15601–15612.
Foo, P., Warren, W.H., Duchon, A., and Tarr, M.J. (2005). Do humans integrate routes into a cognitive
map? Map- versus landmark-based navigation of novel shortcuts. Journal of Experimental Psychology:
Learning, Memory and Cognition, 31, 195–215.
Forssberg, H., and Nashner, L.M. (1982). Ontogenetic development of postural control in man: adaptation
to altered support and visual conditions during stance. Journal of Neuroscience, 2, 545–52.
REFERENCES 155

Franchak, J.M., and Adolph, K.E. (2010). Visually guided navigation: head-mounted eye-tracking of
natural locomotion in children and adults. Vision Research, 50, 2766–74.
Gallistel, C.R. (1990). The organization of learning. MIT Press, Cambridge, MA.
Gibson, J.J. (1979). The ecological approach to visual perception. Houghton Mifflin, Boston, MA.
Godoi, D., and Barela, J.A. (2008). Body sway and sensory motor coupling adaptation in children: effects of
distance manipulation. Developmental Psychobiology, 50, 77–87.
Gori, M., Del Viva, M., Sandini, G., and Burr, D. (2008). Young children do not integrate visual and haptic
form information. Current Biology, 18, 694–98.
Gori, M., Sandini, G., Martinoli, C., and Burr, D. (2010). Poor haptic orientation discrimination in
nonsighted children may reflect disruption of cross-sensory calibration. Current Biology, 20, 223–25.
Gu, Y., Angelaki, D.E., and Deangelis, G.C. (2008). Neural correlates of multisensory cue integration in
macaque MSTd. Nature Neuroscience, 11, 1201–1210.
Heinrichs, M.R. (1994). An analysis of critical transition points of barrier crossing actions. Unpublished
doctoral thesis, University of Minnesota.
Hermer, L., and Spelke, E. (1994). A geometric process for spatial reorientation in young children. Nature,
370, 57–59.
Hermer, L., and Spelke, E. (1996). Modularity and development: the case of spatial reorientation.
Cognition, 61, 195–232.
Higgins, C., Campos, J.J., and Kermoian, R. (1996). Effect of self-produced locomotion on infant postural
compensation to optic flow. Developmental Psychobiology, 32, 836–41.
Hirabayashi, S., and Iwasaki, Y. (1995). Developmental perspective of sensory organization on postural
control. Brain and Development, 17, 111–113.
Howard, I.P., and Templeton, W.B. (1966). Human spatial orientation. John Wiley and Sons, London.
Huttenlocher, J., and Lourenco, S.F. (2007). Coding location in enclosed spaces: is geometry the principle?
Developmental Science, 10, 741–46.
Huttenlocher, J., Newcombe, N., and Sandberg, E. (1994). The coding of spatial location in young children.
Cognitive Psychology, 27, 115–47.
Jouen, F., Lepecq, J., Gapenne, O., and Bertenthal, B. (2000). Optic flow sensitivity in neonates. Infant
Behavior and Development, 23, 271–84.
Kaernbach, C. (1991). Simple adaptive testing with the weighted up-down method. Perception and
Psychophysics, 49, 227–29.
Keating, M.B., McKenzie, B.E., and Day, R.H. (1986). Spatial localization in infancy: position constancy in
a square and circular room with and without a landmark. Child Development, 57, 115–24.
Kingsnorth, S., and Schmuckler, M.A. (2000). Walking skill vs walking experience as a predictor of barrier
crossing in toddlers. Infant Behavior and Development, 23, 331–50.
Klatzky, R.L., Loomis, J.M., Golledge, R.G., Cicinelli, J.G., Doherty, S., and Pellegrino, J.W. (1990).
Acquisition of route and survey knowledge in the absence of vision. Journal of Motor Behavior, 22, 19–43.
Klevberg, G.L., and Anderson, D.I. (2002). Visual and haptic perception of postural affordances in children
and adults. Human Movement Science, 21, 169–86.
Knill, D.C. (2007). Robust cue integration: a Bayesian model and evidence from cue-conflict studies with
stereoscopic and figure cues to slant. Journal of Vision, 7, 5–24.
Kontsevich, L.L., and Tyler, C.W. (1999). Bayesian adaptive estimation of psychometric slope and
threshold. Vision Research, 39, 2729–37.
Körding, K.P., and Wolpert, D.M. (2006). Bayesian decision theory in sensorimotor control. Trends in
Cognitive Sciences, 10, 319–26.
Körding, K.P., Beierholm, U., Ma, W.J., Quartz, S., Tenenbaum, J.B., and Shams, L. (2007). Causal
inference in multisensory perception. PLoS One, 2, e943.
Landy, M.S., Maloney, L.T., Johnston, E.B., and Young, M. (1995). Measurement and modeling of depth
cue combination: in defense of weak fusion. Vision Research, 35, 389–412.
156 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

Langston, R.F., Ainge, J.A., Couey, J.J., et al. (2010). Development of the spatial representation system in
the rat. Science, 328, 1576–80.
Lawson, K.R. (1980). Spatial and temporal congruity and auditory–visual integration in infants.
Developmental Psychology, 16, 185–92.
Learmonth, A.E., Nadel, L., and Newcombe, N.S. (2002). Children’s use of landmarks: implications for
modularity theory. Psychological Science, 13, 337–41.
Lee, D.N., and Aronson, E. (1974). Visual propriceptive control of standing in human infants. Perception
and Psychophysics, 15, 529–32.
Lee, S.A., and Spelke, E.S. (2008). Children’s use of geometry for reorientation. Developmental Science, 11,
743–49.
Lee, S.A., Shusterman, A., and Spelke, E.S. (2006). Reorientation and landmark-guided search by young
children: evidence for two systems. Psychological Science, 17, 577–82.
Lew, A.R., Bremner, J.G., and Lefkovitch, L.P. (2000). The development of relational landmark use in 6–12
month old infants in a spatial orientation task. Child Development, 71, 1179–90.
Lishman, J.R., and Lee, D.N. (1973). The autonomy of visual kinaesthesis. Perception, 2, 287–94.
Loomis, J.M., Klatzky, R.L., Golledge, R.G., Cicinelli, J.G., Pellegrino, J.W., and Fry, P.A. (1993). Nonvisual
navigation by blind and sighted: assessment of path integration ability. Journal of Experimental
Psychology: General, 122, 73–91.
MacNeilage, P.R., Banks, M.S., Berger, D.R., and Bulthoff, H.H. (2007). A Bayesian model of the
disambiguation of gravitoinertial force by visual cues. Experimental Brain Research, 179, 263–90.
McKenzie, B.E., Day, R.H., and Ihsen, E. (1984). Localization of events in space: young infants are not
always egocentric. British Journal of Developmental Psychology, 2, 1–9.
McNaughton, B.L., Battaglia, F.P., Jensen, O., Moser, E.I., and Moser, M.B. (2006). Path integration and
the neural basis of the ‘cognitive map’. Nature Reviews Neuroscience, 7, 663–78.
Meredith, M.A., and Stein, B.E. (1983). Interactions among converging sensory inputs in the superior
colliculus. Science, 221, 389–91.
Meredith, M.A., and Stein, B.E. (1986). Visual, auditory, and somatosensory convergence on cells in
superior colliculus results in multisensory integration. Journal of Neurophysiology, 56, 640–62.
Miller, J. (1982). Divided attention: evidence for coactivation with redundant signals. Cognitive Psychology,
14, 247–79.
Moreau, T., Helfgott, E., Weinstein, P., and Milner, P. (1978). Lateral differences in habituation of
ipsilateral head-turning to repeated tactile stimulation in the human newborn. Perceptual and Motor
Skills, 46, 427–36.
Morgan, M.L., Deangelis, G.C., and Angelaki, D.E. (2008). Multisensory integration in macaque visual
cortex depends on cue reliability. Neuron, 59, 662–73.
Moser, E.I., Kropff, E., and Moser, M.B. (2008). Place cells, grid cells, and the brain’s spatial representation
system. Annual Review of Neuroscience, 31, 69–89.
Nardini, M., Atkinson, J., Braddick, O., and Burgess, N. (2006). The development of body, environment,
and object-based frames of reference in spatial memory in normal and atypical populations. Cognitive
Processing, 7, 68–69.
Nardini, M., Jones, P., Bedford, R., and Braddick, O. (2008). Development of cue integration in human
navigation. Current Biology, 18, 689–93.
Nardini, M., Thomas, R.L., Knowland, V.C., Braddick, O.J., and Atkinson, J. (2009). A viewpoint-independent
process for spatial reorientation. Cognition, 112, 241–48.
Nardini, M., Bedford, R., and Mareschal, D. (2010). Fusion of visual cues is not mandatory in children.
Proceedings of the National Academy of Sciences of the United States of America, 107, 17041–46.
Nardini, M., Bales, J., Zughni, S., and Mareschal, D. (2011). Differential development of audio-visual
integration for saccadic eye movements and manual responses. Journal of Vision, Vision Sciences Society
Meeting, 2011, 11, Article 775.
REFERENCES 157

Neil, P.A., Chee-Ruiter, C., Scheier, C., Lewkowicz, D.J., and Shimojo, S. (2006). Development of
multisensory spatial integration and perception in humans. Developmental Science, 9, 454–64.
Newcombe, N., Huttenlocher, J., Bullock Drummey, A., and Wiley, J.G. (1998). The development of spatial
location coding: place learning and dead reckoning in the second and third years. Cognitive
Development, 13, 185–200.
Newcombe, N.S., Ratliff, K.R., Shallcross, W.L., and Twyman, A.D. (2010). Young children’s use of features
to reorient is more than just associative: further evidence against a modular view of spatial processing.
Developmental Science., 13, 213–20.
Otto, T.U., and Mamassian, P. (2010). Divided attention and sensory integration: the return of the race
model. Journal of Vision, 10, 863.
Pagel, B., Heed, T., and Röder, B. (2009). Change of reference frame for tactile localization during child
development. Developmental Science, 12, 929–37.
Peterka, R.J., and Black, F.O. (1990). Age-related changes in human posture control: sensory organization
tests. Journal of Vestibular Research, 1, 73–85.
Peterson, M.L., Christou, E., and Rosengren, K.S. (2006). Children achieve adult-like sensory integration
during stance at 12-years-old. Gait and Posture., 23, 455–63.
Pick, H.L., Warren, D.H., and Hay, J.C. (1969). Sensory conflict in judgments of spatial direction.
Perception and Psychophysics, 6, 203–205.
Pufall, P.B., and Dunbar, C. (1992). Perceiving whether or not the world affords stepping onto and over: a
developmental study. Ecological Psychology, 4, 17–38.
Raab, D.H. (1962). Statistical facilitation of simple reaction times. Transactions of the New York Academy of
Sciences, 24, 574–90.
Rider, E.A. and Rieser, J.J. (1988). Pointing at objects in other rooms: young children’s sensitivity to
perspective after walking with and without vision. Child Development, 59, 480–94.
Riecke, B.E., van Veen, H.A., and Bulthoff, H.H. (2002). Visual homing is possible without landmarks: a
path integration study in virtual reality. Presence: Teleoperators and Virtual Environments, 11, 443–73.
Rieser, J.J. (1979). Spatial orientation of six-month-old infants. Child Development, 50, 1078–87.
Rieser, J.J., and Rider, E.A. (1991). Young children’s spatial orientation with respect to multiple targets
when walking without vision. Developmental Psychology, 27, 97–107.
Rinaldi, N.M., Polastri, P.F., and Barela, J.A. (2009). Age-related changes in postural control sensory
reweighting. Neuroscience Letters, 467, 225–29.
Romo, R., Hernandez, A., and Zainos, A. (2004). Neuronal correlates of a perceptual decision in ventral
premotor cortex. Neuron, 41, 165–73.
Schmuckler, M.A. (1996). The development of visually-guided locomotion: barrier crossing by toddlers.
Ecological Psychology, 8, 209–36.
Schmuckler, M.A., and Gibson, E.J. (1989). The effect of imposed optical flow on guided locomotion in
young walkers. British Journal of Developmental Psychology, 7, 193–206.
Schmuckler, M.A., and Tsang, H.Y. (1997). Visual-movement interaction in infant search. In Studies in
Perception and Action IV (eds. M.A. Schmuckler and J.M. Kennedy), pp. 233–36. Lawrence Erlbaum,
Mahwah, NJ.
Schmuckler, M.A., and Tsang-Tong, H.Y. (2000). The role of visual and body movement information in
infant search. Developmental Psychology, 36, 499–510.
Shumway-Cook, A., and Woollacott, M.H. (1985). The growth of stability: postural control from a
development perspective. Journal of Motor Behavior, 17, 131–47.
Simons, D.J., and Wang, R.F. (1998). Perceiving real-world viewpoint changes. Psychological Science,
9, 315–20.
Sparto, P.J., Redfern, M.S., Jasko, J.G., Casselbrant, M.L., Mandel, E.M., and Furman, J.M. (2006). The
influence of dynamic visual cues for postural control in children aged 7–12 years. Experimental Brain
Research, 168, 505–16.
158 THE DEVELOPMENT OF MULTISENSORY BALANCE, LOCOMOTION, ORIENTATION, AND NAVIGATION

Tan, U., and Tan, M. (1999). Incidences of asymmetries for the palmar grasp reflex in neonates and hand
preference in adults. Neuroreport, 10, 3253–56.
Taube, J.S. (2007). The head direction signal: origins and sensory-motor integration. Annual Review of
Neuroscience, 30, 181–207.
Twyman, A.D., and Newcombe, N.S. (2010). Five reasons to doubt the existence of a geometric module.
Cognitive Science, 34, 1315–56.
Wall, M.B., and Smith, A.T. (2008). The representation of egomotion in the human brain. Current Biology,
18, 191–94.
Wall, M.B., Lingnau, A., Ashida, H., and Smith, A.T. (2008). Selective visual responses to expansion and
rotation in the human MT complex revealed by functional magnetic resonance imaging adaptation.
European Journal of Neuroscience, 27, 2747–57.
Wallace, M.T., and Stein, B.E. (1997). Development of multisensory neurons and multisensory integration
in cat superior colliculus. Journal of Neuroscience, 17, 2429–44.
Wallace, M.T., and Stein, B.E. (2001). Sensory and multisensory responses in the newborn monkey
superior colliculus. Journal of Neuroscience, 21, 8886–94.
Wallace, M.T., Wilkinson, L.K., and Stein, B.E. (1996). Representation and integration of multiple sensory
inputs in primate superior colliculus. Journal of Neurophysiology, 76, 1246–66.
Wang, R.F., and Simons, D.J. (1999). Active and passive scene recognition. Cognition, 70, 191–210.
Wang, R.F., and Spelke, E.S. (2002). Human spatial representation: insights from animals. Trends in
Cognitive Sciences, 6, 376–82.
Wann, J.P., Mon-Williams, M., and Rushton, K. (1998). Postural control and co-ordination disorders: the
swinging room revisited. Human Movement Science, 17, 491–513.
Wattam-Bell, J., Birtles, D., Nystrom, P., et al. (2010). Reorganization of global form and motion
processing during human visual development. Current Biology, 20, 411–415.
Wertheimer, M. (1961). Psychomotor coordination of auditory and visual space at birth. Science, 134, 1692.
Wills, T.J., Cacucci, F., Burgess, N., and O’Keefe, J. (2010). Development of the hippocampal cognitive map
in preweanling rats. Science, 328, 1573–76.
Wismeijer, D.A., Erkelens, C.J., van, E.R., and Wexler, M. (2010). Depth cue combination in spontaneous
eye movements. Journal of Vision, 10, 25.
Woollacott, M., Debu, B., and Mowatt, M. (1987). Neuromuscular control of posture in the infant and
child: is vision dominant? Journal of Motor Behavior, 19, 167–86.
Zupan, L.H., Merfeld, D.M., and Darlot, C. (2002). Using sensory weighting to model the influence of canal,
otolith and visual cues on spatial orientation and eye movements. Biological Cybernetics, 86, 209–30.
Zwart, R., Lebedt, A., Fong, B., deVries, H., and Savelsbergh, G. (2005). The affordance of gap crossing in
toddlers. Infant Behavior and Development, 28, 145–54.
Chapter 7

The unexpected effects of experience


on the development of multisensory
perception in primates
David J. Lewkowicz

7.1 Introduction
Whenever we encounter objects and the events that they participate in, we can see, hear, feel,
smell, and/or taste them. In addition, during each of these encounters, our bodies provide a con-
stant flow of proprioceptive, kinesthetic, and vestibular information that often bears some rela-
tionship to our actions directed toward the objects. Thus, even though the specific mélange of
sensory inputs that we experience at any moment in time may differ across different situations,
our normal sensory experiences are always multisensory in nature (J.J. Gibson 1966; Marks 1978;
Stein and Meredith 1993; Welch and Warren 1980; Werner 1973).
The rich and varied combination of everyday multisensory experiences presents a considerable
challenge to developing infants. Unless they can integrate the diverse types of sensory information
in the different modalities, they might end up in William James’ classic state of ‘blooming, buzz-
ing confusion’. Fortunately, most of the objects and events in our life are specified by highly
redundant multisensory perceptual cues (J.J. Gibson 1966, 1979) and, as a result, infants have the
opportunity to perceive a world of coherent objects and events rather than one specified by a col-
lection of disparate sensations (E.J. Gibson 1969). Having the opportunity to experience multi-
sensory coherence and having the ability to do so are two different issues. Indeed, it is theoretically
reasonable to posit a priori that infants may not be able to take advantage of the multisensory and
redundant cues that are all around them because they come into the world with a highly imma-
ture nervous system and are perceptually inexperienced. In other words, it is reasonable to expect
that infants’ ability to perceive multisensory coherence is highly immature, that it takes time for
it to develop, and that experience is likely to contribute to its emergence.
In this chapter I review what is currently known about the development of multisensory per-
ception in infancy and show that the ability to perceive multisensory coherence does, indeed, take
time to emerge. In addition, I review some of our recent data showing for the first time that expe-
rience contributes in an unexpected but critical way to the emergence of multisensory perceptual
skills in human infants and developing monkeys. These data demonstrate that young infants are
broadly tuned to the multisensory inputs and that as a result they treat audiovisual inputs as
coherent regardless of whether they represent a native or a non-native species or a native or non-
native language. Furthermore, these data indicate that multisensory perceptual tuning narrows
during the first year of life and that, as it does, the ability to perceive multisensory coherence of
non-native signals declines. Finally, these findings suggest that multisensory perceptual narrow-
ing may be a recent evolutionary phenomenon because young vervet monkey infants do not
160 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

exhibit narrowing and thus continue to perceive the coherence of the faces and vocalizations of
another species.

7.2 Dealing with a multisensory world


Multisensory perceptual inputs provide two kinds of cues: modality-specific and amodal.
Modality-specific cues are unique to a particular modality and include such perceptual attributes
as colour, pitch, aroma, taste, pressure, and temperature. In contrast, amodal cues provide equiv-
alent information about particular objects or events regardless of the modality in which they are
detected and include such perceptual attributes as intensity, duration, tempo, rhythmic pattern,
shape, and texture. Importantly, amodal attributes can be detected in a single sensory modality.
For example, we can perceive the intensity, duration, or rhythm of a stimulus event either by just
listening to it or by seeing it. In addition, regardless of whether we listen to it or hear it, we can tell
that it is the same thing. In contrast, perception of the multisensory coherence of modality-
specific attributes requires concurrent access to such attributes in the different modalities and an
active process of association that relies on their temporal and/or spatial contiguity.
Evolution has taken advantage of the multisensory redundancy that amodal and temporally
and spatially contiguous modality-specific cues provide, resulting in the emergence of nervous
systems that possess sophisticated mechanisms for the detection of multisensory coherence (Foxe
and Schroeder 2005; Ghazanfar and Schroeder 2006; J.J. Gibson 1979; Maier and Schneirla 1964;
Marks 1978; Stein and Meredith 1993; Stein and Stanford 2008). Two sets of findings illustrate
especially well the pervasive nature of multisensory interaction and its fundamental role in behav-
ioral functioning. One is a set of findings on multisensory redundancy effects in various species
and across the lifespan that shows that detection, discrimination, and learning of multisensory as
opposed to unisensory information is more robust (Bahrick et al. 2004; Lewkowicz and Kraebel
2004; Partan and Marler 1999; Rowe 1999). The other set of findings comes from studies of vari-
ous multisensory illusions. These also demonstrate the pervasive nature of multisensory interac-
tion in that in those cases when inputs in different modalities conflict, the nervous system attempts
to resolve the conflict and, in the process, produces surprising perceptual experiences.
Multisensory redundancy effects have been reported in adults of many different species as well
as in human infants. For example, it has been found that human adults exhibit an 11% increase
in speech intelligibility when they can see the face while listening to someone’s voice (Macleod
and Summerfield 1987) and that adults’ ability to understand difficult speech content is enhanced
when the face is visible (Arnold and Hill 2001). Similarly, studies have shown that adult animals
exhibit better learning and discrimination when they have access to multisensory as opposed to
unisensory inputs (Partan and Marler 1999; Rowe 1999; Shams and Seitz 2008). At the neural
level, studies have shown that some proportion of multisensory neurons in the superior colliculus
of the cat and monkey increase their firing rate dramatically in response to multisensory as
opposed to unisensory inputs (Meredith and Stein 1983) and that this ‘superadditivity’ is reflected
in faster behavioral orienting to concurrent and co-located auditory and visual localization cues
than to visual-only cues (Stein et al. 1989).
Like adults, human infants take advantage of multisensory redundancy in that they exhibit bet-
ter detection, learning, and discrimination of multisensory as opposed to unisensory events
(Bahrick et al. 2004; Lewkowicz and Kraebel 2004). For example, as early as two months of age
infants exhibit generally faster responsiveness to co-located auditory and visual cues than to each
of these cues alone and by eight months they exhibit adult-like audiovisual facilitation effects
(Neil et al. 2006). Similarly, infants exhibit greater responsiveness to temporally varying static
visual stimuli presented together with temporally varying sounds than to the same visual stimuli
DEALING WITH A MULTISENSORY WORLD 161

in the absence of the sounds (Lewkowicz 1988a, 1988b). Infants also exhibit greater responsive-
ness to moving objects presented together with their impact sounds than to such objects pre-
sented without impact sounds (Bahrick and Lickliter 2000; Bahrick et al. 2002; Lewkowicz 1992b,
2004). Finally, infants respond more to talking and singing faces than to silently talking and sing-
ing faces (Lewkowicz 1996a, 1998). Importantly, however, no tests have been done so far to
determine when temporally based redundancy effects first become adult-like in infancy.
As far as multisensory illusions are concerned, the McGurk and the spatial ventriloquism
effects are the best known and demonstrate the power of the visual modality to affect responsive-
ness to auditory input. In the McGurk effect (McGurk and Macdonald 1976), adults’ perception
of an audible syllable changes dramatically in the presence of a conflicting visual syllable. For
example, when a visual /va/ is dubbed onto an audible /ba/, adults tend to hear a /va/ rather than
a /ba/. Infants also exhibit evidence of the McGurk effect but not until five months of age
(Rosenblum et al. 1997). In the ventriloquism effect (Bertelson and Radeau 1981; Pick et al. 1969;
Welch and Warren 1980), the perceived location of a sound can be changed dramatically by pre-
senting a concurrent dynamic visual stimulus at another location. For example, when adults hear
a speech sound or a beep at one location and see a talking face or see a flashing light at another
location, they mis-localize the sound and perceive it to be closer to the visual stimulus. Interestingly,
no studies to date have investigated the ventriloquism effect in infancy although it is not likely to
operate during the early months of life for a number of reasons. First, auditory spatial acuity
thresholds are significantly higher (Morrongiello 1988; Morrongiello et al. 1990) making it diffi-
cult for infants to localize sounds precisely. Second, the extent of the visual field is initially quite
small and increases markedly during the early months of life (Delaney et al. 2000). This makes it
difficult for infants to localize visual stimuli during the early months of life. Compounding the
problem is the fact that spatial resolution, saccadic localization, smooth pursuit, and the ability to
localize moving objects are all initially poor and change markedly during the early months of life
(Colombo 2001). Together, these auditory and visual system immaturities make it unlikely that
young infants experience the kind of visual capture that is typical of the spatial ventriloquism
effect in adults because these effects depend on relatively small audio-visual (A-V) disparities
(Recanzone 2009). That is, in order for the visual stimulus to ‘capture’ the location of the auditory
stimulus, subjects must have the ability to precisely localize visual stimuli. Infants certainly do not
have this ability and, if anything, their auditory spatial discrimination ability is probably better,
especially given that audition begins to function during the last trimester of pregnancy while
vision does not begin to function until birth. Even so, auditory spatial detection abilities are still
relatively poor in infancy as well. As a result, the multisensory spatial contiguity window, like the
multisensory temporal contiguity window (Lewkowicz 1996b), is probably much larger in infancy
than later in life. If so, this would suggest that infants might actually integrate spatially discrepant
auditory and visual stimuli over greater spatial discrepancies than adults and thus, paradoxically,
may experience the ventriloquism effect most of the time. So far, this prediction has not been
tested experimentally.
Whereas the McGurk and the ventriloquism effects illustrate the power of the visual modality
to markedly affect auditory perception, the ‘bounce’ and the ‘flash-beep’ illusions illustrate the
power of the auditory modality to radically change our perception of visual input. In the bounce
illusion (Sekuler et al. 1997), adults who watch two identical objects moving in silence along the
same path toward and then through each other report that the objects pass through each other in
nearly 80% of the trials (i.e. they report that the objects bounce in approximately 20% of the tri-
als). In contrast, when they watch the same objects but this time hear a brief sound each time the
objects overlap, they report that the objects now bounce against one another in approximately
65% of the trials. Studies investigating the developmental emergence of this spatiotemporal
162 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

multisensory illusion have found that it emerges in infancy. They have found that 4-month-old
infants do not exhibit it but that 6- and 8-month-old infants do (Scheier et al. 2003). Finally,
in the flash-beep illusion, when adults see a single flash of light, they incorrectly report seeing
multiple flashes when they hear multiple auditory beeps (Shams et al. 2000). Developmental stud-
ies (Innes-Brown et al. 2011) have shown that 8-17 year-old children also exhibit this illusion but
that their responsiveness is immature relative to that found in adults.
In sum, the findings of multisensory redundancy effects and the various multisensory illusions
make it clear that multisensory interaction is the norm rather than the exception in human
behavior. The fact that some evidence of these types of multisensory interaction has also been
found in infancy is further testament to the multisensory character of perception and the behav-
iors that it mediates and supports. In the subsequent sections of this chapter I first consider the
theoretical issues confronting those interested in the development of multisensory perception,
then discuss the theoretical frameworks proposed in the past to deal with these issues, and end
by reviewing some recent findings from our laboratory suggesting the operation of a hitherto
unacknowledged process (multisensory narrowing), and propose a reformulation of the extant
theoretical frameworks.

7.3 The developmental challenge and underlying processes


The fundamental importance of multisensory perception raises interesting questions about its
developmental emergence. The principal question is this: given that the infant nervous system is
immature and given that infants are perceptually and cognitively inexperienced, to what extent
might infants be able to perceive the coherence of multisensory inputs and what process(es) might
underlie this ability? This central question was clearly recognized by two classic theoretical views.
The first, the developmental integration view, held that newborns do not perceive multisensory
coherence and that this ability only emerges gradually as a result of the child’s active exploration of
the world (Birch and Lefford 1963, 1967; Piaget 1952). In contrast, the second, the developmental
differentiation view, did not consider the early developmental limitations to be as much of a barrier
to multisensory processing and, thus, proposed that some multisensory perceptual abilities are
present at birth—presumably because of an inborn sensitivity to the invariant structure of the per-
ceptual array—and that other more sophisticated abilities emerge later as a result of continuous
perceptual learning and differentiation (E.J. Gibson 1969, 1984). Both of these theoretical views
were proposed at a time when no empirical evidence was available. This obviously made it difficult
to assess the validity of either view. Since then, however, a large body of empirical evidence has
accumulated and it has made it clear that the neural and behavioural limitations and the relative lack
of experience play a central role in the development of multisensory processing but that, at the same
time, infants do come into the world with some rudimentary multisensory processing abilities
(Lewkowicz 1994, 2000a, 2002; Lickliter and Bahrick 2000, Walker-Andrews 1997). Moreover, it is
now clear that it is not necessary to posit innate sensitivity to multisensory correspondences because,
as discussed next, organisms have lots of opportunities for experiencing concurrent multisensory
inputs during prenatal life (Turkewitz 1994). Thus, the most reasonable view that has emerged from
the body of empirical evidence to date is that developmental integration and developmental differ-
entiation processes operate side-by-side during early development (Lewkowicz 2002).
It is important to note that the developmental integration and developmental differentiation
views focused mostly on the period after birth. The fact is, however, that multisensory perception
begins prior to birth in all mammalian species and prior to hatching in all avian species because
of the prenatal onset of sensory function (Bradley and Mistretta 1975; Gottlieb 1971; LeCanuet
and Schaal 1996). In addition, in all avian and mammalian species, all but the visual modality
THE DEVELOPMENTAL CHALLENGE AND UNDERLYING PROCESSES 163

develop and have their functional onset according to an invariant sequential order (Gottlieb
1971). Specifically, the tactile modality begins to function first as early as the first trimester of
gestation, followed by the vestibular, the chemosensory, and finally the auditory modality. The
fact that all but the visual modality have their functional onset prior to birth/hatching—even if
their structural and functional properties are immature at first—means that the fetus begins to
receive sensory inputs. This, in turn, means that the developing organism is subject to a barrage
of multisensory inputs during prenatal life and must begin to process the relations among them.
Although it might be reasonable to assume that this barrage of prenatal sensory stimulation
may be confusing and disruptive to an immature and inexperienced fetus, Turkewitz and Kenny
(Turkewitz 1994; Turkewitz and Kenny 1982, 1985) have proposed that this is not necessarily
the case. They have suggested that the sequential emergence of each sensory modality actually
facilitates the development of each modality because it is subject to reduced competitive influ-
ences from the other as yet non-functional modalities. In other words, the sequential emergence
of the different sensory modalities is assumed to have beneficial effects on the ultimate organiza-
tion of multisensory function. Indeed, empirical evidence from animal studies that have tested
this theoretical proposal directly has provided support for it (Honeycutt and Lickliter 2003).
Although the sequential onset of sensory function prior to birth ensures that the developing
fetus is not completely overwhelmed by multisensory inputs, there is little doubt that the fetus is
subject to a great deal of concurrent and redundant multisensory stimulation. As a result, there is
every reason to believe that this kind of prenatal experience has important effects on the organiza-
tion and emergence of multisensory perceptual abilities after birth. For example, it is likely that
the months of concurrent prenatal sensations in the different sensory modalities contribute in an
indirect way (at least in the human case) to the appearance of responsiveness to A-V temporal
synchrony relations at birth (Lewkowicz et al. 2010) and perhaps even to visual responsiveness to
affective vocal stimulation at birth (Mastropieri and Turkewitz 1999). That is, experience with
concurrent multisensory inputs in modalities other than vision may set up neural circuits that are
generally responsive to temporal coherence, regardless of the modality of stimulation. Studies
with birds provide evidence in support of these predictions by showing that prenatal multisensory
stimulation has marked effects on the postnatal emergence of unisensory and multisensory
responsiveness (Lickliter 1993, 2005). For example, Jaime and Lickliter (2006) found that embryos
exposed to temporally synchronous and spatially contiguous auditory and visual stimuli prior to
hatching prefer spatially contiguous audiovisual maternal cues after hatching but that embryos
that did not receive exposure to such stimuli did not exhibit a preference for the contiguous hen
and her call over either the hen alone or the call alone. Similarly, Lickliter et al. (2002, 2004) have
demonstrated that bobwhite quail who are exposed to temporally concurrent auditory and visual
stimulation prior to hatching exhibit more robust learning and memory for multisensory stimu-
lation following hatching.
Although there is now little doubt that a good deal of multisensory perceptual development occurs
prior to birth and that this is advantageous from the newborn’s point of view, the advent of visual
function at birth creates a potentially serious new problem for infants. Now, they must begin to dis-
cover how visual input—poor as it may be due to the highly immature state of the visual system at
birth—corresponds to the diverse sensory inputs in the other modalities. Added to this problem is
the fact that the prenatal sensory experiences of the other modalities occurred in an aquatic environ-
ment and that these modalities are still structurally and functionally immature at birth, though obvi-
ously more mature than vision. Interestingly, however, this may actually be a happy circumstance,
from a developmental limitations view because vision can begin to link up with the other sensory
modalities on the basis of some very simple perceptual cues (e.g., intensity, temporal synchrony) and
this can then further bootstrap the development of multisensory perception.
164 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

7.4 The development of multisensory perception


The general developmental picture seems to be that infants start out life being able to perceive
certain types of low-level multisensory relations (e.g. intensity, temporal synchrony) and that as
they grow and gain increasing perceptual experience they gradually acquire the ability to respond
to higher-level types of multisensory relations (e.g. rhythm, affect, gender). This general develop-
mental picture is illustrated by several findings. In terms of low-level relations, it has been found
that young infants can perceive A-V intensity relations (Lewkowicz and Turkewitz 1980), A-V
temporal synchrony relations (Bahrick 1983; Lewkowicz 1986, 1992a, 1992b, 1996b, 2000b, 2003,
2010; Lewkowicz et al. 2010), and isolated A-V phonemic relations (Kuhl and Meltzoff 1982;
Patterson and Werker 1999, 2003; Walton and Bower 1993). Importantly even though younger
infants can detect low-level multisensory relations, it has been found that their ability to detect
them is limited. For example, 4-, 6-, and 8-month-old infants can only detect a desynchronization
of an audiovisual event when the event is relatively simple (i.e. not rhythmically patterned)
whereas 10-month-old infants can detect the desynchronization regardless of whether the event
is complex or not (Lewkowicz 2003). One finding that at first blush seems to challenge this gen-
eral developmental picture is that newborn infants choose to look more at their mother’s face
than a stranger’s face if they were first exposed to their mother’s voice (Sai 2005). This remarkable
finding might be interpreted as reflecting newborns’ detection of some higher-level multisensory
relations. Unfortunately, however, during the test trials in this study infants only saw silent faces.
As a result, they did not have to perform any relational processing and could simply have based
their choice on an association that they formed between their mother’s face and voice prior to the
test. In terms of higher-level relations, it has been found that only 7–8 month-old infants can
perceive amodal affect (Walker-Andrews 1986) and gender (Patterson and Werker 2002) and that
only infants above six months of age can perceive spatiotemporal synchrony (Scheier et al. 2003)
and crossmodal duration (Lewkowicz 1986). Moreover, only 6.5–7-month-old infants can
bind modality-specific attributes such as colour and taste (Reardon and Bushnell 1988) and
8-month-old but not younger infants can perceive the spatial coincidence of auditory and visual
localization cues in an adult-like manner (Neil et al. 2006).
The general pattern of improvement in human infants’ ability to detect various types of multi-
sensory relations has clear parallels in findings from animal studies related to the neural and
behavioral mechanisms underlying multisensory processing early in life and the effects of neural
plasticity. For example, studies in neonatal cats and monkeys have found that the multisensory
cells in the superior colliculus do not integrate the way they do in adult animals and that they only
begin to do so gradually over the first weeks of life (Wallace and Stein 1997, 2001; see also Chapter
14 by Wallace et al.). Similarly, studies in rats have found that appropriate alignment of auditory
and visual maps in the superior colliculus depends on their usual spatial co-registration (Wallace
et al. 2006). This is consistent with the findings from earlier studies in ferrets and barn owls
showing that the spatial tuning and calibration of the neural map of auditory space depends on
concurrent visual input (King et al. 1988; Knudsen and Brainard 1991). In a similar vein, studies
examining the effects of early experience on behavioral responsiveness in birds (i.e. the bobwhite
quail) have demonstrated that the ability to perceive the audible and visible attributes of the
maternal hen as belonging together depends on pre- and post-hatching experience with concur-
rent auditory, visual, and tactile stimulation arising from the embryo’s own vocalizations as well
as from its broodmates and its mother (Lickliter and Banker 1994; Lickliter et al. 1996). Finally,
studies in which visual input has been re-routed to the auditory cortex in newborn ferrets via the
medial geniculate body of the thalamus have shown that the neurons in the primary auditory
cortex become responsive to visual input and become organized into the kinds of orientation
DEVELOPMENTAL BROADENING 165

modules found in visual cortex. Moreover, the re-routed animals exhibit visually appropriate
behavioral responsiveness (Sharma et al. 2000; von Melchner et al. 2000). In sum, the data from
human and animal studies indicate that multisensory perceptual abilities improve with develop-
ment enabling infants to perceive increasingly more complex types of multisensory relations.
They also indicate that experience and neural plasticity contribute critically to this overall process.
In the rest of the chapter I discuss the different ways in which experience can contribute to this
process and show that experience not only facilitates the emergence of multisensory responsive-
ness but that it also contributes to its narrowing and the formation of multisensory perceptual
expertise.

7.5 Developmental broadening


The overall pattern of findings on the emergence of multisensory perceptual abilities is consistent
with the general conventional theoretical view about the course of development and with the
specific conventional theoretical view about the course of perceptual development (E.J. Gibson
1969; Gottlieb 1996; Piaget 1952; Werner 1973). According to the general view of development,
relatively primitive functional capacities, that greatly limit the organism’s adaptive responsive-
ness, appear first and are then followed by a gradual increase in the organism’s structural com-
plexity and functional capacity. Likewise, according to the specific view of perceptual development,
infants gradually learn to differentiate new stimulus properties, patterns, and distinctive features
as they develop (E.J. Gibson 1969; Werner 1973). In essence, both of these conventional views
hold that development is a progressive process that leads to a general broadening of functional
capacity.
With specific regard to perceptual development, the developmental broadening view is sup-
ported by a wealth of empirical evidence. For example, newborn infants have very poor visual
acuity and spatial resolution skills, cannot discriminate between different affective expressions
that the human face can convey, do not understand that objects are bounded and that they have
an independent existence, cannot segment speech it into its meaningful components, cannot
understand the meanings inherent in the temporal structure of events, do not perceive depth, do
not have a fear of heights, and cannot self-locomote. Of course, all of these various skills emerge
and continually improve with development. A good case in point is the development of face per-
ception (Simion et al. 2007). At birth, infants exhibit weak and relatively unstable preferences for
faces (Morton and Johnson 1991; Nelson 2001), by 2 months they exhibit more stable prefer-
ences, and by 6–7 months they begin to respond to facial affect (Ludemann and Nelson 1988) and
begin to categorize faces on the basis of gender (Cohen and Strauss 1979). Similarly, although
newborn infants possess rudimentary auditory perceptual abilities that make it possible for them
to recognize their mother’s voice (Decasper and Fifer 1980) and to distinguish different languages
on the basis of their overall rhythmic attributes (Nazzi et al. 1998), they do not have a functional
lexicon. By 1 month of age infants begin to exhibit basic speech perception abilities that enable
them to perceive sounds in a language-relevant way (Eimas et al. 1971), and over the next months
they begin to acquire various phonetic discrimination abilities that permit them to segment
speech into linguistically meaningful components (Jusczyk 1997). Importantly, the ability to seg-
ment speech does not mean that infants are fully capable of detecting complex aspects of sequences
when they are young. For example, even though 2-month-old infants can learn the statistical rela-
tions linking the adjacent members of a sequence of meaningless visual stimuli (Kirkham et al.
2002), 4-month-old infants do not perceive invariant ordinal relations in sequences composed of
abstract moving objects and their impact sounds even though they can perceive the statistical
relations inherent in such multisensory (Lewkowicz and Berent 2009) as well as in visual-only
166 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

sequences (Marcovitch and Lewkowicz 2009). Indeed, it is not until 7–8 months of age that
infants begin to extract the statistical properties of auditory speech (Saffran et al. 1996) and can
learn simple syntactic rules linking auditory speech syllables (Marcus et al. 1999).

7.6 Developmental narrowing


Even though the principle of developmental broadening is supported by a wealth of evidence,
there is now a good deal of evidence that developmental narrowing and a concurrent decline in
plasticity play an important role in development too. The earliest acknowledgement of the impor-
tance of narrowing in behavioural development can be found in the concept of behavioural
canalization first introduced into the psychological literature by Holt (1931) to account for the
emergence of organized motor activity patterns during foetal development. According to Holt,
the initially diffuse sensorimotor activity observed in early foetal life was canalized into organized
motor patterns through the process of behavioural conditioning. Kuo (1976) expanded Holt’s
limited concept of canalization by proposing that narrowing of behavioural potential was not
merely the result of the individual’s history of reinforcement but of the individual’s entire devel-
opmental history, context, and experience. Subsequently, Gottlieb (1991) provided especially
convincing evidence in support of Kuo’s concept of canalization in his studies of the development
of species identification in ducks. In a remarkable set of experiments, Gottlieb demonstrated that
mallard hatchlings’ socially affiliative responses toward their conspecifics are determined not only
by exposure to the vocalizations of their siblings and their mother but to their own vocalizations
as well. He found that as embryos vocalize prior to hatching they learn the critical features of their
species-specific call as well as not to respond to the social signals of other species. In this way, they
narrow their perceptual repertoire.

7.7 Perceptual narrowing


A growing body of evidence from studies of human infants has yielded evidence consistent with
the concept of canalization/narrowing. Overall, this body of evidence shows that perceptual
responsiveness is relatively broad during the first months of life and that, as a result, young infants
respond to native as well as non-native perceptual inputs across different domains (e.g. face,
speech, and music perception). The evidence also shows that as infants grow and acquire experi-
ence with their native ecology, their responsiveness to non-native inputs declines while their
responsiveness to native inputs improves.

7.7.1 Narrowing in speech perception


The best-known and earliest evidence of perceptual narrowing effects in humans comes from
studies of speech perception. It shows that young infants can perceive native and non-native
phonetic contrasts but that older infants can only perceive native contrasts. Streeter (1976)
provided the earliest evidence of broad perceptual tuning by showing that 2-month-old Kikuyu
infants learning the Kikuyu language could discriminate a phonologically relevant voice-onset-
time contrast (e.g. the contrast between a /ba/ and a /pa/) despite the fact that this contrast is
not relevant in their native language. Similarly, Aslin et al. (1981) showed that English-learning
6–12-month-old infants can discriminate phonologically relevant and phonologically irrelevant
voice-onset-time contrasts. Werker and Tees (1984) provided the first direct evidence of per-
ceptual narrowing in the speech domain by showing that 6–8-month-old but not 10–12-month-
old English-learning infants can discriminate non-native consonants such as the Hindi retroflex
/Da/ and the dental /da/ and the Thompson glottalized velar /k’i/ versus the uvular /q’i/.
PERCEPTUAL NARROWING 167

Subsequent cross-linguistic studies have provided similar behavioural (Best et al. 1995) and
neural evidence of narrowing in infant response to native and non-native consonants (Rivera-
Gaxiola et al. 2005) as well as vowels (Cheour et al. 1998; Kuhl et al. 1992).

7.7.2 Narrowing of face perception


Following initial work showing that human and monkey adults are better at recognizing faces from
their own species than faces from other species (Dufour et al. 2006; Pascalis and Bachevalier 1998),
Pascalis and colleagues (Pascalis et al. 2002, 2005) demonstrated that this specialization emerges
during infancy. They found that 6-month-old infants can discriminate human faces and monkey
faces but that 9-month-old infants can only discriminate human faces. Similar evidence of narrow-
ing comes from investigations of the ‘other race effect’ (ORE). The ORE reflects adults’ greater
difficulty in distinguishing the faces of people from other races than the faces of people from their
own race (Chiroro and Valentine 1995). The assumption is that this is due to greater exposure to
individuals from one’s own race than to individuals from a different race. Sangrigoli and de Schonen
(2004) demonstrated that the developmental roots of this specialization can be found in infancy by
showing that as early as 3 months of age, Caucasian infants can discriminate different Caucasian
faces but not different Asian faces. Importantly, when infants were given more extensive experience
with the other-race faces by first being familiarized with three different faces within each race, they
did discriminate different Caucasian faces and different Asiatic faces. Kelly et al. (2005) replicated
the ORE in 3-month-old infants and, in addition, showed that the ORE is absent in newborn
infants. This suggests that selective exposure to the faces from one’s own ethnic group during the
first months of life leads to specialization for such faces. In subsequent studies, Kelly et al. (2007)
have shown that the ORE develops slowly between 3 and 9 months of age and Kelly et al. (2009)
demonstrated that the ORE is also present in non-Caucasian (i.e. Chinese) infants. Finally, evidence
indicates that perceptual specialization for certain types of faces early in life also includes specializa-
tion for speech-related utterances that can be seen whenever a face is talking. This evidence
comes from a study showing that 4- and 6-month-old infants can distinguish between two faces
silently uttering native and non-native speech sounds but that 8-month-old infants no longer make
such discriminations unless they are raised in a bilingual environment (Weikum et al. 2007).

7.7.3 Narrowing in music perception


As in the speech- and face-processing domains, investigators have reported evidence of narrow-
ing in the music-perception domain. For example, Hannon and Trehub (2005a) tested adults’
and infants’ discrimination of folk melodies that had either simple or complex metrical struc-
tures. Simple metrical structure, defined by simple duration ratios of inter-onset intervals of
sounds (e.g. 2:1), predominates in North American music whereas complex metrical structure,
defined by complex duration ratios (e.g. 3:2), predominates in many other musical cultures (e.g.
in the Balkans). Hannon and Trehub found that North American adults successfully detected
differences in melodies based on alterations of simple meters but did not when the differences
were based on alterations of complex meters characteristic of Balkan music. In contrast, adults of
Bulgarian or Macedonian origin detected differences in melodies when they differed in terms of
simple and complex metrical structure. Similar to the adults of Bulgarian or Macedonian origin,
6-month-old infants detected melody differences regardless of whether the differences were based
on simple or complex metrical structure. Subsequently, Hannon and Trehub (2005b) showed
that the simple metrical bias exhibited by North American adults is based on selective experience,
in that 12-month-old infants who grew up in a Western musical environment responded to musical
rhythms in the same way that Western adults did.
168 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

7.7.4 The role of experience in perceptual narrowing


Specific experience during an organism’s sensitive period—the period of time during early devel-
opment when the organism is most open to experience (Gottlieb 1992; Michel and Tyler 2005;
Turkewitz and Kenny 1982)—can have one of four effects (Aslin and Pisoni 1980; Gottlieb 1992,
1976, 1981). When specific experience is available it can (1) induce the development of a new
perceptual ability, (2) facilitate the development of an incompletely developed ability, (3) main-
tain the ability at its optimum level into later stages of development, and when a particular expe-
rience is missing this can (4) lead to the decline or loss of the ability.
Given these four possible effects, how might experience contribute to perceptual narrowing
effects? It is relatively well known that the ubiquity of native perceptual inputs induces, maintains,
and facilitates responsiveness to such inputs. What is less clear is whether the absence of non-
native inputs leads to a permanent loss of the ability to respond to the latter types of inputs or just
to a decline in responsiveness. Current opinion is that it is the latter and that the uneven experi-
ence that infants have with native as opposed to non-native inputs leads to perceptual re-organi-
zation rather than a permanent loss in responsiveness to non-native inputs (Lewkowicz and
Ghazanfar 2009; Werker and Tees 2005).
If perceptual re-organization results from the absence of specific perceptual input then extra
experience with non-native input should help maintain responsiveness to it. Indeed, English-
learning infants who are exposed to natural Mandarin Chinese during 12 play sessions between
nine and 10 months of age can discriminate a Mandarin Chinese phonetic contrast that does not
occur in English better than control infants who are equivalently exposed to books and toys but
only hear English (Kuhl et al. 2003). In addition, the usually observed decline in responsiveness
to non-native phonetic contrasts was accompanied by an improvement in response to native
phonetic contrasts during the first year of life (Kuhl et al. 2006). Similar effects of experience
have been found in infant response to non-native faces. For example, Pascalis and colleagues
(Pascalis et al. 2005) demonstrated that the decline in the infant ability to discriminate non-
native faces can be prevented with continued experience with such faces. Infants who were
exposed to monkey faces at home during the 3-month period between 6 and 9 months of age
continued to discriminate monkey faces at 9 months of age. Furthermore, Scott and Monesson
(2009) found that infants who are exposed to individual monkey faces together with unique
names for each face maintain their ability to discriminate monkey faces at 9 months of age
whereas infants who are only exposed to the monkey faces along with a category name (i.e. mon-
key) do not. Although no studies to date have directly manipulated exposure to the faces of other
races (by providing extra exposure at older ages), studies with older children suggest that such
experience can prevent the decline in responsiveness to those kinds of faces as well (Sangrigoli
et al. 2005). What is currently not known is whether, like in infant perception of native speech,
infants’ ability to respond to native faces improves while their ability to respond to non-native
faces declines.
Finally, consistent with the findings of the positive effects of extra experience with non-
native inputs on responsiveness to non-native speech and faces, Hannon and Trehub (2005b)
found that 12-month-old infants could discriminate complex Balkan meters following 2-weeks’
exposure to them. Particularly interesting was the finding that the plasticity observed in the
12-month-old infants is not retained into adulthood because the same 2-week exposure to
Balkan music in North American adults was not effective in restoring successful discrimina-
tion. Thus, as is evident in adults’ ability to learn complex non-native complex rhythms through
formal musical instruction, some plasticity is retained into adulthood but not at the level found
in infancy.
PERCEPTUAL NARROWING 169

7.7.5 Multisensory perceptual narrowing


It is reasonable to hypothesize that perceptual narrowing may be a general developmental process
and thus that it also exerts its effects on the development of multisensory perception. A first indi-
rect hint that this might be the case came from a study by Lewkowicz and Turkewitz (1980). The
findings from this study indicated that 3-week-old infants as well as adults can match auditory
and visual stimuli on the basis of their intensity. Curiously, however, the adult participants
reported that they found the matching task ‘bizarre’. At the time it was not clear how to interpret
the adults’ description of the task but the subsequent reports of unisensory perceptual narrowing
raised the possibility that the adults’ response reflected the fact that they normally perceive the
coherence of multisensory inputs on the basis of higher-level perceptual attributes and thus that
responsiveness to low-level attributes (i.e. intensity) declines during development.
To test the theoretical possibility that narrowing extends to infant response to multisensory
inputs, my colleagues and I recently embarked on a series of studies. In these studies, we asked
whether human infants and non-human primates exhibit multisensory perceptual narrowing
(MPN) as might be expected from the indirect results of Lewkowicz and Turkewitz (1980) and
from the body of unisensory narrowing effects reviewed earlier.

7.7.6 Narrowing of cross-species multisensory perception in infancy


We began our studies by hypothesizing that the broad perceptual tuning that is known to exist in
unisensory responsiveness should enable young infants to match non-native faces and vocaliza-
tions and that this type of unexpected multisensory responsiveness should decline in the later
months of infancy. We also began our studies knowing that infants can match human faces and
vocalizations starting shortly after birth (Kuhl and Meltzoff 1982, 1984; Patterson and Werker
1999, 2002, 2003) and, critically, that rather than narrow with age, this ability improves in that
older infants can match faces and vocalizations on the basis of higher-level amodal cues such as
gender and affect (Poulin-Dubois et al. 1994, 1998; Walker-Andrews 1986, 1991).
In our initial study (Lewkowicz and Ghazanfar 2006), we tested 4-, 6-, 8-, and 10-month-old
infants who saw pairs of movies in which the same macaque monkey’s face could be seen on side-
by-side monitors mouthing a ‘coo’ call on one monitor and a ‘grunt’ call on the other; in silence
first and then in the presence of one of the two calls. Crucially, the onset of the audible call was
synchronized with the onset of both visible calls but its offset was synchronized only with the
matching visible call. This means that the corresponding facial gestures and vocalizations were
related on the basis of their synchronous onsets and offsets as well as in terms of their durations.
To determine whether infants made A-V matches, we compared the proportion of time infants
looked at a given visible call in the presence of the matching audible call (in-sound condition)
versus the proportion of time they looked at the same call in the absence of the audible call (silent
condition). We found that the 4- and 6-month old infants looked significantly longer at the
matching visible call in the in-sound condition than in the silent condition but that the 8- and
10-month-old infants did not (Fig. 7.1). These findings were consistent with our predictions and
provided the first evidence of MPN.

7.7.7 Mechanisms of cross-species multisensory perception


Because the matching visible and audible calls in the Lewkowicz and Ghazanfar (2006) study were
related in terms of their synchronous onsets and offsets and in terms of their durations, in follow-
up studies we investigated which cues were critical for successful matching (Lewkowicz et al. 2008).
To do so, we repeated the Lewkowicz and Ghazanfar (2006) study, but this time we presented the
170 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

0.6
Mean proportion of looking (s)

0.5

0.4

0.3 Silent
In sound
0.2

0.1

0.0
4 months 6 months 8 months 10 months
Age
Fig. 7.1 Cross-species intersensory matching in 4–10 month-old infants. The figure shows the mean
proportion of looking time directed at the matching visible call out of the total amount of looking
time directed at both calls in the silent and the in-sound conditions. Error bars indicate standard
errors of the mean.

audible call 666 ms prior to the onset of the two visible calls. In this way, the matching visible and
audible calls were no longer synchronized but still corresponded in terms of their duration.
In contrast, the non-corresponding visible and audible calls did not correspond at all. This time,
we found that neither the younger nor the older infants performed multisensory matching. This
indicated that A-V temporal synchrony mediates multisensory matching in the younger infants,
that it is no longer sufficient for matching at the older ages, and that duration did not play a role in
matching. If duration alone had mediated matching then infants would have made matches even
if the audible and visible calls were out of synchrony because the matching audible and visible calls
were still related in terms of their durations. Interestingly, the infants’ failure to match on the
basis of duration is consistent with prior findings showing that infants do not make intersensory
matches of duration when the durations are not synchronous but do when they are synchronous
(Lewkowicz 1986).
In addition to questions about the role of synchrony cues, the Lewkowicz and Ghazanfar (2006)
study raised two other questions. First, might the older infants’ failure to make matches be due to
unisensory processing deficits? This is likely in the visual modality because, as indicated earlier,
responsiveness to non-native faces declines during infancy. It is also likely in the auditory modality
although no studies to date have reported whether responsiveness to non-native vocalizations per
se declines. Second, does the decline in cross-species multisensory perception reflect the waning
influence and early dominance of multisensory perceptual mechanisms that rely primarily on the
detection of multisensory temporal synchrony relations and the concurrent cumulative effects of
perceptual experience that gradually enables infants to discover higher-level multisensory
attributes? To answer these two questions, Lewkowicz et al. (2008) examined the data from the
silent paired-preference trials and found that both the 4–6-month-old and the 8–10-month-old
infants exhibited differential looking at the two silent visible monkey calls, indicating that the
decline in multisensory responsiveness could not have been due to a decline in visual responsiveness.
In addition, Lewkowicz et al. (2008) conducted a discrimination study in which 8–10-month-old
infants were habituated to one of the audible monkey calls and then tested with the other one.
Results indicated that infants discriminated between the two audible calls. Thus, the failure of the
older infants to match the monkey faces and vocalizations was not due to a unisensory processing
deficit in either sensory modality. Finally, Lewkowicz et al. (2008) examined whether 12- and
PERCEPTUAL NARROWING 171

18-month-old infants might be able to match the monkey faces and vocalizations that were
presented in the original Lewkowicz and Ghazanfar (2006) study and found that they did not
match. This finding suggests that the ability to perform cross-species multisensory matching does
not return once it has declined and is consistent with the findings reviewed earlier on infant
response to non-native speech, faces, and music.

7.7.8 Cross-species multisensory perception at birth


As indicated earlier, perceptual tuning is likely to be broad starting at birth. This is because sen-
sory/perceptual functions are so crude at birth that newborns are not able to detect the higher-
level attributes of multisensory inputs and, in addition, are perceptually inexperienced. As a
result, it is probable that sensitivity to cross-species multisensory relations is present at birth. This
prediction is made all the more likely by three facts:
1. Multisensory temporal synchrony mediates cross-species perception in young infants (Lewko-
wicz et al. 2008).
2. Detection of A-V synchrony only requires perception of energy onsets and offsets in the two
modalities (Lewkowicz 2010).
3. The neural mechanisms of multisensory temporal synchrony detection are probably operational
at birth because, despite the fact that such mechanisms are widely distributed in the brain, some
of them are subcortical (Bushara et al. 2001) and thus likely to be present at birth.
To determine whether newborns can make cross-species multisensory matches, Lewkowicz
et al. (2010) repeated the Lewkowicz and Ghazanfar (2006) study, but this time by testing new-
born infants. In the first experiment findings showed that newborns did indeed match monkey
audible and visible calls (Fig. 7.2) indicating that infants are broadly tuned to multisensory signals
from birth onwards. The findings of successful matching at birth then raised the obvious question
about the underlying mechanism. Given that matching in older infants was found to be driven by
temporal synchrony cues (Lewkowicz et al. 2008), we hypothesized that newborns also rely on
synchrony to make the multisensory matches and that they probably do so by relying on nothing
more than energy onsets and offsets. To test this hypothesis, in the second experiment we substi-
tuted a complex tone for the natural audible call but, critically, preserved the temporal features
of the audible call by ensuring that the tone had the same duration as the natural call and that
its onsets and offsets were synchronous with the matching visible call. Once again, findings
indicated that newborns performed multisensory matching in that they looked longer at the vis-
ible monkey call in the presence of the synchronous tone than in its absence (Fig. 7.2). What was
most remarkable is that newborns matched despite the fact that the audible stimulus lacked any
identity information and that the correlation between the dynamic variations in the facial ges-
tures and the amplitude and formant structure inherent in the natural audible call was now
absent. This finding clearly shows that the broad multisensory perceptual tuning found early in
life is based on responsiveness to low-level perceptual cues.

7.7.9 Narrowing of multisensory perception of audiovisual speech


The multisensory narrowing findings reviewed so far suggest that perceptual narrowing is a gen-
eral pan-sensory process. If so, then cross-language multisensory responsiveness might also nar-
row during infancy (see also Chapter 9). To test this prediction, Pons et al. (2009) examined
Spanish-learning and English-learning infants’ ability to match visible and audible versions of a
/ba/ and a /va/. These two particular phonemes were chosen because the /v/ sound does not exist
in the Spanish language (it is a homophone of the /b/ sound). Thus, the phonetic distinction
172 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

0.8
a b
Silent
In sound

Mean proportion of looking at matching call


0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
Experiment 1 Experiment 2
Fig. 7.2 Cross-species intersensory matching in newborn infants. The figure shows the mean
proportion of looking-time directed at the matching silent visible call out of the total amount of
looking-time directed at both calls in the silent and the in-sound conditions. (a) Looking at monkey
visible vocalizations in the presence of naturalistic monkey vocalizations; (b) looking at monkey
visible vocalizations in the presence of complex tones that were synchronized with their respective
visible vocalizations and, thus, also matched them in terms of their durations. The fundamental
frequency of each tone was the average fundamental frequency of the two naturalistic
vocalizations. Error bars indicate standard errors of the mean.

between a /ba/ and /va/ is not discriminable to a Spanish speaker. This makes the Spanish lan-
guage the ideal test bed for investigating cross-language MPN. That is, if phonetic tuning is broad
at first and then narrows, then young Spanish-learning infants should be able to match a visible /
ba/ with an audible /ba/ and a visible /va/ with an audible /va/ whereas older Spanish-learning
infants should not. In contrast, English-learning infants, for whom /ba/ and /va/ are both phone-
mically relevant, should be able to make the matches at both ages. Pons et al. tested 6-month-old
and 11-month-old Spanish-learning infants and compared their performance to 6- and 11-month-
old English-learning infants.
The experiment consisted of six trials. The first two were silent paired-preference trials during
which infants saw side-by-side faces of the same person mouthing a /ba/ on one side and a /va/ on the
other side in silence. The next four trials alternated between familiarization trials during which
infants heard one of the two audible syllables in the absence of visual stimulation and the same silent
paired-preference trials as administered before. As predicted, and as depicted in Fig. 7.3, the 6-month-
old but not the 11-month-old Spanish-learning infants looked significantly longer at the matching
visible syllable following auditory familiarization than prior to familiarization. In contrast, and as
expected, the English-learning infants exhibited intersensory matching at both ages.
To determine whether the narrowing persists into adulthood, we also tested Spanish- and English-
speaking adults in a forced-choice multisensory matching task. We presented them with a total of
16 trials during which they first heard one of the two possible syllables (/va/ or /ba/) repeated twice
and, one second later, saw the two silent visible syllables presented side-by-side twice. As expected,
the Spanish-speaking subjects’ choices of matching face were at chance (51.5%) whereas the English-
learning adults’ choices were correct on 92% of the trials. Together, the infant and adult findings
PERCEPTUAL NARROWING 173

60

50
40

30
Preferential looking times (%)

20
10
0

−10
−20

−30

−40
−50

6-month-olds 11-month-olds 6-month-olds 11-month-olds


Spanish English
Fig. 7.3 Looking-time difference scores to the matching face (percentage of total time infants
looked at the matching face during the test trials minus percentage of total time they looked at it
during the silent trials). Open circles represent each infant’s difference score and black circles with
error bars represent the mean difference score and the standard error of the mean for each group
(Reproduced in colour in the colour plate section).

from this study provided the first evidence of MPN in infant response to non-native speech and
suggest that narrowing is a domain-general as well as a pan-sensory developmental process.
Findings from one of our more recent studies (Lewkowicz and Hansen-Tift 2012) provide addi-
tional support for MPN in the speech domain. They indicate that selective attention to the mouth
as opposed to the eyes of a talker is differentially affected by native versus non-native speech at 12
months of age—a point in development when narrowing is complete. Infants exposed to non-
native speech looked longer at the mouth than at the eyes of the talker whereas infants exposed to
native speech did not.

7.7.10 Cross-species multisensory matching in non-human primates


The final question that we asked with respect to MPN was whether multisensory perceptual
narrowing is an evolutionarily old or new process. To do so, we (Zangenehpour et al. 2009) tested
young vervet monkeys with the same monkey faces and vocalization that were presented by
Lewkowicz and Ghazanfar (2006) and using a similar procedure. Prior findings have shown that
macaque monkeys (Macaca mulatta, Macaca fuscata), capuchins (Cebus apella) and chimpanzees
(Pan troglodytes) all are able to perceive the relationship between vocalizing faces and corresponding
sounds (Adachi et al. 2006; Ghazanfar and Logothetis 2003; Ghazanfar et al. 2007; Izumi and Kojima
2004; Jordan et al. 2005; Parr 2004). Thus, we tested two age groups (23–38 weeks of age and 39–65
weeks) of vervet monkeys and asked whether they could match the faces and vocalizations of
macaque monkeys or whether they, like human infants, exhibit narrowing effects. The specific ages
were chosen to ensure that the vervets were well past the point at which their ability to make cross-
species matches should have declined. Two alternative predictions were tested. If MPN is an evolu-
tionarily conserved characteristic of early development, and if it occurs regardless of the rate of
174 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

(a) 23–38 weeks old 39–65 weeks old (b)


1.00 3.00
Duration of looking time (s)

Duration of looking time (s)


0.80
2.00
0.60

0.40
1.00

0.20

0.00 0.00
Match Non-match
ch

ch

ch

ch
at

at

at

at
M

-m

-m
on

on
N

Fig. 7.4 Cross-species intersensory matching in vervet monkeys. The figure shows the mean
duration of looking-time directed at the matching and non-matching visible rhesus calls.
(a) Looking durations at the matching and non-matching calls in the presence of the naturalistic
rhesus vocalizations; (b) looking durations at the matching and non-matching calls in the presence
of complex tones that were synchronized with their respective visible vocalizations and thus also
matched them in terms of their durations. The fundamental frequency of each tone was the
average fundamental frequency of the two naturalistic vocalizations. Error bars indicate standard
errors of the mean.

neural development, then vervets should not match the rhesus faces and vocalizations because they
are past the age of narrowing. If, however, MPN reflects the lengthening of the period of neural and
behavioral development—as is the case in humans relative to other non-human primates—then
vervets may not exhibit narrowing because neural development in monkeys occurs four times faster
than in humans. In other words, the much faster course of neural development in monkeys may
make their nervous system less open to the effects of early perceptual experience.
The results indicated that the vervets matched the rhesus audible and visible calls and, thus, were
consistent with the second prediction. The matching data were, however, somewhat surprising in
that the vervets exhibited greater looking at the non-matching visual call in the presence of the audi-
ble call than at the matching one (Fig. 7.4a). Additional experiments revealed that this effect was due
to the fear-inducing effects of the audible call. This was supported by the finding that when the
spectral information that is normally available in the natural call was replaced by a complex tone
stimulus whose duration matched that of the natural call and whose onset and offset correspond to
the matching visible call, the vervets switched their preference to the matching visual call (Fig 7.4b).
The fear hypothesis was further confirmed by measurements of pupil size. When the vervets listened
to the natural call they exhibited significantly larger pupil size when looking at the matching visible
call in the presence of the corresponding natural call than when looking at the non-matching visible
call. In contrast, when the vervets listened to the tone, they showed no difference in pupil size.
Because pupil size reflects arousal, the pupil data suggested that the greater looking at the non-
matching visible call in the presence of the natural call reflected intersensory matching. Together,
the findings from the vervet monkeys suggest that MPN is a recent evolutionary innovation and that
it reflects the effects of neoteny—the lengthening of the early developmental period in humans as
opposed to vervets. This lengthening presumably makes humans more open to the effects of early
REFERENCES 175

experience. If so, this suggests that other kinds of perceptual and cognitive skills that differentiate
between humans from non-human primates may have developmental roots (Gould 1977).

7.8 Conclusions
There is no longer any doubt that infants come into the world with some rudimentary multisensory
perceptual skills and that as they grow these skills improve and broaden in scope. The processes
that contribute to this broadening are neural growth and differentiation and the perceptual
experience that infants acquire as they grow. Paradoxically, however, experience also contributes
to the narrowing of multisensory responsiveness through the highly selective exposure that
infants have to native as opposed to non-native multisensory inputs. Furthermore, as our vervet
monkey study shows, it appears that MPN may be a recent evolutionary innovation and that it
is the direct result of the neotenous nature of humans as opposed to other primate species. If this
is the case then it is possible that the great apes, whose developmental period is more similar
in length to humans, might also undergo perceptual narrowing in both unisensory and multisen-
sory responsiveness.
One of the questions that the narrowing work raises concerns the role that the neural substrate
plays in perceptual narrowing. Some investigators have proposed that a ‘selectionist’ or regressive
mechanism of neural development partly accounts for the regressive nature of perceptual nar-
rowing (Nelson 2001; Scott et al. 2007). This view is based on theories that propose that neuronal
networks are initially diffuse and that subsequently more modularized networks emerge as expe-
rience ‘prunes’ exuberant connections. The problem with this view is that, except for some rela-
tively isolated cases, for the most part, the nervous system increases greatly in size and the number
of synapses increases enormously during development. This suggests that the selectionist view is
incorrect (Purves et al. 1996). Therefore, a more reasonable view is that narrowing is mediated by
a selective elaboration of synapses whose relevance is determined by postnatal experience and not
the selective pruning of irrelevant synapses (Lewkowicz and Ghazanfar 2009). Regardless of what
specific neural mechanism(s) underlie the MPN phenomenon, there is no doubt that MPN is
clear evidence of the intricate interplay between neural growth and differentiation, on the one
hand, and perceptual experience, on the other. Needless to say, it is the interaction between these
two processes that contributes to the ultimate emergence of multisensory perceptual expertise.

Acknowledgements
The writing of this chapter was supported by a grant from the National Science Foundation
(BCS-0751888).

References
Adachi, I., Kuwahata, H., Fujita, K., Tomonaga, M., and Matsuzawa, T. (2006). Japanese macaques form a
cross-modal representation of their own species in their first year of life. Primates, 47, 350–54.
Arnold, P., and Hill, F. (2001). Bisensory augmentation: A speech-reading advantage when speech is clearly
audible and intact. British Journal of Psychology, 92, 339–55.
Aslin, R.N., and Pisoni, D.B. (1980). Some developmental processes in speech perception. In Child
phonology: perception (eds. G. Yeni-Komshian, J.F., Kavangh, and C.A. Ferguson), pp. 67–96. Academic
Press, New York.
Aslin, R.N., Pisoni, D.B., Hennessy, B.L., and Percy, A.J. (1981). Discrimination of voice onset time by
human infants: New findings and implications for the effects of early experience. Child Development,
52, 1135–45.
176 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

Bahrick, L.E. (1983). Infants’ perception of substance and temporal synchrony in multimodal events. Infant
Behavior and Development, 6, 429–51.
Bahrick, L.E., and Lickliter, R. (2000). Intersensory redundancy guides attentional selectivity and perceptual
learning in infancy. Developmental Psychology, 36, 190–201.
Bahrick, L.E., Flom, R., and Lickliter, R. (2002). Intersensory redundancy facilitates discrimination of
tempo in 3-month-old infants. Developmental Psychobiology, 41, 352–63.
Bahrick, L.E., Lickliter, R., and Flom, R. (2004). Intersensory redundancy guides the development of
selective attention, perception, and cognition in infancy. Current Directions in Psychological Science,
13, 99–102.
Bertelson, P., and Radeau, M. (1981). Cross-modal bias and perceptual fusion with auditory-visual spatial
discordance. Perception and Psychophysics, 29, 578–84.
Best, C.T., Mcroberts, G.W., Lafleur, R., and Silver-Isenstadt, J. (1995). Divergent developmental patterns
for infants’ perception of two nonnative consonant contrasts. Infant Behavior and Development, 18,
339–50.
Birch, H.G., and Lefford, A. (1963). Intersensory development in children. Monographs of the Society for
Research in Child Development, 25, 1–47.
Birch, H.G., and Lefford, A. (1967). Visual differentiation, intersensory integration, and voluntary motor
control. Monographs of the Society for Research in Child Development, 32, 1–87.
Bradley, R., and Mistretta, C. (1975). Fetal sensory receptors. Physiological Reviews, 55, 352–82.
Bushara, K.O., Grafman, J., and Hallett, M. (2001). Neural correlates of auditory-visual stimulus onset
asynchrony detection. Journal of Neuroscience, 21, 300–304.
Cheour, M., Ceponiene, R., Lehtokoski, A., et al. (1998). Development of language-specific phoneme
representations in the infant brain. Nature Neuroscience, 1, 351–53.
Chiroro, P., and Valentine, T. (1995). An investigation of the contact hypothesis of the own-race bias in
face recognition. The Quarterly Journal of Experimental Psychology A: Human Experimental Psychology,
48A, 879–94.
Cohen, L.B., and Strauss, M.S. (1979). Concept acquisition in the human infant. Child Development, 50,
419–24.
Colombo, J. (2001). The development of visual attention in infancy. Annual Review of Psychology, 52,
337–67.
Decasper, A.J., and Fifer, W.P. (1980). Of human bonding: newborns prefer their mothers’ voices. Science,
208, 1174–76.
Delaney, S.M., Dobson, V., Harvey, E.M., Mohan, K.M., Weidenbacher, H.J., and Leber, N.R. (2000).
Stimulus motion increases measured visual field extent in children 3.5 to 30 months of age. Optometry
Visual Science, 77, 82–9.
Dufour, V., Pascalis, O., and Petit, O. (2006). Face processing limitation to own species in primates: A
comparative study in brown capuchins, Tonkean macaques and humans. Behavioural Processes, 73,
107–113.
Eimas, P.D., Siqueland, E.R., Jusczyk, P., and Vigorito, J. (1971). Speech perception in infants. Science, 171,
303–306.
Foxe, J.J., and Schroeder, C.E. (2005). The case for feedforward multisensory convergence during early
cortical processing. Neuroreport, 16, 419.
Ghazanfar, A.A. and Logothetis, N.K. (2003). Facial expressions linked to monkey calls. Nature, 423, 937–38.
Ghazanfar, A.A., and Schroeder, C.E. (2006). Is neocortex essentially multisensory? Trends in Cognitive
Sciences, 10, 278–85.
Ghazanfar, A., Turesson, H., Maier, J., Van Dinther, R., Patterson, R., and Logothetis, N. (2007). Vocal-tract
resonances as indexical cues in rhesus monkeys. Current Biology, 17, 425–30.
Gibson, E.J. (1969). Principles of perceptual learning and development. Appleton-Century-Crofts, New York.
REFERENCES 177

Gibson, E.J. (1984). Perceptual development from the ecological approach. In Advances in developmental
psychology, Vol. 3 (eds. M.E. Lamb, A.L. Brown, and B. Rogoff), pp. 243–86. Lawrence Erlbaum
Associates, Hillsdale, NJ.
Gibson, J.J. (1966). The senses considered as perceptual systems. Houghton-Mifflin, Boston, MA.
Gibson, J.J. (1979). An ecological approach to perception. Houghton-Mifflin, Boston, MA.
Gottlieb, G. (1971). Ontogenesis of sensory function in birds and mammals. In The biopsychology of
development (eds. E. Tobach, L.R. Aronson, and E. Shaw), pp. 67–128. Academic Press, New York.
Gottlieb, G. (1976). The roles of experience in the development of behavior and the nervous system.
In Development of neural and behavioral specificity (ed. G. Gottlieb), pp. 25–54. Academic Press,
New York.
Gottlieb, G. (1981). Roles of early experience in species-specific perceptual development. In Development of
perception: Psychobiological perspectives (R.N. Aslin, J.R. Alberts, and M.R. Petersen), pp. 5–44.
Academic Press, New York.
Gottlieb, G. (1991). Experiential canalization of behavioral development: results. Developmental Psychology,
27, 35–39.
Gottlieb, G. (1992). Individual development and evolution: The genesis of novel behavior. Oxford University
Press, New York.
Gottlieb, G. (1996). Developmental psychobiological theory. In Developmental science. Cambridge studies in
social and emotional development (eds. R.B. Cairns, and G.H. Elder, Jr.), pp. 63–78. Cambridge
University Press, New York.
Gould, S. (1977). Ontogeny and phylogeny. Belknap press, Cambridge, MA.
Hannon, E.E., and Trehub, S.E. (2005a). Metrical categories in infancy and adulthood. Psychological
Science, 16, 48–55.
Hannon, E.E., and Trehub, S.E. (2005b). Tuning in to musical rhythms: infants learn more readily than
adults. Proceedings of the National Academy of Sciences USA, 102, 12639–43.
Holt, E.B. (1931). Animal drive and the learning process. Holt, New York.
Honeycutt, H., and Lickliter, R. (2003). The influence of prenatal tactile and vestibular stimulation on
auditory and visual responsiveness in bobwhite quail: A matter of timing. Developmental Psychobiology,
43, 71–81.
Innes-Brown, H., Barutchu, A., Shivdasani, M.N., Crewther, D.P., Grayden, D.B., and Paolini, A.G. (2011).
Susceptibility to the flash-beep illusion is increased in children compared to adults. Developmental
Science, 14(5), 1089–1099. doi:10.1111/j.1467-7687.2011.01059.
Izumi, A., and Kojima, S. (2004). Matching vocalizations to vocalizing faces in a chimpanzee (Pan
troglodytes). Animal Cognition, 7, 179–84.
Jaime, M., and Lickliter, R. (2006). Prenatal exposure to temporal and spatial stimulus properties affects
postnatal responsiveness to spatial contiguity in bobwhite quail chicks. Developmental Psychobiology,
48, 233.
Jordan, K., Brannon, E., Logothetis, N., and Ghazanfar, A. (2005). Monkeys match the number of voices
they hear to the number of faces they see. Current Biology, 15, 1034–38.
Jusczyk, P.W. (1997). The discovery of spoken language. MIT Press, Cambridge, MA.
Kelly, D.J., Quinn, P.C., Slater, A.M., et al. (2005). Three-month-olds, but not newborns, prefer own-race
faces. Developmental Science, 8, F31–36.
Kelly, D.J., Quinn, P.C., Slater, A., Lee, K., Ge, L., and Pascalis, O. (2007). The other-race effect develops
during infancy: Evidence of perceptual narrowing. Psychological Science, 18, 1084–89.
Kelly, D.J., Liu, S., Lee, K., et al. (2009). Development of the other-race effect during infancy: evidence
toward universality? Journal of Experimental Child Psychology, 104, 105–114.
King, A.J., Hutchings, M.E., Moore, D.R., and Blakemore, C. (1988). Developmental plasticity in the visual
and auditory representations in the mammalian superior colliculus. Nature, 332, 73–76.
178 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

Kirkham, N.Z., Slemmer, J.A., and Johnson, S.P. (2002). Visual statistical learning in infancy: Evidence for
a domain general learning mechanism. Cognition, 83, B35–42.
Knudsen, E.I., and Brainard, M.S. (1991). Visual instruction of the neural map of auditory space in the
developing optic tectum. Science, 253, 85–87.
Kuhl, P.K., and Meltzoff, A.N. (1982). The bimodal perception of speech in infancy. Science, 218, 1138–41.
Kuhl, P.K., and Meltzoff, A.N. (1984). The intermodal representation of speech in infants. Infant Behavior
and Development, 7, 361–81.
Kuhl, P.K., Williams, K.A., Lacerda, F., Stevens, K.N., and Lindblom, B. (1992). Linguistic experience alters
phonetic perception in infants by 6 months of age. Science, 255, 606–608.
Kuhl, P.K., Tsao, F.M., and Liu, H.M. (2003). Foreign-language experience in infancy: effects of short-term
exposure and social interaction on phonetic learning. Proceedings of the National Academy of Sciences
USA, 100, 9096–9101.
Kuhl, P.K., Stevens, E., Hayashi, A., et al. (2006). Infants show a facilitation effect for native language
phonetic perception between 6 and 12 months. Developmental Science, 9, F13–21.
Kuo, Z.Y. (1976). The dynamics of behavior development: an epigenetic view. Plenum, New York.
LeCanuet, J., and Schaal, B. (1996). Fetal sensory competencies. European Journal of Obstetrics and
Gynecology, 68, 1–23.
Lewkowicz, D.J. (1986). Developmental changes in infants’ bisensory response to synchronous durations.
Infant Behavior and Development, 9, 335–53.
Lewkowicz, D.J. (1988a). Sensory dominance in infants: I. Six-month-old infants’ response to auditory-
visual compounds. Developmental Psychology, 24, 155–71.
Lewkowicz, D.J. (1988b). Sensory dominance in infants: II. Ten-month-old infants’ response to auditory-
visual compounds. Developmental Psychology, 24, 172–82.
Lewkowicz, D.J. (1992a). Infants’ response to temporally based intersensory equivalence: the effect of
synchronous sounds on visual preferences for moving stimuli. Infant Behavior and Development, 15,
297–324.
Lewkowicz, D.J. (1992b). Infants’ responsiveness to the auditory and visual attributes of a sounding/
moving stimulus. Perception and Psychophysics, 52, 519–28.
Lewkowicz, D.J. (1994). Development of intersensory perception in human infants. In The development of
intersensory perception: comparative perspectives (eds. D.J. Lewkowicz, and R. Lickliter), pp. 165–203.
Lawrence Erlbaum Associates, Hillsdale, NJ.
Lewkowicz, D.J. (1996a). Infants’ response to the audible and visible properties of the human face. I: Role
of lexical-syntactic content, temporal synchrony, gender, and manner of speech. Developmental
Psychology, 32, 347–66.
Lewkowicz, D.J. (1996b). Perception of auditory-visual temporal synchrony in human infants. Journal of
Experimental Psychology: Human Perception and Performance, 22, 1094–1106.
Lewkowicz, D.J. (1998). Infants’ response to the audible and visible properties of the human face: II.
Discrimination of differences between singing and adult-directed speech. Developmental Psychobiology,
32, 261–74.
Lewkowicz, D.J. (2000a). The development of intersensory temporal perception: an epigenetic systems/
limitations view. Psychological Bulletin, 126, 281–308.
Lewkowicz, D.J. (2000b). Infants’ perception of the audible, visible and bimodal attributes of multimodal
syllables. Child Development, 71, 1241–57.
Lewkowicz, D.J. (2002). Heterogeneity and heterochrony in the development of intersensory perception.
Cognitive Brain Research, 14, 41–63.
Lewkowicz, D.J. (2003). Learning and discrimination of audiovisual events in human infants: the
hierarchical relation between intersensory temporal synchrony and rhythmic pattern cues.
Developmental Psychology, 39, 795–804.
Lewkowicz, D.J. (2004). Perception of serial order in infants. Developmental Science, 7, 175–84.
REFERENCES 179

Lewkowicz, D.J. (2010). Infant perception of audio-visual speech synchrony. Developmental Psychology, 46,
66–77.
Lewkowicz, D.J., and Berent, I. (2009). Sequence learning in 4 month-old infants: do infants represent
ordinal information? Child Development, 80, 1811–23.
Lewkowicz, D.J., and Ghazanfar, A.A. (2006). The decline of cross-species intersensory perception in
human infants. Proceedings of the National Academy Sciences USA, 103, 6771–74.
Lewkowicz, D.J., and Ghazanfar, A.A. (2009). The emergence of multisensory systems through perceptual
narrowing. Trends in Cognitive Sciences, 13, 470–78.
Lewkowicz, D.J., and Hansen-Tift, A.M. (2012). Infants deploy selective attention to the mouth of a talking
face when learning speech. Proceedings of the National Academy of Sciences, 109(5), 1431–1436.
doi:10.1073/pnas.1114783109
Lewkowicz, D.J., and Kraebel, K. (2004). The value of multimodal redundancy in the development of
intersensory perception. In Handbook of multisensory processing (eds. G. Calvert, C. Spence, and B.E.
Stein), pp. 655–78. MIT Press, Cambridge, MA.
Lewkowicz, D.J., and Turkewitz, G. (1980). Cross-modal equivalence in early infancy: auditory-visual
intensity matching. Developmental Psychology, 16, 597–607.
Lewkowicz, D.J., Sowinski, R., and Place, S. (2008). The decline of cross-species intersensory perception in
human infants: underlying mechanisms and its developmental persistence. Brain Research, 1242, 291–302.
Lewkowicz, D.J., Leo, I., and Simion, F. (2010). Intersensory perception at birth: newborns match non-
human primate faces and voices. Infancy, 15, 46–60.
Lickliter, R. (1993). Timing and the development of perinatal perceptual organization. Developmental Time
and Timing (eds. G. Turkewitz, and D.A. Devenny), pp. 105–24. Lawrence Erlbaum Associates,
HIllsdale, NJ.
Lickliter, R. (2005). Prenatal sensory ecology and experience: implications for perceptual and behavioral
development in precocial birds. Advances in the Study of Behavior, 35, 235–74.
Lickliter, R., and Bahrick, L.E. (2000). The development of infant intersensory perception: advantages of a
comparative convergent-operations approach. Psychological Bulletin, 126, 260–80.
Lickliter, R., and Banker, H. (1994). Prenatal components of intersensory devleopment in precocial birds.
In The development of intersensory perception: comparative perspectives (eds. D.J. Lewkowicz and R.
Lickliter), pp. 59–80. Lawrence Erlbaum Associates, Hillsdale, NJ.
Lickliter, R., Lewkowicz, D.J., and Columbus, R.F. (1996). Intersensory experience and early perceptual
development: the role of spatial contiguity in bobwhite quail chicks’ responsiveness to multimodal
maternal cues. Developmental Psychobiology, 29, 403–416.
Lickliter, R., Bahrick, L.E., and Honeycutt, H. (2002). Intersensory redundancy facilitates prenatal perceptual
learning in bobwhite quail (Colinus virginianus) embryos. Developmental Psychology, 38, 15–23.
Lickliter, R., Bahrick, L.E., and Honeycutt, H. (2004). Intersensory redundancy enhances memory in
bobwhite quail embryos. Infancy, 5, 253–69.
Ludemann, P.M., and Nelson, C.A. (1988). Categorical representation of facial expressions by 7-month-old
infants. Developmental Psychology, 24, 492–501.
Macleod, A., and Summerfield, Q. (1987). Quantifying the contribution of vision to speech perception in
noise. British Journal of Audiology, 21, 131–41.
Maier, N.R.F., and Schneirla, T.C. (1964). Principles of animal psychology, New York, Dover Publications.
Marcovitch, S., and Lewkowicz, D.J. (2009). Sequence learning in infancy: the independent contributions
of conditional probability and pair frequency information. Developmental Science, 12, 1020–25.
Marcus, G.F., Vijayan, S., Rao, S., and Vishton, P. (1999). Rule learning by seven-month-old infants.
Science, 283, 77–80.
Marks, L. (1978). The unity of the senses. Academic Press, New York.
Mastropieri, D., and Turkewitz, G. (1999). Prenatal experience and neonatal responsiveness to vocal
expressions of emotion. Developmental Psychobiology, 35, 204–214.
180 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

McGurk, H., and Macdonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 229–39.
Meredith, M.A., and Stein, B.E. (1983). Interactions among converging sensory inputs in the superior
colliculus. Science, 221, 389–91.
Michel, G.F., and Tyler, A.N. (2005). Critical period: a history of the transition from questions of when, to
what, to how. Developmental Psychobiology. Special Issue: Critical Periods Re-examined: Evidence from
Human Sensory Development, 46, 156–62.
Morrongiello, B.A. (1988). Infants’ localization of sounds in the horizontal plane: estimates of minimum
audible angle. Developmental Psychology, 24, 8–13.
Morrongiello, B.A., Fenwick, K.D., and Chance, G. (1990). Sound localization acuity in very young infants:
An observer-based testing procedure. Developmental Psychology, 26, 75–84.
Morton, J., and Johnson, M.H. (1991). CONSPEC and CONLERN: A two-process theory of infant face
recognition. Psychological Review, 98, 164–81.
Nazzi, T., Bertoncini, J., and Mehler, J. (1998). Language discrimination by newborns: toward an
understanding of the role of rhythm. Journal of Experimental Psychology: Human Perception and
Performance, 24, 756–66.
Neil, P.A., Chee-Ruiter, C., Scheier, C., Lewkowicz, D.J., and Shimojo, S. (2006). Development of
multisensory spatial integration and perception in humans. Developmental Science, 9, 454–64.
Nelson, C.A. (2001). The development and neural bases of face recognition. Infant and Child Development,
10, 3–18.
Parr, L.A. (2004). Perceptual biases for multimodal cues in chimpanzee (Pan troglodytes) affect recognition.
Animal Cognition, 7, 171–78.
Partan, S., and Marler, P. (1999). Communication goes multimodal. Science, 283, 1272–3
Pascalis, O., and Bachevalier, J. (1998). Face recognition in primates: a cross-species study. Behavioural
Processes, 43, 87–96.
Pascalis, O., Haan, M.D., and Nelson, C.A. (2002). Is face processing species-specific during the first year of
life? Science, 296, 1321–23.
Pascalis, O., Scott, L.S., Kelly, D.J., et al. (2005). Plasticity of face processing in infancy. Proceedings of the
National Academy of Sciences USA, 102, 5297–300.
Patterson, M.L., and Werker, J.F. (1999). Matching phonetic information in lips and voice is robust in
4.5-month-old infants. Infant Behavior and Development, 22, 237–47.
Patterson, M.L., and Werker, J.F. (2002). Infants’ ability to match dynamic phonetic and gender
information in the face and voice. Journal of Experimental Child Psychology, 81, 93–115.
Patterson, M.L., and Werker, J.F. (2003). Two-month-old infants match phonetic information in lips and
voice. Developmental Science, 6, 191–96.
Piaget, J. (1952). The origins of intelligence in children, New York, International Universities Press.
Pick, H.L., Warren, D.H., and Hay, J.C. (1969). Sensory conflict in judgments of spatial direction.
Perception and Psychophysics, 6, 203–205.
Pons, F., Lewkowicz, D.J., Soto-Faraco, S., and Sebastián-Gallés, N. (2009). Narrowing of intersensory
speech perception in infancy. Proceedings of the National Academy of Sciences USA, 106, 10598–602.
Poulin-Dubois, D., Serbin, L.A., Kenyon, B., and Derbyshire, A. (1994). Infants’ intermodal knowledge
about gender. Developmental Psychology, 30, 436–42.
Poulin-Dubois, D., Serbin, L.A., and Derbyshire, A. (1998). Toddlers’ intermodal and verbal knowledge
about gender. Merrill-Palmer Quarterly, 44, 338–54.
Purves, D., White, L., E., and Riddle, D., R. (1996). Is neural development darwinian? Trends in
Neurosciences, 19, 460–64.
Reardon, P., and Bushnell, E.W. (1988). Infants’ sensitivity to arbitrary pairings of color and taste. Infant
Behavior and Development, 11, 245–50.
Recanzone, G.H. (2009). Interactions of auditory and visual stimuli in space and time. Hearing Research,
258, 89–99.
REFERENCES 181

Rivera-Gaxiola, M., Silva-Pereyra, J., and Kuhl, P. (2005). Brain potentials to native and non-native speech
contrasts in 7- and 11-month-old American infants. Developmental Science, 8, 162–72.
Rosenblum, L.D., Schmuckler, M.A., and Johnson, J.A. (1997). The McGurk effect in infants. Perception
and Psychophysics, 59, 347–57.
Rowe, C. (1999). Receiver psychology and the evolution of multicomponent signals. Animal Behaviour, 58,
921–31.
Saffran, J.R., Aslin, R.N., and Newport, E.L. (1996). Statistical learning by 8-month-old infants. Science,
274, 1926–28.
Sai, F.Z. (2005). The role of the mother’s voice in developing mother’s face preference: evidence for
intermodal perception at birth. Infant and Child Development, 14, 29–50.
Sangrigoli, S., and De Schonen, S. (2004). Recognition of own-race and other-race faces by three-month-old
infants. Journal of Child Psychology and Psychiatry, 45, 1219–27.
Sangrigoli, S., Pallier, C., Argenti, A.M., Ventureyra, V.A.G., and De Schonen, S. (2005). Reversibility of the
other-race effect in face recognition during childhood. Psychological Science, 16, 440–44.
Scheier, C., Lewkowicz, D.J., and Shimojo, S. (2003). Sound induces perceptual reorganization of an
ambiguous motion display in human infants. Developmental Science, 6, 233–44.
Scott, L.S., and Monesson, A. (2009). The origin of biases in face perception. Psychological Science, 20,
676–80.
Scott, L.S., Pascalis, O., and Nelson, C.A. (2007). A domain general theory of the development of perceptual
discrimination. Current Directions in Psychological Science, 16, 197–201.
Sekuler, R., Sekuler, A.B., and Lau, R. (1997). Sound alters visual motion perception. Nature, 385, 308.
Shams, L., and Seitz, A. (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12, 411–417.
Shams, L., Kamitani, Y., and Shimojo, S. (2000). What you see is what you hear. Nature, 408, 788.
Sharma, J., Angelucci, A. and Sur, M. (2000). Induction of visual orientation modules in auditory cortex.
Nature, 404, 841–47.
Simion, F., Leo, I., Turati, C., Valenza, E., and Dalla Barba, B. (2007). How face specialization emerges in
the first months of life. Progress in Brain Research, 164, 169–86.
Stein, B.E., and Meredith, M.A. (1993). The merging of the senses. Cambridge, MA, The MIT Press.
Stein, B.E., and Stanford, T.R. (2008). Multisensory integration: current issues from the perspective of the
single neuron. Nature Review Neuroscience, 9, 255–66.
Stein, B.E., Meredith, M.A., Huneycutt, W.S. and Mcdade, L. (1989). Behavioral indices of multisensory
integration: orientation to visual cues is affected by auditory stimuli. Journal of Cognitive Neuroscience,
1, 12–24.
Streeter, L.A. (1976). Language perception of 2-month-old infants shows effects of both innate mechanisms
and experience. Nature, 259, 39–41.
Turkewitz, G. (1994). Sources of order for intersensory functioning. In The development of intersensory
perception: comparative perspectives (eds. D.J. Lewkowicz and R. Lickliter), pp. 3–18. Lawrence Erlbaum
Associates, Hillsdale, NJ.
Turkewitz, G., and Kenny, P.A. (1982). Limitations on input as a basis for neural organization and
perceptual development: a preliminary theoretical statement. Developmental Psychobiology, 15,
357–68.
Turkewitz, G., and Kenny, P.A. (1985). The role of developmental limitations of sensory input on sensory/
perceptual organization. Journal of Developmental and Behavioral Pediatrics, 6, 302–306.
von Melchner, L., Pallas, S.L., and Sur, M. (2000). Visual behaviour mediated by retinal projections
directed to the auditory pathway. Nature, 404, 871–76.
Walker-Andrews, A.S. (1986). Intermodal perception of expressive behaviors: relation of eye and voice?
Developmental Psychology, 22, 373–77.
Walker-Andrews, A.S. (1997). Infants’ perception of expressive behaviors: differentiation of multimodal
information. Psychological Bulletin, 121, 437–56.
182 UNEXPECTED EFFECTS OF EXPERIENCE ON THE DEVELOPMENT OF PRIMATES

Walker-Andrews, A.S., Bahrick, L.E., Raglioni, S.S., and Diaz, I. (1991). Infants’ bimodal perception of
gender. Ecological Psychology, 3, 55–75.
Wallace, M.T., and Stein, B.E. (1997). Development of multisensory neurons and multisensory integration
in cat superior colliculus. Journal of Neuroscience, 17, 2429–44.
Wallace, M.T., and Stein, B.E. (2001). Sensory and multisensory responses in the newborn monkey
superior colliculus. Journal of Neuroscience, 21, 8886–94.
Wallace, M.T., Stein, B.E., and Ramachandran, R. (2006). Early experience determines how the senses will
interact: a revised view of sensory cortical parcellation. Journal of Neurophysiology, 101, 2167–72.
Walton, G.E., and Bower, T.G. (1993). Amodal representations of speech in infants. Infant Behavior and
Development, 16, 233–43.
Weikum, W.M., Vouloumanos, A., Navarra, J., Soto-Faraco, S., Sebastián-Gallés, N., and Werker, J.F.
(2007). Visual language discrimination in infancy. Science, 316, 1159.
Welch, R.B., and Warren, D.H. (1980). Immediate perceptual response to intersensory discrepancy.
Psychological Bulletin, 88, 638–67.
Werker, J.F., and Tees, R.C. (1984). Cross-language speech perception: Evidence for perceptual
reorganization during the first year of life. Infant Behavior and Development, 7, 49–63.
Werker, J.F., and Tees, R.C. (2005). Speech perception as a window for understanding plasticity and
commitment in language systems of the brain. Developmental Psychobiology. Special Issue: Critical
Periods Re-examined: Evidence from Human Sensory Development, 46, 233–34.
Werner, H. (1973). Comparative psychology of mental development. International Universities Press,
New York.
Zangenehpour, S., Ghazanfar, A.A., Lewkowicz, D.J., and Zatorre, R.J. (2009). Heterochrony and cross-species
intersensory matching by infant vervet monkeys. PLoS ONE, 4, e4302.
Chapter 8

The role of intersensory redundancy


in early perceptual, cognitive, and
social development
Lorraine E. Bahrick and Robert Lickliter

In the simplest of terms, attention refers to a selectivity of


response. Man or animal is continuously responding to some
events in the environment and not to others that could be
responded to (or noticed) just as well.
(Donald Hebb 1949, p. 4)

8.1 Introduction
The natural environment provides a flux of concurrent stimulation to all our senses, far more
than can be attended to at any given moment in time. Adults are exquisitely skilled at selectively
attending to specific features or aspects of objects and events, picking out information that is
relevant to their needs, goals, and interests, and ignoring irrelevant stimulation. For example, we
easily pick out a friend in a crowd, follow the flow of action in a ball game, and attend to the voice
of the speaker at a cocktail party in the context of competing conversations. We long ago learned
to pick out human speech from non-speech sounds and parse continuous speech into meaningful
words by ignoring variations across speakers, accents, and intonation. Similarly, we have learned
to parse the visual array into coherent objects and surfaces despite variation due to lighting and
shadow, and interruption of surfaces due to occlusion. These remarkable skills, easily taken for
granted by experienced perceivers, develop rapidly across infancy as a result of ongoing experi-
ence with objects and events (Kellman and Arterberry 1998; Lewkowicz and Lickliter 1994). This
rapid perceptual development entails improving attentional allocation and economy of informa-
tion pick-up for relevant aspects of the environment by attending to meaningful variability while
ignoring meaningless variability (E.J. Gibson 1969, 1988; E.J. Gibson and Pick 2000; Ruff and
Rothbart 1996).
A great deal of research and theory has been devoted to understanding how perception and
learning develop across the first years of life. In contrast, little developmental research has focused
on the processes that guide selective attention to relevant aspects and levels of stimulation in
the first place (see Driver 2001; Pashler 1998; Spence and Driver 2004, for useful reviews of
adult-based research). Like perceptual development, the progressive honing of selective attention
is also the result of ongoing experience with objects and events and provides the basis for further
perceptual learning and exploratory activity. In contrast with later development, the early
184 ROLE OF INTERSENSORY REDUNDANCY IN EARLY PERCEPTUAL, COGNITIVE, AND SOCIAL DEVELOPMENT

development of attentional selectivity is thought to be more influenced by the infant’s sensitivity


to salient properties of stimulation such as contrast, movement, intensity (e.g. Kellman and
Arterberry 1998; Lewkowicz and Turkewitz 1980), and intersensory redundancy (overlapping
information across auditory, visual, tactile, and/or proprioceptive stimulation for properties
of objects and events; see Bahrick 2010; Bahrick and Lickliter 2002). In this chapter, we explore
the powerful role of intersensory redundancy in guiding and shaping early selective attention and,
in turn, perception and learning. We review recent empirical and theoretical efforts to better
understand what guides the allocation of selective attention during early development and we
briefly discuss the implications of early selective attention for perceptual, cognitive, and social
development.

8.2 Perceiving unitary multisensory events in infancy: a


basic bootstrapping problem
The newborn infant faces a significant developmental challenge following birth: how to become
increasingly economical and efficient at attending to multisensory stimulation that is unitary
(coherent across the senses and originating from a single event) and relevant to their needs and
actions, while ignoring stimulation that is less relevant. This is a particularly challenging task, as
the environment provides far more stimulation from multiple objects and events than can be
attended to at any given time, each providing stimulation to multiple sense modalities concur-
rently. The infant must attend to variations in incoming stimulation that are meaningful, rele-
vant, and coherent (e.g. coordinated changes in the face and voice of a single speaker amidst
unrelated changes in other objects, people, and events nearby; goal-directed human actions
amidst irrelevant movements of people, objects, and events) and ignore other variations that are
relatively meaningless (differences in lighting and shadow across cohesive objects, variations in
speaker voice or intonation across the same phoneme). What factors might determine which
information is selected and attended to by young infants and which information is typically
ignored during early development?
Evidence accumulated over several decades of infancy research suggests that selective attention
is more stimulus-driven during early postnatal development and with experience becomes
increasingly endogenous and modulated by top-down processes, including the individual’s goals,
plans, and expectations (see Colombo 2001; Haith 1980; Johnson et al. 1991; Ruff and Rothbart
1996). Thus, for experienced perceivers, prior knowledge, categories, goals, plans, and expecta-
tions typically guide information pick-up (e.g. Bartlett 1932; Chase and Simon 1973; Neisser
1976; Schank and Ableson 1977). What we know and what we expect to happen influence
where we allocate our attention and what information we pick up in the present as well as in
future encounters. What guides this process in young infants, who have little prior knowledge or
experience to rely on in the first months of postnatal life?

8.3 The salience of amodal information during


early development
Amodal information is information that is not specific to a particular sense modality. Rather, it is
information that can be conveyed redundantly across multiple senses, including fundamental
aspects of stimulation such as time, space, and intensity. A large body of research has indicated
that the detection of amodal information such as temporal synchrony, rhythm, tempo, and inten-
sity is a cornerstone of early perceptual development (see Bahrick 2004, in press; Bahrick and
SELECTIVE ATTENTION: THE FOUNDATION FOR PERCEPTION, LEARNING, AND MEMORY 185

Lickliter 2002; Lewkowicz 2000; Lewkowicz and Lickliter 1994). The finding that infants are adept
at perceiving amodal information is consistent with J.J. Gibson’s (1966, 1979) ecological view of
perception, which proposed that the different forms of stimulation available to the senses are not
a problem for perception, but rather provide an important basis for perceiving unitary objects
and events, such as a person speaking or a ball bouncing. Gibson proposed that our senses work
together as a unified perceptual system. For example, by attending to and perceiving amodal
information, there is no need to learn to integrate stimulation across the senses in order to per-
ceive unified objects and events, as proposed by constructivist accounts of early perceptual
and cognitive development (e.g. Piaget 1952, 1954). Perceiving amodal relations, combined with
an increasing sensitivity to the statistical regularities of the environment, effectively ensures
that young inexperienced perceivers preferentially attend to unified multimodal events, such as
people speaking, dogs barking, or keys jingling.
Temporal synchrony is the most fundamental type of amodal information. Temporal synchrony
refers to the simultaneous co-occurrence of stimulation across the senses (e.g. audiovisual) with
respect to onset, offset, and duration of sensory patterning. It is a higher-order, global amodal
property, in that it can be detected only by abstracting information across different sense modali-
ties (e.g. audible and visual changes) over time. Thus, it is inherently relational and abstract.
Furthermore, it facilitates the detection of nested amodal properties such as rhythm, tempo, and
duration across the senses (Bahrick 1992, 1994, 2001; E.J. Gibson 1969). Temporal synchrony has
been proposed as the ‘glue’ that effectively binds stimulation across the senses (see Bahrick and
Lickliter 2002; Bahrick and Pickens 1994; Lewkowicz 2000). For example, by attending to audio-
visual synchrony, the sounds and sights of a single person speaking will be perceived together as a
unified event. Detecting this synchronous information can prevent the accidental association of
unrelated but concurrent sensory stimulation, such as nearby conversations. The ‘ventriloquism
effect’ (Alais and Burr 2004; Radeau and Bertelson 1977; Warren et al. 1981) illustrates the power-
ful role of synchronous amodal information in guiding perception. Because the ventriloquist
moves the dummy’s mouth and body in synchrony with his own speech sounds, he or she creates
amodal information, which promotes the illusion that the dummy is speaking even though the
sound actually emanates from the ventriloquist’s mouth. Amodal information (audiovisual
temporal synchrony, rhythm, tempo, and intensity changes common to the dummy’s movements
and the sounds of speech) promotes the perception of a unitary event—the dummy speaking—
and effectively overrides information about the source or location of the sound. Young infants
show similar sensitivity to synchronous amodal information, even in the first months of life (e.g.
Bahrick 1988; Lewkowicz 1996; Morrongiello et al. 1998). Importantly, once infant attention is
focused on a ‘unitary’ audiovisual event, further perceptual differentiation of the unitary event can
then be promoted. This sequence sets the stage for coherent perceptual processing and in turn
provides a foundation for early cognitive and social development.

8.4 Selective attention: the foundation for perception,


learning, and memory
Attention entails exploratory behaviour such as orienting, eye movements, and active interaction
with the environment (e.g. reaching, head turning). These behaviours, provide continuous
and contingent feedback to our multiple senses. An obvious but nonetheless important insight is
that selective attention to stimulation generated from exploratory activity provides the basis for
what is perceived, learned, and remembered. In turn, what is perceived, learned, and remem-
bered, influence what is attended to in subsequent bouts of exploration, in continuous cycles of
186 ROLE OF INTERSENSORY REDUNDANCY IN EARLY PERCEPTUAL, COGNITIVE, AND SOCIAL DEVELOPMENT

attention ´  perception ´  learning ´  memory ´  attention, and so on. Figure 8.1


illustrates this dynamic system of influences and the fundamental role of selective attention for
perception, learning, and memory. Moreover, action is tightly coupled with these processes,
as exploratory activity provides new stimulation for attention, perception, learning, and memory
across continuous feedback loops (see Fig. 8.1; Adolph and Berger 2005; E.J. Gibson 1988; E.J.
Gibson and Pick 2000; von Hofsten 1983, 1993). This cycle can be characterized as a system of
dynamic, interactive influences that evolve over time, with concurrent changes in neurodevelop-
ment that go hand-in-hand with perception and action (see Adolph and Berger 2006; E.J. Gibson
1988; Thelen and Smith 1994, for discussion of such systems).
Surprisingly little scientific effort has been devoted to the study of attentional selectivity in
infancy (see Colombo 2001; Ruff and Rothbart 1996 for overviews), despite its obvious impor-
tance for perceptual, cognitive, social, and linguistic development. However, in recent years
investigators working at the neural, physiological, and behavioural levels of analysis have begun
to provide new insights into the nature and processes that guide attentional allocation to unimo-
dal and multimodal stimulation during early development (e.g. Hollich et al. 2005; Hyde et al.

Stimulation available
for exploration:
environmental and
self-stimulation

Action/
exploratory
activity
Perception

Attention Learning

Memory

Fig. 8.1 The critical role of selective attention in the development of perception, learning and
memory is depicted in two interrelated, concurrent feedback loops: (a) the attention–perception–
learning–memory system, and (b) the attention–perception–action system. The arrows represent the
primary direction of the flow of information. Selective attention to stimulation that results from
exploratory activity provides the basis for what is perceived, what is perceived provides the basis for
what is learned, and in turn what is remembered. This sequence in turn affects what is attended to
next and in subsequent encounters with similar stimulation. Perception is tightly coupled with
action via selective attention to the stimulation generated from exploratory activity in a continuous
feedback loop. (Reproduced from Theories of Infant Development, Bremner, and A. Slater, The
development of perception in a multimodal environment, Bahrick, L.E., pp. pp. 90–120 © 2004,
John Wiley & Sons, Ltd., with permission.)
THE INTERSENSORY REDUNDANCY HYPOTHESIS 187

2009; Reynolds et al. 2010; Richards et al. 2010). This work emphasizes the salience of multimodal
stimulation for early attention allocation. It is clear that infants quickly establish efficient patterns
for selectively attending to relevant and coherent aspects of the environment, and these patterns
become increasingly efficient with experience, eventually evolving into the expert patterns of
adult selective attention. A central issue for developmental science is to uncover what principles
govern this process. We have proposed and provided support for the intersensory redundancy
hypothesis (IRH), a framework describing four general principles that we think guide this devel-
opmental process (Bahrick 2010; Bahrick and Lickliter 2000, 2002; Bahrick et al. 2004a; Lickliter
and Bahrick 2004). These principles are all an outcome of infants’ sensitivity to intersensory
redundancy in attentional allocation, perceptual processing, and learning and memory during
the first months of postnatal life. A large body of research indicates that intersensory redundancy
promotes attention and perceptual processing of some properties of stimulation at the expense of
others, particularly in early development when attentional resources are most limited. We have
argued that this has a profound effect on the nature and trajectory of early development (Bahrick
and Lickliter 2002).

8.5 The intersensory redundancy hypothesis: a framework


for early perceptual development
Intersensory redundancy is provided by an event when the same amodal information (rhythm,
tempo, intensity changes) is simultaneously available and temporally synchronized across two or
more sense modalities. For example, when the rhythm and tempo of speech can be perceived by
looking and by listening, the rhythm and tempo are redundantly specified. Most naturalistic,
multimodal events provide intersensory redundancy for multiple properties (e.g. tempo, rhythm,
duration, intensity). By definition, only amodal properties (as opposed to modality-specific
properties) can be redundantly specified across the senses. Typically, a given event (such as a
person speaking) also provides non-redundant modality-specific information, such as the appear-
ance of the face, the colour of clothing, and the specific acoustic qualities of the voice. What
guides selective attention to these various properties of events during bouts of exploration?
Infant-based research consistently indicates that redundancy across the senses (both global and
nested amodal properties) promotes attention to redundantly specified properties of objects and
events at the expense of other (non-redundantly specified) stimulus properties, particularly in
early development when attentional resources are most limited (e.g. Bahrick and Lickliter 2000,
2002; Bahrick et al. 2010; Lewkowicz 2000; Lickliter and Bahrick 2004). Later in development,
attention is extended to less salient, non-redundantly specified properties. Factors such as com-
plexity, familiarity, the length of exploratory time, and the level of expertise of the perceiver can
affect the speed of progression through this salience hierarchy.
The IRH (Bahrick and Lickliter 2000, 2002) is a model of how selective attention guides early
perceptual development. It provides a framework for understanding how and under what condi-
tions attention is allocated to amodal versus modality-specific aspects of stimulation. The IRH
addresses how young infants, with no prior knowledge of the world, rapidly come to perceive
unitary events and attend to stimulation that is relevant to their needs and actions. Although the
IRH is primarily a framework for describing the early development of attention and intermodal
perception, the principles can also apply across the lifespan, particularly when attentional
resources are limited (for example, for difficult tasks or conditions of high cognitive load).
The IRH consists of four specific predictions. Two predictions address the nature of selective
attention to different properties of objects and events. The remaining two are developmental
predictions that address implications of the IRH across the life span. The first prediction describes
188 ROLE OF INTERSENSORY REDUNDANCY IN EARLY PERCEPTUAL, COGNITIVE, AND SOCIAL DEVELOPMENT

the salience of redundantly specified amodal properties in multimodal synchronous stimulation


(intersensory facilitation). The second describes the salience of non-redundantly specified,
modality-specific properties in unimodal stimulation (unimodal facilitation). The third predic-
tion holds that across development infants become more efficient and flexible processors, leading
to detection of both redundantly and non-redundantly specified properties in unimodal and
multimodal stimulation. The fourth prediction holds that intersensory and unimodal facilitation
are most pronounced for tasks of relatively high difficulty in relation to the expertise of the
perceiver, and thus are likely apparent under some conditions across the lifespan.
These four predictions have been supported by empirical studies with both human and
non-human animal infants and across non-social and social domains. Below we describe the four
predictions of the IRH in more detail and briefly review the research findings that support each
prediction. Table 8.1 provides a summary of the convergent research findings from our labs that
have supported each of the four predictions to date.

8.5.1 Prediction 1: intersensory facilitation


Redundantly specified, amodal properties are highly salient and detected more easily in bimodal
synchronous stimulation than are the same amodal properties in unimodal stimulation.

According to the first and most fundamental prediction of the IRH, intersensory redundancy (the
synchronous alignment of stimulation across two or more senses) recruits infant attention to
redundantly-specified properties of events (amodal properties such as tempo, rhythm, duration,
and intensity), effectively causing them to become ‘foreground’ and other stimulus properties to
become ‘background’ in a given bout of exploration. For example, intersensory redundancy has
been shown to be so salient that it allows young infants to selectively attend to one of two super-
imposed events while ignoring the other. When the soundtrack to one film (e.g. a hand striking
the keys of a toy xylophone) is presented to 4-month-old infants, they can selectively follow the
flow of action even when the film is superimposed on another film (e.g. a toy slinky or a hand-
clapping game). In other words, the sound-synchronized film appears to ‘pop out’ and become
foreground while the silent film becomes background, indicated by the fact that infants respond
to the background film as novel when given a novelty preference test (Bahrick et al. 1981).
Research from our respective labs (see Table 8.1) has demonstrated that intersensory redun-
dancy promotes enhanced attention and perceptual discrimination and learning in human and
non-human animal infants (Bahrick and Lickliter 2000, 2002; Bahrick et al. 2002a; Flom and
Bahrick 2007, 2010; Lickliter et al. 2002, 2004). For example, in the domain of non-social events,
human infants detect the rhythm and tempo of a toy hammer tapping when they experience the
synchronous sights and sounds together (providing intersensory redundancy), but not when they
experience the rhythm or tempo in one sense modality alone or when the sights and sounds are
presented out of synchrony (providing no intersensory redundancy, see Bahrick and Lickliter
2000; Bahrick et al. 2002a). In particular, our habituation studies demonstrate that infants show
visual recovery to a change in tempo or a change in rhythm only in the context of redundant
stimulation (i.e. when they can see and hear the hammer tapping in synchrony). In contrast, they
show no visual recovery to this change in tempo or rhythm in the context of non-redundant
stimulation (i.e., unimodal visual or asynchronous audiovisual hammer tapping). Our data also
suggest evidence of intersensory facilitation for detection of tempo changes in adult participants
(Bahrick et al. 2009a). Research from other laboratories has also provided support for intersen-
sory facilitation. For example, four-month-old infants have been shown to detect the serial order
of events in synchronous audiovisual but not unimodal auditory or unimodal visual stimulation
(Lewkowicz 2004). Seven-month-old infants can detect numerical information in audiovisual
Table 8.1 Studies providing support for the intersensory redundancy hypothesis. Most studies listed under ‘Developmental change’ also include specific age
groups which provide support for ‘Intersensory facilitation’ or ‘Unimodal facilitation’ as well. Thus these studies are entered again under the respective cate-
gories. Under findings, type of stimulation is defined as: A, unimodal auditory (soundtrack accompanied by static image); V, unimodal visual (dynamic visual,
no soundtrack); AV, bimodal dynamic audiovisual; Sync, synchronous (soundtrack temporally aligned with video); Async, asynchronous (sound unsystemati-
cally out of phase with video). All infant studies were conducted using infant-control habituation procedures, and some also included two-choice preference
phases. Discrimination is inferred from visual recovery to a change in infant-control habituation procedures, as well as from preference data. All studies test-
ing bobwhite quail were conducted using two-choice preference procedures. Studies testing children and adults were conducted using forced-choice judg-
ments. Recognition memory is inferred from two-choice novelty preference data and forced-choice judgments

Prediction Study Event property Domains assessed Stimuli Subjects Findings


1. Intersensory Bahrick and Rhythm Discrimination Non-social Infants: 5 months Discriminate rhythm change in AV-sync but not
facilitation Lickliter (2000) (hammer tapping) V, A, or AV-async
Bahrick et al. Tempo Discrimination Non-social Infants: 3 months Discriminate tempo change in AV-sync but
(2002a) (hammer tapping not V or A
fast vs slow)
Lickliter et al. Multiple Discrimination, Social Quail embryos Learn and remember call in AV-sync but not A
(2002) temporal recognition memory (maternal and chicks or AV-async across prenatal to postnatal life
properties call A vs B)

THE INTERSENSORY REDUNDANCY HYPOTHESIS


Lickliter et al. Multiple Discrimination, long- Social Quail embryos Learn and remember AV-sync but not A
(2004) temporal term recognition (maternal and chicks across 4 days
properties memory call A vs B)
Lickliter et al. Multiple Discrimination, Social Quail embryos Discriminate a maternal call with AV-sync→A
(2006) temporal recognition memory, (maternal and chicks exposure, but not with A→AV-sync exposure or
properties educating attention call A vs B) AV-async →A exposure
Castellanos Tempo Discrimination, Non-social Infants: 2 months Discriminate tempo in V habituation following
et al. (2006) educating attention (hammer tapping AV-sync but not V pre-exposure
fast vs slow)
Flom and Affect Discrimination Social Infants: 3, 4, 3 months, no discrimination
Bahrick (2007) (women 5, 7 months 4 months, discriminate AV-sync
speaking- happy, 5 months, discriminate AV-sync, A
sad, angry) 7 months, discriminate AV-sync, A, V
(Continued )

189
190
ROLE OF INTERSENSORY REDUNDANCY IN EARLY PERCEPTUAL, COGNITIVE, AND SOCIAL DEVELOPMENT
Table 8.1 (continued) Studies providing support for the intersensory redundancy hypothesis

Prediction Study Event property Domains assessed Stimuli Subjects Findings


Bahrick, Todd Tempo Discrimination, Non-social Adults More correct responses (same/different
et al. (2009) recognition memory (hammer tapping, judgments) for AV-sync than V; main effect of
four tempos) task difficulty
Jaime et al. Synchrony Discrimination, Social (maternal Quail embryos Onset AV-sync sufficient to enhance learning of
(2010) educating attention call A vs B) and chicks maternal call
Bahrick et al. Prosody of Discrimination; Social (women Infants: 4 months Discriminate approval vs prohibition in AV-sync
(submitted) speech categorization speaking: but not A or AV-async
approval vs
prohibition)
Vaillant-Molina Object-affect Discrimination, Social (contingent Infants: 5.5 Social referencing: discriminate affective
and Bahrick relations recognition memory robot vs pony) months expressions and the object to which they refer in
(submitted) AV-sync but not V; prefer to touch objects
paired with happy expression
Castellanos Prosody of Discrimination, Social (women Infants: 3 months Discriminate prosody in A habituation following
and Bahrick speech educating speaking AV-sync but not A or AV-async pre-exposure
(2008) attention -approval vs
prohibition)
2. Unimodal Bahrick et al. Pitch of voice Discrimination Social 3, 4 months 3 months, discriminate voice in A but not
facilitation (2005) (women AV-sync
speaking) 4 months, discriminate voice in A and AV-sync
Bahrick et al. Orientation Discrimination Nonsocial Infants: 3,5, 8 3 months and 5 months, discriminate
(2006) (hammer tapping months orientation in V but not AV-sync
up vs down) 8 months, discriminate in V and AV-sync
3 months, controls discriminate in AV-async
Vaillant et al. Pitch Discrimination Social (maternal Quail embryos Detect a change in pitch following A but not
(2009) call pitch A vs B) and chicks AV-sync exposure
Bahrick, Facial Discrimination, Social Children: 4 yrs Face discrimination and memory in V and
Krogh- configuration recognition memory (women AV-async but not AV-sync
Jesperson, et al. speaking)
(submitted)
Flom and Orientation Discrimination, Non-social Infants:3, 5, 9 Memory across a 1-month delay for V but not
Bahrick (2010) long-term memory (hammer tapping months AV-sync at 5 months
up vs down)
3. Bahrick and Rhythm and Discrimination Nonsocial Infants: 3, 5, Older infants discriminate tempo and rhythm
Developmental Lickliter (2004) tempo (hammer tapping) 8 months change in AV-sync and V; younger infants
change discriminate in AV-sync but not V
Bahrick et al. Pitch of voice Discrimination Social (women Infants: 3, 3 months, discriminate voice in A but not
(2005) speaking) 4 months AV-sync
4 months, discriminate voice in A and AV-sync
Bahrick et al. Orientation Discrimination Non-social Infants: 3, 3 months and 5 months, discriminate V but not
(2006) (hammer tapping 5, 8 months AV-sync
up vs down) 8 months, discriminate V and AV-sync
3 months, controls discriminate AV-async
Flom and Affect Discrimination Social Infants: 3, 4, 3 months, no discrimination
Bahrick (2007) (women 5, 7 months 4 months, discriminate AV-sync
speaking-happy, 5 months, discriminate AV-sync, A

THE INTERSENSORY REDUNDANCY HYPOTHESIS


sad, angry) 7 months, discriminate AV-sync, A, V
Flom and Orientation Discrimination, Non-social Infants: 3, 5, Memory across a 1-month delay emerged by
Bahrick (2010) long-term memory (hammer tapping 9, months 5 months for V (but not AV-sync) and by 9 months
up vs down) for V and AV-sync; memory was expressed as