Académique Documents
Professionnel Documents
Culture Documents
CITATION
Matheson, H., White, N., & McMullen, P. (2014, October 13). Accessing Embodied Object
Representations From Vision: A Review. Psychological Bulletin. Advance online publication.
http://dx.doi.org/10.1037/bul0000001
Psychological Bulletin
2014, Vol. 141, No. 1, 000
Nicole White
Dalhousie University
University of Toronto
Patricia McMullen
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Dalhousie University
Theories of embodied cognition (EC) propose that object concepts are represented by reactivations of
sensorimotor experiences of different objects. Abundant research from linguistic paradigms provides
support for the notion that sensorimotor simulations are involved in cognitive tasks like comprehension.
However, it is unclear whether object concepts, as accessed from the visual presentation of objects, are
embodied. In the present article we review a large body of visual cognitive research that addresses 5 main
predictions of the theory of EC. First, EC accounts predict that visual presentation of manipulable objects,
but not nonmanipulable objects, should activate motor representations. Second, EC predicts that sensorimotor activity is necessary to perform visual cognitive tasks such as object naming. Third, EC posits
the existence of distinct neural ensembles that integrate information from action and vision. Fourth, EC
predicts that relationships between visual and motor activity change throughout development. Fifth, EC
predicts that the visual presentation of objects or actions should prime performance cross-modally. We
summarize findings from neuroimaging, neuropsychology, neurophysiology, development, and behavioral paradigms. We show that while much of the research published so far demonstrates that there is a
relationship between visual and motoric representations, there is no evidence supporting a strong form
of EC. We conclude that sensorimotor simulations may not be required to perform visual cognitive tasks
and highlight a number of directions for future research that could provide strong support for EC in visual
cognitive paradigms.
Keywords: visual object recognition, vision and action, theory of embodied cognition
Since the cognitive revolution of the 1960s, informationprocessing theories have dominated experimental psychology, providing models for understanding human cognition and shaping the
disciplines ideas about what constitutes mental processes. These
ideas have their roots in computational metaphors (e.g., the mind
as software executed in the brains hardware). Whether explicitly
or implicitly, most theories in cognitive psychology assume that
the brain is like a computer, processing sensory input with symbolic algorithms and producing a meaningful output (i.e., behavior). These ideas are perhaps most strongly reflected in the seminal
works of Fodor (1983) and Marr (2010). Classic applications of
this framework have resulted in models of cognitive processing
that are largely modular, with different processes specialized for
particular types of information and transforming it in particular
ways. An important feature of these theories is the notion that the
sensorimotor systems used to initially perceive the world are not
used to represent the world; that is, the cognitive processes carried
Heath Matheson, Department of Psychology and Neuroscience, Dalhousie University; Nicole White, Department of Psychology, University of
Toronto; Patricia McMullen, Department of Psychology and Neuroscience,
Dalhousie University.
Correspondence concerning this article should be addressed to Heath
Matheson, Department of Psychology, Dalhousie University, Life Sciences
Center, Halifax, Nova Scotia, B3H 4J1. E-mail: heathmatheson@dal.ca
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
EC hypotheses.1 The category that we focus on here is the conceptualization hypothesis (Shapiro, 2011). According to researchers endorsing this hypothesis, cognitive processes are not only
constrained by the body and its modal sensory systems, but cognitionidentifying objects, reasoning about people and places,
solving emotional problems and planning for the futurealso
comprises reactivating the same sensory and motor regions involved in perception. According to this view of embodied cognition, to know that an object lying on the table is a hammer, we
must simulate (or reactivate) our experiences with a hammer
what it looks like, the sounds it makes, how to use it. Additionally,
in the future, when thinking about a hammer, we use these simulations as the basis of cognitive processing. Indeed, all of our
thoughts about hammers will rely on these modality-specific simulations in some way. By extension (and perhaps this is the most
radical consequence of the conceptualization hypothesis), even our
abstract thoughts rely on these simulationsthoughts about hammering our argument home or what it means to hammer away at
a task (e.g., Lakoff & Johnson, 1999). It is the conceptualization
hypothesis that has attracted the most attention in experimental
cognitive psychology and cognitive neuroscience and underlies the
present review.
Models of Embodiment
Though there is no single agreed-upon neuropsychological
model of EC, there are at least two that provide a strong theoretical
foundation.2 Fundamentally, both of these models share the assumption that Hebbian processes of association (see Hebb, 1949,
p. 62) underlie the functional organization of the brain. That is, the
particular collections of neurons activated by the experience of a
particular event (e.g., the visual, tactile, and auditory experiences
of hammering) become functionally connected by virtue of being
active at the same time and are thus more likely to be activated
together at a later time (see also Wennekers, Garagnani, & Pulvermller, 2006). There is now compelling neuroscientific evidence for such associative processes. Research in molecular neuroscience supports the notion that neural connections change
dynamically as a result of functional coactivation with other neuronal populations (see Seung, 2012, for a discussion).
Meyer and Damasio (2009) developed one model they call the
convergence divergence zone (CDZ) framework. This model is
organized into functional hierarchies. At the lowest level are neural
ensembles that process basic sensory information in the different
modalities. For instance, distinct neural ensembles have evolved to
1
The first is what Shapiro (2011) calls the replacement hypothesis. For
researchers endorsing this hypothesis, mental representations (i.e., amodal
symbol manipulation) are not needed for cognitive processes; consequently, cognition is not modeled as symbolic and algorithmic. For instance, robots that have an embodied neural architecture, in which
cognition is modeled as a dynamic, online interaction with the environment
without any explicit representational system, can navigate complex environments much more effectively than robots based on traditional information processing architectures (see Brooks, 1991). The second hypothesis
described by Shapiro is the constitution hypothesis. Here, researchers
believe that while mental representations exist and are necessary, cognitive
processes carried out on representations comprise both the body and the
environment; thus the body and environment are both critical components
of knowledge (e.g., Clark & Chalmers, 1998). For instance, when a person
uses a grocery list to remember which items to pick up on the way home,
the constitution hypothesis would argue that the grocery list is a form of the
cognitive process. Although these two embodied cognitive hypotheses are
intriguing, and they have obvious consequences for understanding cognitive science in the broader sense, they have had relatively little impact on
experimental cognitive psychology and cognitive neuroscience; such lines
of thought are more concerned with artificial intelligence and basic questions about the ontology of the concept of cognition.
2
Within cognitive psychology, ideas about embodiment have appeared
throughout its history. Allport (1985), was one of the first cognitive
psychologists to posit that the same neural elements that are involved in
coding the sensory attributes of a (possibly unknown) object presented to
eye or hand or ear also make up the elements of the auto-associated
activity-patterns that represent familiar object-concepts in semantic memory (p. 53). Further, Gibsons (1986) affordance hypothesis is often
described as an early form of embodiment. Indeed, ideas of embodiment
can be traced to James (1890), who suggested that never is the body felt
all alone, but always together with other things (Chapter 10, para. 16) and
also speculated about the role of the body in generating emotional reactions. Finally, Hommel, Msseler, Aschersleben, & Prinzs (2001) theory
of event coding shares some strong conceptual similarities to the models
discussed here, though it is neuroagnostic and not modality-specific.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Together, the CDZ (Meyer & Damasio, 2009) and perceptualsymbol-systems (Barsalou et al., 2003) models serve as a general
framework for understanding how the brain might represent a
concept like hammer. They are useful in generating specific predictions stemming from the conceptualization hypothesis and
guide much of the review of evidence presented here.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
beyond the scope of the review. Our literature search was conducted between October 2012 and March 2014. In the final review,
we discuss four studies from neuroimaging, six from neuropsychology, two from neurophysiology, two from development, and
three from behavioral studies (summarized in Table 1) and draw
on additional sources to highlight our main criticisms of the
interpretation of these studies.
On the basis of our review of the current evidence, we argue that
findings in diverse areas of psychology and cognitive neuroscience
do not provide evidence for a strong form of EC in visual object
processing. In the next section, we examine studies relevant to EC
and attempt to identify the main problems with interpreting their
results as evidence for embodied object representations. While this
review does not speak against the general utility of EC, it does
provide a comprehensive integration of research examining EC
and object representation and helps to constrain the development
of the theory of embodied cognition as a way of understanding the
mind and brain. We conclude by discussing the types of evidence
that will be needed to support an EC account of object concepts
from vision.
Neuroimaging
One of the most obvious predictions of the EC account of object
representations is that viewing manipulable objects should elicit
activity in both visual and motor cortices, while viewing nonmanipulable objects should elicit activity in visual cortices only. It is
well established that, after reaching primary visual cortex, visual
information is processed in separate streams; Information about
object shape and color is processed in the ventral stream, extending
from V1 into the inferior temporal lobes, while information about
an objects movement, its location in space, and the metrics needed
to grasp it are processed dorsally, in the posterior middle temporal
gyrus and regions of the anterior inferior parietal lobe (see Milner
& Goodale, 1995). Despite interactions between these two streams
during higher visual processes (Matheson & McMullen, 2010), the
notion that there are separate ventral and dorsal cortical streams is
generally taken as evidence that different object features are encoded in distributed neural networks (e.g., form vs. motion or
action; see Chao, Haxby, & Martin, 1999; see also ThompsonSchill, 2003). This claim was supported by PET research (Martin,
Wiggs, Ungerleider, & Haxby, 1996) showing that although naming images of animals and tools activated overlapping ventral
visual cortical structures, there were category-specific activations
in the occipital lobe in response to animals and unique premotor
activations in response to tools. An EC account of object representation is consistent with this pattern of brain activity.
Additional research has shown that visual presentation of manipulable objects activates motor regions. PET studies have shown
activation in premotor cortex in response to tools during passive
viewing, naming, and describing a tools function, suggesting
motor involvement in representing objects (e.g., Grafton, Fadiga,
Arbib, & Rizzolatti, 1997). fMRI results have further revealed
frontoparietal activity during passive viewing of manipulable objects (e.g., Chao & Martin, 2000). Dorsal regions are activated
when participants imagine manipulating visually presented objects, including the inferior parietal cortex, the prefrontal cortex,
the motor cortex (in some cases), and the supplementary motor
area (see Grzes & Decety, 2001). Similar findings have been
Note.
EC embodied cognition.
Behavior
Craighero et al., 1996
Development
Kalnine & Bonthoux, 2008
Object naming
Main finding
Lhermitte, 1983
Object naming
Neuropsychology
Negri et al., 2007; Papeo et al.,
2010
Neuroimaging
Martin et al., 1996
Study
Table 1
Summary of Studies of Interest Relevant to the Main Predictions of EC Reviewed
Relevance to EC
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Conclusion
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
reported in an upright/inverted orientation judgment task of manipulable objects (Grzes & Decety, 2002; see also Devlin et al.,
2002, for a relevant meta-analysis). Overall, these neuroimaging
studies are consistent with the theory of ECs prediction of motor
involvement for manipulable objects but not nonmanipulable objects.
However, the neuroimaging findings are subject to an important
limitation in that they are strictly correlational. Findings of motor
activation during the visual presentation of manipulable objects
cannot be used to argue that such activity is necessary for object
perception; activity in motor regions may be inconsequential for
performing cognitive tasks like object naming (see also Mahon &
Caramazza, 2008). Alternatively, observed frontoparietal activity
might simply reflect a covert, automatic preparation for action on
an object (e.g., grasping) that has nothing to do with the cognitive
task (i.e., object naming). Thus, the neuroimaging evidence does
not allow us to make strong claims about the role of motor activity
in cognitive tasks and provides only correlational evidence.
Neuropsychology
Results from studies of the cognitive performance of braindamaged individuals are relevant to the predictions of embodied
object representations. Specifically, damage to modality specific
cortices used during simulations should impair cognitive performance (e.g., object naming). It is well documented that lesions to
different regions of the cortex can result in category-specific
deficits in object recognition. Warrington and Shallice (1984), for
example, described patients who had deficits in identifying animals but relatively intact identification of tools. The reverse pattern has also been shown (e.g., Gonnerman, Andersen, Devlin,
Kempler, & Seidenberg, 1997; see also Hillis & Caramazza, 1991,
for a double dissociation between manipulable objects and nonmanipulable objects; see Capitani, Laiacona, Mahon, & Caramazza, 2003, for a discussion of the interpretation of these dissociations). A double dissociation can suggest that object
representations are distributed over different cortical regions and
provides evidence that different experiences with manipulable and
nonmanipulable objects result in distinct neural representations
(see also Masullo et al., 2012). However, given the nature and
extent of the lesions reported in these studies (i.e., clearly defined
lesions to motor regions are not always present), double dissociations do not provide strong support for the notion that motor
regions are necessary for processing manipulable objects.
The importance of sensorimotor activity in processing manipulable objects is more comprehensively addressed in studies of
patients with visual apraxia. Apraxia is a multifaceted disorder
resulting in impaired object use (i.e., a deficit in action production)
after lesions to different cortical regions (see Petreska, Adriani,
Blanke, & Billard, 2007, for a detailed review; see De Renzi,
Faglioni, & Sorgato, 1982). Importantly, the EC account of object
representation makes a clear prediction about the effects of apraxic
lesions on object processing. Specifically, if motor simulations
play a role in representing knowledge about an object, then apraxic
patients unable to use intact simulations due to cortical damageshould show deficits in visual cognitive tasks (e.g., naming,
categorizing, etc.) pertaining to manipulable objects. This prediction has been supported by Negri et al. (2007), who reported four
patients who showed deficits in a simple object-naming task while
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Neurophysiology
The discovery of mirror neurons in the monkey frontal lobe (i.e.,
neurons that respond during both the performance and observation
of action) has prompted much speculation about the brains capacity to simulate other peoples actions, and it is often argued that
mirror neurons are the basis of action understanding (see Rizzolatti
& Craighero, 2004). Indeed, mirror neurons appear to validate
many assumptions made by EC and serve as a neurophysiological
basis for simulation-based models such as CDZ and perceptual
symbols systems (see Gallese & Sinigaglia, 2011).
However, the role of mirror neurons in representing object
concepts (rather than action concepts) is unclear. There is a subclass of visuomotor neurons called canonical neurons in area F5 of
the macaque frontal lobe. These neurons preferentially respond to
the presentation of objects or the performance of specific actions
on those objects (see Murata et al., 1997). For instance, cells that
fire when the animal performs precision grip movements also fire
during the visual presentation of small objects that afford precision
grips; similarly, those that fire during power grasps also respond
preferentially to the visual presentation of larger objects. These
findings suggest that canonical neurons code movements for in-
Development
One prediction of the EC is that experience with manipulable
and nonmanipulable objects shapes embodied representations over
a lifetime. According to one group, as a child accumulates experience with objects, reliance on multiple modalities for representation may actually decrease as more specific representations form
(see Mounoud, Duscherer, Moy, & Perraudin, 2007). Lexical
research has shown that manipulability is important in conceptual
processing throughout development (e.g., Borghi & Caramelli,
2003), and research with physical objects shows that children as
young as 4 years prefer to categorize novel objects on the basis of
function rather than visual similarity (e.g., Kemler Nelson, Frankenfield, Morris, & Blair, 2000). However, very little developmental research has explored the nature of visual object representations
over time. Kalnine and Bonthoux (2008) have shown that children as young as 5 years old are able to detect that a picture of a
screwdriver matches with a picture of a screw (i.e., thematic
relationships) faster for manipulable objects than for nonmanipulable objects (e.g., castle matches with knight), a finding they
suggest reflects differential contributions of action experience to
concept representation. Mounoud et al. (2007) showed that in
children as young as 5 years old the presentation of an action can
prime object categorization (e.g., the presentation of a person
sawing results in faster categorization of a saw as a tool), but this
priming decreases as children age. According to Mounoud et al.,
this suggests that the role of experience in grounding conceptual
representations of objects is actually greater in early development,
when interactions with the world dominate experience (see also
Perraudin & Mounoud, 2009).
The developmental literature indirectly supports the notion that
action and object representations are associated early in development. However, given the dearth of research in this area, the exact
developmental trajectory of purported embodied object representations remains unknown. Further, we argue that the assumption of
stronger embodiment in early development is flawed. That is, as
one accumulates experience and correlated sensorimotor activity
occurs more frequently for manipulable objects (i.e., correlations
between actions and visual presentation of objects), models based
6
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Behavioral Paradigms
Increasingly, EC is influencing behavioral investigations of
visual cognition. It is now well established that the visual presentation of an object can affect both action production (e.g., grasping) and object recognition (e.g., naming). For instance, Craighero,
Fadiga, Umilta`, & Rizzolatti (1996) had participants grasp bars
that were oriented 45 from upright. On some of the trials, a visual
prime of a bar oriented in the same way (i.e., a congruent visual
prime) was shown before the grasp, and on others a visual prime
of a bar oriented in the opposite way (i.e., an incongruent visual
prime) was shown. The authors reported that grasping was faster
on congruent trials and interpreted this as evidence that the visual
presentation of a graspable object automatically primes related
motor actions. This type of result is consistent with a body of
research that suggests a direct coupling between object perception
and action. In a more specific demonstration of the coupling
between object perception and action, Masson, Bub, and Breuer
(2011) showed that manual actions toward objects are facilitated
by the presentation of visual objects whose handles are orientated
congruently with the targets. For example, visually presenting a
picture of a beer mug with its handle oriented vertically facilitated
vertical grasps to a grasping apparatus. These results suggest
further that the presentation of a visual object automatically activates action representations (see also Bub & Masson, 2010; Bub,
Masson, & Cree, 2008; Tipper, Paul, & Hayes, 2006; Tucker &
Ellis, 1998; Tucker & Ellis, 2001).
The behavioral literature suggests that the visual presentation of
an object can automatically activate actions, influencing motor
responses in cognitive tasks. These findings are consistent with EC
and support one of its main predictions: namely, that visual presentation of an object will activate motor activity associated with
that object. However, a growing body of behavioral research
suggests that an EC interpretation of these results may not be
correct. Specifically, stimulusresponse compatibility effects (e.g.,
facilitated responding with the hand that is compatible with the
orientation of an objects handle) may be better explained by the
location of visual attention. For instance, Matheson, White, and
McMullen (2014a) had participants make upright/inverted or category judgments about left- and right-oriented tools and animals.
They showed that participants were faster at responding with the
hand that was compatible with the orientation of the tools handle
in the upright/inverted judgment task. However, responses were
also faster with the hand that was compatible with the animal
heads. Facilitated responses to animal heads or tails are not predicted by EC (because humans do not often manipulate large
animals) and suggest a more domain-general mechanism.
Importantly, in some tasks, motor facilitation effects are not
restricted to the manipulable part of the object. For instance,
Anderson, Yamagishi, and Karavia (2002) presented participants
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Discussion
Summary and Conclusion
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
10
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
in linguistic paradigms and studies of action understanding (Barsalou, 2008). Given the discrepancies between these literatures and
the literature of embodied visual object processing, we suggest that
the theory of embodied cognition and the simulation processes it
posits might not apply equally across all cognitive domains. Indeed, it appears that embodied effects may apply differentially to
different domains, with a strong embodiment in linguistic domain,
weaker in action understanding, and weakest still in visual cognitive tasks (see also Meteyard et al., 2012, for ideas concerning
embodiment and the depth of processing). We suggest that the
relative reliance on sensorimotor simulations changes depending
on how far removed the stimulus is from direct experience. With
visual objects, simulations are less important because much of the
visual information is available in view; in contrast, with words,
much more elaborate simulations are required to fully make sense
of the stimulus and make use of it in cognitive tasks. Overall, this
hypothesis suggests that simulation/retroactivation processes may
be more useful for some cognitive tasks than others. This conclusion is reached by Zwaan (2014), who suggests that embodied
simulations may be more important for recognizing a word that
refers to a tool, rather than identifying a tool that is in view.
Though this is speculative, this proposal suggests that future research will need to explore the ways in which embodied effects are
manifested in different cognitive domains and opens the possibility
of domain-specific embodiment.
11
References
Allport, D. A. (1985). Distributed memory, modular subsystems and dysphasia. In S. D. Newman & R. Epstein (Eds.), Current perspectives in
dysphasia (pp. 207244). New York, NY: Churchill Livingstone.
Anderson, S. J., Yamagishi, N., & Karavia, V. (2002). Attentional processes link perception and action. Proceedings of the Royal Society of
London Series B: Biological Sciences, 269, 12251232. doi:10.1098/
rspb.2002.1998
Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617 645. doi:10.1146/annurev.psych.59.103006.093639
Barsalou, L. W. (2010). Grounded cognition: Past, present, and future.
Topics in Cognitive Science, 2, 716 724. doi:10.1111/j.1756-8765.2010
.01115.x
Barsalou, L. W., Simmons, W. K., Barbey, A. K., & Wilson, C. D. (2003).
Grounding conceptual knowledge in modality-specific systems. Trends
in Cognitive Sciences, 7, 84 91. doi:10.1016/S1364-6613(02)00029-3
Binder, J. R., & Desai, R. H. (2011). The neurobiology of semantic
memory. Trends in Cognitive Sciences, 15, 527536. doi:10.1016/j.tics
.2011.10.001
Borghi, A. M., & Caramelli, N. (2003). Situation bounded conceptual
organization in children: From action to spatial relations. Cognitive
Development, 18, 49 60. doi:10.1016/S0885-2014(02)00161-2
Brooks, R. A. (1991, September 13). New approaches to robotics. Science,
253, 12271232. doi:10.1126/science.253.5025.1227
Bub, D. N., & Masson, M. E. J. (2010). Grasping beer mugs: On the
dynamics of alignment effects induced by handled objects. Journal of
Experimental Psychology: Human Perception and Performance, 36,
341358. doi:10.1037/a0017606
Bub, D. N., Masson, M. E. J., & Cree, G. S. (2008). Evocation of functional
and volumetric gestural knowledge by objects and words. Cognition,
106, 2758. doi:10.1016/j.cognition.2006.12.010
Buccino, G., Sato, M., Cattaneo, L., Roda`, F., & Riggio, L. (2009). Broken
affordances, broken objects: A TMS study. Neuropsychologia, 47,
3074 3078. doi:10.1016/j.neuropsychologia.2009.07.003
Buxbaum, L. J., Kyle, K. M., & Menon, R. (2005). On beyond mirror
neurons: Internal representations subserving imitation and recognition of
skilled object-related actions in humans. Cognitive Brain Research, 25,
226 239. doi:10.1016/j.cogbrainres.2005.05.014
Byrge, L., Sporns, O., & Smith, L. B. (2014). Developmental process
emerges from extended brain body behavior networks. Trends in Cognitive Sciences, 18, 395 403. doi:10.1016/j.tics.2014.04.010
Capitani, E., Laiacona, M., Mahon, B., & Caramazza, A. (2003). What are
the facts of semantic category-specific deficits? A critical review of the
clinical evidence. Cognitive Neuropsychology, 20, 213261. doi:
10.1080/02643290244000266
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
12
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
13
This document is copyrighted by the American Psychological Association or one of its allied publishers.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
14