26.

Aesthetic Computing
by Paul A. Fishwick

The phrase "Aesthetic Computing" while taken literally applies the philosophical area of
aesthetics to the :eld of computing, and work in the area is broadly de:ned as such;
however, in my operational de:nition for the work we do in my research lab and in teaching,
aesthetic computing is treated as embodied formal language. The purpose of aesthetic
computing is to deliver knowledge and practice of formal languages using aesthetic
products as a vehicle. Aesthetic Computing is founded on an increasing collection of
literature on the role of the body in learning, speci:cally in mathematics. This foundation is
then applied to the :eld of computing whose formal language elements are extensions of
mathematics. There are two questions that this new area raises:

Q1: How can embodied cognition be situated within formal languages?

Q2: How can embodied cognition result in novel computer interfaces for formal
languages?

Q1 surfaces a host of sub-questions revolving around theory, philosophy, and analysis.

Asking this question raises issues of motivation: 1) Why am I interested in this topic? 2) How
is the area of aesthetic computing built on top of embodied cognition and philosophy? 3)
Who has worked in this area (e.g., the literature)? Q1 is not enough, however. It is one matter
to analyze and develop theory, but another to ask oneself, "How can this theory be
transformed into practice?" That is the essence of Q2. What should we be doing, practicing,
and creating to take embodied cognition of mathematics and computing to the next level?
We need to build a new generation of human-computer interfaces that are informed by
embodied principles and use these principles as design elements for interacting with formal
languages. A potential, and vital, third question would revolve around the effects on such
computer interfaces on learning via assessment and scienti:cally-based research methods.
This represents an area that aesthetic computing needs to investigate; however, most work
to date is based on theory construction and engineering the novel interfaces.

The Aesthetic Computing Hypothesis is that given

the embodied nature of cognition, we should realize
this embodiment through novel human-computer
interfaces for learning formal languages.
26.1 Context for area of aesthetic computing
I pose two questions as a means to provide context for the area of aesthetic computing: 1)
Why is the term "Aesthetic Computing" being treated as "Embodied Formal Languages?" and
2) What are "Embodied Formal Languages"? For the :rst question, we must revisit the roots
of the word "aesthetics." The original Greek de:nition of aesthetics, (aisthetikos),
stems from another Greek word aisthanomai, meaning "I perceive, feel, sense." At the core of
aesthetics, then, lies the body, and its interactions in forming concepts and knowledge: a
esthetics as embodiment. Aesthetics is, in breadth and depth, a much richer enterprise
above this level (Kelly 1998), yet we maintain a view of aesthetics that is body-based, even
though Diffey (1995) notes that the term 'aesthetic' has largely lost its perceptual sense
except in the word 'anaesthetic,' but retains its senses of "beautiful' and 'artistic.'" As far as to
why "Formal Languages" are used to characterize "Computing," we note that the bulk of
theory of automata and computing is situated within linguistics although a subset of
general linguistics that requires a formally well-de:ned speci:cation and treatment.

Let us now consider the de:nitions of embodiment and formal language. Embodiment
suggests the perception/action feedback loop present when the body interacts with its
environment. So, it seems clear that an embodied approach to anything would involve
sensorimotor functions using the mouse, keyboard, multi-touch displays as well as
donning a head-mounted display or using a tactile feedback device. Human-Computer
Interaction is chock-full of approaches that leverage such technologies. But, embodiment is
a much deeper concept than sensory stimuli and physical manipulation. We have a sense of
presence with certain advanced technologies such as multi-user virtual environments (i.e.,
achieving different types of presence, including social). We also have a sense of presence
when reading a book since the book situates our "mind's body" within the narrative (ref.
narrative psychology in Beck et al. 2011). Thus, embodiment can be measured objectively
by hardware used to enable the senses, or subjectively through a presence instrument on the
human subject. Embodiment should not be viewed as a rejection of abstraction, but rather as
a complement to it (Devlin 2006).

Formal languages de:ne a category of language that is arti:cial, such as a programming

language. These languages stem from formal grammars which can be based on text,
shapes, or diagrams. FORTRAN, Java, and Perl are examples of formal languages, but so are
the eXtensible Markup Language (XML), Uni:ed Modeling Language (UML), data structures,
Morse code, and dynamic model structures used for simulation (Fishwick 1995, Fishwick
2007b). Formal languages are frequently speci:ed using grammars such as the Backus-Naur
Form (BNF) and need not be text-based. For example, one can have formal audiovisual
languages and also graph grammars. All formal language structures can be de:ned
hierarchically using levels of abstraction (e.g., 3 :nite state machine levels governing an
underlying set of ordinary differential equations, which in turn are translated into the
programming language Java, and then further into byte code). Languages, therefore, are
frequently de:ned in long chains of speci:cation and translation. Each language has its own
target functionality, culture, and adherents. Ghezzi and Jazayeri (1997) provide general
concepts of speci:cation for programming languages.

26.2 Personal Experiences and In@uences

26.2.1 Art
It is easy to take the idea of embodied cognition for granted since it seems like something so
naturalthat the body plays a central role in cognition. However, an adherence to
embodiment tends to change your worldview when looking at objects. As an amateur artist, I
collected many posters and prints of historically well-known artists. In middle school, I was
strongly inluenced by Thomas Gainsboroughs work, in particular Figure 1.

Copyright: pd (Public Domain (information that is common property and contains no original
authorship)).

Figure 26.1: Mr. and Mrs. Andrews, oil on canvas, Thomas Gainsborough, 1750. The National
Gallery, London, UK

I imagined that with myself as an avatar , I could enter the painting, walk the wheat :eld,
examine the trees, and engage in social discourse with Mr. and Mrs. Andrews. This led to a
series of imaginary conversations and observations in world. The key point here is the
reading of this work as a form of embodied experience. The Gainsborough painting was
not a remote object of study for me, but rather an example of virtual reality, a time machine
an illusion that allowed me to immerse myself within the world of 18th century England. This
approach is an example of Deweys art as experience (Dewey 1934) and relates to Graus
(Grau 2004) argument about artists as the :rst virtual reality creators. The approach stresses
that when we approach an object, we can interpret it dynamically via a bodily simulation with
all of the perceptual and motor-based actions that the body affords. This way of thinking and
acting can be applied to all objects and media, including mathematics and computing.

26.2.2 Mathematics
In elementary school, like hoards of other students throughout the world, I was taught the
elements of arithmetic its methods and laws, with many examples that were exercised
using rote memorization and intense practice. Doing mathematics was highly action-based,
but the action was limited to solving multiple problems over extended periods of time. After
the basic elements of arithmetic came algebra. Let's consider the following mathematical
expression containing arithmetic with a sliver of algebra:

X = 2 * (3 + 4)
We have all been subject to such mathematical objects as they are critical to an educated
public. Learning all components of this equation was not easy one had to understand the
concept of a variable, operations of multiplication and addition, followed by the concept of a
parenthetically-delimited group. Order of operations is also critical, as suggested by the
group. So, for example, I can add 3 to 4 and then multiply by 2 to obtain 14, which was then
set to X as an equivalence. Certain laws of arithmetic were useful in transforming
expressions such as this one. The Law of Distribution states that x(y + z) = xy + xz where
x , y , and z are numbers, and the multiplication is implicit rather than being de:ned
explicitly using * as in the above equation. The teacher would de:ne the law of distribution
and give us many useful examples as a means to reinforce our understanding of the law and
how it can be employed in symbolic manipulation. Such patterns of equivalence drove a
static pattern-matching type of approach to mathematics.

However, during the ensuing lessons, I found it convenient to create an arti:cial method of
solution that involved treating the numbers and symbols as physical objects. In mathematics
education, this kind of process is termed rei:cation (Sfard 1994) and is related to
constructivism (Piaget 1950) and constructionism (Papert 1980), where students create
their own knowledge through a combination of ideas and life experiences. I used a virtual
manipulation of the above expression by representing the distributive law through analogy
and metaphor:

Grab the " 2 " object, which when juxtaposed with the "
biomechanical state where the " 2
* " operator, provides a
" is pushed inward toward the group object
de:ned within the parentheses "(...)". The " 2 " is pushed gradually and then when it
reaches the edge of the spatial boundary denoted by "(", it moves through it to the
other side and splits in a biological fashion into two clones that are attached to
the " 3 " and to the " 4 ," respectively. This cloning activity results in the expression
(2 * 3 + 2 * 4) . The sub-expressions 2 * 3 and 2 * 4 are evaluated through
further bodily activity. Pushing the 2 and 3 into the * , for example, results in
multiplications. Similar reactions occur to perform the + operation last, as
dictated by the learned order of operations. The result is then placed manually in a
box with an X printed on it.

Mathematics then, for me, had become akin to a full-body sport rather than simple

operations requiring a collection of static text-based rules and patterns. The virtual
manipulations might involve other embodied activities, where I might have "launched the "
over a wall that bounds the parenthetical expression. While this is a personal experience, it is
by no means unique, as Sfard observes in her dialogue with Thompson (Thompson and
Sfard 1994), where she notes the propensity for similar mental imagery: "My work with
mathematicians brought lots of further evidence that, indeed, the inner world of a
mathematizing person may look very much like a material, populated with objects which wait
to be combined together, decomposed, moved and tossed around." Arzarello (2004) explains
the difference between natural versus formal mathematical presentations, and surfaces the
importance of gesture in using naturalistic explanations and interpretations in addition, or on
the path, to the formal. The previous embodied description would be termed natural. Goldin
and Kaput (1996) overview the effects of media on mathematical representation by noting
"..changes in physical media that permit external representations to be action rather than
display representations give these representations one characteristic of powerful internal
representations." Hadamard (1996) studied mathematical thought which echoed similar
cognitive processing. This action-based narrative on mathematical symbols was not limited
to the distributive law for me. For example in an expression such as , something interesting
happens when moving numbers through the equals sign. There is a virtual line or plane that
intersects at a right angle to the . When a number such as is dragged through this vertical
plane, the number lips its sign on the other side with a mirror-like effect, resulting in . The
laws of commutativity and associativity have similar pseudo-physical, material, behaviors
that can be used to understand and process arithmetic expressions.

The problem with my early experiences with embodied sense of symbol manipulation is that
none of the books (or teachers) explained mathematics in this way, and I, and likely many
others, were forced to keep these somewhat peculiar cinematic episodes to ourselves.
Whether this type of thinking is common requires more scienti:c studies and relection upon
the nature of mathematics. At the University of Florida, we have developed a web-based
interactive tool that allows anyone to manipulate expressions in this fashion. We have also
previously explored similar embodied representations involving a sense of presence in a
virtual environment (Fishwick and Park 2008a).

My purpose of relaying this experience is to emphasize the importance of the body in

understanding formal languages such as mathematics. Lakoff and Nunez (2001) presented
a landmark compilation of mathematical metaphors that build on top of the philosophy of
embodied cognition (Johnson 1987, Varela et al. 1991, Barsalou 2010). In particular,
Johnson's image schemata such as containment, attraction, and equilibrium were integral
aspects of my arithmetic experience. The literature in embodied thinking centers thought
and knowledge on the body and is informed not only by areas such as conceptual metaphor
(Lakoff and Johnson 1999, Lakoff and Johnson 2003), but also by subsequent empirical
studies of the brain (Feldman and Narayanan 2004, Feldman 2006). Even more generally,
language-based narratives appear to contain an embodied basis (Speer et al. 2007, Mar and
Oatley 2008) de:ning natural language in terms of simulation. Reading a story about
grasping or running can result in a cognitive simulation of these events and activities, as if
the reader had been physically active. Going back in time to when the Method of Loci
lourished (Yates 1966), we note that the act of memorizing a set of facts was turned into a
rich, embodied process rather than viewed as mere associative retrieval. The area of situated
learning and cognition (Brown et al. 1989, Lave and Wenger 1991) meshes well with the
embodied approach in terms of its goals and methods: learning by doing.

In closing the discussion of an embodied mathematics, we should note that the concepts of
"action", "interaction", and "process" can be framed within standard mathematical notation
containing explicit aspects of functional composition, dynamics, and procedure (i.e.,
embodied-types of thought). For example, the aesthetics of geometry and shape can be
constructed generatively (Leyton 2001, Leyton 2006) and dynamically via Blum's wave
propagation-based medial axis (Leymarie 2006). We can also use mathematics to create a
formal representation of mathematical metaphors (Guhe et al. 2009), thus making a loop:
grounding metaphors on mathematical expressions, where the metaphors themselves are
formally de:ned.
The embodied approach has profound implications for mathematics, and by extension for
applied mathematics, and computing since computing is a direct outgrowth of mathematics,
and formulas such as the one described earlier are common objects found in software
"expressions." If our thought is embodied, then:

We should investigate the variety of metaphors used within mathematics and

computing, and also their origins and cultural associations.
We should leverage the metaphorical, and embodied, substrate of language by creating
new human-computer interfaces that reinforce and amplify this experience.
We should bring to bear other disciplines for whom "the body" is a natural component,
such as the arts and humanities (Slingerland 2008), thus forming new interdisciplinary
collaborations that span the academy.

26.2.3 Programming
The embodied approach was extended from mathematics into learning programming and
data structures. Programming, in particular, is known to be rich in metaphor. Loops are just
that: patterns of cyclic behavior small objects moving around a closed path as these
objects perform other tasks. Sequential behavior is sometimes a movement along a spatial
path, and functions are machines that take product inputs and produce outputs. Papert
(1980) in his explanation of the LOGO language reinforces the importance of embodiment in
a term he calls syntonicity, where he notes We have stressed the fact that using the Turtle
as metaphorical carrier for the idea of angle connects it :rmly to body geometry. Petre and
Blackwell (1999) performed studies on programmers, and results indicate metaphorical
reasoning involving objects, motion and general embodied interaction. Metaphors such as
these are not only present in all programming languages, but also in the theory of
computation on which the theory of computing is based. For example, the Turing machine is
an excellent example: a machine envisioned by Alan Turing in the 1930s consisting of a tape
read/write head and an in:nite tape. This metaphor may have been because of the extensive
use of magnetic tape at the time. In the previous century, Charles Babbage used a "mill" in
his computing engine. Interestingly, in the vast history of computing where these historical
concepts are discussed (Ifrah 2002), most programming and computing was analog and
embodied by de:nition and implementation. It is only relatively recently that the evolution
from analog to digital has simultaneously sped up our computations, facilitated a computer
revolution, but also disembodied our relations to computing.

26.2.4 Media
Media theorists have provided a host of approaches in understanding the evolution of media.
McLuhan (1964) places importance, not only on the message created through a modulated
medium, but on the medium itself which affects the message. McLuhan employs the
example of a light bulb which he claims is a "medium without a message." However, the light
bulb can host a binary digit, and perhaps more in the case of multi-way switch bulbs in a
means not unlike Morse code manipulated through signal lamps. Bolter and Grusin (2000)
present a theory of media forms undergoing gradual alteration, generally technology-driven,
causing us to examine issues of immediacy (seeing beyond the medium to the target
signi:ed) and hypermediacy (being aware and relecting on the medium). New media studies
place speci:c importance on materiality, the medium, and embodiment. Manovich (2002, p.
317), when he considers the "loop as a narrative engine," with a loop being de:ned as a
common programming structure enabling index-based iteration, asks "Can the loop be a new
narrative form appropriate for the computer age?"

Popular media have signi:cantly shaped my thought process underlying aesthetic

computing. For example, Tron (Kallay 2011), which debuted in 1982, is noteworthy because it
was created based on a highly innovative screenplay which included a large piece of
software, namely an "operating system," that could be experienced directly. Programs were
bodies, and the operating system was composed of a city-like space with lighted, moving
vehicles and interacting programs. Tron is fairly unique in this way within the science
:ction/fantasy genre. Other more recent cinematic offerings, while impressive and engaging,
tend to ignore the "program." For example, on Star Trek: The Next Generation, we were
introduced to the Holodeck where one could experience an ultimate virtual reality with full
sensory simulation. A user would stop at the outside of the Holodeck and say "Computer.
Load Holodeck Program A-3" or some such phrase, and then the Holodeck would load this
program and the user would enter. However, we never actually experienced the program itself
only its inputs and outputs. Similarly, in The Matrix, we have a rich embodied experience of
human characters that , in reality, are stored inside of a network of luid-:lled pods.

Despite our familiarity and utility with text-based process descriptions, it is remarkable and
ironic that a hyper-real environment such as the Matrix affording real-time synthetic
interactions and simulacra would have to be programmed by strange-looking rivulets of
green rain, which are not obvious to anyone, presumably except for the operator well trained
in this postmodern descendant of cuneiform script. This semiotic condition presents a stark
contrast: practically unlimited full-sensory simulation on one hand produced by the program,
and what amounts to glori:ed typewriter symbols on the other de:ning the program itself. It
is as if one provides you with a highly maneuverable hypersonic jet plane to ly with the
caveat that you need to pilot the plane by tapping on a straight key to produce Morse code
dots and dashes. One would expect that, just perhaps, the capabilities that form programs
and data might avail themselves of the practically unlimited human-computer interface that
the Matrix provides. Rotman (2000, p. 67) poses the question that forms this concern, "What
if language is no longer con:ned to inscriptions on paper and chalkboards but becomes
instead the creation of pixel arrangements on a computer screen?"

26.3 Aesthetic Computing: Turning Computers Inside-

Out
Computers have shrunk in size, and increased in number, considerably over the past half-
century. We are familiar with news stories about how ever smaller and thinner computers
and software are now ubiquitous in our culture to the point where we carry or wear them in
our daily routines. The decrease in size and increase in number creates a situation where
computing effects most consumer products. For example, the digital video recorder enables
time and place shifting for movies and television shows. What is just as interesting is
exploring how computing affects us and our thinking. Turkle (2004) explains this
psychological phenomenon and closes with the phrase "we are all computer people now."

Turkle's argument has signi:cant rami:cations for computing, and I would go one step
further to suggest that the way in which our thinking is changing culturally surfaces deep
abstract concepts in computing to us as we use these devices: from number, to information
structure, to process. Digital watches and video recorders (DVRs) are good examples. Most
digital watches are multi-function. These watches contain the ability to act as a way to tell
time, set a stop watch, or wake up to an alarm. To use the watch, you have to learn how to
navigate a menu by repeatedly pressing a mode button. In each mode, there are sub-
functions re:ning that mode's interaction. This experience of mode-button pressing directly
maps to a fundamental theoretical structure in computing called a :nite state machine
(Hopcroft et al. 2000). It is not just that the :nite state machine is embedded within the
watch's silicon, but also that the human wearing the watch becomes aware of this virtual
machine's structure and its components through the experience of using the watch. The
state machine changes how the wearer thinks, even though the wearer is probably unaware
of the formal mathematical notation of a state machine. The watch's software internals
become embedded within our psychology and culture. A similar process occurs within most
other household appliances such as the DVR, however, the state machines in DVRs are more
complex than in watches yet to understand how to navigate the hierarchical menus, one
has to become fully aware of a new type of thinking (Negroponte 1996). Effects of
computing on thought (e.g., neo-millennial/digital native learning styles) have also been
covered in the context of learning (Dieterle et al. 2007).

Experience with computing artifacts is a form of information representation, where the

de:nition of "representation" is expanded as a form of interaction, rather than as a static
object in the form of a sign. If the raw elements of computing information, data, and
software are changing the way that we think and entering into our popular culture, it is
natural to suggest that aesthetics of these raw elements can and should play a central role in
computing. Aesthetics has evolved from the embodied, sensory, de:nition to a more
comprehensive one offered by Kelly (1998), a "critical relection on art, culture, and nature."
Aesthetics within computing results in new interaction modalities for computing artifacts
such as formal languages. Given the preponderance of new ways to connect human with
computer, there are many opportunities for creative representation. We categorize and study
these new ways using the phrase aesthetic computing.

26.4 Why Aesthetic Computing?

Representation targets of aesthetic computing include terms such as data, information,
software, and code. I use these terms somewhat interchangeably because of semantic
overlaps. Data can be atomic or in the form of a structure. Code usually refers to software
which encompasses both data as well as process. Information theory tells us that all of this
is a form of information since information can be decoded as atomic, structural, or
procedural. I prefer terms such as code, software, or information when referring to the
"computing" part of aesthetic computing since these terms encompass broader categories
of items that can be represented, whereas the term "data" in common parlance tends to
denote non-procedural forms of information.
The argument for aesthetic computing involves emerging areas of computing which have
changed:

Our relationship to each other and to nature. These aspects include ubiquitous
(Green:eld 2006, Gershenfeld et al. 2004) and pervasive computing, customization and
personalization of interfaces, and the new modalities for human-nature interaction as
mediated through computing (e.g., the virtual reality continuum spanning physical,
virtual, and augmented reality). Shared and customized interfaces for information
visualization (Viegas et al. 2007), code sharing (Reas and Fry 2007), assisted with "remix
culture" (Lessig 2008) create a networked, customized (Pine 1999) representational
space.
Our thought patterns, allowing computing artifacts such as information and software to
permeate our experience. Salomon (1990) makes an argument for computing changing
thought, resulting in cognitive residues from human-computer interaction. These studies
are consistent with Turkle (2005).
The importance of experience in computing in human-computer interaction (HCI).
Cockton (2011) and Hassenzahl (2011) describe the shift in HCI from etciency, alone,
to experience, and Lwgren (2011) emphasizes the importance of interaction a core
aspect of experience. The emphasis on experience is related conceptually to
embodiment (Lakoff and Nunez 2001, Johnson 2007) as a basis for cognition. The
relevance of aesthetics in HCI is discussed by Tractinsky et al. (2000) and Norman
 (2004). Dourish (2001) lays out a philosophical foundation for embodiment in HCI
through its beginnings in phenomenology.
Our need as computer scientists to interact more frequently with artists and designers
since they represent the creative component of aesthetic inquiry, and so experience-
based representations for the diffusing computing artifacts need to be studied with the
help of artist-scientist collaborations (Buxton 1988, Malina 2011).

26.5 History of the Aesthetic Computing Field

I have been teaching a course in aesthetic computing since 2000, and information on the
most recent course can be found in (Fishwick 2012). A preliminary paper was published on
the concept (Fishwick 2002). A Dagstuhl seminar on Aesthetic Computing (Fishwick and
Bertelsen 2002) was co-organized in Germany (Dagstuhl 2011) by myself, Roger Malina, and
Christa Sommerer during the summer of 2002. This interaction resulted in several
publications (Fishwick et al. 2005, Fishwick 2006, Fishwick, 2007a, Fishwick 2008b). Kelly et
al. (2009) represents the most recent published workshop in the area. The use of the word
"aesthetics" and "programs" can be found in several contexts, including Mohr (2011) and
Nake (2009) who were early investigators in the aesthetics of interaction through the use of
computer programs as a means of artistic expression. Knuth (1992) developed literate
programming and made note of the importance of aesthetics in programming. Knuth's
interest in aesthetics went beyond the purely cognitive, and included artistic forms of
typography and layout design for programs. For Knuth, it would seem that computing was an
embodied experience.

Aesthetic computing is unusual in that aesthetics is intended to be applied to computing

rather than in the inverse direction: using computing to create artistic products. Examples of
aesthetic computing, therefore, capture a kind of "boomerang effect" where elements of
computer graphics, ubiquitous computing, and mixed reality interfaces can be used to
interactively represent that which formed these technologies namely the information and
software.

In terms of academic curricula, Aesthetic Computing has been taught for a decade at the
University of Florida in the form of two classes, which are usually combined: CAP 4403
(undergraduate) and CAP 6402 (graduate). The combined classes began as part of the
Digital Arts & Sciences (DAS) programs (Fishwick 2012) designed and developed to connect
computing with the arts. The class has undergone several stages since 2000:

(2000-2005) Representational alternatives to software artifacts from numbers and

expressions, to data structures and programs. There was one physical project, with the
other two projects resulting in digital representations. The physical product was
 exhibited in several gallery areas on and off campus allowing passers-by to comment
and explore.
(2006-2009) Alternative representations for mass media and communications. This
emphasis required students to employ representational creativity, but with the idea of
starting with a contemporary news story and then mining this issue for the software
artifacts in the story that were to be represented. The physical project was eventually
dropped since many of the computer science students in the mid to later years had
minimal design and art backgrounds.
(2010-2011) Representation using web mining and APIs. This was an effort to create
more automation in the representational process with students :nding sources of
information and then, mostly using APIs, to translate this information into creative
representations. Most students use data as their information, but others used more
complex web structures (e.g. XML) as sources.
(2012) A focus on representation of data structures, mathematical models, and dynamic
models and programs. The end product is either an interactive game or video production
whose goal is to facilitate education of computing concepts for early-age groups and
non-computing specialists. This is the current incarnation of the class (Fishwick 2012).

We use the term aesthetics in the spirit of Kelly's de:nition, but also extend the concept of
"critical inquiry" to include the creative aspect of design and art. This is only natural, for
engaging in critical inquiry presupposes and requires the creative act. Studies in aesthetics
are numerous (Audi 1999, Kivy 2004) often with underlying attempts to :nd universal
attributes of beauty (Scruton 2011). My view on aesthetics is one that focuses on that which
is generated as a result of cultural inquiry, which is to say the vast diversity of design and art
forms. This "aesthetics as diversity" approach is similar in spirit to Hogarth (Burke 1943) with
the associated phrase, "unity in variety."

26.6 Toward Software as Embodied Experience

26.6.1 Introduction
Partial justi:cation for the use of embodiment as a form of representation is based on
educational learning styles (Dede 2005). Also, our ongoing research indicates a signi:cant
correlation between presence and memory in a virtual environment (Fishwick et al. 2010)
with results currently in the journal submission phase. Recent mixed reality memory studies
such as (Ikei and Ota 2008) indicate positive effects on memory in an augmented
environment. Instruments and studies on memory performance within virtual environments
are being continually re:ned and investigated. Parsons and Rizzo (2008) introduce a test of
validity for a virtual environment cognitive instrument called VRCPAT. Johnson and Adamo-
Villani (2010) note signi:cant effects of immersion on short term spatial memory. Embodied
interaction with technology provides us with an understanding of internal logic, software, and
process usually through pure experience. For example, we learn the state machine of a DVR
through repetitive DVR use. While a large population may require this learning, not everyone
may be required to take representation to the next step: from interaction to relection and
rei:cation. The latter steps, however, have potential utility in entertainment (the arts, games)
as well as in education.

26.6.2 Audiovisual Explorations: Steampunk Obesity Machine

Let's consider one such artifact, which is de:ned by a system dynamics model found in
systems science and simulation. Figures 2 and 3 are two different representations of a
System Dynamics low graph (Forrester 1991) capturing the temporal nature of human
metabolism.

Copyright Inderscience Publishing. All Rights Reserved. Reproduced with permission. See
section "Exceptions" in the copyright terms below.

Figure 26.2: A System Dynamics low graph with two levels (i.e., stocks) and three rates

The diagram in Figure 2 represents a virtual machine based on the analogy of luid low. Fluid
starts from source node (left-most "cloud" icon) and proceeds to low through a system of
levels separated by rates to a sink node (right-most cloud icon). More generally, the luid low
can be construed as a kinetic energy low since luid velocity is the dominant low variable.
At the start of the machine, at the left, luids pour into metabolism and food intake to
suggest that the more energy, the higher the Fitness Level, but also the higher the Weight.
The rate variable, Metabolism, is proportional to a functional combination of Fitness Level,
Exercise, and Nutrition. The nature of this precise formula is not present in the model since
the model is an abstract representation of the dynamics. The solid curve arrows relect luid
low through the system, and the dashed curve arrows relect control settings to change the
rates on the valves. Figure 2 is a hypothetical example, and is not put forth as an accurate or
valid simulation model of nutrition, but rather to demonstrate that similar diagrammatic
models are widely used in science and engineering. These types of models were originally
implemented as physical, analog computers although their more frequent existence today is
as digital models with a diagrammatic front end authoring capability. The MONIAC, or
"Phillips Machine," is one such example (Swade 2000, Ryder 2009) from the analog
computing era.

Figure 3 shows the same model which is a synthetic rendition of Figure 2, rei:ed using a
"steampunk machine" since its structure is reminiscent of the cyberpunk aesthetic that
continues to be popular since its inception in Gibson's work (Cavallaro 2001). Steampunk
culture has connotations of "reclaiming tech for the masses" (Grossman 2009). Water is
pumped using steam-power underneath the wooden loor. This water shoots out of two
brass ori:ces that represent the two valve-icons in Figure 2. Water :lled glass containers
represent the level quantities, and wood/brass control rods connect everything together as in
Figure 2. The human avatar on the left is demonstrating the machine in action to us, or we
may become the avatar. The natural question is why anyone might want to construct such a
machine when Figure 2 might do. For the answer to this, we have additional questions to ask,
with possible use-cases.
Copyright Inderscience Publishing. All Rights Reserved. Reproduced with permission. See
section "Exceptions" in the copyright terms below.

Figure 26.3: A steampunk obesity machine isomorphic to Figure 2

Figure 2, and the equations that map to this diagram, are most often used by scientists
familiar with the system dynamics method. It is unlikely that these scientists have any
interest in structures such as Figure 3 mainly because they are comfortable and familiar with
more formal representations. However, the vast majority of the population may require
additional motivation if they are to understand, and be motivated or inluenced by, the more
formal representations. Therefore, the machine for Figure 3 is appropriate for education and
entertainment. It is easy to imagine the machine in Figure 3 being engaging especially with
game-like features that required certain goals such as stabilizing the water level in the
Weight container.

26.6.3 Visual Representations of Data

There are numerous additional examples of artworks that, if used as guidance, can lead to
aesthetic computing products useful for education. The vast majority of examples are the
encoding and presentation of data rather than of program or model. It is logical given that
data repositories and accessibility are expanding rapidly and that they represent the simplest
and easiest to grasp forms of information. Consider the model of a single number shown in
Figure 4.

Copyright Oto Godfrey. All Rights Reserved. Reproduced with permission. See section
"Exceptions" in the copyright terms below.

Figure 26.4: The relative size of the U.S. debt if it reaches 15 trillion dollars. The large
rectangular block represents stacks of one hundred dollar bills

This encoding of number as a stack of one hundred dollar bills is given context by familiar
objects whose size is known through pictures or experience (e.g., the Statue of Liberty, a
football :eld, a truck). One might take this same approach to representing other analog
representations of monetary amounts through choosing different familiar objects. A
participants engagement can have both artistic and mathematical consequences. For
example, we can imagine performing operations on numbers in this type of representation
much as we have done manually in the past with quipus and abaci.

Consider Huff's prime number series (Huff 2006) with two example encodings of prime
factors shown in Figure 5.
Copyright Kenneth A. Huff. All Rights Reserved. Reproduced with permission. See section
"Exceptions" in the copyright terms below.
Copyright Kenneth A. Huff. All Rights Reserved. Reproduced with permission. See section
"Exceptions" in the copyright terms below.

Figure 26.5 A-B: Prime number factorization encodings (EPF: 2003:V:A:997141 and 2000.24)

The two encodings in Figure 5 are pieces of :ne art, but could also be potentially used to
motivate students to appreciate prime factorization through puzzle-making. For example,
consider where one might provide to someone a visual encoded integer and then ask that
person to identify the number and factors. Figure 6 shows two additional examples of
information presence: Levin's infoviz gratti for data, and Living Light. The gratti is a
deliberate mechanism for surfacing numbers of societal relevance in public places. Living
Light is a permanent outdoor pavilion in Seoul, South Korea. The pavilion's purpose is to
allow spectators to visualize environment levels such as air quality. As pervasive computing
extends into the future, most lat surfaces become display surfaces opening up numerous
possibilities for bringing information into our daily lives. Figure 7 shows a model of a city
which is turned into a computer program-like artifact, or automaton, whose output is a
musical score.
Copyright Golan Levin. All Rights Reserved. Reproduced with permission. See section
"Exceptions" in the copyright terms below.

Copyright David Benjamin and Soo-In Yang. All Rights Reserved. Reproduced with
permission. See section "Exceptions" in the copyright terms below.

Figure 26.6 A-B: Leftmost: Infoviz Gratti/Adjustable Pie-Chart Stencil by Golan Levin.
Rightmost: Living Light by David Benjamin and Soo-In Yang
Copyright Akko Goldenbeld. All Rights Reserved. Reproduced with permission. See section
"Exceptions" in the copyright terms below.

Figure 26.7: Pianola City Music

26.6.4 Textual Representations

The examples so far have been mainly visual; however, it is often desirable to use thinking
similar to that described earlier for mathematical expressions, but to extend this to software
for example. Emerging areas in the humanities such as software studies (Fuller 2008) and
critical code studies (Marino 2006) situate the need for studying formal languages using
some dimension of hermeneutics. These areas also provide opportunity for creation of new
human-computer interfaces. For example, we may treat software as a full hyper-mediated
structure (Roth et al. 1994, Anderson et al. 2000). One can then, through an embodied
approach facilitated by link interaction, treat formal language-based constructs as
hypermedia.

26.7 Art and Design as Creative In@uences for

Embodied Formal Languages
The provided examples including Figures 3 through 7 are related to aesthetic computing in
different ways. Since aesthetic computing is embodied in formal languages with an
educational goal as a :nal end product, I will overview how these examples might achieve
that goal. Table 1 contains 5 columns: column 1 refers to a previously described image or
product; column 2 is the original medium; column 3 is a hypothesized goal for the last 5 rows
(i.e., since the original intention is not known but assumed); column 4 is an example
repurposing of the original product for a formal language goal (column 5). Let's consider the
3rd row. The product has been designed to a highly compelling and attractive display of the
national debt. This creative use can be recast as a new way to learn number sense. The
formal language products are only examples and have not been constructed by anyone,
however, the original art and designs are dually inspirational for their original goal or
purpose, and for a form that leverages their embodied characteristics for the purpose of
formal language instruction.

Example Original Aesthetic Goal Formal Formal

Product (hypothesized) Language Language Goal
Product
(example)

Personal Typographic To illustrate Game with To teach laws

Experience Image elegance of the moving of arithmetic
(Arithmetic) mathematical operators and
form operands

Steampunk Raster Image To create Video To teach

Obesity art work steampunk illustrating System
Machine genre-related functional Dynamics
(Figure 2) imagery mechanism Methodology
and control

US National Raster Image To illustrate the A tactile set of To teach

Debt (Figure 3) art work magnitude of blocks and number sense
the US debt objects
using scale

Prime Number Raster Image To celebrate An adventure To teach about

Factorization art work organic forms game using prime numbers
(Figure 4) using prime encodings as and
number 3D puzzles factorization
encoding

Infoviz Gratti Gratti and To present An alternate To teach

(Figure 5a) Template in societal reality game concept of
Outdoor information to (hunting for percentage
location the public gratti)

Living Light Outdoor To present A kinetic To teach data

(Figure 5b) sculpture environmental sculpture structures
 information to
the public

Pianola City Indoor kinetic To explore an Indoor kinetic To teach

Music (Figure art work architecture- object concept of a
6) music interface data search via
sound

Hyper- Web-based To represent Hyper- To encourage

mediated computational cultural mediated learning of how
software literature knowledge software/code to code
engineering

Table 26.1: An aesthetic transformation to formal language learning objectives

Table 1 portrays aesthetic computing through repurposing existing art works, but this
procedure is optional. Formal language-based products that capture the essence of
embodied interaction can be designed directly from initial design, to detailed design, and
onto an implementation. The Steampunk Obesity Machine (Table 1, Row 2) is a case in point.
Even though a poster board image (Figure 3) was part of a curated art exhibit (Harn 2011),
the image was meant as a preliminary design for a virtual machine to teach System
Dynamics concepts. The machine has not yet been constructed.

26.8 Embodied Computing using Serious Gaming

The discussion of aesthetic computing and the interpretation of it via embodied formal
language would be incomplete without reference to video and console game cultures. Two
examples are illustrative: logic circuits in the game Minecraft (2011) and the game called
Code Hero (PrimerLabs 2011). Minecraft is a block game where players move around a
space and build blocks using a mining metaphor. Some of the procedural capabilities within
the game have engaged members of the community to create basic circuits, leading up to
full-ledged computers out of the logic circuitry. Since Minecraft is highly interactive, and
invokes a sense of presence to boot, this type of hacking is consistent with the concepts in
aesthetic computing: players are working together to form circuits through embodied
interaction. Primer Labs recently created a game called Code Hero where the play learns a
programming scripting language such as Javascript. It is the means for this pedagogy,
however, which places it squarely in the embodied realm: a player has a gun that shoots
code at a target object, thus causing that object to react to the code. This is, in actuality, a
rei:ed form of data low in a manner similar to the capabilities within lambda calculus and
languages based on that formalism such as Lisp (e.g., consider the map functions where a
function can accept another function as input and then apply that function to arguments,
producing output). Figure 8 shows a Minecraft arithmetic logic unit (ALU) described by
Ganapati (2010), and Figure 9 shows a snapshot from Code Hero.

Copyright . All Rights Reserved. Used without permission under the Fair Use Doctrine (as
permission could not be obtained). See the "Exceptions" section (and subsection
"allRightsReserved-UsedWithoutPermission") on the page copyright notice.

Figure 26.8: Arithmetic Logic Unit built with redstone in an immersive play space using the
Minecraft game engine
Copyright Primer Labs. All Rights Reserved. Used without permission under the Fair Use
Doctrine (as permission could not be obtained). See the "Exceptions" section (and
subsection "allRightsReserved-UsedWithoutPermission") on the page copyright notice.

Figure 26.9: The Code Hero Game: talking to the Ada Lovelace avatar prior to entering a
space to learn how to script code by shooting scripts at objects.

26.9 Collaborative Roles, Usability, and Experience

Aesthetic Computing begins with a formal language construct such as a number, data,
model, or software. Then the challenge is to represent this construct through embodiment.
We noted that "embodiment" can be as simple as pure rei:cation without representation of
existing objects when we demonstrated the ability to grab hold of numbers and move them
toward operators. However, rei:cation can also suggest object representation as in Figures 4
through 7. I need to address the "who" and "why" aspects of aesthetic computing.

First, who is going to be creating these representations? In the case of collaboration, I

recommend teams of humanist scholars, artists, and computer scientists. Humanist
scholars bring to bear different philosophies and theories which can help shape the resulting
representation. The artist has the creative perspective and tools to create the representation,
and the computer scientist can serve two roles: to help construct tools used by the humanist
and artist in the extraction of information and in enabling the interaction that ensues through
externalizing embodiment in the human-computer interface.

Second, who is going to use the representations? Students in my aesthetic computing class
are often initially confused why one would construct anything but diagrams. This confusion
is expected, but we must be careful when de:ning usability: usable for whom and for what
purpose? We need to identify 1) the goal of the representation, and 2) the end target users.
Goals for the embodied representations are education, arts, and entertainment (e.g., cinema,
visual and performing arts, :ction). Target users may be any grade level in school or some
segment of the general public. From a psychological perspective, a broad view of "usability"
can encompass user goals including: increased valence, motivation, and attitudinal change,
as well as improved short or long term memory. Mathematicians and computer scientists
are not the target, as these populations are adept at using existing notations. Aesthetic
Computing is less stressed on information extraction and more on the use of entertainment,
arts, and humanities on formal languages with the largest practical effects being in
education. Thus the target users are formal and informal learners of all elements of formal
language-based instruction (e.g., mathematics, computer science).

The roles of participants in aesthetic computing will likely be different given the interests of
each party. For the computer scientist, for example, Figure 5 serves as a design template for
the creation of special effects and interactive games for the purpose of expressing elements
of prime numbers and the factorization process into these numbers. The artist's work is a
medium through which this aspect of formal language is creatively expressed. The goals of
the artist and computer scientist are clearly different, but the means (i.e., representations of
prime numbers) are common. This difference in ends, with similar means, plays out in the
other examples. For instance, Perl poetry (i.e., poetry created using the programming
language, Perl) may be an aesthetic product to the writer a valid end in itself. To the
computer scientist, this product represents a medium in which to express a different end
the formal language "message." Therefore, aesthetic computing by its arrangement of words
comprising this phrase is focused on computing the learning of formal languages.
However, aesthetic products play a key role in this learning activity and allow for the artist,
scholar, and computer scientist to collaborate with different intentions and goals.

Other areas related to aesthetic computing are information visualization (Card et al. 1999,
Ward et al. 2010), and software visualization (Eades and Zhang 1996, Stasko et al. 1998,
Zhang 2007, Diehl 2007); however, the goals of these areas are generally quite different than
for aesthetic computing. In information visualization, the goal is etcient communication of
data and information, whereas for aesthetic computing, the goal is education through highly
embodied, and interactive, aesthetic products in the forms of art and entertainment. As such,
Aesthetic Computing fosters a deeper experience than building representations meant for
immediate consumption (e.g., newspaper diagrams and maps). Readers will observe that the
use of metaphor is rich within the high level interactions with computers. We are an interface
culture (Johnson 1997). However, the metaphors used on the "desktop," for instance, have
not yet made their way into the core of mathematics and computing. Efforts such as
computational thinking (Wing 2006) are a move in the right direction.

Laurel (1991) presciently captures a prerequisite for aesthetic computing in her "Computing
as Theatre." However, Laurel was mainly constructing a case for human-computer interaction
as a complex theatrical production, involving many of the same elements found in theatre.
The use of computing, and its associated interaction phenomena, are like theatre. However,
what we :nd is that as we break open the lid of the black box containing the atomic
elements of normally hidden data, formulas, code, and models is that computing is theatre
all the way down.

26.10 Toward a Method of Aesthetic Computing

While it is interesting to pose ideas and directions, a procedural method is something that
can help to forge a discipline even if only as a general guide. Fishwick (2007a) was an initial
attempt at this process with a small example of code that was represented as a collection of
rooms in a building, complete with a partial narrative for context. Figure 10 serves as a basis
for describing the approach used in (Fishwick 2012):

Courtesy of Paul Fishwick. Copyright status: Unknown (pending investigation). See section
"Exceptions" in the copyright terms below.

Figure 26.10: Aesthetic Computing Method

We begin (in the top left of Figure 10) with a formal language construct that is to be
conveyed to non-specialists in mathematics and computing with the goal of broadening the
exposure of computing concepts. The asterisks denote current emphases in (Fishwick
2012). Target users will depend on the type of formal language. If the goal is number sense,
and the numbers are fairly simple, we may be looking at elementary school children. If the
formal language is simple algebraic formulas, we may be looking at 8th grade mathematics.
More complex mathematical and computing structures may require higher grades, including
universities and in postgraduate, informal learning contexts. One of the desirable outcomes
of this approach to representation, though, is to expose very young children to seemingly
complex data structures and programs by using games and video as motivational media. I
expect that the approaches may serve as 1) scaffolding for later, more traditional, instruction
and notations, and 2) secondary devices (e.g., puzzles) to reemphasize concepts that some
learners :nd ditcult using standard notations. The goal is not to eliminate standard
notations as this would be counterproductive. Representation is divided, in Figure 10, into
two components: methods that achieve representation and technologies that support
embodiment. End products that emphasize, or surface, embodiment can vary. A good piece
of :ction can create a strong sense of presence and virtual embodiment, whereas a weak
interactive game may be left ignored if not well designed.

26.11 New Connections

A primary goal of mine in fostering aesthetic computing is to link disciplines especially
those in computing to the humanities and arts. As evidenced by designers and humanist
scholars, artifacts such as "code" and "data" are now being interpreted and recreated. There
are many reasons for this. Perhaps, the ubiquitous computing trend is the most signi:cant
driver software is everywhere and so, by natural extension, cultural. I welcome the artists,
designers, and humanists into the "formal languages" space and hope that through
collaborations and interdisciplinary discussions and critique that we might re-humanize core
elements of computing, and perhaps even mathematics.

26.12 Disciplinary and Technical Challenges

The area of aesthetic computing is not without its challenges. The goal is to leverage
embodiment theories toward building new computer-based interfaces for learning formal
languages. Disciplines that I have covered have sub-areas that are all targeted toward this
goal, but signi:cant challenges remain for each area:

Mathematics: the literature in mathematics education, and in the application of cognitive

linguistics within mathematics learning, is well-founded and supports aesthetic
computing. This body of knowledge, however, is more focused on analysis and theory
construction rather than, through analogy, building new interfaces in mathematics
education to take full advantage of the embodiment theories through realization. Some
 efforts in virtual manipulatives are a good start, but this work should expand to employ
the next generation of interface capabilities that stress embodiment (e.g., multi-touch
displays, body tracking, mixed/virtual reality technology).
Computing: the literature in computing education provides fairly easy-to-use interfaces
for seeing the results of executing programs; however, the programs are often limited to
the canonical alphanumeric notation with all of the human interaction being in the
program execution rather than inside the program. Efforts at software visualization
move in the direction required by aesthetic computing, and yet, there is a much wider set
of possibilities for representation if the goal is to teach non-specialists especially
through immersion, situated learning, and interactive games. Diagrams are :ne for
communication, but if the goal is to explore deeply embodied approaches for learning,
additional media and newer interfaces as recommended for mathematics education
should be more thoroughly investigated.
Humanities: the work in cognitive linguistics, and resulting embodiment theories, ground
the work in aesthetic computing, but as with the work in the philosophy of mathematics
learning, there is little corresponding effort in realizing these theories in a human-
computer interface. Conversely, the work in cultural theory production is recently
targeting code speci:cally as a new type of literacy (e.g., critical code studies). And yet,
this production tends to avoid linguistic analysis and instead focuses on socio-historical
analysis. New, embodied, interfaces for code can build off of the scholarly analysis, but
these interfaces should also be informed by key facts of semiotics (e.g., analogy and
metaphor) which lie at the foundation of formal languages. There appears to be a bias
toward textual notation rather than exploring broader forms of embodied literacy, which
would include textual notation as one dimension.
Art & Design: works of art have traditionally treated formal languages as black boxes,
tools needed to create art or designs. Unlike in the humanities, where code has become
subject material, in art, code tends to be treated purely as a tool, whether embedded in
package or programmed via a text-based development environment. The only exception
to this observation would be in typography within graphic design, where the subject
material is the text. More explorations are required so that formal languages become the
active subjects of artwork.

Each one of these four areas has some common challenges. Observing that analogy is the
engine of metaphor in scienti:c practice, aesthetic computing products can be created with
an increased attention to analogy. Another observation is that with the exception of Art &
Design, there is a classical focus on alphabetic notation. Such notation serves us well and
has enriched our formal languages. However, there are other types of notations that exercise
more of the bodys sensorimotor functions. Diagrams are a good place to start in seeing this
transition since with diagrams spatial metaphors for text-based notations abound, but we
should not limit our embodied explorations to diagrams.
A primary aesthetic computing challenge is technological. It is still relatively expensive to
build new interfaces based on the types of products described by the :gures previously
shown. 3D modeling as a real-time technical interface capability is nowhere near the
futuristic landscapes of Tron, the Matrix, and the Holodeck. Modeling and animating in three
dimensions remains a major challenge compared with diagrammatic approaches, and even
diagram-based software modeling (e.g., model-driven architecture) struggles for acceptance
in the marketplace of software engineering solutions because of the relative ease of using
textual symbols. Human-computer interaction solutions are expanding in scope and
capability, but we still are a long way from being able to easily and inexpensively become
embodied in our formal language constructs.

26.13 Summary: The Argument for Embodied Formal

Language
This chapter began with personal experiences in mathematics and then moved on to
discussions of embodied cognition, along with some examples of where aesthetic
computing could be applied. The area of aesthetic computing rests primarily on the
foundation of embodiment whether we believe that our bodily interactions form our
thought. This assumption of embodiment runs deep in philosophy. We all recognize that we
have body and mind, and most would agree that the latter is the effect of the former. It is only
fairly recently, though, that literature has arisen to indicate a strong relationship to the extent
that thought itself, even for abstract objects, is embodied. The theory that undergirds
embodiment is compelling, but we have the nagging question about how this theory can
change what we do and how we act. If I imagine that I am imagining grabbing and pushing a
number through a pseudo-biological membrane during arithmetical operations, I want to
build a human-computer interface that reinforces this mental sequence by infusing theory
into practice. This perceived need matches the aesthetic computing hypothesis stated at the
start of the chapter: Given the embodied nature of cognition, we should realize this
embodiment through novel human-computer interfaces for learning formal languages.

Achieving this realization involves a more thorough understanding of the interplay among
disciplines and how embodiment theories in those disciplines interact and connect. The
realization also requires a host of newer virtuality continuum technologies that allow us to
achieve what Biocca refers to as degrees of progressive embodiment (Biocca 1997). The
technologies and their characteristics are overviewed for virtual reality by Sherman and Craig
(2002), and by Bowman et al. (2004), and for augmented reality by Bimber and Raskar
(2005).

26.14 Where to Learn More about Aesthetic Computing

For a thorough understanding of computing as a discipline, and its artifacts which are
represented in aesthetic computing, the 1998 ACM Computing Classi:cation System (CCS
1998) serves as a good starting point. Even though my treatment of aesthetics is based on
its original, perceptual de:nition, Kelly (1998) collects that which erupted from this kernel in
philosophy and the arts in four volumes. Even though information visualization is centered
on etcient communication (e.g., reading the equivalent of a diagram in a newspaper), some
archives such as infosthetics curated by Vande Moere (2011) are broader and contain a wide
variety of potential use cases from etcient communication to experience, education, and
play. For text-based representations, HASTAC (2011) serves as a high level repository of
bloggers and projects, many of which are associated with digital humanities. The reader is
encouraged to review articles cited in this chapter.

26.15 Acknowledgments
I would like to :rst acknowledge all individuals who have participated in this journey from its
inception, including my colleagues in the arts, natural and social sciences, computer science,
and mathematics. Students in my Aesthetic Computing class have had to put up with these
ideas, and they have produced wonderful products that I could never have imagined. Thanks
to the following colleagues who took time to make very good critical remarks on earlier
forms of this manuscript: Sophia Acord (University of Florida), Michael Kelly (University of
North Carolina at Charlotte), Mads Sgaard (Interaction Design), and Kang Zhang (University
of Texas at Dallas). I take responsibility for any errors and omissions.

26.16 References
Anderson, Kenneth M., Taylor, Richard N. and Whitehead, E. James (2000): Chimera:
hypermedia for heterogeneous software development environments. In ACM Transactions
on Information Systems, 18 (3) pp. 211-245

Audi, Robert (ed.) (1999): The Cambridge Dictionary of Philosophy. Cambridge University
Press

Barsalou, Lawrence W. (2010): Grounded Cognition: Past, Present, and Future. In Topics in
Cognitive Science, 2 (4)

Beck, Dennis, Fishwick, Paul A., Kamhawi, Rasha, Coffey, Amy Jo and Henderson, Julie
(2011): Synthesizing Presence: A Multidisciplinary Review of the Literature. In Journal of
Virtual Worlds Research, 3 (3)

Bimber, Oliver and Raskar, Ramesh (2005): Spatial Augmented Reality: Merging Real and
Virtual Worlds. A K Peters, Ltd
Biocca, Frank (1997): The Cyborg's Dilemma: Progressive Embodiment in Virtual
Environments. In Journal of Computer-Mediated Communication, 3 (2)

Bolter, Jay David and Grusin, Richard (2000): Remediation: Understanding New Media. The
MIT Press

Bowman, Doug A., Kruijff, Ernst, LaViola, Joseph J. and Poupyrev, Ivan (2004): 3D User
Interfaces: Theory and Practice. Addison-Wesley Professional

Brown, John Seely, Collins, Allan and Duguid, Paul (1989): Situated Cognition and the Culture
of Learning. InEducational Researcher, 18 (1)

Burke, Joseph (1943): Hogarth and Reynolds: A contrast in English art theory. Oxford
University Press

Buxton, Bill (2008): The Role of the Artist in the Laboratory. In: (ed.). "Meisterwerke der
Computer Kunst". pp. 29-32

Card, Stuart K., Mackinlay, Jock D. and Shneiderman, Ben (eds.) (1999): Readings in
Information Visualization: Using Vision to Think. Academic Press

Cavallaro, Dani (2001): Cyberpunk & Cyberculture: Science Fiction and the Work of William
Gibson. The Athlone Press

Cockton, Gilbert (2012). Usability Evaluation. Retrieved 4 November 2013 from [URL to be
de:ned - in press]

Dede, Chris (2005): Planning for Neomillennial Learning Styles Neomillennial Learning Styles
and Mediated Immersion. In EDUCAUSE Quarterly, 28 (1) pp. 18-21

Devlin, Keith (2006): The Useful and Reliable Illusion of Reality in Mathematics. In: Towards a
New Epistemology of Mathematics. Workshop at GAP.6 Freie Universitt Berlin 2006.

Dewey, John (1934): Art as Experience. Perigee Trade

Diehl, Stephan (2007): Software Visualization: Visualizing the Structure, Behaviour, and
Evolution of Software.Springer

Dieterle, Edward, Dede, Chris and Schrier, Karen (2007): "Neomillennial" Learning Styles
Propagated by Wireless Handheld Devices. In: Lytras, Miltiadis D. and Naeve, Ambjorn (eds.).
"Ubiquitous and Pervasive Knowledge and Learning Management: Semantics, Social
Networking and New Media to Their Full Potential". pp. 35-66

Diffey, T. J. (1995): A note on the meanings of the term 'aesthetic'. In The British Journal of
Aesthetics, 35 (1) pp. 61-66

Dourish, Paul (2001): Where the Action Is: The Foundations of Embodied Interaction. MIT
Press

Eades, Peter and Zhang, Kang (1996): Software Visualization. Series on Software
Engineering and Knowledge Engineering. World Scienti:c Pub Co Inc

Feldman, Jerome (2006): From Molecule to Metaphor: A Neural Theory of Language. A

Bradford Book

Feldman, Jerome and Narayanan, Srinivas (2004): Embodied meaning in a neural theory of
language. In Brain and Language, 89 (2) pp. 385-392

Fishwick, Paul A. (2012). Aesthetic Computing Class. Retrieved 1 January 2012 from
University of Florida: http://www.cise.ul.edu/~:shwick/ac/2012

Fishwick, Paul A. (ed.) (2006): Aesthetic Computing. The MIT Press

Fishwick, Paul A. (2007a): Customized Visual Computing: The Aesthetic Computing Method.
In: Ferri, Fernando (ed.). "Visual Languages for Interactive Computing: De:nitions and
Formalizations (Premier Reference Source)". pp. 425-435

Fishwick, Paul A. (ed.) (2007b): Handbook of Dynamic System Modeling. Chapman and Hall -
CRC

Fishwick, Paul A. (2002): Aesthetic Programming: Crafting Personalized Software. In

Leonardo, 35 (4) pp. 383-390

Fishwick, Paul A. (1995): Simulation Model Design and Execution: Building Digital Worlds.
Prentice Hall

Fishwick, Paul A. (2008): Software Aesthetics: From Text and Diagrams to Interactive Spaces
Paul A. Fishwick. InInternational Journal of Arts and Technology, 1 (1) pp. 1-13

Fishwick, Paul A. (2012b): A Decade of Digital Arts and Sciences at the University of Florida.
In Leonardo, pp. 211-216
Fishwick, Paul A. and Bertelsen, Olav W. (2002). Dagstuhl Seminar Report Number
348.http://www.dagstuhl.de/Reports/02/02291.pdf

Fishwick, Paul A. and Park, Yuna A. (2008): A 3D Environment for Exploring Algebraic
Structure and Behavior. In:Ferdig, Richard E. (ed.). "Handbook of Research on Effective
Electronic Gaming in Education Set of 3". Information Science Referencepp. 546-559

Fishwick, Paul A., Coffey, Amy J., Kamhawi, Rasha and Henderson, Julie (2005): An
experimental design and preliminary results for a cultural training system simulation. In:
Simulation Conference WSC, Proceedings of the 2010 Winter 2005. pp. 799-810

Fishwick, Paul A., Diehl, Stephan, Prophet, Jane and Lowgren, Jonas (2005b): Perspectives
on Aesthetic Computing. In Leonardo, 38 (2) pp. 133-141

Fuller, Matthew (ed.) (2008): Software Studies: A Lexicon (Leonardo Book Series). The MIT
Press

Gershenfeld, Neil, Krikorian, Rat and Cohen, Danny (2004): The Internet of things. In
Scienti:c American, 291 (4) pp. 76-81

Ghezzi, Carlo and Jazayeri, Mehdi (1997): Programming Language Concepts. Wiley

Goldin, Gerald A and Kaput, James J (1996): A Joint Perspective on the Idea of
Representation in Learning and Doing Mathematics. In: Steffe, Leslie P., Nesher, Pearla, Cobb,
Paul, Goldin, Gerard A. and Greer, Brian (eds.). "Theories of Mathematical Learning".
Routledgepp. 397-431

Grau, Oliver (2004): Virtual Art: From Illusion to Immersion (Leonardo Books). The MIT Press

Green:eld, Adam (2006): Everyware: The Dawning Age of Ubiquitous Computing. New Riders
Publishing

Grossman, Lev (2009). Steampunk: Reclaiming Tech for the Masses. Retrieved 19 April 2012
from Times Magazine: http://www.time.com/time/magazine/article/0,9171,1...

Guhe, Markus, Smaill, Alan and Pease, Alison (2009): A formal cognitive model of
mathematical metaphors. In:Proceedings of the 32nd annual German conference on
Advances in arti:cial intelligence 2009. pp. 323-330

Hadamard, Jacques (1996): The Mathematician's Mind. Princeton University Press

Hassenzahl, Marc (2011). User Experience and Experience Design. Retrieved 4 November
2013 from [URL to be de:ned - in press]

Hopcroft, John E., Motwani, Rajeev and Ullman, Jeffrey D. (2000): Introduction to Automata
Theory, Languages, and Computation (2nd Edition). Addison Wesley

Huff, Kenneth A. (2006): Visually Encoding Numbers using Prime Factors. In: Fishwick, Paul
A. (ed.). "Aesthetic Computing (Leonardo Books)". The MIT Press

Ifrah, Georges (2002): The Universal History of Computing: From the Abacus to the Quantum
Computer. Wiley

Ikei, Yasushi and Ota, Hirofumi (2008): Spatial Electronic Mnemonics for Augmentation of
Human Memory. In2008 IEEE Virtual Reality Conference, 53 (14) pp. 217-224

Johnson, Mark (2007): The Meaning of the Body: Aesthetics of Human Understanding.
University of Chicago Press

Johnson, Steven A. (1997): Interface Culture. Basic Books

Johnson, Mark (1987): Body in the Mind: The Bodily Basis of Meaning, Imagination and
Reason. University of Chicago Press

Johnson, Eric and Adamo-Villani, Nicoletta (2010): A Study of the Effects of Immersion on
Short-term Spatial Memory. In Virtual Reality, 71 pp. 1-44

Kallay, William (2011): The Making of Tron: How Tron Changed Visual Effects and Disney
Forever. William Kallay

Kelly, Michael (ed.) (1998): Encyclopedia of Aesthetics. Oxford University Press, USA

Kelly, Michael, Vesna, Victoria, Fishwick, Paul A., Moere, Andrew Vande and Huff, Kenneth
(2009): The state of aesthetic computing or info-aesthetics: curated panel discussion. In:
ACM SIGGRAPH 2009 Art Gallery 2009. pp. 43-

Kivy, Peter (ed.) (2004): The Blackwell Guide to Aesthetics. Wiley-Blackwell

Knuth, Donald E. (1992): Literate Programming. Center for the Study of Language and
Information

Lakoff, George and Johnson, Mark (2003): Metaphors We Live By, 2nd edition. University of
Chicago Press

Lakoff, George and Johnson, Mark (1983): Metaphors We Live by. New York, Basic Books

Lakoff, George and Johnson, Mark (1999): Philosophy in the Flesh: The Embodied Mind and
Its Challenge to Western Thought. New York, Basic Books

Lakoff, George and Nunez, Rafael (2001): Where Mathematics Comes From: How the
Embodied Mind Brings Mathematics into Being. Basic Books

Laurel, Brenda K. (1991): Computers as Theatre. Reading, MA, Addison-Wesley Publishing

Lave, Jean and Wenger, Etienne (1991): Situated Learning: Legitimate Peripheral
Participation (Learning in Doing: Social, Cognitive and Computational Perspectives).
Cambridge University Press

Lessig, Lawrence (2008): Remix: Making Art and Commerce Thrive in the Hybrid Economy.
Penguin Press HC, The

Leymarie, Frederic Fol (2006): Aesthetic Computing and Shape. In: Fishwick, Paul A. (ed.).
"Aesthetic Computing". The MIT Presspp. 258-288

Lowgren, Jonas (2008). Interaction Design - brief intro. Retrieved 4 November 2013 from
[URL to be de:ned - in press]

Malina, Roger (2011). The Strong Case for Art-Science Interaction. Retrieved 19 March 2012
from thoughtmesh: http://vectors.usc.edu/thoughtmesh/publish/120.php

Manovich, Lev (2001): The Language of New Media. The MIT Press

Mar, Raymond A. and Oatley, Keith (2008): The Function of Fiction is the Abstraction and
Simulation of Social Experience. In Perspectives on Psychological Science, 3 (3) pp. 173-192

Marino, Mark C. (2006). Critical Code Studies. Retrieved 18 April 2012 from Electronic Book
Review: http://www.electronicbookreview.com/thread/electro...

McLuhan, Marshall (1964): Understanding Media: The Extensions of Man. The MIT Press

Mohr, Manfred (2011). Rlexions Sur Une sthetique Programme. Retrieved 18 March 2012
from Bitforms Gallery, New York: http://www.bitforms.com/press-releases/405-2011-ma...

Nake, Frieder (2009): The semiotic engine: notes on the history of algorithmic images in
Europe. In Art Journal, pp. 76-89

Negroponte, Nicholas (1996): Being Digital. Vintage

Norman, Donald A. (2004): Emotional Design: Why We Love (Or Hate) Everyday Things. Basic
Books

Papert, Seymour (1980): Mindstorms: Children, Computers and Powerful Ideas. New York,
Basic Books

Parsons, Thomas D and Rizzo, Albert A (2008): Initial validation of a virtual environment for
assessment of memory functioning: virtual reality cognitive performance assessment test.
In CyberPsychology and Behavior, 11 (1) pp. 17-25

Petre, Marian and Blackwell, Alan (1999): Mental Imagery in Program Design and Visual
Programming. InInternational Journal of Human-Computer Studies, 51 (1) pp. 7-30

Piaget, Jean (1950): The Psychology of Intelligence. Routledge

Pine, B. Joseph (1999): The Experience Economy: Work Is Theater & Every Business a Stage.
Harvard Business Press

Pine, B. Joseph and Davis, Stan (eds.) (1999): Mass Customization: The New Frontier in
Business Competition.Harvard Business School Press

Reas, Casey and Fry, Ben (eds.) (2007): Processing: A Programming Handbook for Visual
Designers and Artists.The MIT Press

Ritchie, Andrew C. (1968): English Painters, Hogarth to Constable. Ayer Co Pub

Roth, Tina, Aiken, Peter and Hobbs, Scarlette (1994): Hypermedia Support for Software
Development: A Retrospective Assessment. In Hypermedia, 6 (3) pp. 149-173

Rotman, Brian (2000): Mathematics as Sign: Writing, Imagining, Counting (Writing Science).
Stanford University Press

Ryder, William (2009): A System Dynamics View of the Phillips Machine. In Proceedings of
the 27th International Conference of the System Dynamics Society,

Salomon, Gavriel (1990): Cognitive Effects With and Of Computer Technology. In

Communication Research, 17 (1) pp. 26-44
Samuel P. Harn Museum of Art (2011). Art in Engineering. Museum Nights at the Harn
Museum of Art. Retrieved 19 April 2012 from Samuel P. Harn Museum of Art:
http://www.harn.ul.edu/octmuseumnight.pdf

Scruton, Roger (2011): Beauty: A Very Short Introduction. Oxford University Press

Sfard, Anna (1994): Rei:cation as the birth of metaphor. In For the Learning of Mathematics,
14 (1) pp. 44-55

Sfard, Anna and Thompson, Patrick W. (1994): Problems of rei:cation: Representations and
mathematical objects. In Proceedings of the Annual Meeting of the International Group for
the Psychology of Mathematics Education North America Plenary Sessions, 1

Sherman, William R. and Craig, Alan B. (2002): Understanding Virtual Reality: Interface,
Application, and Design (The Morgan Kaufmann Series in Computer Graphics). Morgan
Kaufmann

Slingerland, Edward (2008): What Science Offers the Humanities: Integrating Body and
Culture (New Approaches to European His). Cambridge University Press

Speer, Nicole K., Zacks, Jeffrey M. and Reynolds, Jeremy R. (2007): Human brain activity
time-locked to narrative event boundaries. In Psychological Science, 18 (5) pp. 449-455

Stasko, John T., Domingue, John B., Brown, Marc H. and Price, Blaine A. (eds.) (1998):
Software Visualization. The MIT Press

Swade, Doron (2000): The Phillips Machine and the history of computing. In: Leeson, Robert
(ed.). "A. W. H. Phillips: Collected Works in Contemporary Perspective". Cambridge University
Presspp. 120-126

Thompson, Patrick W. and Sfard, Anna (1994): Problems of rei:cation: Representation and
Mathematical objects. In:Proceedings of the Annual Meeting of the International Group for
the Psychology of Mathematics Education - North America 1994, Baton Rouge, USA.

Tractinsky, Noam, Katz, A. S. and Ikar, D. (2000): What is Beautiful is Usable. In Interacting
with Computers, 13 (2) pp. 127-145

Tufte, Edward R. (1983): The Visual Display of Quantitative Information. Cheshire, CT,
Graphics Press

Turkle, Sherry (2004): How Computers Change the Way We Think. In Higher Education, 50
(21)

Varela, Francisco J., Thompson, Evan and Rosch, Eleanor (1991): The Embodied Mind:
Cognitive Science and Human Experience. Cambridge, Massachusetts, MIT Press

Vigas, Fernanda B., Wattenberg, Martin, Ham, Frank van, Kriss, Jesse and McKeon, Matt
(2007): ManyEyes: a Site for Visualization at Internet Scale. In IEEE Transactions on
Visualization and Computer Graphics, 13 (6) pp. 1121-1128

Ward, Matthew O., Grinstein, Georges and Keim, Daniel (2010): Interactive Data Visualization:
Foundations, Techniques, and Applications. A K Peters Ltd

Wing, Jeannette M. (2006): Computational thinking. In Communications of the ACM, 49 (3)

pp. 33-35

Yates, Frances A. (1966): The Art of Memory. University of Chicago Press

Zhang, Kang (2007): Visual Languages and Applications. Springer

26.16 Commentary by Michael Kelly

The masters of information have forgotten about poetry,

where words may have a meaning quite different from
what the lexicon says, where the metaphoric spark is
always one jump ahead of the decoding function.

-- J. M. Coetzee, Diary of a Bad Year [1]

26.16.1 Fishwick's Encyclopedia Entry on Aesthetic Computing

Paul Fishwick has a well-developed, impressive research and pedagogical platform at the
University of Florida from which hes been exploring one of the particular versions of
aesthetic computing, although, being involved since its inception, he also has a general
sense of the :eld. In fact, there would hardly be such a :eld without him. The question I want
to ask is how much his particular projects are inluencing his general conception of aesthetic
computing and whether hes achieved the appropriate, if ditcult editorial balance here [2].

Fishwick begins with a pedagogical focus: personal experiences in mathematics that led to
discoveries and explorations of embodied cognition. In particular, he analyzes the aesthetic
transformation to formal language, using the concept of embodied knowledge, understood
as a perception-action feedback loop based on the idea that embodiment is a form of
representation, not just an insigni:cant step in the process of a strictly cognitive mode of
representation. From there he argues, with a rich set of projects, that The purpose of
aesthetic computing is to deliver knowledge and practice of formal languages using
aesthetic products as a vehicle. In short, the examples of teaching abstract mathematical
concepts that led Fishwick to aesthetic computing have continued to have structural as well
as thematic roles throughout his entry and have largely determined his conception of the
:eld. The result is an excellent but partial picture of aesthetic computing that, if taken for the
whole, would be misleading.

Fishwick acknowledges a broader conception of aesthetic computing in the Why Aesthetic

Computing? section (26.4). But he does not adequately clarify it or show concretely how it
informs or otherwise relates to his research. Has he perhaps changed his view of aesthetic
computing over the last decade? Back in 2006, he made a number of statements about
aesthetics that point to a conception of aesthetic computing broader than what is generally
evident in his Encyclopedia entry (Fishwick 2006). For example, he says that aesthetics
reaches beyond classic concepts such as symmetry and invariance and encompasses the
wide range of aesthetic de:nitions and categories normally associated with making art. Yet
now he seems to limit aesthetic computing to often classic concepts speci:cally relevant to
his projects. Quoting the Preface to the Encyclopedia of Aesthetics, he embraces the idea of
aesthetics as the philosophical analysis of the beliefs, concepts, and theories implicit in the
creation, experience, interpretation, or critique of art. However, when he references this
same idea in the Encyclopedia entry, he seems to pull back from its full implications for
aesthetic computing. Fishwick also says earlier that aesthetics has logical as well as
material aspects, so it can extend to computing as well as art. Taking discrete mathematics
as an example, he claims that aesthetic computing encompasses notions of formal
language, geometry, and topology, and from such claims he concludes that aesthetic
computing corresponds naturally with mathematical formalism. In his current research and
Encyclopedia entry, Fishwick focuses mostly on this last sense of aesthetics and develops,
albeit very well, only the narrower view of aesthetic computing it implies.

Yet, the case for a broader conception of aesthetic computing can be made from within
Fishwicks own projects because he argues that aesthetic computing rests primarily on the
foundation of embodiment, which is itself a very important research topic in aesthetics and a
number of disciplines (e.g., cognitive psychology, affective computing, philosophy of mind,
etc.). But even here Fishwicks sense of embodiment is mostly cognitive and pedagogical
because its linked principally to formal languages. This may seem like an appropriate link
because computing is so much about formal languages. But isnt the whole point of
aesthetic computing to develop and sustain a richer conception of computing? With a richer
conception in mind, in effect, the art historian and theorist Caroline A. Jones offers a more
art-centric and aesthetics-informed account of embodiment that is focused on the impact of
computerized technology on the human body, on the techno-human. [3]. She begins by
arguing that the best way for the critique of our techno-culture to keep pace with the speed
of technological innovation is to take up these technologies in the service of aesthetics,
which provides a site for questioning how our bodies are interacting with technologies at
the present moment. Aesthetics provides contemplative space for such a critique because it
buys us time and space to encounter and relect on embodied experience in an ever more
technologized world. That is, aesthetics sets up critique within computing to examine how
human-computer interactions impact our bodies. The goal of such critique is not merely to
understand all the computer-generated bodily interactions that have been experienced
already but to explore which ones could be experienced, and, moreover, which ones we
would prefer to experience going forward. In the end, a major advantage of Joness account
of embodiment is that she makes it clear that this kind of critical thinking internal to
computing already has a name with a long tradition: aesthetics. By making the links among
embodiment, computing, and aesthetics explicit, she offers broader conceptions of
computing and aesthetic computing alike.

Joness account of embodiment, which is explored by a number of artists and theorists

involved in the Sensorium exhibit or catalog, also dovetails well with the aesthetics of
participatory art practices that have developed recently in contemporary art, which would
also help to broaden aesthetic computing [4]. Participatory art is, in brief, the convergence of
various art forms that emerged in avant-garde modernism or contemporary art: interactive
art, installation art, performance art, conceptual art, new media art, public art, socially
engaged art, etc. Such convergence has altered the aesthetics of contemporary art in ways
(e.g., agency is collective, form is participatory, interactions are transformative) that resonate
in computing, too, as it becomes ever more ubiquitous, participatory, collaborative, social,
and interactive. Since a central concern about aesthetic computing is how aesthetics is
relevant to computing, it would help this cause to examine the most recent developments in
the aesthetics of contemporary art. This does not mean that the aesthetics of classical or
modern art are not relevant, but since participatory art is emerging in part because of the
impact that computing has already had on the production and reception of contemporary art,
participatory art is an excellent area to explore while developing aesthetic computing.

In addition to the editorial imbalance, my other principal concern with Fishwicks

Encyclopedia entry is that he regards aesthetics primarily as a means (a vehicle): aesthetic
computing is embodied formal language with an educational goal as a :nal end product. As
a result, the critical thinking core of aesthetics seems to be lost. For example, although
Fishwick identi:es some of his own aesthetic norms in 26.7 (only some of which strike me
as aesthetic) and his assumptions about aesthetics (e.g., that a principal concern is still the
universal attributes of beauty), he doesnt analyze them critically [5]. For example, the unity
in variety concept he endorses is a strategy in 18th century British aesthetics (developed by
Francis Hutcheson and others) to identify a property of an object that accounts for its beauty
without violating the shared principle among empiricists and rationalists that beauty itself is
not a property of any object. How does such a concept or strategy help to clarify the
diversity of computing or to negotiate between the conceptual nature of aesthetics and the
empirical practices of computing? Moreover, as Jones argues, aesthetics can also help to
determine the ends of computing by clarifying and critiquing its aesthetic and related norms,
so it shouldnt be viewed primarily as the means to achieve ends determined before
aesthetic computing was introduced.

To be fair, even if Fishwicks approach to aesthetic computing is narrow in the ways I have
described, it may be that the :eld :rst has to develop through particular (and thus narrow)
projects. Perhaps only then can we initiate a relective equilibrium between the general :eld
of aesthetic computing and the multivarious, particular projects that Fishwick and others are
engaged in. Even though I think the general and particular have to be developed
simultaneously from the start, Fishwick has clearly made important contributions to
aesthetic computing in this Encyclopedia and his research.

With the same relective equilibrium in mind, Id now like to clarify my understanding of a
broader conception of aesthetic computing because I appealed to it while critiquing Fishwick
[6].

26.16.2 Aesthetics in Computing

John Maeda (a computer scientist, designer, and President of the Rhode Island School of
Design) once created Palm Paintings: small, shallow boxes painted in various abstract
styles with a Palm computer built into each one serving as its visible center. His stated
purpose was to enable us to think, from the inside, about what the painting signi:ed. I take
it that his point was not necessarily that signi:cation is located materially inside the work of
art but, more provocatively, that our critical thinking about the work should take place as the
work is being made, if the critical thinking is to be truly inside the work and not merely added
as an extra after the fact. The mode of critical thinking here is aesthetics since the key
normative issues in art are aesthetic, making aesthetics an integral part of (Maedas)
painting.

In a reciprocal gesture, now imagine that we were to embed aesthetics into the design and
production of all the artifacts associated with computers databases, programs, networks,
data visualizations, games, etc [7]. The purpose would again be to think about what they
signify and, prospectively, what else we might want them to signify in the future (as well as
what other effects besides signi:cation we would like to see). The computing artifacts with
embedded aesthetics could be marked in some way to distinguish them from others. We
could then hope to learn about ubiquitous computing from the inside, as it is being
developed, not merely when it is already being used by people in society.

This reciprocal gesture is not imaginary because, as Fishwick has established, theres been
an aesthetic turn in a number of areas of computing, leading to the introduction of new
sub:elds such as aesthetic computing, computational aesthetics, database aesthetics,
digital aesthetics, information aesthetics, network aesthetics, or software studies [8]. The
diverse names, introduced by collaborative research teams of computer scientists and
others (e.g., artists, philosophers, art historians), are distinguished by where or, in the spirit of
Nelson Goodman, when aesthetics is introduced into computing [9]. That is, if we think of the
computer stack, the various layers of computing (with bits and hardware at the bottom and
user interaction at the top), the choice of name here is a function of when aesthetic norms
:rst enter computing. If aesthetic norms are involved in structuring databases, for example,
then we have database aesthetics; if they inluence how we give form to information, then we
have information aesthetics; if theyre part of how we organize networks of people
participating in various social media, then we have network aesthetics and so on within the
layers of the computing stack. The lower the layer on which aesthetic norms are implicitly
present, the greater the ripple effect the critique of these norms will have on the higher layers
of computing [10].

In this light, aesthetic computing is the one name among all the options that, in principle,
encompasses the entire computing stack and thus best captures the full breadth and depth
of the aesthetic turn in computing. In exploring more what aesthetics adds to computing, I
want to emphasize that aesthetic computing is not merely about the aesthetics of
computing (merely the design of programs or products, or merely an external critique of the
aesthetic norms of computing). Following Maedes Palm Paintings, what I envision is
aesthetics in computing, albeit with an anticipatory eye to its ethical and social-political
impact rather than only its internal structure (i.e., not merely computational aesthetics).

26.16.3 What is Aesthetic Computing?

Aesthetics is critical thinking about the norms, concepts, values, or principles guiding or
emerging from the production, experience, or reception of art, culture, or nature. Besides
referring to the range of theory and practice associated with computer programming,
databases, computation, software, operating systems, and hardware (everything from digits
to gadgets), the term computing (as distinct from computer science) captures the
recognition that computer science operates in a broad social (moral-political) context.
Aesthetic computing is a preferred way to operationalize this recognition because it is
critical thinking about the complex set of norms shaping all layers of computing that are, in
turn, shaping this moral-political-social context [11].

Despite all the various names for aesthetic computing, there is a common thread running
through all the versions or iterations of it. The thread is the recognition among people
involved in computing that there are aesthetic norms implicit in the decisions or judgments
made on all layers of computing. Accordingly, the main tasks of aesthetic computing are (1)
to identify the genealogy and current status of the largely implicit aesthetic norms of
computing and to render them explicit; (2) to critique the aesthetic norms with an eye to their
moral-political-social implications for users; and (3) to help make decisions or judgments in
the future about which aesthetic norms to abandon, revise, or sustain in computing, given (1)
and (2), and of course given the technical norms within computing.

We can get a clearer picture of the need for aesthetic computing and its tasks by considering
Zadie Smiths review of David Finchers :lm, The Social Network, and of Jaron Laniers book,
You Are Not A Gadget: A Manifesto [12]. What the :lm and book have in common, on Smiths
analysis, is the claim that Different software embeds different philosophies, and these
philosophies, as they become ubiquitous, become invisible. Not only is software not neutral,
and not only are there important norms embedded in it (e.g., personhood, privacy, sociality),
but software in use also enacts these norms (i.e., puts them in practice). Software does not
merely copy our existing norms about ourselves and the world, however, it also enacts new
norms and, in doing so, makes a world. The problem, as Smith sees it, is that these invisibly
embedded and enacted norms are not discussed critically in advance; rather, they are
embedded and enacted by the programmers, most strikingly in the case of Facebook
because 800+ million users have had little or no say about its norms. Smith rightly points out
that there are ethical issues involved in this case: Why these norms rather than others? Why
this format for connecting people with one another rather than another format? What is the
quality of the connection? Why this privacy policy? For example, users are expected to give
up their privacy to a large extent, and they seem to do so willingly, albeit while reducing
themselves to :t the software they are using, according to Lanier, so much so that their life
is turned into a database.

Smiths analysis is relevant to aesthetic computing not only because she points to the
invisibility of the norms governing Facebook, the Internet, or the Web, but also because when
she develops her critique of these norms, she often refers to their look or feel. For
example, while we know that it is a mistake to believe that computers can personify human
relationships, we know this instinctively only by feeling the affective consequences of this
mistaken belief, which Facebook embodies: We know that having two thousand Facebook
friends is not what it [friendship] looks like. What is this look that we feel and that enables
us in turn to know that certain norms embedded and enacted in Facebook may be
problematic? We come to learn that Facebook is doing something to us through the
invisibility of its underlying norms and, if our continued critical relection is successful, well
come to learn what Facebook is doing to us and, moreover, whether there are any
alternatives. To succeed, we need to render visible the invisible norms operating in Facebook
so that well have a good reason for at times feeling discomfort at the world theyre making
[in Facebook]. This kind of critical thinking is precisely what aesthetic computing offers
because one of its main tasks is to render explicit the implicit norms of computing.

But let me return to the question: Why aesthetics? We might :rst ask, why philosophy? Smith
answers this second question by emphasizing that its the idea of Facebook that
disappoints, not merely the implementation of its idea. To analyze its idea, we need
philosophy to counter what she sees as a general cultural tendency in the Anglo-American
world to race ahead with technology and hope the ideas will look after themselves. We
need to examine the idea of Facebook and all the other ideas enacted on the Web and
Internet before, in Laniers words, we become locked in them, or entrapped in somebody
elses careless thought, which means that we are locked into the invisible norms shaping
these ideas and, once those norms are enacted on the Web or Internet, shaping our world
and us. But why turn to aesthetics in particular to examine these ideas involving ethics (e.g.,
security), metaphysics (e.g., personhood or virtual reality), etc.? Returning to Smiths
discussion of the look of Facebook, and remembering Joness account of embodiment, the
closest we come to experiencing the invisible norms that are enacted in software on the Web
or Internet is by experiencing the affects they create on us, the users. Many of these affects
are visible, but they involve all the senses (hearing and, increasingly, the tactile), just as
works of art do and just as our aesthetic experiences of everyday life do. Aesthetics brings
the affective dimensions of our experiences of computing to the fore, and it does so in a way
that provides a basis for critique of the sort that Smith, Jones, and Lanier are exploring.
These critiques are examples of aesthetic computing in action.

To take another kind of example clearly internal to computing, there has been an "aesthetic
turn" in the area of human-computer interaction (HCI) because some researchers believe it is
important to obtain a fuller picture of the user now that computer interfaces are more
interactive, participatory, immersive, and ubiquitous [13]. In a word, they need to understand
the user in affective, moral, and political as well as cognitive terms in order, in turn, to create
the right (i.e., effective, usable) interfaces. So aesthetics comes into the picture as the notion
of usability becomes normatively more complex. Why turn to aesthetics? A major reason is
that aesthetics has a long history of critiquing the particular kinds of affective and cognitive
interactions and modes of participation constitutive of our experiences of art, and these
critiques are relevant to the critiques of the affective-cognitive experiences of the user in
human-computer interactions [14]. These interactions (with their own modes of
participation) also have moral and political dimensions because users have to be treated
fairly (e.g., in matters of access, whether for economic or disability reasons) and their
political or cultural beliefs have to be respected. Here, too, aesthetics has a history of
critiquing works of art in relation to moral-political as well as aesthetic considerations. The
aesthetic turn here, whether in HCI or in any other :eld of computing, is therefore not a
narrowing of moral-political-social impact to aesthetic questions; rather, aesthetics provides
a philosophical structure for thinking critically about norms that are moral, political, social,
and aesthetic at the same time [15].

26.16.4 Aesthetic Computing and Science

Although the critique of aesthetic norms in computing with an eye to their moral-political-
social impact is a relatively new process, we can more easily appreciate its relevance and
importance if we see it as an augmentation of the existing practices of critical thinking in
computing [16]. That is, computing has always critically analyzed its normativity, even if the
norms have been understood mostly in technical terms (e.g., what is most etcient or
effective). The emergence of aesthetic computing stems from the recognition within
computing that its norms are more than technical, as we saw in the case of HCI. So aesthetic
computing is principally an outgrowth and re:nement of the recognition of the complex
normativity always already operative within computing. This is an important point to
emphasize because some computer scientists may view aesthetics the way they at times
view ethical, political, or other issues seemingly external to computing: those issues are not
relevant to what they do qua scientists (given their methodologies, aims, etc.) and thus to
give such issues methodological credibility can only place constraints on science. However,
if aesthetics (and the related normative) questions are understood as emerging from within
computing, scientists no longer need to be concerned that aesthetics is constraining
computing.

Yet researchers may still worry that aesthetic computing will change computer science in
ways that would make it less scienti:c, especially if Roger Molina is right that the strong
claim of aesthetic computing is that it will generate new objectives that "would not naturally
have evolved within the computing sciences and, moreover, that will redirect the future
development of computing. [17]. That is, the transition from implicit to explicit aesthetic
norms on the layers of the computing stack may have the result that we will change
technical as well as aesthetic norms and then change the objectives of computing on that
basis. But, again, if computing is normative and the self-critique of normativity is part of
science, the only real change resulting from aesthetic computing is that the aesthetic norms
always already part of computing will now be explicit and critically examined. How can
computing not bene:t from more self-critique, since the revision of its internal norms is part
of the engine that has driven progress in modern science, on its own terms? For example, as
computing becomes more conscious of the design issues that could contribute to
environmental sustainability, that may change certain objectives of computing but it would
not make computing less scienti:c, for if it were to become less scienti:c, it could not
contribute to sustainability. In short, aesthetic computing shows how seemingly external
norms are actually internal to computing.

The issue of the status of computing as a science is worth dwelling on even longer because
it can stop the discussion of aesthetic computing cold. Some may still worry that aesthetics
involves taste and is thus subjective. In this light, to integrate aesthetics into computing
would be to introduce subjectivity into an otherwise objective science. However, what is
actually happening here is that computer scientists are recognizing (a) that the normative
complexity of computing has already shaped their idea of science, making room for a more
interdisciplinary approach to computing, and (b) that computing is more than a science, not
only because its moral-political-social impact entails too many nontechnical issues that
scientists need to understand in order to develop computing internally, but also because the
implicit nontechnical norms of computing are already shaping its development in ways that
need to be analyzed critically for the sake of computing as well as for our sakes as we live
and work with computers [18]. In short, the aesthetic turn in computing is a way to critique
its nontechnical norms in order to strengthen its status as science at this stage of its
development. Why aesthetics? Again, because it is a long-standing :eld of philosophy that
has developed a variety of ways to think critically about aesthetic norms as they are related
to moral-political-social as well as technical norms.

26.16.5 Open Aesthetic Properties and Objects

Now, if aesthetic norms are always already a part of computing, why is aesthetic computing
barely a decade old, though aesthetics has its origins in the eighteenth century and
computers have been around for decades already? One explanation, according to Fishwick,
is that computing had to develop to a certain stage before its connection to aesthetics could
clearly emerge: We have had to wait for the technology to become available to leverage the
arts, especially in the :elds of HCI, ubiquitous computing, augmented reality, and virtual
reality [19]. Yet if aesthetics is so obviously relevant to computing, why was this delay
necessary? After all, aesthetics is a form of critical thinking and computing has relied on
critical thinking to evolve, so why didnt aesthetic computing emerge earlier? Another
explanation why its emergence has been slow, besides the worries about science being
constrained or becoming subjective, is that too many people in computer science seem to
have rather narrow, sometimes outdated ideas about aesthetics and thus have not been able
to see its relevance to computing or, when they have seen the relevance, theyve not been
able to get from the narrow ideas to what they hope aesthetics could contribute to
computing.

Too many people today still assume (and some philosophers still believe) that aesthetics is
principally concerned with making disinterested judgments of the quality of beauty inherent
in a class of unique and autonomous objects called works of art. However, aesthetics has no
unique set of objects, not only because so many things can be works of art, as the history
of modern art has taught us, but because aesthetics is as much about people, experience,
and value as it is about objects or things. And beauty is no longer a principal concern in
aesthetics because its not a principal concern in art (for a host of reasons analyzed by
others elsewhere) [20]. Moreover, aesthetics is not about the :xed properties of any objects,
whether works of art, natural objects, or artifacts of computing. This does not mean,
intentionally or unwittingly, that aesthetics is merely subjective or that, as we sometimes
hear, beauty is in the eye of the beholder. Aesthetics is not merely subjective any more than it
is merely objective because beauty (understood not merely as a particular aesthetic property
but as a stand-in for the entire set of aesthetic properties) is not in the subject any more than
it is in the object. But where is beauty, if it is not a :xed property of any subject or object? In
the language of eighteenth-century aesthetics, beauty is a relational property, that is, a
property resulting from cognitive and affective relations or interactions among human
subjects or between them and an open-ended set of works of art, natural objects, or artifacts
of computing. In this light, the task of aesthetic computing is to identify, render explicit, and
analyze critically the various conditions technological, social, ontological, psychological,
etc. that make such relations or interactions possible, not just what makes them more
effective, usable, communicable, pleasurable, and the like, though by understanding what
makes them possible well presumably be in a better position to address these other
concerns. Since the interactions here involve humans, and particularly since the interactions
are not only between humans and objects but among humans (hence the need to shift from
interaction to participation), aesthetic norms here are also moral and political. Again,
aesthetics is able to coordinate all the dimensions of these norms better than either ethics or
politics could because aesthetics has a long history of doing just that in the context of art.

On this account, aesthetics is a natural ally of computing because computing also tratcs in
objects lacking :xed properties, as is evident in Lev Manovichs discussion of the word
object in the Introduction to The Language of New Media. Expressions such as Arti:cial
Intelligence, Virtual Reality, Simulation, and Second Life likewise involve computer-based
realities and objects that are not :xed. Also, in the :eld of scienti:c visualization involving,
say, molecular biology, the data that are visualized are inaccessible to human senses since
there is no light at the molecular level. So the data do not constitute objects in the usual
sense of the word and their visualizations have no objective correlates. This means that
there is no single objective way to visualize molecular data, no essential visualization of
them just waiting to be discovered by a computer scientist (though any visualization is
always constrained by scienti:c methodologies and goals). Moreover, this means these
objects remain invisible even after they have been visualized, so it makes no sense to say
that the visualization of molecular data have :xed properties (other than in the broadest
sense of data properties i.e., qua numbers and codes). Looking at this description of
scienti:c visualization, computer scientists working in scienti:c (and other forms of)
visualization should feel at home in aesthetics because molecular (and other) data are very
similar to contemporary works of art: they too are not (necessarily) objects; they are more
conceptual than sensuous, even when they assume sensuous form(s); and they are not
imitations of objective realities against which they can be judged, so they can take numerous
forms, subject to the limits of visualization and the methodological structures and goals of
science (or art).

Now, if aesthetic computing is as much about human interactions as about objects or

properties, a key question here is what makes these interactions aesthetic. How can we
delimit the open-ended range of human-computer interactions and isolate those that are
speci:cally aesthetic, especially if beauty is not a :xed property and is actually an effect of
these interactions rather than a criterion for identifying them? [21] This question is both
easier and harder to answer in the case of aesthetic computing than it might be in aesthetics
more generally; easier, because the interactions have to involve some computing activities,
artifacts, or the like, which, for the most part, are easier to identify than works of art, which
have proven to be very elusive in recent years; yet harder too, because what is aesthetic
about human interactions involving computers? The answer to this last question is that the
norms implicitly embedded and enacted in the various layers of computing are what
introduce the aesthetic dimension (hence database aesthetics, information aesthetics, etc.,
depending on which layer of the computing stack is involved).

The open-ended nature of aesthetic computing may create consternation among some
computer scientists, or at least that has been my experience while researching, lecturing, or
teaching about aesthetic computing. For there is a tendency to expect that aestheticians
should provide objective norms (concepts, criteria, or the like) that can then serve as
practical guides for researchers in computing (the :eld of criticism in computing
sometimes embodies this tendency). If followed, however, this tendency would make
aesthetics a :eld external to computing that is then applied to it. By contrast, Ive proposed a
model of aesthetic computing that operates only within computing by rendering explicit the
aesthetic norms that are always already implicit and operative in the layers of the computing
stack. Any new norms will have to emerge from within computing practices, just as new
norms are introduced within artistic practices. In the end, aesthetics is either internal to
computing or has little critical relevance to it.
26.16.6 Endnotes
1. J. M. Coetzee, Diary of a Bad Year (New York: Penguin, 2008). See also Jaron Lanier:
Information systems need to have information in order to run, but information
underrepresents reality You Are Not A Gadget: A Manifesto (New York: Knopf, 2006).
2. As the Editor of the Encyclopedia of Aesthetics (New York: Oxford University Press, 1998),
Im not suggesting that the general should exclude the particular its all a matter of
balance
3. Caroline A. Jones, Introduction, in Jones, Ed. Sensorium: Embodied Experience,
Technology, and Contemporary Art (Cambridge: MIT Press, 2006).
4. For more on participatory art, see, e.g., Claire Bishop, Editor, Participation (London &
Cambridge: Whitechapel Gallery MIT Press, 2006); and Arti:cial Hells: Participatory Art
and the Politics of Spectatorship (London: Verso, 2012). Nicolas Bourriaud, Relational
Aesthetics (France: Les Presse Du Reel, 2002). Rudolf Freiling, Editor, The Art of
Participation: 1950 to Now (San Francisco & London: San Francisco Museum of Modern
Art and Thames & Hudson). Pablo Helguera, Education for Socially Engaged Art (New
York: Jorge Pinto Books, 2011). Grant Kester, Conversation Pieces: Community and
Communication in Modern Art (Berkeley: University of California Press, 2004); and The
One and the Many: Contemporary Collaborative Art in a Global Context (Durham: Duke
University Press, 2011). Nato Thompson, Editor, Seeing Power: Art and Activism in the
Age of Cultural Production (New York: Melville House, 2012); and Living as Form: Socially
Engaged Art from 1991-2011 (Cambridge: MIT Press, 2012).
5. To give another example, Lev Manovich argues that people often point positively to the
user-generated content available now online (e.g., anime music videos, political mashups)
as evidence of artistic freedom or creativity on the internet (even enhanced democracy),
yet they fail to relect critically on the fact that this content follows implicitly embedded
and enacted industry templates and conventions or reuses professionally produced
content. Manovich, Art After Web 2.0 in The Art of Participation: 1950 to Now.
6. Since Im a philosopher, its likely inevitable that my perspective on this new :eld is going
to be general. But such generality is also due to the fact that aesthetics is a conceptual
and normative :eld, though it clearly must be linked to the empirical reality of computing if
its going to have any etcacy as a mode of critical thinking that is internal to computing.
7. Mary Flanagan and Helen Nissenbaum have developed values at play, a conception of
critical play that identi:es and transforms the values embedded and enacted in computer
(and other) games. As I see it, their approach is a good example of aesthetic computing
because they render explicit the implicit norms of games. But they do not appeal to
aesthetics. In fact, they seem to shun it, perhaps because Flanagan is an artist and seems
to adopt uncritically the anti-aesthetic stance common in contemporary art, while
Nissenbaum is a philosopher who doesnt yet appreciate the critical value of aesthetics.
This is unfortunate, I think, because aesthetics provides exactly the kind of conceptual
 and critical resources Flanagan and Nissenbaum are developing as they analyze and
create games that embed and enact transformative values. See Flanagan, Critical Play:
Radical Game Design (Cambridge: MIT Press, 2009); and Flanagan and Nissenbaum,
Values at Play (forthcoming).
8. Aesthetic Computing began at a conference in Dagstuhl, Germany, in 2002, from which
emerged a manifesto published in Leonardo in 2003, and an anthology, Aesthetic
Computing, Paul Fishwick, Ed. (Cambridge: MIT Press, 2006) (which I reviewed in
Leonardo On-line Reviews (January 2007):
http://www.leonardo.info/reviews/jan2007/aest_kelly.html
Computational aesthetics, which is also called (or linked to) algorithmic aesthetics or
exact aesthetics, has been traced back to the 1930s; see Gary Green:eld, On the Origins
of the Term Computational Aesthetics; and Florian Hoenig, De:ning Computational
Aesthetics, in Computational Aesthetics in Graphics, Visualization and Imaging, I.
Neumann, M. Sbert, B. Gooch, W. Purgathofer, Editors (2005), pp. 9-12 and 13-18.
Database Aesthetics can be traced back to at least 1999; see Victoria Vesna, Editor,
Database Aesthetics (Minneapolis: University of Minnesota Press, 2007).
For examples of Digital Aesthetics, see Sean Cubitts website:
http://www.ucl.ac.uk/slade/digita/; and Johanna Drucker, SpecLab: Digital Aesthetics and
Projects in Speculative Computing (Chicago: University of Chicago Press, 2009).
Information Aesthetics has an active website: http://infosthetics.com/. See also the
SIGGRAPH Information Aesthetics Showcase in 2009:
http://www.siggraph.org/s2009/galleries_experiences/information_aesthetics/
For an example of Network Aesthetics, see Warren Sack, Network Aesthetics, in
Database Aesthetics, pp. 183-210.
For an example of Software Aesthetics, see Stephan Diehl and Carsten Grg, Aesthetics
and the Visualization and Quality of Software, in Fishwick, Aesthetic Computing, pp. 230-
37. There are also various websites devoted to this topic.
And theres also Visual Aesthetics, discussed extensively elsewhere in this Encyclopedia.
9. Goodman, Art in Action, in Encyclopedia of Aesthetics, pp. 322-25. For an account of
Goodmans relevance to aesthetic computing, see John Lee, Goodmans Aesthetics and
the Language of Computing, in Aesthetic Computing, pp. 29-42.
10. Manovich speaks of the cultural layer in addition to the computing layer, but Im
envisioning aesthetic computing that integrates rather than separates these layers. See
The Language of New Media (Cambridge: MIT Press, 2001).
11. As Fishwick clari:es in his Encyclopedia entry, aesthetic computing is different from
computer or digital art, that is, digital technology applied to the arts. Aesthetic
computing refers to the impact of artistic practices and aesthetic principles on the :eld
of computing, so the inluence lows from art and aesthetics to computing. For example,
computer scientists are looking to learn from artists how to conduct critiques of their
prototypes for new technologies (as artists do of their new works); how best to visualize
 data in scienti:c, information, or knowledge visualization; and how to understand the
balance between form and function or, more typically in computing, beauty and usability in
new technologies, especially those involving user interfaces. As these kinds of inluence
of art on computing are developed, aesthetics is a natural third party since art always
involves some type of aesthetics.
12. Zadie Smith, Generation Why? (Review of The Social Network, a :lm directed by David
Fincher, with a screenplay by Aaron Sorkin; and Jaron Lanier, You Are Not a Gadget: A
Manifesto (New York: Knopf, 2010), in New York Review of Books (November 25, 2010):
http://www.nybooks.com/articles/archives/2010/nov/25/generation-why/?
pagination=false
13. See, e.g., Olav W. Bertelsen and Sren Pold, Criticism as an Approach to Interface
Aesthetics, NordiCHI '04, October 23-27, 2004; Lars Erik Udsen and Anker Helms
Jrgensen, The Aesthetic Turn: Unravelling Recent Aesthetic Approaches to Human-
computer Interaction, Digital Creativity, 16, 4 (2005): 20516; Jeffrey Bardzell, Interaction
Criticism and Aesthetics, Proc. of CHI09. ACM Press (2009), 2357-66, and Jeffrey
Bardzell, Interaction Criticism: An Introduction to the Practice, Interacting with
Computers, 23 (2011) 60421. See also Olav W. Bertelsen: Tertiary Artifacts at the
Interface, in Aesthetic Computing, ed. Paul Fishwick (Cambridge: MIT Press, 2006), pp.
357-368. According to Bertelsen, human-computer interaction requires understanding of
the aesthetics of computing technology, that is, how computing technology is
experienced and experienceable. Input from aesthetic computing is greatly needed in
human-computer interaction (p. 359). In explaining what he has in mind, Bertelsen
analyzes the work of Marx Wartofsky, a philosopher of art and science. I think this is a
very good article in aesthetic computing, even if one does not accept the Wartofsky
framework, because Bertelsen clari:es aesthetics in a way that is philosophically sound,
linked to art and science, and relevant to computing.
14. See, e.g., Kirsten Boehner, Rogrio DePaula, Paul Dourish, and Phoebe Sengers, Affect:
From Information to Interaction, CC 05, Proceedings of the Dicennial Conference on
Critical Computing (New York: ACM Press), pp. 59-68. See also the MIT Lab for Affective
Computing: http://affect.media.mit.edu/
15. While any other discourses or disciplines implicated in this normative complex could
critique its own type of normativity, only aesthetics is able to critique the normativity in all
its complexity. For example, when Ken Goldberg installed Demonstrate (2004) in Sproul
Plaza on the campus of the University of California at Berkeley, his project raised all sorts
of issues and aesthetics is arguably at the center of them all. He set up a robotic
webcamera for six weeks (24/7) that could be manipulated (zooming in, taking
photographs, and the like) by people in remote locations, allowing somebody in Tokyo,
say, to conduct surveillance on people in the Berkeley plaza. Although it was technology
that made this installation possible, it clearly was not just an engineering project because
of the consequences of remote surveillance on unsuspecting people in an open plaza on
 the campus of a public university. There were legal issues, starting with the question of
the privacy rights of the people under surveillance, in particular because, as I understand
it, the camera was not calibrated tightly enough at :rst so it was able to scan beyond the
parameters intended for the project. In addition, because this project was also construed
as an art work, there were also issues of artistic freedom, not only on behalf of Goldberg
(and perhaps the people conducting the surveillance) but for the people in the plaza; for
they were no longer as strictly constrained in their public behavior because they were
participating in a work of art (apparently, some people engaged in or at least simulated
sex acts under the protection of artistic freedom). Finally, the project commemorated the
40th Anniversary of the Berkeley-led Free Speech Movement, so there were important
political issues at stake too because the movement was subjected to surveillance in its
time, albeit without todays more sophisticated technology. Aesthetic critique is able to
make sense of the normative complexity (technical, legal, ethical, political) of a project like
Goldbergs Demonstrate because, again, aesthetics has a long history of analyzing works
of art with this same type of normative complexity.
16. Warren Sack argues, as I understand it, that the recognition of the aesthetic (as well as
other nontechnical) dimensions of software and computing was evident from the early
days of computing. See his website: http://people.ucsc.edu/~wsack/
17. Roger Malina, A Forty-Year Perspective on Aesthetic Computing in the Leonardo Journal,
in Fishwick, Aesthetic Computing, p. 48. The other, weak claim is that aesthetics may help
computer scientists "achieve their [existing] objectives more easily, quickly, or elegantly
(p. 47).
18. For similar developments in other sciences, see, e.g., Aesthetic Science: Connecting
Minds, Brains, and Experience, Arthur P. Shimamura and Stephen E. Palmer, Eds. (New
York: Oxford University Press, 2012).
19. Fishwick, Aesthetic Computing, p. 13.
20. On the fate of beauty in modern art, see, e.g., Arthur C. Danto, The Abuse of Beauty:
Aesthetics and the Concept of Art (Chicago: Open Court Press, 2003); Elizabeth
Prettejohn, Beauty and Art: 1750-2000 (New York: Oxford University Press, 2005); and
Wendy Steiner, Venus in Exile: The Rejection of Beauty in 20th Century Art (Chicago:
University of Chicago Press, 2001).
21. There a long-standing discussion of the open nature of art works in the history of
contemporary aesthetics. See, e.g., Umberto Eco, The Open Work, tr. A. Cancogni
(Cambridge: Harvard University Press, 1989; originally published in 1962).

26.17 Commentary by Roger Malina

Paul Fishwick has been formulating for a number of years his concept of "Aesthetic
Computing", broadly de:ned as the application of the theory and practice of aesthetics to
computing; in the process an "embodied" formal language is advocated. In my view this
approach becomes particularly pertinent if, in the process, the methods and content of
computing as science and engineering is changed and enhanced. I have called this kind of
goal the strong case for art-science interaction [1] where the interaction goes beyond the
demonstrative or pedagogical. It is perhaps no accident that at the Dagstuhl workshop a
manifesto was issued [2] as there was a general feeling by the participants that the possible
contributions of the arts, design and humanities to computer science were not generally
accepted outside a group of enthusiasts.

A number of developments, some of which are referred to by Fishwick are mentioned, but
here I would like to address a number of issues which are included in the concept of
aesthetic computing but go beyond it. Fishwick talks of aesthetic computing addressing the
different elements of formal languages which are number, data, model and software.

Here it is perhaps useful to add Dennings [3] seven principles of computing; these have the
advantage of being process oriented and helps focus areas of possible art and design
intervention:

1. Computation: What can and cannot be computed

2. Communication: Reliably moving information between places
3. Coordination: Effectively using many computers
4. Recollection: Representing, storing and retrieving information from media
5. Automation: Discovering algorithms for information processes
6. Evaluation: Predicting performance of complex systems
7. Design: Structuring software systems for reliability

This signi:cantly broadens the landscape of possible intervention of aesthetic computing

approaches, and indeed many of these areas have been barely engaged by the arts and
humanities to date.

26.17.1 Big Data Transition and the Crisis of Representation.

A major development in the last decade is sometimes referred to as the "big data" transition
[4]. As data volumes and rates continue to grow at accelerating rates scienti:c disciplines go
through transformational changes. Astronomy was perhaps the :rst discipline to make this
transition with the emergence of virtual observatory strategies on both data archives and
software. Fields such as genomics soon followed and now all areas of business and culture
are impacted (see for instance [5]). This has led to what might be called a "crisis of
representation" and the emergence of new disciplines such as infoviz and dataviz. It has
rapidly become apparent that the problem is no longer one of communication of the content
of data via illustration techniques (e.g. [6]) but rather the problem becomes one of
immersion in data which can no longer be thought of as objects but rather as a "luid";
hence strategies of rei:cation referred to by Fishwick are proving inadequate as we enter
media territory. A number of researchers have been seeking to expand the reach of image
science into this new territory that requires semiotic approaches not yet developed. Most
data is never analysed or viewed, and new kinds of "technologies of attention" are required to
help navigate and isolate data that has particular content or meaning. This crisis of
representation is a good area for the arts and humanities to be involved and will be a long
term agenda for aesthetic computing; and as emphasised by Fishwick the key issue is
embodiment or how data is put in forms that are apprehensible via the human senses. A
number of artists have been prominent in exploratory projects for navigating through data
such as Donna Cox [7] and Ruth West [8]. There have been a number of exhibitions that have
sought to display the variety of approaches (e.g. siggraph information aesthetics [9]). It
seems to me that Fishwick pays insutcient attention to these developments which are
rapidly leading to new research areas.

26.17.2 Articcial Life Art, Visual Mathematics and Embodied Code

Paul Fishwick points out briely the area of serious gaming as one area where code can be
seen as being embodied. It seems to me there is a far larger area of algorithmic art, visual
mathematics, arti:cial life art which have much stronger examples of embodiment of code.
The development of algorithmic art by computer art pioneers such as Michael Noll, Roman
Verostko, Harold Cohen to name just a few has already established a 50 year history of
making algorithms apprehensible by human vision and hearing using aesthetic means and
objectives. The area of visual mathematics (see Michele Emmer and his two Leonardo books
on the visual mind [10]) has already provided success stories of how aesthetic methods have
led to scienti:c, or mathematical, discoveries. Following the Santa Fe Institute workshops
that established visibly the bases of arti:cial life [11], and its more recent applications to
synthetic biology, the arts community rapidly picked up the challenge with the development
of arti:cial-life art projects that has led to a proliferation of projects in robotics, virtual
worlds, interactive installations and other ways of bringing code into physical contact with
the human senses. Recently Leonardo Journal published [12] a selection of works as part of
the 10th anniversary of the VIDA arti:cial life competition, the pre-eminent forum for arti:cial
life art practitioners.

26.17.3 Translation as a possible method for Aesthetic Computing

In section 26.10 Fishwick articulates the elements of a method for aesthetic computing. In
recent discussions with colleagues Rainer Schulte and Frank Dufour at the University of
Texas, Dallas [13], I have been impressed by their work that seeks to apply the humanities
based methods of translation studies to the problems posed by computer generated or
mediated forms of creative "writing" (whether text, image, sound or indeed multi-medial and
multi sensorial). There are issues of media "essentialism as pointed out by Fishwick in his
section 26.2.4 on Media, where each medium has speci:cities that allows or prevents
certain concepts to be translated from one medium to another. But in addition - as pointed
out by Fishwick - the limitations of technology may constrain certain form of embodiment.
He gives the example of the experience of early word processing software where the writer
had to stop and wait for the microprocessor to catch up; similarly now certain
microprocessor steps occur at faster rates than the cycle times of the human cortex, so that
the act of translation from code to embodied perception requires the slowing down and time
stretching of phenomena. The act of translation from the culture of the "formal languages of
computing" to the "formal languages of the arts and literature", or vice versa, requires
methodologies from the humanities in order to create meaningful embodiment strategies for
aesthetic computing.

26.17.4 Intimate Science

Elsewhere I have written [14] of the general problem of how to enable cultural appropriation
of scienti:c phenomena inaccessible to the human senses. I have called this the agenda of
intimate science that many artists are now involved in. As pointed out by philosophers many
of the ways that we conceive of the world are built from our experiences from birth onwards.
Our ideas of causality, or more generally of explanatory systems, are fed by our interactions
with the world via our bodies and senses. But science now deals with many phenomena that
are not only beyond the "ampli:cation" or "augmentation" of our senses but are inherently
non-commensurate with the way our senses operate. This problem perhaps emerged most
clearly in the case of quantum mechanics where our basic ontologies are no longer
applicable (objects can be both wave and particle) and concepts of causality (in the case of
entanglement) totally foreign to our experience in the macro world. I would argue that we run
into similar issues in the concept of emergence in complexity science. Certainly we run into
similar issues in making sense of general relativity and the presence of distortions in space-
time and the structure of space itself; there is no way to experience gravity waves as a
human being. It seems to me that Fishwicks agenda for aesthetic computing in a way
transposes the problem of making science intimate into the :eld of computer science.
Computers have different internal logics than human bodies, different teleologies, and only
through the mechanisms of embodiment that he discusses can we begin to imagine the
way that a computer "imagines", The exercise of transposing models into maquettes is one
such process of cultural appropriation.

26.17.5 STEM to STEAM

During the last two years the U.S National Science Foundation, in partnership with the U.S
National Endowment for the Arts, have been organizing a number of workshops that bring
together the research communities in Science and Engineering and the creative communities
in Arts and Design [15]. These initiatives were responding to the sociological fact that there
is a growing body of research practice that bridges Science and Engineering to Art and
Design; in some cases School of Arts and Design :nd themselves engaged very similar
research agendas to Science or Engineering departments with of course different, or
overlapping, outcomes in mind. Sometimes unusual trans-discliplinary collaborations are
involved, in other cases artists and designers :nd themselves in the role of inventors and
technological innovators. This development was :rst recognized in information technology.
The "Mitchell" report "Beyond Productivity" [15] in 2003 laid out the problems and
opportunities. Similarly developments are now occurring in other areas of science and
engineering. As a result of these workshops the NSF has put in place two research contracts
to stimulate network development via the SEAD initiative [16], and a trans disciplinary
documentation platform via the XSEAD contract [17]. An early outcome of these workshops
was the acceleration of the concept of turning STEM into STEAM. After thirty more years of
national efforts in the U.S to develop the STEM workforce pipeline, the U.S is faced with
shortage of trained scientists and engineers. As articulated by the STEM to STEAM
movement (see for instance the congressional testimony organized by John Maeda,
President of the Rhode island School of Design [18]) we need to integrate the Arts and
Design, or more generally the Arts and Humanities, into Science Technology Engineering and
Mathematics education and research strategies. Some of the fastest growing computer
science related programs are in the computer arts, gaming, social media; and as pointed out
by theorists in innovation studies, the process of social and cultural innovation is playing a
stronger and stronger role in successful adoption of disruptive technologies. Expertise in
visualization and image science areas that are grounded in the arts and humanities are
promising areas that emerge from the aesthetic computing agenda that is outlined by
Fishwick. As pointed out by Fishwick our institutions of higher education are badly organized
to address these research agendas; the program that Fishwick has been developing at the
University of Florida is one exemplar of possible approaches.

Fishwick addresses strategies of aesthetic computing for the formal language construct
which are number, data, model and software. If we add Dennings seven principles of
computing namely Computation, Communication, Coordination, Recollection, Automation,
Evaluation, and Design it is clear that aesthetic computing is part of a larger ensemble of
arts and humanities research strategies that offer the opportunity of making major
contributions to computer science in the coming decades. As I write these comments, there
is a large online discussion on "The New Aesthetics", a discussion that credits its source as
James Bridles blog "The New Aesthetics" [20] in May 2011. With a starting point that
computing is now culturally integrated into our way of being in the world, the discussion (see
for instance Ian Bogost [20], for a rebuttal) has been lively - indicating that we are only at the
 beginning of aesthetic computing.

26.17.6 References
1. Malina, Roger (2011), The Strong Case for Art-Science Interaction, Retrieved from
http://vectors.usc.edu/thoughtmesh/publish/120.php
2. Paul Fishwick et al., 2003, 'Aesthetic Computing Manifesto'. Leonardo, 36, Issue No 4,
3. Peter Denning, http://cs.gmu.edu/cne/pjd/GP/GP-site/welcome.html
4. The fourth paradigm, Data Intensive Scienti:c Discovery, EDITED BY Tony Hey, Stewart
Tansley,and Kristin Tolle, 2009 Microsoft Corporation, ISBN 978-0-9825442-0-4.
5. http://datajournalism.stanford.edu/
6. Edward Tufte, http://www.edwardtufte.com/tufte/
7. Donna Cox, http://www.ncsa.illinois.edu/~cox/
8. Ruth West, http://www.atlasinsilico.net/gallery.html
9. 2003 Siggraph Information Aesthetics Show case,
http://www.siggraph.org/s2009/galleries_experiences/information_aesthetics/
10. Michele Emmer, The Visual Mind, Leonardo Books, 1993,
http://leonardo.info/isast/leobooks/books/emmer.html
11. Christopher G Langton (1998). Arti:cial life: an overview. MIT Press.ISBN 0262621126 ,
12. Leonardo Journal Vida Gallery, http://www.leonardo.info/isast/journal/toc411.html
13. http://translation.utdallas.edu/
14. http://www.sonicacts.com/portal/index.php/roger-malina-intimate-science-or-artists-in-
the-dark-universe/
15. Beyond Productivity : http://www.nap.edu/openbook.php?record_id=10671&page=235
16. http://sead.viz.tamu.edu/index.html
17. http://www.slideshare.net/ironman28/xsead-presentation-scalar-11912
18. http://stemtosteam.org/
19. James Bridle, http://new-aesthetic.tumblr.com/
20. Ian Bogost, http://www.wired.com/beyond_the_beyond/2012/04/ian-bogost-the-new-
aesthetic-needs-to-get-weirder/

26.18 Commentary by Sophia Krzys Acord

The hand knows what the mind cannot tell. This classic adage about embodied cognition,
familiar to artists (cf: Sudnow, 1978) and craftsman (I like: Crawford, 2009), plays an
important role in representing the complex ways in which we come to learn and understand
the knowledge and substance at work in building our physical worlds. This adage also
reveals the shortcomings of the perpetuating Cartesian superiority of mind over body, which
privileges language as an authoritative mediator of knowledge. Paul Fishwicks career work
to develop and formulate the novel :eld of Aesthetic Computing demonstrates an exciting
and important way in which notions of sensory and bodily experience can be brought to bear
on the most formal of languages: computer code. In this commentary, I will give a short
de:nition of embodied cognition, a brief background on how thinking aesthetically about
interaction can reveal important dimensions of knowledge-making, and then posit a few
ways in which aesthetic computing may be transformational for the evolving digital
humanities.

Cognition refers to the mental processes involved with gaining knowledge, including those
involved in producing and understanding language. To cite an entry on embodied cognition
from another excellent open-access scholarly encyclopedia, the Stanford Encyclopedia of
Philosophy, we can see cognition as embodied when aspects of an individuals body beyond
the brain play a signi:cant constitutive role in ones ability to intake, process, and develop an
understanding of new knowledge. New research in the cognitive sciences provides further
support for the importance of embodiment in conceptual learning. As philosopher of mind
Alva No (2006) described in summarizing this work, the sense of touch, not vision, should
be our model for thinking about perception; we acquire new content through active inquiry
and exploration. (This is not a new idea of course, Vichean philosophy see Vico, 1725
drew on the work of Aristotle to argue that men can only know what they make.) The
understanding that knowledge is an action something we do in concert with material
objects, bodies, and environments is also supported by much qualitative research in my
home :eld of the sociology of the arts (cf: Acord and DeNora, 2008; Sutherland and Acord,
2007).

Seen from this vantage point, aesthetic computing is a move that :ts into a broader
theoretical paradigm interested in exploring the non- and quasi-cognitive aspects of behavior,
knowledge-production, and interaction, as well as the important roles played by materials,
technologies, and objects in the worlds we make. As MIT social scientist Sherry Turkle
describes in her 2007 edited volume, Evocative Objects, the physical objects in our lives are
anchors of our memories, thoughts, and action; how we interact with them demonstrates
that thought and feeling are linked. Similarly, music sociologist Tia DeNora (2000) pointed
out that aesthetic materials, like the songs we hear, are accomplices in our everyday lives;
they allow us to undertake tasks that we could not accomplish without them. (Any Zumba
instructor will be familiar with this power of music.) Even earlier studies in science and
mathematics support this point. Looking at how shoppers in the grocery store use
mathematics, Jean Lave (1988) demonstrated that cognition is an interactive process
between persons acting and the settings in which their activity is constituted. This has also
been discovered in a range or professions, including: design engineers (Henderson, 1999),
cookie manufacturers (Streeck, 1996), and ship navigators (Hutchins, 1995). As Fishwick
rightly points out, incorporating aesthetic encounters into learning software design reunites
the mind and body of the computer scientist such that the physical coding (or serious
gaming) experience can build understanding of more abstract analytic concepts. Creating
embedded virtual experiences for learning code, or otherwise bringing real world bodily
metaphors into software design, is signi:cant in the student interaction with formal
language. The resources we have at our hands with which to make meaning inluence what
we can know.

In this chapter, Fishwick identi:es the challenge of aesthetic computing as connecting

humans with computers. In an evaluation of one of his University of Florida aesthetic
computing courses, Fishwick, et al. (2005) discovered that many of the undergraduate
students felt that aesthetic computing was time consuming, but particularly useful in
explaining computing concepts to non-engineers. In digging more deeply into the theory
behind aesthetic computing, however, I posit that there are additional important
opportunities to connect humans and computers here to advance knowledge in computer
science and also the humanities disciplines.

As sociologists of science show, producing scienti:c knowledge requires moving from dirty,
fuzzy, and hands-on experiences to abstract and codi:ed representations (cf: Latour and
Woolgar, 1979). As a result, scienti:c results and :ndings are translations of our human
experiences that may distort what it is that we really know. Ong (1982) makes a similar
argument about the technology of written human languages: learning a written language
entails a transformation of consciousness; we begin thinking with words, rather than
speaking our thoughts. While Fishwick cites Mark Johnson and George Lakoffs good work
to show that our embodied experiences are present in language through metaphor, Ong also
demonstrates that written languages risk eliminating processes of embodiment by
positioning words and written language notations as arti:cial mediators of what we know
and how we can express it. How can we exit language to study language concepts?

I see aesthetic computing as offering an opportunity for computer scientists (and their
students) to engage differently with software design by creating a new and embodied
experience to play (cf: Huizinga, 1944) with the concepts upon which formal language
notation is built. By side-stepping formal language notation as a mediator of our knowledge
of mathematical relations, and engaging with the body as a different kind of mediator,
aesthetic computing may enable new ways of thinking about software design. Briely, let us
consider the case of art as an aesthetic activity, in which art-making or participation is a way
to externalize and relect upon felt experiences in order to grasp (but also to extend)
linguistically mediated situations. Art can be a place to work through alternative
constructions and implementations of our understandings; in this way art may enable
healing, conlict resolution, and social movements (Acord and DeNora, 2008). Similarly, as
education scholar Donald Schn (1987) observes of architecture students, engaging in the
hands-on making and interrogation of more abstract architecture concepts through building
physical models creates opportunities for the on-the-spot experimentation, problem-solving,
and tinkering that he terms relection-in-action. Importantly, this embodied tinkering can
create opportunities for questioning and altering language design concepts that may result
in amendments to formal language systems. (The hand may feel something differently than
how the mind classi:ed it.) In this chapter, Fishwick observes, as we break open the lid of
the black box containing the atomic elements of normally hidden data, formulas, code, and
models, we :nd that computing is theatre all the way down. If computing is theatre, the trick
is to treat it as such: a place to engage in embodied play to not simply repeat well-rehearsed
formal language concepts but also to tinker with or improve upon them. This potential for
improvisation is, to me, one of the most exciting potentials of aesthetic computing.

Finally, Fishwick describes aesthetic computing as a terrain to bring together computer

scientists, artists, and humanities scholars to extend our models of meaning-making in the
digital age. Earlier in the chapter, however, Fishwick draws on the work of Sherry Turkle and
the example of the digital watch to discuss how deep abstract concepts in computing (as a
formal language) affect the thinking of users. The idea that formal language shapes (and, by
inference, limits) how we think is particularly frightening to some scholars in the humanities
who would like to use computing technology to expand interpretive possibilities (not delimit
them). (As Jaron Lanier, 2010, notes: we should use gadgets, not the other way around.)
Johanna Drucker, in particular, has written about the tensions between formal languages
based upon mathematics and their use with images and digital humanities data that
question formalized and established regimes of meaning (cf: Drucker, 2001, 2009, 2011).
Aesthetic computing, which enables questioning and play with formal language concepts
themselves (not simply their written notation as code), may well be the solution to creating
genuinely new and open-ended interpretive interfaces. As Dexter, et al. (2011) argue in a
recent special issue in Culture Machine on the digital humanities, The functional role of
aesthetics plays out most richly when contextualized as part of the creative process of
software development (pp. 16-17). In other words, opportunities to transform interactive
computing environments are enhanced when programmers see themselves as embodied. I
posit, then, that to advance the digital humanities we must not only think about code
aesthetically or symbolically, as does the research area of critical code studies, but take a
step further and think differently about the mental/physical constructions upon which coding
is based. In this way, aesthetic computing provides the physical and bodily tools to
dramatically rethink possibilities for cultural computing in the digital age.

26.18.1 References
Acord, Sophia Krzys and DeNora, Tia (2008). Culture and the arts: From art worlds to
arts-in-action. The Annals of the American Academy of Political and Social Science,
619(1): 223-237.
Crawford, Matthew (2009) Shop Class as Soul Craft: An Inquiry into the Value of Work.
New York: The Penguin Press.
DeNora, Tia (2000) Music in Everyday Life. Cambridge: Cambridge University Press.
Dexter, Scott, Melissa Dolese, Angelika Seidel, and Aaron Kozbelt (2011) On the
embodied aesthetics of code. Culture Machine, 12.
Drucker, Johanna (2001) Digital ontologies: The ideality of form in/and code storage: Or:
Can graphesis challenge mathesis? Leonardo, 34(2): 141-145.
--- (2009) SPECLAB: Digital Aesthetics and Projects in Speculative Computing. Chicago,
London: University of Chicago Press.
--- (2011) Humanities approaches to interface theory. Culture Machine, North America,
12.
Fishwick, Paul, Timothy Davis, and Jane Douglas (2005) Model representation with
aesthetic computing: Method and empirical study. ACM Transactions on Modeling and
Computer Simulation, 15(3): 254279.
Henderson, Kathryn (1999) On Line and On Paper: Visual Representations, Visual
Culture, and
Computer Graphics in Design Engineering. Cambridge, MA: The MIT Press.
Huizinga, Johan (1944) Homo ludens: A study of the play element in culture. Translated
by G. Steiner, 1970. London: Paladin.
Hutchins, Edwin (1995) Cognition in the Wild. Cambridge, MA: The MIT Press.
Lanier, Jaron (2010) You Are Not a Gadget: A Manifesto. New York: Alfred A. Knopf.
Latour, Bruno and Steve Woolgar (1979) Laboratory Life: The Construction of Scienti:c
Facts. Beverly Hills: SAGE Publications.
Lave, Jean (1988) Cognition in Practice: Mind, Mathematics, and Culture in Everyday
Life. Cambridge: Cambridge University Press.
No, Alva (2006) Action in Perception. Cambridge, MA: The MIT Press.
Ong, Walter J. (1982) Orality and Literacy: The Technologizing of the Word. New York:
Methuen.
Schn, Donald A. (1987) Educating the Relective Practitioner. San Francisco: Jossey-
Bass Publishers.
Streeck, Jrgen (1996) How to do things with things: Objects trouvs and symbolization.
Human Studies, 19: 365-384.
Sudnow, David (1978) Ways of the Hand: The Organization of Improvised Conduct.
Cambridge, MA: Harvard University Press.
Sutherland, Ian and Sophia Krzys Acord (2007). Thinking with art: From situated
knowledge to experiential knowing. The Journal of Visual Art Practice, 6(2): 125-140.
Turkle, Sherry (ed.) (2007) Evocative Objects: Things We Think With. Cambridge, MA: The
MIT Press.
Vico, Giambattista. (1725) Scienza Nuova. (The First New Science, edited and translated
 by Leon Pompa, Cambridge: Cambridge University Press, 2002.)

26.19 Commentary by Kang Zhang

Fishwicks edited book published in 2006 had formally introduced the concept of Aesthetic
Computing [1]. Based on the then de:nition, aesthetic computing addresses the question
"how can the theory and techniques in the traditional visual art help beautify modern
technology outputs and products and enhance their usability?". It includes the aesthetic
design of computer algorithms, simulation, visualization [4], human-machine interfaces, and
high-tech products, so that users are highly engaged and thus usability is enhanced. An
interesting example of aesthetic computing is to apply Kandinskys aesthetics to Java
programming [3]. Malina [2] highlighted the aesthetic computing activities published in the
Leonardo journal over the last forty years. Fishwicks new chapter on aesthetic computing
takes a more speci:c and operational view on aesthetic computing, focusing on the notion
of embodied formal language. In analogy to artists who were considered to be the :rst
virtual reality creators, computer scientists may interpret objects dynamically via bodily
simulations, possibly within virtual reality environments.

A related new discipline, but conceptually in reverse direction, is computational aesthetics

that aims at answering the question "how can the computer automatically generate various
forms of visually aesthetic expressions?" [6]. In other words, computational aesthetics
investigates how the modern technology helps arts. The technology serves to create tools
that can enhance the expressive power of visual art and heighten human understanding of
aesthetic evaluation, perception, and meaning.

The viewpoints and concepts expressed in this chapter, some of which bear deep roots in
arts, science and technology, are stimulating and of great interest to anyone who is
interested in both computing (or mathematics) and art. Technology has advanced to such a
level that art and design become increasingly important and relevant to science and
technology, whereas the importance of our technical knowledge is decreasing. This trend will
continue and the argument is indirectly supported by a recent discovery that the Internet and
search technology are changing our brain and also how we think, since we no longer need to
memorize, just need to know how to search to :nd information we need.

1. P. Fishwick (Ed.) Aesthetic Computing, MIT Press, Cambridge, 2006.

2. R.F. Malina, A Forty-Year Perspective on Aesthetic Computing in the Leonardo Journal,
in: P. Fishwick (Ed.) Aesthetic Computing, MIT Press, 2006, 43-52.
3. C.B. Price, From Kandinsky to Java (The Use of 20th Century Abstract Art in Learning
Programming), ITALICS, Vo.6, No.4, October 2007, 35-50.
4. K. Zhang, From Abstract Painting to Information Visualization, IEEE Computer Graphics
 and Applications, May/June 2007, 12-16.
5. K. Zhang, Book Review: Aesthetic Computing edited by Paul Fishwick, MIT Press, 2006,
ISBN 0-262-06250-X, Journal of Visual Languages and Computing, Vol.18, No.6,
December 2007, Elsevier Science Inc., New York, 613-616.
6. K. Zhang, S. Harrell, and X. Ji, Computational Aesthetics - On Complexity of Computer-
Generated Paintings, Leonardo Journal, MIT Press, 2012, Vol.45, No.3 (in press).

26.20 Commentary by David J. Therriault

In his chapter on Aesthetic Computing, Paul Fishwick has created a very accessible and
compelling argument for applying embodiment to human-computer interfaces. Fishwicks
case is built by examining the different levels of presence in embodied examples. Can you
imagine yourself in the landscape of a painting, or virtually manipulating objects while doing
math, or becoming a character in a textually described world? Indeed you can, and more
importantly there may be learning bene:ts to doing so. Fishwick provides rich examples of
the use of embodiment in software (e.g., a steampunk obesity machine crafted from barrels
of water to explain dynamic systems) that provide us with a glimpse into the potential of
aesthetic computing. In the balance of this commentary, I briely discuss the cognitive
history and my own experiences researching embodiment, and argue that Fishwicks insights
provides readers with a true glimpse into the future of programing.

Traditional views of cognitive psychology assume that information processing makes use of
abstract symbols. Since the 1950s, the manipulation of abstract/amodal symbols has
become the cornerstone of theories examining memory, reading, and thinking. Let us take as
an example the psychology of reading. Kintschs (1974, 1998) inluential research on reading
comprehension was built from propositions (i.e., abstract idea units). Kintsch also provided
evidence for the psychological reality of propositions, largely shaping our current
understanding of what happens in our minds when we read. However, there was always a
slice of representation missing from our reading experiences (often referenced as the
symbol grounding problem)--a sense of presence or embodiment. How do we explain how
propositions acquire meaning, how do we truly experience what we have read? In many
ways, Fishwick is tackling the same symbol grounding problem (but within a programming
framework).

Embodiment most simply put is attempting to understanding mind through the experiences
and perceptions of our bodies (e.g., perceptual symbols, or grounded cognition).
Interestingly, research examining how the body can inluence our understanding has been
most prominent in discourse psychology, speci:cally text comprehension. Researchers such
as Barsalou (1999), Glenberg (1997), Lakoff and Johnson (1999), and Pecher & Zwaan
(2005) have all argued that an embodied approach to cognition may have advantages over
traditional views of mental representation.

My own introduction to embodiment came during the earlier 2000s working as a post doc
with Rolf Zwaan. It was a wonderful experience, one were we spent the bulk of our time
talking about how we would test the links between cognition and action, over good coffee or
even better scotch, and see these ideas come to life in the lab. Prior to starting my position,
other faculty members had cautioned me that researching grounded cognition was kooky
and that attention on the topic would soon pass.

Since that time, there have been a host of compelling demonstrations of the psychological
reality of embodiment. For example, we know that when listeners hear a story with their eyes
closed they move them as if viewing the story in the real world (Spivey, Richardson, Tyler,
&Young, 2000), that areas of the brain employed when doing a physical task are the same
used when reading about that task (Feldman & Narayanan, 2004), and that even our
judgments of morality can inluence our perception, such as the perceived level of light in a
room (Banerjee, Chatterjee, & Sinha, 2012). Our own research (Kaschak et al. (2002))
provided evidence that the perception of motion makes use of some of the same neural
machinery needed to understand a verbal description of motion.

Research exploring an embodied view of cognition continues to lourish but there is currently
no uni:ed theory. Most researchers strongly advocate for either a symbolic or embodied
view of cognition, but the :eld is moving away from this dichotomy. For example, Louwerse
(2007) argues for exploring the relative contribution of both symbols and embodiment.
Fishwicks chapter represents a truly novel approach to embodiment: applying it to create a
better understanding of such things as system dynamics, number sense, or programming.
This view is also refreshing, in that, embodied computing isnt intended to supplant the rich
symbols tradition in coding. But embodiment may positively augment how we interact with
computers in the future, and who wouldnt want that?

In sum, I :nd Fishwicks chapter a successful venture and one that has implications beyond
undergirding (forgive the pun) the study of aesthetic computing. Programming becomes a
more attractive domain for folks in my discipline to continue the study of mental
representation.

26.20.1 References
Banerjee, P., Chatterjee P., & Sinha, J. (2012). Is it light or dark? Recalling moral behavior
changes perception of brightness. Psychological Science, 23 (4), 407-409.
Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22,
577-660.
 Glenberg, A. M. (1997). What memory is for. Behavioral and Brain Sciences, 20, 1-55.
Kaschak, M. P., Madden, C. J., Therriault, D. J., Yaxley, R. H., Aveyard, M., Blanchard, A., &
Zwaan, R. A. (2005). Perception of Motion Affects Language Processing. Cognition,
94(3), B79-B89.
Kintsch, W. (1974). The representation of meaning in memory. Hillsdale, New Jersey:
Erlbaum
Kintsch, W. (1998). Comprehension: A paradigm for cognition. New York: Cambridge
University Press
Lakoff, G. & Johnson, M. (1999). Philosophy in the lesh: The embodied mind and its
challenge to Western thought. New York: Basic Books.
Louwerse, M. M. (2007). Symbolic or embodied representations: A case for symbol
interdependency. In T. Landauer, D. McNamara, S. Dennis, & W. Kinsch (Eds.). Handbook
of latent semantic analysis (pp. 107-120). Mahwah, NJ: Erlbaum.
Pecher, D. & Zwaan, R. A. (Eds.) (2005). Grounding Cognition: The role of perception and
action in memory, language, and thinking. New York: Cambridge University Press.
Spivey, M. J., Richardson, D. C., Tyler, M. J., & Young, E. E. (2000). Eye movements during
comprehension of spoken scene descriptions. Proceedings of the Twenty-second
Annual Meeting of the Cognitive Science Society, (pp. 487-492), Erlbaum: Mawhah, NJ.

Topics in this book chapter:

Aesthetics User Experience (UX) Design Human-Computer Interaction (HCI)

Design History Visual Representation

Semiotics Computer Supported Cooperative Work

Learnt something new? Share with your friends:

# $ % + '

26.17 References
The Encyclopedia of Human-Computer Interaction, 2nd Ed.

BY JONAS LOWGREN, JOHN M. CARROLL, MARC HASSENZAHL, THOMAS ERICKSON, ALAN BLACKWELL, KEES
OVERBEEKE, CAROLINE HUMMELS, ROBERT SPENCE, MARK APPERLEY, KAREN HOLTZBLATT, HUGH R. BEYER, JESPER
KJELDSKOV, MARGARET M. BURNETT, CHRISTOPHER SCAFFIDI, DAG SVANAES, KRISTINA HOOK, ALISTAIR G. SUTCLIFFE,
ALBRECHT SCHMIDT, GILBERT COCKTON, VICTOR KAPTELININ, CLAYTON M. CHRISTENSEN, ERIC VON HIPPEL, NOAM
TRACTINSKY, BEN CHALLIS, RICHARD SHUSTERMAN, WILLIAM HUDSON, STEVE MANN, BRIAN WHITWORTH, ADNAN

READ OFFLINE

PDF version

TABLE OF CONTENTS
View chapter TOC

AUTHOR(S)

Paul A. Fishwick

Paul Fishwick is Distinguished University Chair of Arts and Technology (ATEC), and Professor of Computer Science.
He has six years of industry experience as a systems analyst working at Newport News Shipbuilding and at NASA
Langley Research Center in Virginia. He was on the faculty at the University of Florida from 1986 to 2012, and was
Director of the Digital Arts and Sciences Programs. His PhD was in Computer and Information Science from the
University of Pennsylvania. Fishwick is active in modeling and simulation, as well as in the bridge areas spanning art,
science, and engineering. He pioneered the area of aesthetic computing, resulting in an MIT Press edited volume in
2006. He is a Fellow of the Society for Computer Simulation, served as General Chair of the Winter Simulation
Conference (WSC), was a WSC Titan Speaker in 2009, and has delivered over 16 keynote addresses at international
conferences. He is Chair of the Association for Computing Machinery (ACM) Special Interest Group in Simulation
View proVle
COMMENTARIES BY

Michael Kelly

I am a Professor of Philosophy at the University of North Carolina at Charlotte, and Editor-in-Chief of the Encyclopedia
of Aesthetics (Oxford UP, 1998). A new expanded, revised, second edition of the Encyclopedia is forthcoming, and will
include entries on aesthetic computing. I organized and participated in a panel discussion on information aesthetics
at SIGGRAPH in 2009. And I organized a two-day symposium on aesthetic computing at UNC Charlotte in 2010 (with
funding from the National Science Foundation). I also initiated a symposium on aesthetic computing and
neuroaesthetics at UC Berkeley while I was a fellow at the Arts Research Center on campus. My most recent
publication is A Hunger for Aesthetics: Enacting the Demands of Art (Columbia UP, 2012), which is a critique of the
anti-aesthetic stance in contemporary art theory and a defense of the relevance of aesthetics to moral-political art.
My earlier book, Iconoclasm in Aesthetics (Cambridge UP, 2003), is a critique of contemporary philosophical
View proVle

Roger Malina

Roger F. Malina is an art-science researcher, astronomer and editor. He is a Distinguished Professor of Arts and
Technology and Professor of Physics at the University of Texas, Dallas where he is developing an Art-Science R and D
and Experimental Publishing program. He is a Directeur de Recherche of the CNRS and former Director of the
Observatoire Astronomique de Marseille Provence at Aix-Marseille University. His scienti:c specialty is in space
instrumentation and big data problems; he was the Principal Investigator for the NASA Extreme Ultraviolet Explorer
Satellite at the University of California, Berkeley. He also has been involved for 25 years with the Leonardo
organization whose mission is to promote and make visible work that explores the interaction of the arts and
sciences and the arts and new technologies. Since 1982 he has been the Executive Editor of the Leonardo
Publications at MIT Press. More recently he has helped set up the Mediterranean Institute for Advanced Studies
View proVle

Sophia Krzys Acord

Sophia Acord is Associate Director of the Center for the Humanities and the Public Sphere at the University of Florida.
She holds Ph.D. and M.Res. degrees in Sociology from the University of Exeter (UK) and a B.A. from Swarthmore
College (USA). Acord's research draws on her background in the sociology of art and science and technology studies
in order to explore the impact of digital technologies on research and knowledge-making in different scholarly
disciplines. Dr. Acord is also a founding editor of the refereed open-access journal Music and Arts in Action (MAiA).

View proVle

Kang Zhang
Kang Zhang is Professor and Director of Visual Computing Lab, Department of Computer Science at the University of
Texas at Dallas. He is also a Board Director of Vital Art and Science Inc., USA. He holds a B.Eng. degree in Computer
Engineering from University of Electronic Science and Technology of China, a Ph.D. degree from University of Brighton,
UK, and an Executive MBA degree from the University of Texas at Dallas. Prior to joining UT-Dallas, he held various
academic positions in the UK, Australia, and China. Dr. Zhang's current research interests include information
visualization, visual languages, aesthetic computing, and managerial aesthetics; and has published over 180 papers
and 6 books in these areas. He is also accomplished artist, having won various awards. Dr Zhang is on the Editorial
Boards of Journal of Visual Languages and Computing, International Journal of Software Engineering and Knowledge
Engineering, and International Journal of Advanced Intelligence. His home page is at www.utdallas.edu/~kzhang
View proVle

David J. Therriault

Dr. Therriault is an Associate Professor in the School of Human Development and Organizational Studies in the
College of Education at the University of Florida. He received his undergraduate degree in psychology from the
University of New Hampshire and his M. A. and Ph.D. in cognitive psychology from the University of Illinois at Chicago.
He was formerly a Postdoctoral Fellow at Florida State University's Psychology Department working with Dr. Rolf
Zwaan from 2001-2004. Dr. Therriault's primary research interests include the representation of text in memory,
comprehending time and space in language, the link between attention and intelligence, the use of perceptual
symbols in language, creativity and problem solving, and educational issues related to these topics.

View proVle