Vous êtes sur la page 1sur 13

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/220605190

Computer Models of Creativity

Article  in  Ai Magazine · July 2009


DOI: 10.1609/aimag.v30i3.2254 · Source: DBLP

CITATIONS

178

1 author:

Margaret A. Boden
University of Sussex
115 PUBLICATIONS   3,155 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Margaret A. Boden on 25 September 2014.

The user has requested enhancement of the downloaded file.


Articles

Computer Models
of Creativity

Margaret A. Boden

C
n Creativity isn’t magical. It’s an aspect of nor-
mal human intelligence, not a special faculty
granted to a tiny elite. There are three forms: reativity is a marvel of the human mind, and an obvious
combinational, exploratory, and transforma- goal for AI workers. Indeed, the proposal that led to the famous
tional. All three can be modeled by AI—in some 1956 Dartmouth Summer School often remembered as the time
cases, with impressive results. AI techniques
of AI’s birth mentioned “creativity,” “invention,” and “discov-
underlie various types of computer art. Whether
computers could “really” be creative isn’t a sci- ery” as key aims for the newly named discipline (McCarthy et
entific question but a philosophical one, to al. 1955, 45, 49ff.). And, 50 years later, Herb Simon—in an e-
which there’s no clear answer. But we do have mail discussion between AAAI Fellows (quoted in Boden 2006,
the beginnings of a scientific understanding of 1101)—cited a paper on creativity (Simon 1995) in answer to
creativity.
the challenge of whether AI is a science, as opposed to “mere”
engineering.
But if its status as an AI goal is obvious, its credibility as a
potential AI achievement is not. Many people, including many
otherwise hard-headed scientists, doubt—or even deny out-
right—the possibility of a computer’s ever being creative.
Sometimes, such people are saying that, irrespective of its per-
formance (which might even match superlative human exam-
ples), no computer could “really” be creative: the creativity lies
entirely in the programmer. That’s a philosophical question that
needn’t detain us here (but see the final section, below).
More to the point, these people typically claim that a com-
puter simply could not generate apparently creative perform-
ance. That’s a factual claim—which, in effect, dismisses AI
research on creativity as a waste of time.
However, it is mistaken: AI scientists working in this area
aren’t doomed to disappointment. It doesn’t follow that they
will ever, in practice, be able to engineer a new Shakespeare or
a neo-Mozart (although the latter goal has been approached
more nearly than most people imagine). But lesser examples of
AI creativity already abound. And, crucially, they help us to
understand how human creativity is possible.

Copyright © 2009, Association for the Advancement of Artificial Intelligence. All rights reserved. ISSN 0738-4602 FALL 2009 23
Articles

What Is Creativity? for P-creativity. Examples discussed below include


drawing scientific generalizations that were first
Creativity can be defined as the ability to generate discovered centuries ago (Langley et al. 1987), or
novel, and valuable, ideas. Valuable, here, has generating music of a type composed by long-dead
many meanings: interesting, useful, beautiful, sim- musicians (Cope 2001, 2006)
ple, richly complex, and so on. Ideas covers many Even P-creativity in computers need not match
meanings too: not only ideas as such (concepts, all the previous achievements of human beings.
theories, interpretations, stories), but also artifacts Years ago, in the early days of AI, Seymour Papert
such as graphic images, sculptures, houses, and jet used to warn AI researchers, and their sceptical crit-
engines. Computer models have been designed to ics, against “the superhuman human fallacy.” That
generate ideas in all these areas and more (Boden is, we shouldn’t say that AI has failed simply
2004). because it can’t match the heights of human intel-
As for novel, that has two importantly different ligence. (After all, most of us can’t do that either.)
meanings: psychological and historical. A psycho- We should try to understand mundane thinking
logical novelty, or P-creative idea, is one that’s new first, and worry about the exceptional cases only
to the person who generated it. It doesn’t matter how much later. His warning applies to AI work on cre-
many times, if any, other people have had that ativity, too. If AI cannot simulate the rich creativi-
idea before. A historical novelty, or H-creative idea, ty of Shakespeare and Shostakovich, it doesn’t fol-
is one that is P-creative and has never occurred in low that it can teach us nothing about the sorts of
history before. processes that go on in human minds—including
So what we need to explain, here, is P-creativi-
theirs—when people think new thoughts.
ty—which includes H-creativity but also covers
more mundane examples. And our explanation
must fit with the fact that creativity isn’t a special Creativity without Magic
faculty, possessed only by a tiny Romantic elite.
Rather, it’s a feature of human intelligence in gen- How is creativity possible? In other words, how is
eral. Every time someone makes a witty remark, it possible for someone—or, for that matter, a com-
sings a new song to his or her sleepy baby, or even puter program—to produce new ideas?
appreciates the topical political cartoons in the dai- At first blush, this sounds like magic: literally,
ly newspaper, that person is relying on processes of producing something out of nothing. Stage magi-
creative thought that are available to all of us. cians seem to do that, when they show us rabbits
To be sure, some people seem to be better at it coming out of hats. But of course it’s not really
than others. Some newspaper cartoonists have an magic at all: members of the Magic Circle know
especially good eye, and brain, for the delectable how it’s done. In the case of creativity, the psy-
absurdities of our political masters. And a few peo- chologist—and the AI scientist—need to know
ple come up with highly valued H-creative ideas how it’s done if there’s to be any hope of modeling
over and over again. Alan Turing is one example it on a computer.
(he did revolutionary work in mathematics, com- If we look carefully at the many examples of
puter science, cryptography, and theoretical biolo- human creativity that surround us, we can see that
gy [Boden 2006, 3.v.b–d, 4.i–ii, 15.iv]). But some there are three different ways in which creativity
people are better at tennis, too. To understand how happens. Novel ideas may be produced by combi-
Wimbledon champions manage to do what they nation, by exploration, or by transformation
do, one must first understand how Jo Bloggs can (Boden 2004).
do what he does at the municipal tennis courts. P- Combinational creativity produces unfamiliar
creativity, whether historically novel or not, is combinations of familiar ideas, and it works by
therefore what we must focus on. making associations between ideas that were pre-
Computer models sometimes aim for, and even viously only indirectly linked. Examples include
achieve, H-creativity. For example, a quarter cen- many cases of poetic imagery, collage in visual art,
tury ago, an AI program designed a three-dimen- and mimicry of cuckoo song in a classical sym-
sional silicon chip that was awarded a patent— phony. Analogy is a form of combinational cre-
which requires that the invention must not be ativity that exploits shared conceptual structure
“obvious to a person skilled in the art” (Lenat and is widely used in science as well as art. (Think
1983). And the AARON program (mentioned of William Harvey’s description of the heart as a
below) that generates beautifully colored drawings pump, or of the Bohr-Rutherford solar system
is described by its human originator as a “world- model of the atom.)
class” colorist. So it’s presumably H-creative—and It is combinational creativity that is usually
it’s certainly capable of coming up with color mentioned in definitions of “creativity” and that
schemes that he himself admits he wouldn’t have (almost always) is studied by experimental psy-
had the courage to use. chologists specializing in creativity. But the other
Often, however, computer models aim merely two types are important too.

24 AI MAGAZINE
Articles

Exploratory creativity rests on some culturally transformation is exploration on a metalevel, so


accepted style of thinking, or “conceptual space.” that there’s no real distinction here [Wiggins
This may be a theory of chemical molecules, a style 2006]. However, in exploratory creativity none of
of painting or music, or a particular national cui- the initial rules of the search space are altered,
sine. The space is defined (and constrained) by a whereas in transformational creativity some are.
set of generative rules. Usually, these rules are We’ll see below, for example, that the style may be
largely, or even wholly, implicit. Every structure varied by GAs, that is, metarules that change oth-
produced by following them will fit the style con- er rules, while remaining unchanged themselves.)
cerned, just as any word string generated by Eng- But combinational creativity is not to be sneezed
lish syntax will be a gramatically acceptable Eng- at. Kurt Schwitters and Shakespeare are renowned
lish sentence. for their H-creative collages and poetic images,
(Style-defining rules should not be confused which depend not on stylistic transformations but
with the associative rules that underlie combina- on associative processes for their origination (and
tional creativity. It’s true that associative rules gen- their interpretation, too). Exploratory creativity,
erate—that is, produce—combinations. But they likewise, is worthy of respect—and even wonder.
do this in a very different way from grammarlike Indeed, the vast majority of what H-creative pro-
rules. It is the latter type that are normally called fessional artists and scientists do involves
“generative rules” by AI scientists.) exploratory, not transformational, creativity. Even
In exploratory creativity, the person moves Mozart and Crick and Watson spent most of their
through the space, exploring it to find out what’s time exploring the spaces created by their (rela-
there (including previously unvisited locations)— tively rare) moments of transformation.
and, in the most interesting cases, to discover both Despite what’s been said above, it must also be
the potential and the limits of the space in ques- said that there’s no clear-cut distinction between
tion. These are the “most interesting” cases exploratory and transformational creativity. That’s
because they may lead on to the third form of cre- because any rule change, however trivial, will
ativity, which can be the most surprising of all. result in structures that weren’t possible before. So
In transformational creativity, the space or style one must decide whether to count superficial
itself is transformed by altering (or dropping) one “tweaking” as part of exploration. Since even the
or more of its defining dimensions. As a result, average Sunday painter may make slight changes
ideas can now be generated that simply could not to the style they’ve been taught, it’s probably best
have been generated before the change. For to do so. And one will still have to judge, in any
instance, if all organic molecules are basically given case, whether the stylistic change is superfi-
strings of carbon atoms, then benzene can’t be a cial or fundamental.
ring structure. In suggesting that this is indeed But if creativity isn’t magic, it’s not immediately
what benzene is, the chemist Friedrich von Kekule obvious that it could be achieved or modeled by
had to transform the constraint string (open curve) the particular types of nonmagic offered by AI. Nor
into that of ring (closed curve). This stylistic trans- is it immediately clear which of the three forms of
formation made way for the entire space of aro- human creativity would be the easiest for AI work
matic chemistry, which chemists would explore to emulate, and which the most difficult.
[sic] for many years.
The more stylistically fundamental the altered
constraint, the more surprising—even shocking—
Computer Combinations
the new ideas will be. It may take many years for That last question has a surprising answer. Con-
people to grow accustomed to the new space and trary to what most people assume, the creativity
to become adept at producing or recognizing the that’s most difficult for AI to model is the combi-
ideas that it makes possible. The history of science, national type. Admittedly, there’s no problem get-
and of art too, offers many sad examples of people ting a computer to make novel combinations of
ignored, even despised, in their lifetimes whose familiar (already stored) concepts. That can be
ideas were later recognized as hugely valuable. done until kingdom come. The problem, rather, is
(Think of Ignaz Semmelweiss and Vincent van in getting the computer to generate and prune
Gogh, for instance. The one was reviled for saying these combinations in such a way that most, or
that puerperal fever could be prevented if doctors even many, of them are interesting—that is, valu-
washed their hands, and went mad as a result; the able. What’s missing, as compared with the human
other sold only one painting in his lifetime.) mind, is the rich store of world knowledge (includ-
Transformational creativity is the “sexiest” of ing cultural knowledge) that’s often involved.
the three types, because it can give rise to ideas Certainly, AI programs can make fruitful new
that are not only new but fundamentally different combinations within a tightly constrained con-
from any that went before. As such, they are often text. For instance, a program designed to solve
highly counterintuitive. (It’s sometimes said that alphabetical analogy problems, of the form If ABC

FALL 2009 25
Articles

goes to ABD, what does MRRJJJ go to? generates com- ition” to be spelled out. Think what world knowl-
binations that are valuable (that is, acceptable to edge would be needed in the database even to rec-
us as answers) and sometimes surprising, and it can ognize the aptness of these combinations, never
even judge one possible answer to be “better” than mind originating them as Shakespeare did.
another (Hofstadter 2002). In this case, for Of course, AI workers could cheat. They could
instance, its answers included MRRJJD, MRRDDD, build a toy system that knew only about worry-
MRRKKK, and MRRJJJJ; and when given XYZ induced sleeplessness, knitting, sleeves, ointment,
instead of MRRJJJ, it sometimes gave the surpris- and dining customs and that—provided also with
ingly elegant WYZ. Again, the chess program Deep the relevant combinational processes—could deci-
Blue came up with novel combinations of moves— pher Shakespeare’s meaning accordingly. And even
in one case, so seemingly uncomputerlike that that would be an achievement. But it wouldn’t
world champion Kasparov accused the computer match the power of human minds to cope with the
scientists of cheating. And programs with chemical huge range of creative combinations that can assail
knowledge can come up with valuable new sug- us in a single day.
gestions for molecules that might have pharma- That power rests in the fact that our memories
ceutical uses. store (direct and indirect) associations of many dif-
But no current AI system has access to the rich ferent kinds, which are naturally aroused during
and subtly structured stock of concepts that any everyday thinking. Shakespeare seemingly had
normal adult human being has built up over a life- access to more associations than the rest of us or to
time. A few systems already have access to a sig- subtler criteria for judging the value of relatively
nificant range of concepts and factual knowledge, indirect associations. But our ability to understand
stored in databases such as Wordnet, Wikipedia, his poetry rests in this mundane fact about human
the CYC encyclopedia, and Google. And future memory. Moreover, this fact is biological (psycho-
programs may also have increased associative and logical), not cultural. The specific associations are
inferential powers, based on the ontology of the learned within a culture, of course—as are the
semantic web. But using huge databases sensibly, socially accepted styles of thought that ground
and aptly, so as to match the combinations gener- exploratory and transformational creativity. But
ated by linguistically—and culturally—sensitive the making of associations doesn’t have to be
human beings is a tall order. Not impossible in learned: it’s a natural feature of associative memo-
principle (after all, we don’t do it by magic), but ry. That’s why combinational creativity is the easi-
extremely difficult to achieve. est of the three types for human beings to achieve.
Consider, for instance, an example of H-creative One of the best current computer models of
combinational creativity from Shakespeare. Mac- combinational creativity is the joke-generating sys-
beth, sleepless because tormented by guilt, says tem JAPE (Binsted, Pain, and Ritchie 1997). One
this: might call it a “toy” system when compared with
the human mind, but it’s more impressive than the
Sleep that knits up the ravelled sleeve of care,
imaginary system just described.
The death of each day’s life, sore labour’s bath,
Balm of hurt minds, great nature’s second course, JAPE’s jokes are based on combination but
Chief nourisher in life’s feast. involve strict rules of structure too. They are pun-
ning riddles, of a type familiar to every eight year
Describing sleep as someone knitting is certainly
old. For example:
an unfamiliar combination of two boringly famil-
What do you call a depressed train? A low-como-
iar ideas. And, assuming that Shakespeare didn’t
tive.
borrow this poetic imagery (like many of his plots)
from some previous writer, such as Petrarch, it’s H- What do you call a strange market? A bizarre bazaar.
creative too. But why is it apt? What kind of murderer has fibre? A cereal killer.
Well, the knitter imagined here is not producing What’s the difference between leaves and a car?
a new garment (the most obvious function of knit- One you brush and rake, the other you rush and
ting), but mending (“knitting up”) a torn sleeve. brake.
And sleep, likewise, mends the worried mind. Sim- Hilarious, these are not. But they’re good enough
ilarly, a bath (line 2) or balming ointment (line 3) for a Christmas cracker.
can cure the soreness caused by one’s daily work. Those four riddles, along with many more, were
Moreover, in Shakespeare’s time, as in ours, the created by JAPE. The program is provided with
second course of a meal (line 3) was the most nour- some relatively simple rules for composing nine
ishing, the one best suited for replenishing the different types of joke. Its joke schemas include:
body. In short, sleep provides desperately needed What kind of x has y? What kind of x can y? What
relief. do you get when you cross x with y?; and What’s
It’s because we know all these things that we can the difference between an x and a y?
intuitively understand and appreciate Shake- The joke-generating rules are only “relatively”
speare’s text. But a computer model needs its “intu- simple—and much less simple than most people

26 AI MAGAZINE
Articles

would expect. That’s par for the course: AI has trying to identify different styles—and they aim
repeatedly shown us unimagined subtleties in our merely for verbal description, not computer imple-
psychological capacities. Think for a moment of mentation. Anyone trying to model exploratory
the complexity involved in your understanding creativity requires not only advanced AI skills but
the jest (above) about the cereal killer, and the also expertise in, and deep insights into, the
rather different complexities involved in getting domain concerned.
the point of the low-comotive or the bizarre bazaar. Despite the difficulties, there has been much
Sounds and spellings, for instance, are crucial for greater success here than in modeling combina-
all three. So making (and appreciating) these rid- tional creativity. In many exploratory models, the
dles requires you to have an associative memory computer comes up with results that are compara-
that stores a wide range of words—not just their ble to those of highly competent, sometimes even
meanings, but also their sounds, spelling, syllabic superlative, human professionals.
structure, and grammatical class. Examples could be cited from, for instance,
JAPE is therefore provided with a semantic net- stereochemistry (Buchanan, Sutherland, Feigen-
work of over 30,000 units, within which new—and baum 1969), physics (Langley et al. 1987, Zytkow
apt—combinations can be made by following 1997), music (Cope 2001, 2006), architecture (Kon-
some subset of the links provided. The network is ing and Eizenberg 1981, Hersey and Freedman
an extended version of WordNet, a resource devel- 1992), and visual art. In the latter category, a good
oped by George Miller’s team at Princeton Univer- example is Harold Cohen’s program, AARON
sity and now exploited in many AI programs. (Cohen 1995, 2002).
WordNet is a lexicon whose words are linked by AARON’s creations have not been confined to
semantic relations such as superordinate, subordi- the laboratory. On the contrary, they have been
nate, part, synonym, and antonym. Dimensions cod- exhibited at major art galleries around the world—
ing spelling, phonology, syntax, and syllable-count and not just for their shock value. Under develop-
had to be added to WordNet by JAPE’s programmer ment since the late-1960s, this program has gener-
so that the program could do its work, for JAPE ated increasingly realistic (though not
uses different combinations of these five aspects of photo-realistic) line drawings, followed by colored
words, in distinctly structured ways, when gener- images. The latter have included paintings, where-
ating each type of joke. in the paint is applied by AARON to its own draw-
It wasn’t enough merely to provide the five ings, using paint brushes (or, more accurately,
dimensions: in addition, rules had to be given to rounded paint pads) of half a dozen different sizes.
enable JAPE to locate appropriate items. That is, Most recently, AARON’s colored images have been
the rules had to define what is appropriate (valu- subtly multicolored prints.
able) for each joke schema. Clearly, an associative It’s especially interesting to note Cohen’s recent
process that obeys such constraints is very differ- remark, “I am a first-class colorist. But AARON is a
ent from merely pulling random combinations out world-class colorist.” In other words, the latest ver-
of the semantic network. sion of AARON outstrips its programmer—much as
The prime reason that JAPE’s jokes aren’t hilari- Arthur Samuel’s checkers player, way back in the
ous is that its associations are very limited, and 1950s, learned how to beat Samuel himself
also rather boring, when compared with ours. But, (Samuel 1959).
to avoid the superhuman human fallacy, we This example shows how misleading it is to say,
shouldn’t forget that many human-generated jokes as people often do, “Computers can’t do anything
aren’t very funny either. Its success is due to the creative, because they can do only what the pro-
fact that its joke templates and generative schemas gram tells them to do.” Certainly, a computer can
are relatively simple. Many real-life jokes are much do only what its program enables it to do. But if its
more complex. Moreover, they often depend on programmer could explicitly tell it what to do,
highly specific, and sometimes fleetingly topical, there’d be no bugs—and no “world-class” color
cultural knowledge—such as what the prime min- prints from AARON surpassing the handmade pro-
ister is reported to have said to the foreign secre- ductions of Cohen himself.
tary yesterday. In short, we’re faced with the A scientific example—or, better, a varied group
“Shakespeare’s sleep” problem yet again. of scientific examples—of exploratory creativity
can be found in the work of Pat Langley and
Simon’s group at CMU (Langley et al. 1987,
Computer Exploration Zytkow 1997). This still-burgeoning set of pro-
Exploratory creativity, too, can be modeled by AI— grams is the BACON family, a dynasty, including
provided that the rules of the relevant thinking close relations and more distant descendants, that
style can be specified clearly enough to be put into has been under development since the mid-1970s
a computer program. Usually, that’s no easy mat- (Boden 2004, 208–222). And it is this body of
ter. Musicologists and art historians spend lifetimes research on which Simon was relying when he

FALL 2009 27
Articles

© Franck Boston

defended AI’s status as a science (see previous text). tion, and “discovered” the ideal gas laws (PV / t =
Among the early achievements of the BACON k).
family were the “discovery” of several important The words discovery and discovered need to be in
laws of classical physics (Boyle’s law, Ohm’s law, scare quotes here because this was P-creativity
Snell’s law, Black’s law, and Kepler’s third law) and rather than H-creativity. Although a few results
some basic principles of chemistry (an acid plus an were historically new (for example, a version of
alkali gives a salt; molecules are composed of iden- Black’s law that is more general than the original
tifiable elements, present in specific proportions; one), most were not.
and the distinction between atomic and molecular Later members of this set of programs were
weight). These generalizations were gleaned from aimed at genuine discovery, or H-creativity. Some,
the experimental data used by human scientists for instance, suggested new experiments, intended
hundreds of years ago (and recorded in their note- to provide new sets of correlations, new observa-
books); initially, the data were cleaned up for tions, with which the program could then work
BACON.1’s benefit, but they were later provided in when testing a theory. Others could introduce new
the original noisy form. The BACON suite also basic units of measurement, by taking one object
reinvented the Kelvin temperature scale, by adding as the standard (human scientists often choose
a constant of 273 to the Celsius value in the equa- water). And Simon foresaw a future in which pro-

28 AI MAGAZINE
Articles

grams modeling scientific creativity could read changing itself. That is, it may contain genetic algo-
papers in the scientific journals, so as to find extra rithms, or GAs (see Boden 2006, 15.vi).
experimental data, and hypotheses, for them- GAs can make random changes in the program’s
selves. (To some extent, that future has already own task-oriented rules. These changes are similar
come: some bioinformatics software, such as for to the point mutations and crossovers that under-
predicting protein structure, can improve accuracy lie biological evolution. Many evolutionary pro-
by reading medical publications on the web. But grams also include a fitness function, which selects
the ideas in more discursive scientific papers are the best members of each new generation of task
less amenable to AI.) programs for use as “parents” in the next round of
There’s an obvious objection here, however. random rule changing. In the absence of an auto-
These programs, including the more recent ones, mated fitness function, the selection must be made
all assume a general theoretical framework that by a human being.
already exists. The physics-oriented BACON pro- Biological evolution is a hugely creative process,
grams, for instance, were primed to look for math- in which major transformations of bodily form
ematical relationships. Moreover, they were have occurred. This has happened as a result of
instructed to seek the simplest relationships first. many small changes, not of sudden saltations, and
Only if the system couldn’t find a numerical con- few if any of those individual changes count as
stant or linear relationship (expressible by a transformations in the sense defined above. (Even
straight-line graph) would it look for a ratio or a small mutations can be damaging for a living
product. But one might say that the greatest cre- organism, and larger—transformational—ones are
ative achievement of the famous scientists mod- very likely to be lethal.) Nevertheless, over a vast
eled here, and of Galileo before them, was to see period of time, the evolutionary process has deliv-
that—or even to ask whether—some observable ered unimaginable changes.
events can be described in terms of mathematics at It’s not surprising, then, that the best prima facie
all, for this was the real breakthrough: not discov- examples of transformational AI involve evolu-
ering which mathematical patterns best describe tionary programming. For example, Karl Sims’s
the physical world, but asking whether any math- (1991) graphics program produces varied images
ematical patterns are out there to be found. (12 at a time) from which a human being—Sims, or
In other words, these programs were explorato- a visitor to his lab or exhibition space—selects one
ry rather than transformational. They were spoon- or two for breeding the next generation. (There’s
fed with the relevant questions, even though they no automatic fitness function because Sims doesn’t
found the answers for themselves. They have been know what visual or aesthetic properties to favor
roundly criticized as a result (Hofstadter and FARG over others.) This system often generates images
1995, 177–179; Collins 1989), because of the that differ radically from their predecessors, with
famous names (BACON and the like) used to label no visible family resemblance.
them. To be sure, they can explore creatively. (And, Sims’s program can do this because its GAs allow
as remarked above, exploration is what human not only small point mutations (leading to minor
chemists and physicists do for nearly all of their changes in color or form) but also mutations in
time.) However, the long-dead scientists whose dis- which (for instance) two whole image-generating
coveries were being emulated here did not merely programs are concatenated, or even nested one
explore physics and chemistry, but transformed inside the other. Since one of those previously
them. evolved programs may itself be nested, several
Could a computer ever do that? hierarchical levels can emerge. The result will be an
image of some considerable complexity. As an
analogy, consider these two trios of sentences:
Stylistic Transformations (1) The cat sat on the mat; The cats sat on the
Many people believe that no computer could ever mat; The dog sat on the porch, and (2) The cat sat
achieve transformational creativity. Given a style, on the mat; Aunt Flossie went into the hospital;
they may admit, a computer can explore it. But if The cat given to me by Aunt Flossie last Christmas
you want it to come up with a new style, don’t before she went into the hospital in the neighbor-
hold your breath! ing town sat on the mat. Clearly, the second trio
After all, they say, a computer does what its pro- displays much greater differences than the first.
gram tells it to do—and no more. The rules and So this program undeniably delivers transforma-
instructions specified in the program determine its tions: images that are fundamentally different
possible performance (including its responses to from their ancestors, sometimes even from their
input from the outside world), and there’s no parents. But whether it delivers transformed styles
going beyond them. as well as transformed items is less clear, for family
That thought is of course correct. But what it resemblance is the essence of style. When we speak
ignores is that the program may include rules for of styles in visual art (or chemistry, or cooking), we

FALL 2009 29
Articles

mean a general pattern of ideas/artifacts that is sus- ries can arise. Those for fashion and art are not. So
tained—indeed, explored—over time by the artist if structural transformation is necessary for a nov-
concerned, and perhaps by many other people too. el idea to be hailed as a triumph of style-changing
But Sims’s program cannot sustain a style, because creativity, it certainly isn’t sufficient.
some equivalent of Aunt Flossie’s trip to the hospi-
tal can complicate the previous image at any time.
In brief, Sims’s program is almost too transfor-
Is Transformational
mational. This lessens the importance of the selec- AI Actually Possible?
tor. Even an automatic fitness function would not
I said, above, that the best prima facie examples of
prevent highly unfit examples from emerging. And
transformational AI involve evolutionary pro-
when human selectors try to steer the system
gramming. Why that cautious “prima facie”?
toward certain colors or shapes, they are rapidly
Sims’s program, after all, does generate radically
disappointed: sooner rather than later, unwanted
transformed images. And Latham’s program gener-
features will appear. This is frustrating for anyone
ates new visual styles, even if the family resem-
seriously interested in the aesthetics of the evolv- blances to the ancestor styles are relatively obvi-
ing images. ous. Moreover, we don’t need to focus only on the
That’s why the sculptor William Latham, a pro- contentious area of art, nor only on cases where a
fessional artist rather than a computer scientist, human selector decides on “fitness.” Even a very
uses evolutionary programming in a less radically early GA program was able to evolve a sorting algo-
transformational way (Todd and Latham 1992). rithm that could put a random set of numbers into
His GAs allow only relatively minor changes to the an increasing series, or order words alphabetically
current image-generating program, such as altering (Hillis 1992). Since then, many other highly effi-
the value of a numerical parameter. Nesting and cient algorithms have been automatically evolved
concatenation are simply not allowed. The result from inferior, even random, beginnings. If that’s
is a series of images that, he admits, he could not not transformation, what is?
possibly have imagined for himself, but that nev- Well, the objection here comes from people who
ertheless carry the stamp of his own artistic style. take the biological inspiration for evolutionary
The transformations, in other words, are relatively programming seriously (Pattee [1985]; Cariani
minor and concern relatively superficial dimen- [1992]; see also Boden [2006, 15.vi.c]). They
sions of the original style (conceptual space). assume that AI is either pure simulation or abstract
It would be possible, no doubt, for an evolu- programming that defines what sort of interac-
tionary program to be allowed to make “Aunt tions can happen between program and world (as
Flossie” mutations only very rarely. In that case, in computer vision, for example). And they offer a
there would be a greater chance of its producing version of the familiar argument that A computer
transformed styles as well as transformed items. can do only what its program tells it to do. Specifical-
Indeed, the minor mutations might then be ly, they argue that genuine transformations can
regarded as exploring the existing style, whereas the arise in a system only if that system interacts pure-
nesting/concatenating mutations might be seen as ly physically with actual processes in the outside
transforming it. world, as biological organisms do.
Whether those stylistic transformations would Their favorite example concerns the origin of
be valued is another matter. By definition, a cre- new organs of perception. They allow that once a
ative transformation breaks some of the currently light sensor has arisen in biology, it can evolve into
accepted rules. It may therefore be rejected out of better and better sensors as a result of genetic
hand—as Semmelweiss and van Gogh knew only mutations that can be approximated in AI pro-
too well. But—as neither of them lived long grams. So an inefficient computer-vision system
enough to find out—even if it is rejected, it may be might, thanks to GAs, evolve into a better one. But
revived later. In biology, nonlethal mutations lead the first light sensor, they insist, can arise only if
to viable organisms, which then compete as natu- some mutation occurs that causes a bodily change
ral selection proceeds. In human thought, social that happens to make the organism sensitive to
selection takes the place of natural selection. So, light for the very first time. The light—considered
since being valuable is part of the very definition of as a physical process—was always out there in the
creative ideas, the identification of “creativity” is world, of course. But only now is it “present” for
not a purely scientific matter but requires socially the organism. One might say that only now has it
generated judgments. passed from the world into the environment.
Putatively creative ideas are evaluated by means That acceptance of light as part of the organ-
of a wide range of socially determined criteria. The ism’s environment depends crucially on physical
criteria for scientific evalutation are relatively processes—both in the world and in the living
straightforward, and also relatively stable—even body. And these processes, they say, have no place
though bitter disputes about new scientific theo- in AI.

30 AI MAGAZINE
Articles

They grant that the generative potential of a involved may be too great, and too various. If so,
computer program is often unpredictable and may doubt about (nonaccidental) genuine transforma-
even be indefinitely variable, as it is for most evo- tions in AI still stands. But that they can some-
lutionary programs. But still, it is constrained by times happen unexpectedly is clear.
the rules (including the GAs) in the program. And
if it interacts with events in the outside world, as a
computer-vision system or a process monitor does,
Computer Models
the types of data to which it is sensitive are preor- and Computer Art
dained. Certainly, they say, improved sensory arti-
All computer models of creativity are aimed at the
facts can result from evolutionary computing. And
production of P-creative ideas, and a few at H-cre-
those improvements may be so surprising that we
ativity too. And many are intended also to throw
naturally classify them as “transformations.” But
some light on creativity in human minds. Some,
(so the argument goes) no fundamentally new
however, function in ways that have no close rela-
capacities can possibly arise.
tion to how the the mind works: it’s enough that
For instance, if the physical parameters foreseen they generate creative outcomes.
by the programmer as potentially relevant don’t Examples of the latter type include most of the
happen to include light, then no artificial eye can AI programs employed in the various forms of
ever emerge. In general, then, there can be no real computer art. (The different types are distin-
transformations in AI. guished, and their varying implications for “cre-
That may be true of AI systems that are pure sim- ativity” outlined, in Boden and Edmonds [2009].)
ulations. But it’s demonstrably not true of all AI One might say that these aren’t really computer
systems—in particular, of some work in so-called “models” at all, but rather computer programs—
embodied AI, for recent research in this area has ones that may sometimes seem to work in creative
resulted in the evolution of a novel sensor: the very ways. (AARON was unusual in that Cohen—
thing that these critics claim can happen only in already a highly successful abstract painter—first
biology. turned to AI techniques in the hope of under-
In brief, a team at the University of Sussex were standing his own creativity better.) Most comput-
using a GA to evolve oscillator circuits—in hard- er artists are interested not in human psychology
ware, not in simulation (Bird and Layzell 2002). To but in the aesthetic value of their program’s per-
their amazement, they ended up with a primitive formance.
radio receiver. That is, the final (GA-selected) cir- That performance may be a stand-alone matter,
cuit acted as a primitive radio antenna (a “radio wherein the computer generates the result all by
wave sensor”), which picked up and modified the itself. Having written the program, the human
background signal emanating from a nearby PC artist then stands back, hands off, to let it run.
monitor. These are cases of generative art, or G-art (Boden
On investigation post hoc, it turned out that the and Edmonds 2009).
evolution of the radio-wave sensor had been driv- Where G-art is involved, it’s especially likely that
en by unforeseen physical parameters. One of the AI system itself—not just its human origina-
these was the aerial-like properties of all printed tor—will be credited with creativity. In evolutionary
circuit boards, which the team hadn’t previously art too (see the following text), much of the cre-
considered. But other key parameters were not ativity may be credited to the program, for here,
merely unforeseen but unforeseeable, for the oscil- the computer produces novel results—images or
latory behavior of the evolved circuit depended melodies, for instance—that the human artist
largely on accidental—and seemingly irrelevant— couldn’t predict, or even imagine. In yet other cas-
factors. These included spatial proximity to a PC es of computer art, such as the interactive art
monitor; the order in which the analog switches described below, some or all of the creativity is
had been set; and the fact that the soldering iron attributed to the programmer or the human par-
left on a nearby workbench happened to be ticipants. The interactive program isn’t designed to
plugged in at the mains. be (or even to appear to be) creative in its own
If the researchers had been aiming to evolve a right, but rather to produce aesthetically attrac-
radio receiver, they would never have considered tive/interesting results in noncreative ways.
switch order or soldering irons. Nor would either The preeminent case of G-art in the visual arts is
of these matters necessarily be relevant outside the AARON, whose programmer tweaks no knobs
specific (physical) situation in which this research while it is running. In music, perhaps the best-
was done. On another occasion, perhaps, arcane known example is the work of the composer David
physical properties of the paint on the surround- Cope (2001, 2006).
ing wallpaper might play a role. So we can’t be sure Cope’s program Emmy (from EMI: Experiments
that even research in embodied AI could confident- in Musical Intelligence) has composed music in
ly aim to evolve a new sensor. The contingencies the style of composers such as Bach, Beethoven,

FALL 2009 31
Articles

Chopin, Mahler … and Scott Joplin, too. Some are over, it moves in extraordinarily lifelike ways, in
pieces for solo instrument, such as a keyboard response to a variety of human movements and
fugue or sonata, while others are orchestral sym- sounds. If someone shouts, for instance, or sud-
phonies. They are remarkably compelling, striking denly pounds the table, the starfish instantly
many musically literate listeners—though admit- “freezes” as a frightened animal might do.
tedly not all—as far superior to mere pastiche. Interactive art isn’t wholly new: Mozart’s dice
That’s sometimes so even when the listener music is one ancient example. (Someone would
approached Emmy’s scores in a highly sceptical throw a die to decide the order in which to play
spirit. For instance, the cognitive scientist Douglas Mozart’s precomposed musical snippets, and the
Hofstadter, a leading figure in the computer mod- result would always be a coherent piece.) But
eling of creativity (Hofstadter and FARG 1995, because of the general-purpose nature of comput-
Rehling 2002), believed it to be impossible that tra- ing, a very wide range of types of interaction can
ditional AI techniques could compose music of be accommodated, many of which were previous-
human quality. But on playing through some ly unimaginable.
Emmy scores for new “Chopin mazurkas,” a genre In computer-based interactive art, the aesthetic
with which Hofstadter, a fine amateur musician, interest is not only, or not even primarily, in the
was already very familiar, he was forced to change intrinsic quality of the results (images and sounds).
his mind (Hofstadter 2001, 38f.). Rather, it is in the nature of the interaction between
Other examples of computer art are not stand- computer and human beings (Boden in press). The
alone, but interactive (Boden and Edmonds 2009; “audience” is seen as a participant in the creation of
Boden in press); that is, the computer’s perform- the artwork—especially if the causal relations
ance is continually affected by outside events between human activity and computer perform-
while the program is running. ance are direct and intelligible. In the latter case,
Those “outside events” may be impersonal mat- one can voluntarily shape the computer’s perform-
ters such as wave movements or weather condi- ance so as to fit one’s own preferences. But whether
tions, but usually they are the movements/actions the the relatively direct cases are more artisticially
of human beings. Given that what the system does interesting than the indirect ones is disputed:
on any occasion depends in part on the actions of there’s no consensus on just what type of interac-
the human audience, the causal relation may be tions are best from an aesthetic point of view.
obvious enough for the person to choose what As for evolutionary art, two pioneering exam-
effect to have on the program’s performance. ples (Sims and Latham) have been mentioned
Sometimes, however, such predictions are impos- already. Today, young computer artists are increas-
sible: the audience affects what happens but, per- ingly using evolutionary techniques in their work.
haps because of the complexity of the causality One main reason is the potential for surprise that
involved, they don’t know how. They may not this (randomness-based) approach provides.
even realize that this is happening at all—for Another is its connection with A-Life, and with life
instance, because of a built-in delay between itself. Some evolutionary artists even claim that
(human-generated) cause and (computer-generat- their work, or something like it, may one day gen-
ed) effect. erate “real” life in computers (Whitelaw 2004).
One interactive program, written by Ernest (They are mistaken, because computers—although
Edmonds, was chosen by the curators of a Wash- they use energy, and can even budget it—don’t
ington, D.C., art gallery to be run alongside the metabolize [Boden 1999].)
works of Mark Rothko, Clyfford Still, and Kenneth Additional types of computer art exist, which
Noland, in a 2007 exhibition celebrating the 50th can’t be discussed here. But there is a debilitating
anniversary of the “ColorField” painters. (So much occupational hazard that faces all who work in this
for the view that computer art can’t really be art— area, whichever subfield they focus on. Namely,
see below.) many members of the general public simply refuse
An interactive artwork commissioned for the point-blank to take their work seriously.
huge millennial exhibition filling London’s new- Consider Emmy, for instance. I said, above, that
built Millennium Dome was described by a Times Emmy composes music in the style of Bach and oth-
journalist as “the best bit of the entire dome.” This er composers. I should rather have said that it com-
was Richard Brown’s captivating Mimetic Starfish. posed such music, for in 2005, Cope destroyed the
The starfish is a purely virtual creature: a visual musical database that had taken him 25 years to
image generated by a self-equilibrating neural net- build and that stored musical features characteris-
work that’s attached to sensors in the vicinity. The tic of the composers concerned (Cope 2006, 364).
image is projected from the ceiling down onto a His reason was twofold. First, he had found over
marble table, and appears to onlookers to be a large the years that many people dismissed Emmy’s
multicolored starfish trapped inside it. But despite compositions (sometimes even refusing to hear
being “trapped” inside the table, it moves. More- them at all), failing to take them seriously because

32 AI MAGAZINE
Articles

of their nonhuman origin. Second, even those who The scientific questions offer more hope.
did appreciate Emmy’s scores tended to regard Enough advance has already been made in com-
them not as music but as computer output. As such, putational psychology and computer modeling to
they were seen as infinitely reproducible—and make it reasonable to expect a scientific explana-
devalued accordingly (Boden 2007). Now, howev- tion of creativity. Optimists might even say that it’s
er, Emmy has a finite oeuvre—as all human com- already on the horizon. This doesn’t mean that
posers, beset by mortality, do. we’ll ever be able to predict specific creative ideas,
Some of Emmy’s detractors would be equally any more than physicists can predict the move-
adamant in dismissing every other example of ments of a single grain of sand on a windswept
computer art. They might admit that the Mimetic beach. Because of the idiosyncracy and (largely
Starfish, for example, is both beautiful and—for a hidden) richness of individual human minds, we
while—intriguing. But they would regard it as a can’t even hope to explain all creative ideas post
decorative gimmick, not as art. For them, there can hoc. But, thanks in part to AI, we have already
be no such thing as computer art. Despite the fact begun to understand what sort of phenomenon cre-
that there is always a human artist somewhere in ativity is.
the background, the mediation of the computer in Still something of a mystery, perhaps. And cer-
generating what’s actually seen or heard under- tainly a marvel. But not—repeat, not—a miracle.
mines its status as art.
This isn’t the place to attempt a definition of the References
notoriously slippery concept of art. (But remember Binsted, K.; Pain, H.; and Ritchie, G. D. 1997. Children’s
that a computer artwork was chosen by tradition- Evaluation of Computer-Generated Punning Riddles.
al gallery curators for inclusion in Washington’s Pragmatics and Cognition 5(2): 305–354.
“ColorField” exhibition; see above.) In other Bird, J., and Layzell, P. 2002. The Evolved Radio and Its
words, it’s not the place for discussing whether Implications for Modelling the Evolution of Novel Sen-
sors. In Proceedings of Congress on Evolutionary Computa-
computer art is “really” art.
tion, CEC-2002, 1836–1841. Pisacataway, NJ: Institute of
But we can’t wholly ignore a closely related
Electrical and Electronics Engineers.
question, which is often in people’s minds when
Boden, M. A. In press. Aesthetics and Interactive Art. In
they deny the possibility of computer art. Namely,
Art, Body, Embodiment, ed. R. L. Chrisley, C. Makris, R. W.
can a computer “really” be creative? Clowes, and M. A. Boden. Cambridge: Cambridge Schol-
ars Press.
But Are Computers Boden, M. A. 2007. Authenticity and Computer Art. Spe-
cial Issue on Creativity, Computation, and Cognition, ed.
Creative, Really? P. Brown. Digital Creativity. 18(1): 3–10.
Whether a computer could ever be “really” creative Boden, M. A. 2006. Mind as Machine: A History of Cogni-
is not a scientific question but a philosophical one. tive Science. Oxford: Oxford University Press.
And it’s currently unanswerable, because it Boden, M. A. 2004. The Creative Mind: Myths and Mecha-
involves several highly contentious—and highly nisms, 2nd ed. London: Routledge.
unclear—philosophical questions. Boden, M. A. 1999, Is Metabolism Necessary? British Jour-
These include the nature of meaning, or inten- nal for the Philosophy of Science 50(2): 231–248.
tionality; whether a scientific theory of psycholo- Boden, M. A., and Edmonds, E. A. 2009. What Is Gener-
gy, or consciousness, is in principle possible; and ative Art? Digital Creativity 20(1–2): 21–46.
whether a computer could ever be accepted as part Buchanan, B. G.; Sutherland, G. L.; and Feigenbaum, E.
of the human moral community. Indeed, you can A. 1969, Heuristic DENDRAL: A Program for Generating
Explanatory Hypotheses in Organic Chemistry. In
ignore creativity here, for many philosophers argue
Machine Intelligence 4, ed. B. Meltzer and D. M. Michie,
that no naturalistic explanation of any of our psy- 209–254. Edinburgh: Edinburgh University Press.
chological capacities is possible, not even an expla-
Cariani, P. 1992. Emergence and Artificial Life. In Artificial
nation based in neuroscience. In short, the philo- Life II, ed. C. J. Langton, C. G. Taylor, J. D. Farmer, and S.
sophical respectability of “strong” AI, and of Rasmussen, 775–797. Reading, MA: Addison-Wesley.
cognitive science in general, is hotly disputed. Cohen, H. 1995. The Further Exploits of AARON Painter.
These are among the very deepest questions in In Constructions of the Mind: Artificial Intelligence and
philosophy. I’ve discussed them elsewhere (Boden the Humanities, ed. S. Franchi and G. Guzeldere. Special
2004, chapter 11; Boden 2006, chapter 16). I’ve edition of Stanford Humanities Review, 4(2): 141–160.
also argued that the ultrasceptical, postmodernist Cohen, H. 2002. A Million Millennial Medicis. In Explo-
view is irrational and fundamentally self-defeating rations in Art and Technology, ed. L. Candy and E. A.
(Boden 2006, 1.iii.b, 16.vi–viii). But there’s no Edmonds, 91–104. London: Springer.
knock-down refutation on either side. That being Collins, H. M. 1989. Computers and the Sociology of Sci-
so, even the youngest readers of AI Magazine entific Knowledge, Social Studies of Science, 19: 613–624.
shouldn’t expect to see these questions to be defin- Cope, D. 2006. Computer Models of Musical Creativity.
itively answered in their lifetimes. Cambridge, Mass.: The MIT Press.

FALL 2009 33
Articles

J. M. 1987. Scientific Discovery: Computational Explorations


of the Creative Process. Cambridge, MA: The MIT Press.

ACM IUI'10 Todd, S. C., and Latham, W. 1992. Evolutionary Art and
Computers. London: Academic Press.
Conference on Intelligent User Interfaces Lenat, D. B. 1983. The Role of Heuristics in Learning by
Discovery: Three Case Studies. In Machine Learning: An
Hong Kong, China; 7-10 February 2010 Artificial Intelligence Approach, ed. R. S. Michalski, J. G.
Carbonell, and T. M. Mitchell, 243–306. Palo Alto, CA:
Tioga.
IUI is where the Human-Computer Inter- McCarthy, J.; Minsky, M. L.; Rochester, N.; and Shannon,
action (HCI) community and the Artificial C. E. 1955. A Proposal for the Dartmouth Summer
Intelligence (AI) community meet. Research Project on Artificial Intelligence. AI Magazine
27(4): 12–14.
Conference Co-Chairs: Pattee, H. H. 1985. Universal Principles of Measurement
Charles Rich (Worcester Polytechnic Inst., USA); and Language Functions in Evolving Systems. In Com-
Qiang Yang (Hong Kong U. of Sci. & Tech., plexity, Language, and Life: Mathematical Approaches, ed. J.
China) Casti and A. Karlqvist, 168–281. Berlin: Springer-Verlag.
Rehling, J. A. 2002. Results in the Letter Spirit Project. In
Program Co-Chairs: Creativity, Cognition, and Knowledge: An Interaction, ed. T.
Marc Cavazza (U. Teesside, UK); Dartnall. London: Praeger, 273–282.
Michelle Zhou (IBM Research, China)
Samuel, A. L. 1959. Some Studies in Machine Learning
Using the Game of Checkers. IBM Journal of Research and
Development 3: 211–229.
Simon, H. A. 1995. Explaining the Ineffable: AI on the
For more information, the conference website is: Topics of Intuition, Insight, and Inspiration. In Proceed-
ings of the Fourteenth International Joint Conference on Arti-
http://www.iuiconf.org/ ficial Intelligence, vol. 1, 939–948. San Francisco: Morgan
Kaufmann Publishers.
(in cooperation with AAAI) Sims, K. 1991. Artificial Evolution for Computer Graph-
ics. Computer Graphics. 25(4): 319–28.
Whitelaw, M. 2004. Metacreation: Art and Artificial Life.
London: The MIT Press.
Wiggins, G. 2006. Searching for Computational Creativ-
ity. New Generation Computing 24(3): 209–222.
Cope, D. 2001. Virtual Music: Computer Synthesis of Musi- Zytkow, J. ed. 1997. Machine Discovery. London: Kluwer
cal Style. Cambridge, Mass.: The MIT Press. Academic.
Hersey, G., and Freedman, R. 1992. Possible Palladian Vil-
las Plus a Few Instructively Impossible Ones. Cambridge,
MA: The MIT Press.
Margaret A. Boden (research professor of cognitive sci-
Hillis, W. D. 1992. Co-Evolving Parasites Improve Simu-
ence at the University of Sussex) was the founding dean
lated Evolution as an Optimization Procedure. In Artifi-
of Sussex University’s School of Cognitive and Comput-
cial Life II, ed. C. J. Langton, C. G. Taylor, J. D. Farmer,
ing Sciences. She is a Fellow (and former vice-president)
and S. Rasmussen, 313–324. Reading, MA: Addison-Wes-
of the British Academy, a member of the Academia
ley.
Europaea, a Fellow of the Association for the Advance-
Hofstadter, D. R. 2002. How Could a COPYCAT Ever be ment of Artificial Intelligence, a Fellow of the European
Creative? In Creativity, Cognition, and Knowledge: An Inter-
Coordinating Committee for Artificial Intelligence, a Life
action, ed. T. Dartnall, 405–424. London: Praeger.
Fellow of the UK’s Society for Artificial Intelligence and
Hofstadter, D. R., and FARG, The Fluid Analogies the Simulation of Behaviour, a member of Council of the
Research Group. 1995. Fluid Concepts and Creative Analo- Royal Institute of Philosophy, and the former chairman
gies: Computer Models of the Fundamental Mechanisms of of Council of the Royal Institution of Great Britain.
Thought. New York: Basic Books. Boden has lectured widely, to both specialist and general
Hofstadter, D. R. 2001. Staring Emmy Straight in the audiences, in North and South America, Europe, India,
Eye—And Doing My Best Not to Flinch. In Virtual Music: the former USSR, and the Pacific. She has also appeared
Computer Synthesis of Musical Style, ed. D. Cope, 33–82. on many radio and TV programs in the UK and else-
Cambridge, Mass.: The MIT Press. where. Her work has been translated into 20 foreign lan-
Koning, H., and Eizenberg, J. 1981. The Language of the guages. She was awarded an OBE in 2001 (for “services to
Prairie: Frank Lloyd Wright’s Prairie Houses. Environment cognitive science”), and beside her Cambridge ScD, she
and Planning, B, 8(4): 295–323. has three honorary doctorates (from Bristol, Sussex, and
Langley, P. W.; Simon, H. A.; Bradshaw, G. L.; and Zytkow, the Open Universities).

34 AI MAGAZINE

View publication stats

Vous aimerez peut-être aussi