Académique Documents
Professionnel Documents
Culture Documents
It’s probably pretty clear to you that some of this is nonsense, and that
some of this is real. But if you aren’t deeply immersed in the �eld, it can
be hard to guess which is which. And while there are endless primers
on the Internet for people who want to learn to program AI, there
aren’t many explanations of the ideas, and the social and ethical
challenges they imply, for people who aren’t and don’t want to be
software engineers or statisticians.
I’ll give you one spoiler to start with: most of the hardest challenges
1 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
1 -
2 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
There’s a system with some sensors that can perceive the world—these
can be anything from video cameras and LIDAR to a web crawler
looking at documents. There’s some other system which can act on the
world, by doing anything from driving a car to showing ads to sorting
3 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
To connect the two, you need a box that takes the perceptions of the
world, and comes out with advice about what what is likely to happen
if you take various courses of action. That box in the center is called a
“model,” as in “a model of how the world works,” and that box is the AI
part.
The diagram above has some extra words in it, which are ones you may
hear when professionals discuss AI. “Features” are simply some
distillation of the raw perceptions, the parts of those perceptions which
the designers of the model thought would be useful to include. In some
AI systems, the features are just the raw perceptions—for example, the
color seen by every pixel of a camera. Such a huge number of features
is good for the AI in that it doesn’t impose any preconceptions of what
is and isn’t important, but makes it harder to build the AI itself; it’s only
in the past decade or so that it’s become possible to build computers big
enough to handle that.
“Predictions” are what comes out the other end: when you present the
model with some set of features, it will generally give you a bunch of
possible outcomes, and its best understanding of the likelihood of each.
If you want an AI to make a decision, you then apply some rule to that
—for example, “pick the one most likely to succeed,” or “pick the one
least likely to cause a catastrophic failure.” That �nal rule, of weighing
possible costs and bene�ts, is no less important to the system than the
model itself.
Now, you could imagine a very simple “model” that gives rules that are
just �ne for many uses: for example, the mechanical regulator valves on
old steam engines were a kind of simple “model” which read the
pressure in on one end, and if that pressure pushed a lever beyond
some set point, it would open a valve. It was a simple rule: if the
pressure is above the set point, open the valve; otherwise, close it.
The reason this valve is so simple is that it only needs to consider one
input, and make one decision. If it had to decide something more
4 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
What makes AI models special is that they are designed for this. Inside
any AI model are a bunch of rules to combine features, each of which
depends on one of hundreds, thousands, or even millions of individual
knobs, telling it how much to weigh the signi�cance of each feature
under di�erent circumstances. For example, in one kind of AI model
called a “decision tree,” the model looks like a giant tree of yes/no
questions. If the AI’s job were to tell tuna from salmon, the very �rst
question may be “is the left half of the picture darker than the right
half?,” and by the end of it it would look like “given the answers to the
past 374 questions, is the average color of pixels in this square more
orange or red?” The “knobs” here are the order in which questions are
asked, and what the boundaries between a yes and a no for each of
them are.
�xed, and the model can be put to use, connected to real actuators.
The advantage that ML models have over humans doing the same task
isn’t speed; an ML model typically takes a few milliseconds to make a
decision, which is roughly what a human takes as well. (You do this all
the time while driving a car) Their real advantage is that they don’t get
bored or distracted: an ML model can keep making decisions over
di�erent pieces of data, millions or billions of times in a row, and not
get any worse (or better) at it. That means you can apply them to
problems that humans are very bad at—like ranking billions of web
pages for a single search, or driving a car.
(Humans are terrible at driving cars: that’s why 35,000 people were
killed by them in the US alone in 2015. The huge majority of these
crashes were due to distraction or driver error—things that people
normally do just �ne, but failed to do just once at a critical moment.
Driving requires tremendous awareness and the ability to react within
a small fraction of a second, something which if you think about it is
kind of amazing we can do at all. But worse, it requires the ability to
consistently do that for hours on end, something which it turns out we
actually can’t do.)
For example, look at the following six pictures, and �gure out the key
di�erence between the �rst three and the second three:
• If you guessed “the �rst three have carpet in them,” you’re right!
You would also be right, of course, if you had guessed that the �rst
6 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
three were pictures of grey cats, and the second three were pictures of
white cats. But if you had used these images to train your Grey Cat
Detector, you might get excellent performance when the model tries to
rate your training pictures, and terrible performance in the real world,
because what the model actually learned was “grey cats are cat-shaped
things which sit on carpets.”
2 -
Problems where both the goals, and the means to achieve those goals,
are well-understood don’t even require AI. For example, if your goal is
“tighten all the nuts on this car wheel to 100 foot-pounds,” all you need
is a mechanism that can tighten and measure torque, and stops
tightening when the torque reaches 100. This is called a “torque
wrench,” and if someone o�ers you an arti�cially intelligent torque
wrench the correct �rst question to ask them is why would I want that.
7 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
These are the steam relief valves of AI; all you need is a simple
mechanism.
AI shines in problems where the goals are understood, but the means
aren’t. This is easiest to do when:
The number of stimuli or decisions is still so big that you can’t just
write down the rule; and separately, that
These things are harder than they seem. For example, pick up an object
sitting next to you right now—I’ll do it with an empty soda can. Now
do that again slowly, and watch what your arm did.
The fact that we can walk without falling on our faces in confusion is a
lot more amazing than it seems. Next time you walk across the room,
pay attention to the exact path you take, each time you bend or move
your body or put your foot down anywhere except directly in front of
you. “Motion planning,” as this problem is called in robotics, is really
hard.
8 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
This is one of the tasks which is so hard that our brains have a double-
digit percentage of their mass dedicated to nothing else. That makes
them seem far easier to us than they actually are. Other tasks in this
category are face recognition (a lot of our brain is dedicated not to
general vision, but speci�cally to recognizing faces), understanding
words, identifying 3D objects, and moving without running into things.
We don’t think of these as hard because they’re so intuitive to us—but
they’re intuitive because we have evolved specialized organs to do
nothing but be really good at those.
For this narrow set of tasks, computers do very poorly, not because they
do worse at them than they do at similar tasks, but because we’re
intuitively so good at them that our baseline for what constitutes
“acceptable performance” is very high. If we didn’t have a huge chunk
of our brain doing nothing but recognizing faces, people would look
about as di�erent to us as armadillos do—which is just what happens
to computers.
(Conversely, the way humans are wired makes other tasks arti�cially
easy for computers to get “right enough.” For example, human brains
are wired to assume, in case of doubt, that something which acts more-
or-less alive is actually animate. This means that having convincing
dialogues with humans doesn’t require understanding language in
general; so long as you can keep the conversation on a more-or-less
focused topic, humans will autocorrect around anything unclear. This
is why voice assistants are possible. The most famous example of this is
ELIZA, a 1964 “AI” which mimicked a Rogerian psychotherapist. It
would understand just enough of your sentences to ask you to tell it
more about various things, and if it got confused, it would fall back on
safe questions like “Tell me about your mother.” While it was half
meant as a joke, people did report feeling better after talking to it. If
you have access to a Google Assistant-powered device, you can tell it
“OK Google; talk to Eliza” and see for yourself.)
9 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
minutes from now,” and can attribute that decision to a much later
consequence. You had to spend a lot of time understanding the
mechanics of the game before that connection became understandable
to you; AI’s have the same problem.
We’ve talked about cases where the goals and means are understood,
and cases where the goals but not the means are understood. There’s a
third category, where AI can’t help at all: problems where the goal itself
isn’t well understood. After all, if you can’t give the AI a bunch of
examples of what is and isn’t a good solution look like, what’s it going
to learn from?
. . .
10 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
attacked in the past few years, which is why we’re now attacking
problems like self-driving cars or self-�ying airplanes.
Winning at board games isn’t very useful in its own right, but it opens
up the path to indirect goals in an unpredictable environment, like
planning your �nancial portfolio. This is a harder problem, and we
haven’t yet made major inroads on it, but I would expect us to get good
at these over the next decade.
And �nally you have the hardest case, of unde�ned goals. These can’t
be solved by AI at all; you can’t train the system if you can’t tell it what
you want it to do. Writing a novel might be an example of this, since
there isn’t a clear answer to what makes something a “good novel.” On
the other hand, there are speci�c parts of that problem where goals
could be de�ned—for example, “write a novel which will sell well if
marketed as horror.”
3 -
11 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
I’m starting with this problem because it’s been discussed a lot in public
in the past few years, and the discussion has often been remarkably
intelligent, and shows o� the kinds of question we really need to ask.
First of all, there’s a big caveat to this entire question: this problem
matters very little in practice, because the whole point of self-driving
cars is that they don’t get into this situation in the �rst place. Children
rarely appear out of nowhere; mostly when that happens, either the
driver was going too fast for their own re�exes to handle a child
jumping out from behind an obstruction they could see, or the driver
was distracted and for some reason didn’t notice the chid until too late.
These are both exactly the sorts of things that an automatic driver has
no problem with: looking at all the signals around at once, for hours on
end, without getting bored or distracted. A situation like this one would
become vanishingly rare, and that’s where the lives saved come from.
But “almost never” isn’t the same thing as “never,” and we have to
accept that sometimes this will happen. When it does, what should the
car do? Should it prioritize the life of its passengers, or of pedestrians?
This isn’t a technology question: it’s a policy question, and in the form
above, it’s been boiled down to its simple core. We could agree on
either answer (or any combination) as a society, and we can program
the cars to do that. If we don’t like the answer, we can change it.
There’s one big way in which this is di�erent from the world we inhabit
today. If you ask people what they would do in this situation, they’ll
give a wide variety of answers, and caveat them with all sorts of “it
depends”es. The fact is that we don’t want to have to make this
decision, and we certainly don’t want to publicly admit if our decision
is to protect ourselves over the child. When people actually are in such
situations, their responses end up all over the map.
12 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
that split-second between when you see oncoming disaster and when it
happens, we recognize that we can’t make rational decisions. We will
end up both holding the driver accountable for their decision, and
recognizing it as inevitable, no matter what they decide. (Although we
might hold them much more accountable for decisions they made
before that �nal split-second, like speeding or driving drunk.)
2. Polite �ctions
Machine-learned models have a very nasty habit: they will learn what
the data shows them, and then tell you what they’ve learned. They
obstinately refuse to learn “the world as we wish it were,” or “the world
as we like to claim it is,” unless we explicitly explain to them what that
is—even if we like to pretend that we’re doing no such thing.
In mid-2016, high school student Kabir Alli tried doing Google image
searches for “three white teenagers” and “three black teenagers.” The
results were even worse than you’d expect.
13 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
The fact is that these biases do exist in our society, and they’re re�ected
in nearly any piece of data you look at. In the United States, it’s a good
bet that if your data doesn’t show a racial skew of some sort, you’ve
14 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
This kind of honesty can force you to be very explicit. A good recent
example was in a technical paper about “word debiasing.” This was
about a very popular ML model called word2vec which learned various
relationships between the meanings of English words—for example,
that “king is to man, as queen is to woman.” The authors of this paper
found that it contained quite a few examples of social bias: for
example, it would also say that “computer programmer is to man, as
homemaker is to woman.” The paper is about a technique they came up
with for eliminating that bias.
This is all good work, but it’s important to recognize that the key step in
this—of identifying which male/female splits should be removed—
was a human decision, not an automatic process. It required people to
literally articulate which splits they thought were natural and which
ones weren’t. Moreover, there’s a reason the original model derived
those splits; it came from analysis of millions of written texts from all
over the world. The original word2vec model accurately captured
people’s biases; the cleaned model accurately captured the raters’
15 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
The risk which this highlights is the “naturalistic fallacy,” what happens
when we confuse what is with what ought to be. The original model is
appropriate if we want to use it to study people’s perceptions and
behavior; the modi�ed model is appropriate if we want to use it to
generate new behavior and communicate some intent to others. It
would be wrong to say that the modi�ed model more accurately
re�ects what the world is; it would be just as wrong to say that because
the world is some way, it also ought to be that way. After all, the
purpose of any model—AI or mental—is to make decisions. Decisions
and actions are entirely about what we wish the world to be like; if they
weren’t, we would never do anything at all.
Many people suspected that this issue was the same one as the one that
caused HP’s face-tracking webcams to not work on Black people six
years earlier: that the training data for “faces” had been composed
exclusively of white people. This was the �rst thing we suspected as
well, but it we quickly crossed it o� the list: the training data included a
wide range of people of all races and colors.
The �rst problem was that face recognition is hard. Di�erent people
look so vividly di�erent to us precisely because a tremendous fraction
of our brain matter is dedicated to nothing but recognizing people’s
faces; we’ve spent millions of years evolving tools for nothing else. But
16 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
if you compare how di�erent two di�erent faces are in to how di�erent,
say, two di�erent chairs are, you’ll see that faces are tremendously
more similar than you would guess—even across species.
In fact, we discovered that this bug was far from isolated: the system
was also prone to misidentifying white faces as dogs and seals.
And this goes to the second problem, which is the real heart of the
matter: ML systems are very smart in their domain, but know nothing
at all about the broader world, unless they were taught it. And when
trying to think about all the ways in which di�erent pictures could be
identi�ed as di�erent objects—this AI isn’t just about faces— nobody
thought to explain to it the long history of Black people being
dehumanized by being compared to apes. That context is what made
this error so serious and harmful, while misidentifying someone’s
toddler as a seal would just be funny.
This problem doesn’t just manifest in AI: it also manifests when people
are asked to make value judgments across cultures. One particular
challenge for this is when detecting harassment and abuse online. Such
questions are almost entirely handled by humans, rather than AI’s,
because it’s extremely di�cult to set down rules that even humans can
use to judge these things. I spent a year and a half developing such
rules at Google, and consider it to be one of the greatest intellectual
challenges I’ve ever faced. To give a very simple example: people often
say “well, an obvious rule is that if you say n****r, that’s bad.” I
challenge you to apply that rule to the di�erent meanings of the word
in (1) nearly any of Jay-Z’s songs, (2) Langston Hughes’ poem “Christ
in Alabama,” (3) that routine by Chris Rock, (4) that same routine if he
had performed it in front of a white audience, (5) and that same
routine if Ted Nugent had performed it, verbatim, to one of his
audiences, and come up with a coherent explanation of what’s going
on. It’s possible; it’s far from simple. And those are just �ve examples
17 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
Even with teams of people coming up with rules, and humans, not AI’s,
enforcing them, cultural barriers are a huge problem. A reviewer in
India won’t necessarily have the cultural context around the meaning
of a racial slur in America, nor would one in America have cultural
context for one in India. But the number of cultures around the world is
huge: how do you express these ideas in a way that anyone can learn
them?
The lesson is this: often the most dangerous risks in a system come, not
from problems within the system, but from unexpected ways that the
system can interact with the broader world. We don’t yet have a good
way to manage this.
(The third problem in the Gorilla Incident—for those of you who are
interested—is a problem of racism in photography. Since the �rst days
of commercial �lm, the standards for color and image calibration have
included things like “Shirley Cards,” pictures of standardized models.
These models were exclusively white until the 1970’s—when furniture
manufacturers complained that �lm couldn’t accurately capture the
brown tones of dark wood! Even though modern color calibration
standards are more diverse, our standards for what constitute “good
images” still overwhelmingly favor white faces rather than black ones.
As a result, amateur pictures of white people with cell phone cameras
turn out reasonably well, but amateur pictures of black people—
especially dark-skinned people—often come out underexposed. Faces
are reduced to vague blobs of brown with eyes and sometimes a mouth,
which unsurprisingly are hard for image recognition algorithms to
make much sense of. Photography director Ava Berkofsky recently gave
an excellent interview on how to light and photograph Black faces
well.)
If you’ve been reading carefully so far, you may have spotted a few ways
this could go horribly, terribly, wrong. And that’s exactly what
happened across the country, as revealed by a 2016 ProPublica exposé.
But we don’t have a magic light that goes o� every time someone
commits a crime, and we certainly don’t have training examples where
the same person was given two di�erent sentences at once and turned
out two di�erent ways. So the COMPAS model was trained on a proxy
for the real, unobtainable data: given the information we know about a
person at the time of sentencing, what is the probability that this
person will be convicted of a crime? Or phrased as a comparison
between two people, “Which of these two people is most likely to be
convicted of a crime in the future?”
If you know anything at all about the politics of the United States, you
can answer that question immediately: “The Black one!” Black people
are tremendously more likely to be stopped, arrested, convicted, and
given long sentences for identical crimes than white people, so an ML
model which looked at the data and, ignoring absolutely everything
else, always predicted that a Black defendant is more likely to be
convicted of another crime in the future, would in fact be predicting
quite accurately.
But what the model was being trained for wasn’t what the model was
being used for. It was trained to answer, “who is more likely to be
convicted,” and then asked “who is more likely to commit a crime,”
without anyone paying attention to the fact that these are two entirely
di�erent questions.
There are obviously many problems at play here. One is that the courts
took the AI model far too seriously, using it as a direct factor in
sentencing decisions, skipping human judgment, with far more
con�dence than any model should warrant. (A good rule of thumb,
also recently encoded into EU law, is that decisions with serious
consequences of people should be sanity-checked by a human—and
that there should be a human override mechanism available.) Another
problem, of course, is the underlying systemic racism which this
exposed: the fact that Black people are more likely to be arrested and
convicted of the same crimes.
20 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
But there’s an issue speci�c to ML here, and it’s one that bears
attention: there is often a di�erence between the quantity you want to
measure, and the one you can measure. When these di�er, your ML
model will become good at predicting the quantity you measured, not
the quantity for which it was meant to be a proxy. You need to very
carefully reason about the ways in which these are similar and di�er
before trusting your model.
First, we should understand why it’s hard to do this; second, and more
importantly, we should understand why we expect it to be easy to do,
and why this expectation is wrong. And third, we can look at what we
can actually do.
For any situation the model encounters, the only “explanation” it has of
what it’s doing is “well, the following thousand variables were in these
states, and then I saw the letter ‘c,’ and I know that this should change
the probability of the user talking about a dog according to the
following polynomial…”
21 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
What people are good at, it turns out, isn’t explaining how they made
decisions: it’s coming up with a reasonable-sounding explanation for
their decision after the fact. Sometimes this is perfectly innocent: for
example, we identify some fact which was salient for us in the decision-
making process (“I liked the color of the car”) and focus on that, while
ignoring things which may have been important to us but were
invisible. (“My stepfather had a hatchback. I hated him.”) It can also
have deeper motivations: to resolve cognitive dissonance by explaining
how we did or didn’t want something anyway (“the grapes were
probably sour, anyway”), or to avoid thinking too closely about
something we may not want to admit. (“The �rst candidate sounded
just like I did when I graduated. That woman was good, but she felt
di�erent… she wouldn’t �t as well working with me.”)
22 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
But when we ask for explanations, what we’re really often interested in
is which facts were both salient (in that changing them would have
changed the outcome materially) and mutable (in that changes to them
are worth discussing). For example, “you were shown this job posting;
had you lived ten miles west, you would have seen this one instead”
may be interesting in some context, but “you were shown this job
posting; had you been an emu, you would instead have been shown a
container of mulga seeds” is not.
23 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
People are rightly terri�ed of such devices, and would be even more
terri�ed if they heard more of the stories of people who already live
under the perpetual threat of death coming suddenly out of a clear sky.
24 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
The simplicity comes because this question is far from new. Much of the
point of military discipline is to create a �ghting force which does not
try to think too autonomously during battle. In countries whose
militaries are descended from European systems, the role of enlisted
and noncommissioned o�cers is to execute on plans; the role of
commissioned o�cers is to decide on which plans to execute. Thus, in
theory, the decision responsibility is entirely on the shoulders of the
o�cers, and the clear demarcation of areas of responsibility between
o�cers based on rank, area of command, and so on, determines who is
ultimately responsible for any given order.
Perhaps this is the last important lesson of the ethics of AI: many of the
problems we face with AI are simply the problems we have faced in the
past, brought to the fore by some change in technology. It’s often
valuable to look for similar problems in our existing world, to help us
understand how we might approach seemingly new ones.
4 -
These are rarely new problems; rather, the formal process of explaining
our desires to a computer—the ultimate case of someone with no
cultural context or ability to infer what we don’t say—forces us to be
explicit in ways we generally aren’t used to. Whether this involves
making a life-or-death decision years ahead of time, rather than
delaying it until the heat of the moment, or whether it involves taking a
25 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
long, hard look at the way our society actually is, and being very
explicit about which parts of that we want to keep and which parts we
want to change, AI pushes us outside of our comfort zone of polite
�ctions and into a world where we have to discuss things very
explicitly.
Every one of these problems existed long before AI; AI just made us talk
about them in a new way. That might not be easy, but the honesty it
forces on us may be the most valuable gift our new technology can give
us.
26 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
27 di 28 01/03/18, 23:41
Asking the Right Questions About AI – Yonatan Z... https://medium.com/@yonatanzunger/asking-the-...
28 di 28 01/03/18, 23:41