Vous êtes sur la page 1sur 124


CACM.ACM.ORG OF THE 10/2018 VOL.61 NO.10

Human-Level Intelligence
or Animal-Like Abilities?

Computing within Limits

Transient Electronics Take Shape
Q&A with Dina Katabi
Formally Verified Software
in the Real World Association for
Computing Machinery
Communications of the ACM
China Region
Special Section
A collection of articles spotlighting how computing is
transforming the China region, and the leading-edge industry,
academic, and government initiatives currently underway, is
coming to CACM this Fall. This section includes articles on Big
Trends and Hot Topics by leading practitioners and academics
from the China region, including:
† Cloud Infrastructure for World’s Largest Consumer Market
† How AI-powered FinTech is Improving People’s Lives
† The Future of Artificial Intelligence in China
† Supercomputers as SuperData and SuperAI Machines
† Last-mile Delivery and Autonomous Vehicles
† Quantum Communication
† And much more!

This special section is the first

in a series coming to ACM’s
flagship magazine; each will
feature articles authored by
the region’s leading computing
professionals in a particular
geographic region, highlighting
the most exciting computing
advances and innovation.

Departments News Viewpoints

5 Cerf’s Up 20 Technology Strategy and Management

The Internet in the 21st Century The Business of Quantum Computing
By Vinton G. Cerf Considering the similarities
of quantum computing
6 Letters to the Editor development to the early years
Hennessy and Patterson of conventional computing.
on the Roots of RISC By Michael A. Cusumano

8 BLOG@CACM 23 Privacy and Security

Can We Use AI for Global Good? A Pedagogic Cybersecurity Framework
Amir Banifatemi observes how A proposal for teaching
the AI for Good Summit the organizational, legal,
“allowed us to start a dialogue, and international aspects
find a common frame of reference, of cybersecurity.
and decide how our steps 11 By Peter Swire
would be smart and structured.”
11 Floating Voxels Provide 27 Kode Vicious
31 Calendar New Hope for 3D Displays The Obscene Coupling
In search of holograms that Known as Spaghetti Code
114 Careers can be viewed from any angle. Teach your junior programmers
By Chris Edwards how to read code.
By George V. Neville-Neil
Last Byte 14 Transient Electronics Take Shape
Advances in materials science 29 Viewpoint
120 Q&A and chemistry are leading Building the Universal Archive
Reaping the Benefits to self-destructing circuits of Source Code
of a Diverse Background and transient electronics, which A global collaborative project
Earlier this year, ACM named could impact many fields. for the benefit of all.
Dina Katabi of the Massachusetts By Samuel Greengard By Jean-François Abramatic,
Institute of Technology’s Computer Roberto Di Cosmo,
Science and Artificial Intelligence 17 The Dangers of Automating and Stefano Zacchiroli
Laboratory recipient of the 2017 ACM Social Programs
Prize in Computing for her creative Is it possible to keep bias Watch the authors discuss
their work in this exclusive
contributions to wireless systems. out of a social program driven Communications video.
By Leah Hoffmann by one or more algorithms? https://cacm.acm.org/
By Esther Shein universal-archive-of-

32 Viewpoint
Are CS Conferences (Too)

Closed Communities?
Assessing whether newcomers
have a more difficult time
achieving paper acceptance
at established conferences.
By Jordi Cabot,
Javier Luis Cánovas Izquierdo,
and Valerio Cosentino
Association for Computing Machinery
Advancing Computing as a Science & Profession

2 COMMUNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

10/2018 VOL. 61 NO. 10

Practice Contributed Articles Review Articles

44 56 86

36 The Mythos of Model Interpretability 56 Human-Level Intelligence 86 Computing within Limits

In machine learning, or Animal-Like Abilities? The future of computing research
the concept of interpretability is What just happened in relies on addressing an array
both important and slippery. artificial intelligence and how of limitations on a planetary scale.
By Zachary C. Lipton it is being misunderstood. By Bonnie Nardi, Bill Tomlinson,
By Adnan Darwiche Donald J. Patterson, Jay Chen,
44 The Secret Formula for Choosing Daniel Pargman, Barath Raghavan,
the Right Next Role Watch the author discuss and Birgit Penzenstadler
his work in the exclusive
The best careers are not defined Communications video.
by titles or résumé bullet points. https://cacm.acm.org/
By Kate Matsudaira intelligence-or-animal- Research Highlights

47 Mind Your State for 95 Technical Perspective

Your State of Mind 68 Formally Verified Software A Control Theorist’s View on Reactive
The interactions between in the Real World Control for Autonomous Drones
storage and applications Verified software secures By John Baillieul

can be complex and subtle. the Unmanned Little Bird

By Pat Helland autonomous helicopter against 96 Fundamental Concepts of Reactive
mid-flight cyber attacks. Control for Autonomous Drones
Articles’ development led by By Gerwin Klein, June Andronick, By Luca Mottola and Kamin Whitehouse
Matthew Fernandez, Ihor Kuz,
Toby Murray, and Gernot Heiser 105 Technical Perspective
The Future of MPI
78 The Productivity Paradox in By Marc Snir
Health Information Technology
About the Cover: New York State healthcare providers 106 Enabling Highly Scalable Remote
This month’s cover was
inspired by a Judea Pearl increased their use of the technology Memory Access Programming
quote that contends the but delivered only mixed results for with MPI-3 One Sided
vision system of an eagle
outperforms anything their patients. By Robert Gerstenberger,
created in the lab, yet By Quang “Neo” Bui, Sean Hansen, Maciej Besta, and Torsten Hoefler
the eagle cannot build a
telescope or microscope. Manlu Liu, and Qiang (John) Tu
Adnan Darwiche uses
this quote as a jumping-off
point to argue what AI is
and is not doing today
(p. 56). Cover illustration
by Hugh Syme.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF THE ACM 3

Trusted insights for computing’s leading professionals.

Communications of the ACM is the leading monthly print and online magazine for the computing and information technology fields.
Communications is recognized as the most trusted and knowledgeable source of industry information for today’s computing professional.
Communications brings its readership in-depth coverage of emerging areas of computer science, new trends in information technology,
and practical applications. Industry leaders use Communications as a platform to present and debate various technology implications,
public policies, engineering challenges, and market trends. The prestige and unmatched reputation that Communications of the ACM
enjoys today is built upon a 50-year commitment to high-quality editorial content and a steadfast dedication to advancing the arts,
sciences, and applications of information technology.

ACM, the world’s largest educational STA F F EDITORIAL BOARD ACM Copyright Notice
and scientific computing society, delivers DIRECTOR OF PU BL ICATIONS E DITOR- IN- C HIE F Copyright © 2018 by Association for
resources that advance computing as a Scott E. Delman Andrew A. Chien Computing Machinery, Inc. (ACM).
science and profession. ACM provides the cacm-publisher@cacm.acm.org eic@cacm.acm.org Permission to make digital or hard copies
computing field’s premier Digital Library Deputy to the Editor-in-Chief of part or all of this work for personal
and serves its members and the computing Executive Editor Lihan Chen or classroom use is granted without
profession with leading-edge publications, Diane Crawford cacm.deputy.to.eic@gmail.com fee provided that copies are not made
conferences, and career resources. Managing Editor S E NIOR E DITOR or distributed for profit or commercial
Thomas E. Lambert Moshe Y. Vardi advantage and that copies bear this
Executive Director and CEO Senior Editor notice and full citation on the first
Vicki L. Hanson Andrew Rosenbloom page. Copyright for components of this
Deputy Executive Director and COO Senior Editor/News work owned by others than ACM must
Patricia Ryan Co-Chairs be honored. Abstracting with credit is
Lawrence M. Fisher
Director, Office of Information Systems William Pulleyblank and Marc Snir permitted. To copy otherwise, to republish,
Web Editor
Wayne Graves Board Members to post on servers, or to redistribute to
David Roman
Director, Office of Financial Services Monica Divitini; Mei Kobayashi; lists, requires prior specific permission
Rights and Permissions
Darren Ramdin Michael Mitzenmacher; Rajeev Rastogi; and/or fee. Request permission to publish
Barbara Ryan
Director, Office of SIG Services François Sillion from permissions@hq.acm.org or fax
Editorial Assistant
Donna Cappo Jade Morris (212) 869-0481.
Director, Office of Publications VIE W P OINTS
Scott E. Delman Art Director Co-Chairs For other copying of articles that carry a
Andrij Borys Tim Finin; Susanne E. Hambrusch; code at the bottom of the first or last page
Associate Art Director John Leslie King; Paul Rosenbloom or screen display, copying is permitted
Margaret Gray Board Members provided that the per-copy fee indicated
Assistant Art Director Stefan Bechtold; Michael L. Best; Judith Bishop; in the code is paid through the Copyright
Cherri M. Pancake
Mia Angelica Balaquiot Andrew W. Cross; Mark Guzdial; Haym B. Hirsch; Clearance Center; www.copyright.com.
Production Manager Richard Ladner; Carl Landwehr; Beng Chin Ooi;
Elizabeth Churchill
Bernadette Shade Francesca Rossi; Loren Terveen; Subscriptions
Advertising Sales Account Manager Marshall Van Alstyne; Jeannette Wing; An annual subscription cost is included
Yannis Ioannidis
Ilia Rodriguez Susan J. Winter in ACM member dues of $99 ($40 of
Past President
Alexander L. Wolf which is allocated to a subscription to
Chair, SGB Board Columnists Communications); for students, cost
Jeff Jortner David Anderson; Michael Cusumano; P R AC TIC E is included in $42 dues ($20 of which
Co-Chairs, Publications Board Peter J. Denning; Mark Guzdial; Co-Chairs is allocated to a Communications
Jack Davidson and Joseph Konstan Thomas Haigh; Leah Hoffmann; Mari Sako; Stephen Bourne and Theo Schlossnagle subscription). A nonmember annual
Members-at-Large Pamela Samuelson; Marshall Van Alstyne Board Members subscription is $269.
Gabriele Anderst-Kotis; Susan Dumais; Eric Allman; Samy Bahra; Peter Bailis;
Renée McCauley; Claudia Bauzer Mederios; C O N TAC T P O IN TS Terry Coatta; Stuart Feldman; Nicole Forsgren; ACM Media Advertising Policy
Elizabeth D. Mynatt; Pamela Samuelson; Copyright permission Camille Fournier; Jessie Frazelle; Communications of the ACM and other
Theo Schlossnagle; Eugene H. Spafford permissions@hq.acm.org Benjamin Fried; Tom Killalea; Tom Limoncelli; ACM Media publications accept advertising
SGB Council Representatives Calendar items Kate Matsudaira; Marshall Kirk McKusick; in both print and electronic formats. All
Sarita Adve; Jeanna Neefe Matthews calendar@cacm.acm.org Erik Meijer; George Neville-Neil; advertising in ACM Media publications is
Change of address Jim Waldo; Meredith Whittaker at the discretion of ACM and is intended
BOARD C HA I R S acmhelp@acm.org to provide financial support for the various
Letters to the Editor activities and services for ACM members.
Education Board C ONTR IB U TE D A RTIC LES
letters@cacm.acm.org Current advertising rates can be found
Mehran Sahami and Jane Chu Prey Co-Chairs
by visiting http://www.acm-media.org or
Practitioners Board James Larus and Gail Murphy
W E B S IT E by contacting ACM Media Sales at
Terry Coatta and Stephen Ibaraki Board Members
http://cacm.acm.org (212) 626-0686.
William Aiello; Robert Austin; Kim Bruce;
REGIONA L C O U N C I L C HA I R S Alan Bundy; Peter Buneman; Carl Gutwin;
WEB BOARD Single Copies
ACM Europe Council Yannis Ioannidis; Gal A. Kaminka;
Chair Single copies of Communications of the
Chris Hankin Ashish Kapoor; Kristin Lauter; Igor Markov;
James Landay ACM are available for purchase. Please
ACM India Council Bernhard Nebel; Lionel M. Ni; Adrian Perrig;
Board Members contact acmhelp@acm.org.
Abhiram Ranade Marie-Christine Rousset; Krishan Sabnani;
Marti Hearst; Jason I. Hong;
ACM China Council m.c. schraefel; Ron Shamir; Alex Smola;
Jeff Johnson; Wendy E. MacKay COMMUN ICATION S OF THE ACM
Wenguang Chen Josep Torrellas; Sebastian Uchitel;
(ISSN 0001-0782) is published monthly
AU T H O R G U ID E L IN ES Hannes Werthner; Reinhard Wilhelm
by ACM Media, 2 Penn Plaza, Suite 701,
PUB LICATI O N S BOA R D http://cacm.acm.org/about- New York, NY 10121-0701. Periodicals
Co-Chairs communications/author-center RES E A R C H HIGHLIGHTS
postage paid at New York, NY 10001,
Jack Davidson; Joseph Konstan Co-Chairs
and other mailing offices.
Board Members Azer Bestavros and Shriram Krishnamurthi
Phoebe Ayers; Edward A. Fox; Board Members
2 Penn Plaza, Suite 701, New York, NY POSTMASTER
Chris Hankin; Xiang-Yang Li; Martin Abadi; Amr El Abbadi; Sanjeev Arora;
10121-0701 Please send address changes to
Sue Moon; Michael L. Nelson; Michael Backes; Maria-Florina Balcan;
T (212) 626-0686 Communications of the ACM
Sharon Oviatt; Eugene H. Spafford; David Brooks; Stuart K. Card; Jon Crowcroft;
F (212) 869-0481 2 Penn Plaza, Suite 701
Stephen N. Spencer; Divesh Srivastava; Alexei Efros; Bryan Ford; Alon Halevy;
New York, NY 10121-0701 USA
Robert Walker; Julie R. Williamson Gernot Heiser; Takeo Igarashi; Sven Koenig;
Advertising Sales Account Manager
Greg Morrisett; Tim Roughgarden;
Ilia Rodriguez
ACM U.S. Public Policy Office Guy Steele, Jr.; Robert Williamson; Printed in the USA.
Adam Eisgrau, Margaret H. Wright; Nicholai Zeldovich;
Director of Global Policy and Public Affairs Andreas Zeller
Media Kit acmmediasales@acm.org
1701 Pennsylvania Ave NW, Suite 300,
Washington, DC 20006 USA
T (202) 659-9711; F (202) 667-1066 Association for Computing Machinery S P EC IA L S EC TIONS
(ACM) Co-Chair
Computer Science Teachers Association 2 Penn Plaza, Suite 701 Sriram Rajamani A

Jake Baskin New York, NY 10121-0701 USA Board Members



Executive Director T (212) 869-7440; F (212) 869-0481 Tao Xie; Kenjiro Taura; David Padua




4 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

cerf’s up

DOI:10.1145/3275378 Vinton G. Cerf

The Internet in the 21st Century

After working on DARPA-funded projects
from 1967–1982, including the design
and implementation of the ARPANET and
Internet, I left DARPA to go into the private
sector to design and build MCI Mail. apply this global system to our daily The Secretary-General of the Unit-
At that time, I handed the architec- challenges. Perhaps more seriously, ed Nations has commissioned a High-
tural reins of the Internet to David D. many people are finding ways to do Level Panel on Digital Cooperation. I
Clark and Jonathan B. Postel as chief harmful things through the Internet consider this to be an aptly named
Internet architect and deputy Internet medium. Headlines highlighting abus- effort. The charge to the panel is to
architect, respectively. Since that time, es abound: Identity theft; electronic consider these matters and to make
Clark and Postel went on to make funds transfer and automated teller recommendations to deal with them
deeply significant contributions to machine heists; point-of-sale terminal in an internationally cooperative
the Internet’s evolution. Postel as the hacks; theft of personal information fashion. It is clearly unlikely the panel
Internet Assigned Numbers Author- including credit cards, passwords, and will solve the problems in general,
ity and RFC editor and member of the other personal information; malware but it may be able to surface imple-
Internet Architecture Board; Clark as and denial-of- service attacks; bullying; mentable, international, or transna-
the chairman of the Internet Architec- misinformation; election interference; tional actions that would reduce the
ture Board (earlier: Internet Activities and the exacerbation of social ten- vulnerabilities currently being ex-
Board) and as a leader in articulating sions. The list is longer and would take ploited by individuals, organizations,
Internet design principles. Sadly, Jon up the rest of this column. and nation states.
Postel passed away 20 years ago, October Responses to these abuses have At the national level, only a small
16, 1998,a just as the Internet Corpora- been sporadic at best. Two-factor au- percentage of businesses and indi-
tion for Assigned Names and Numbers thentication would remediate many viduals are well equipped to defend
(ICANN) was forming. He was to have penetration scenarios but is not widely themselves in the hazardous online
been its chief technology officer. More adopted. Operating system and ap- world. People must be trained to de-
recently, David Clark has produced two plication software weaknesses are not tect and reject phishing attacks and
wide-ranging and deep books about the adequately addressed. Corporate at- be more vigilent about cyber hygiene.
Internet. One book will be published tention to these risks is unevenly ap- More information sharing between
this month, Designing an Internet,b and plied and incentives to do better are in the national security apparatus and
the other, International Relations in the short supply. The social unrest accom- private-sector enterprises seems
Cyberage (The Co-Evolution Dilemma), panying deliberate misinformation called for, especially as vulnerabilities
will be published later by MIT Press. campaigns is finally reaching policy and their remedies become appar-
These two works capture the depth awareness and is leading to demands ent. That such a practice would ben-
and breadth of thought the Internet for response, but legislators are often efit from international cooperation
now demands of us on technical and poorly equipped to produce imple- seems likely but fraught with details
policy grounds. As new methods for mentable regulations. ACM has an about implementation. I am looking
exercising the network arrive (think active US-ACM Public Policy Commit- forward to reading both of Clark’s vol-
smartphones and the Internet of tee and other ACM Councils are being umes in the expectation that he and
Things), we are finding new ways to drawn into discussions about these his co-authors will throw light in the
problems but there is, as yet, little con- dark places that have developed in our
sensus on effective responses. Varying 21st-century Internet.
a https://tools.ietf.org/html/rfc2468 societal norms and conditions make
b D.D. Clark. Designing an Internet (Information Vinton G. Cerf is vice president and Chief Internet Evangelist
Policy). The MIT Press, Cambridge, MA, Oct.
for a wide range of possible reactions,
at Google. He served as ACM president from 2012–2014.
30, 2018. ISBN-10: 0262038609; ISBN-13: 978- some of which strike me as excessive
0262038607 and hostile to human rights. Copyright held by author.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF THE ACM 5

letters to the editor


Hennessy and Patterson

Computer Vision
on the Roots of RISC
Institute for Computational and Experimental Research in Mathematics

February 4 – May 10, 2019

Organizing Committee:
Y. Amit, University of Chicago A.M.
WARDING ACM’S 2017 Certainty Is Unobtainable” (Feb. 2018)
R. Basri, Weizmann Institute Turing Award to John included a number of misleading state-
A. Berg, University of NC Hennessy and David Pat- ments, the most important that: “Mean-
T. Berg, University of NC terson was richly deserved while, Gödel’s results were based on
P. Felzenszwalb, Brown Univ. and long overdue, as de- first-order logic, but every moderately
B. Fux Svaiter, IMPA
scribed by Neil Savage in his news sto- powerful first-order theory is inconsis-
S. Geman, Brown University
B. Gidas, Brown University ry “Rewarded for RISC” (June 2018). tent. Consequently, computer science
D. Jacobs, University of MD RISC was a big step forward. In their is changing to use higher-order logic.”
O. Veksler, Univ of W. Ontario acceptance speech, Patterson also Computer science is based on logic,
graciously acknowledged the contem- mostly first-order logic, and program-
Program Description: porary and independent invention mers make their coding decisions us-
Computer vision is an of the RISC concepts by John Cocke, ing logic every day. The most important
inter-disciplinary topic
another Turing laureate, at IBM, as results of logic (such as Kurt Gödel’s
crossing boundaries
between computer science,
described by Radin.1 Unfortunately, Incompleteness Theorems) are taught
statistics, mathematics, Cocke, who was the principal inven- in theory courses and are the funda-
engineering, and cognitive tor but rarely published, was not in- mentals on which computer science
science. Research in cluded as an author, and it would and software engineering are based. No
computer vision involves have been good if Savage had men- inconsistencies have ever been found in
the development and tioned his contribution. any of the standard first-order theories
evaluation of computa-
It is noteworthy that RISC archi- used in logic, ranging from moderately
tional methods for image
tectures depend on and emerged powerful to very powerful, and none are
from optimizing compilers. So far as believed to be inconsistent.
The focus of the program I can tell, all the RISC inventors had Harvey Friedman, Columbus, OH, USA,
will be on problems that
strong backgrounds in both architec- and Victor Marek, Lexington, KY, USA
involve modeling, machine
learning and optimization. ture and compilers.
The program will also
bridge a gap between 1. Radin, G. The 801 minicomputer. IBM Journal of
Author Responds:
theoretical approaches and Research & Development (1983), 237–246. Powerful first-order theories of intelligent
practical algorithms, Fred Brooks, Chapel Hill, NC, USA information systems are inconsistent
involving researchers with because these systems are not compact,
a variety of backgrounds.
thus violating a fundamental principle
Associated Workshops: No Inconsistencies in Fundamental of first-order theories. Meanwhile, the
First-Order Theories in Logic properties of self-proof of inferential
• Theory and Practice
in Machine Learning Referring to Martin E. Hellman’s Tur- completeness and formal consistency in
and Computer Vision ing Lecture article “Cybersecurity, Nu- higher-order mathematical theories are
(February 18 - 22, 2019) clear Security, Alan Turing, and Illogical the opposite of incompleteness and the
• Image Description for Logic” (Dec. 2017), Carl Hewitt’s letter self-unprovability of consistency Gödel
Consumer and to the editor “Final Knowledge with showed for first-order theories. Differing
Overhead Imagery properties between higher-order and
(February 25 - 26, 2019) first-order theories are reconciled by
• Computational Imaging It is noteworthy that Gödel’s “I’mUnprovable” proposition’s
(March 18 - 22, 2019) nonexistence in higher-order theories.
• Optimization Methods in
RISC architectures First-order theories are not foundational
Computer Vision and depend on and to computer science, which indeed relies
Image Processing on the opposite of Gödel’s results.
(April 29 - May 3, 2019) emerged from Carl Hewitt, Palo Alto, CA, USA

icerm.brown.edu optimizing compilers.

Brown University
121 S. Main Street, 11th floor More Accurate Text Analysis
Providence, RI 02903 for Better Patient Outcomes
David Gefen et al.’s article “Identifying
Patterns in Medical Records through

6 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

ComputerVision.CACM Ad.indd 1 8/3/18 2:55 PM

letters to the editor

an inaccurately coded disease con-

Coming Next Month in COMMUNICATIONS

cerns nonalcoholic fatty liver disease
A disease mention (NAFLD), a common yet underdocu-
could even lack mented disease often mentioned in
notes without ICD codes indicated.
any meaning at all, Given also subjective and idiosyncrat-
as it is just part ic physician billing styles, a patient re-
cord might include a code for NAFLD,
of a template though the code might indicate just
generated by a biopsy, despite greater odds that
the patient’s liver is functioning nor-
an electronic health mally. Incorporating codes without CHINA REGION
records system of associated dates likewise limits their
true meaning and thus reduces their
a particular provider’s applicability in association studies Industry and academic
care system. based on text. A code in a patient’s leaders from the region
problem list (a standard record in- share their insights on
dicating the most important health many of the big trends
problems a patient might be facing)
has a very different meaning from the and hot topics generating
same code appearing on the same pa- excitement throughout
tient’s doctor-noted encounter-diag- China’s computing
Latent Semantic Analysis” (June 2018) nosis record. community.
endorsed the latent semantic analy- To improve classification, accu-
sis (LSA) method of text analysis due racy of text-processing methods fo-
to its ability to identify links among cused on health care (such as LSA, as A Look at
mentions of medical terms, includ- Gefen et al. explored) would strongly the Design of Lua
ing the strengths of their relative as- benefit from much more specific
sociations. In practice, however, a representations of keywords to more Skill Discovery
single-keyword mention in a clinical accurately indicate or negate a con- in Virtual Assistants
narrative note might not represent the dition rather than incorporate only
true factual meaning of such a men- single keywords. For instance, in-
tion. Moreover, a disease may be men- stead of noting “hypertension,” a Modern Debugging:
tioned in the context of being ruled one-keyword mention, as in Gefen The Art of Finding
out as a diagnosis or only in the con- et al.’s Figure 1, the methods should a Needle in a Haystack
text of documenting family history. A use specific non-negated and time-
disease mention could even lack any dependent expressions like “Current
Software Challenges
meaning at all, as it is just part of a visit: Hypertension is in excellent
template generated by an electronic control” or in the context of a cardiac- for the Changing
health-records system of a particu- related condition, as in Gefen et al.’s Storage Landscape
lar provider’s care system. And many Figure 2, “No evidence of coronary ar-
clinical-narrative notes include con- tery disease.” Corp to Cloud: Google’s
tent that has been copied and pasted LSA and other advanced tech-
Virtual Desktops
from other notes, possibly inflating niques have the potential to truly
the importance of certain mentions represent the level of strength in
thus incorporated into the applicable the connections among textual con- Tracking and
machine-learning algorithms. cepts. However, to deliver accurate Controlling Microserve
Even incorporating standard In- results that most serve the patient, Dependencies
ternational Classification of Dis- the features within them must be
eases (ICD) codes, as defined and more descriptive. Such features
published by the World Health Orga- should thus be based on commonly
nization, into text-processing meth- used multi-keyword expressions and
ods, as Gefen et al. discussed, could their variations.
be misleading. Uri Kartoun, Cambridge, MA, USA
For a variety of everyday conditions
(such as insomnia), such codes do not
Communications welcomes your opinion. To submit a Plus the latest news about
indicate definitively the existence or sensing earthquakes with
Letter to the Editor, please limit yourself to 500 words or
nonexistence of a particular condi- less, and send to letters@cacm.acm.org. optical fiber, the impact of
tion. Another example of ICDs yield- GDRP, and AI explained.
ing potentially misleading results for © 2018 ACM 0001-0782/18/10

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF THE ACM 7

The Communications Web site, http://cacm.acm.org,
features more than a dozen bloggers in the BLOG@CACM
community. In each issue of Communications, we’ll publish
selected posts or excerpts.

Follow us on Twitter at http://twitter.com/blogCACM

DOI:10.1145/3264623 http://cacm.acm.org/blogs/blog-cacm

Can We Use AI
for Global Good?
Amir Banifatemi observes how the AI for Good Summit
“allowed us to start a dialogue, find a common frame of reference,
and decide how our steps would be smart and structured.”

Amir Banifatemi and communication technology (ICT), private-sector decision-makers. Four

Validating in partnership with the XPRIZE breakthrough tracks—looking at sat-
Beneficial AI Foundation, the Association for Com- ellite imagery, healthcare, smart cit-
https://cacm.acm. puting Machinery (ACM), and 32 sister ies, and trust in AI—set out to propose
org/blogs/blog- UN agencies. The 500+ attendees AI strategies with supporting projects
cacm/229283-validating- consisted of a diverse set of multi- to advance sustainable development.
beneficial-ai/fulltext stakeholders with wide-ranging Teams were guided in this effort by an
July 3, 2018 expertise—from the individual UN expert audience representing indus-
Can the diverse artificial intelligence agencies (including everything from try, academia, government, and civil
(AI) community come together to UNESCO and UNICEF to The World society. Each track proposed projects,
build an infrastructure to advance Health Organization, The World as well as introducing existing and fu-
the United Nation’s sustainable de- Bank, and UNHCR), AI researchers, ture obstacles to the attendees, who
velopment goals (SDGs, https://sus- public- and private-sector decision- then worked collaboratively to take
tainabledevelopment.un.org/sdgs) makers, potential financial partners promising strategies forward.
around the world? Can global projects and sponsor organizations. The results were demonstrative
be developed that begin to address The focus of the 2018 edition of of a strong momentum and multi-
pressing issues surrounding some of the AI for Good Summit was to bring stakeholder interest in collaboration
our greatest humanitarian challenges together stakeholders prepared to to identify AI-based solutions with
to help all? propose practical projects to tackle action at their core. The AI for Good
Those were the goals of the second topics within the 17 SDGs. Inspired Summit has achieved agreement on
annual AI for Good Global Summit, by the XPRIZE incentive model, the a community-oriented approach to
the leading United Nations platform goal was to present actual proposals support 35 projects, fast-tracked so
for dialogue on Artificial Intelligence in front of attendees to validate fea- they can be realized in as quickly as six
held in Geneva, Switzerland, over sibility, timing, and how meaningful months through a two- or three-year
three days in May. next steps can be identified. In short, window. Priority projects coming out
The conference was organized by setting actual solutions in motion. of each of the event tracks included:
the International Telecommunica- As part of the summit design, AI ˲˲ Developing Data and AI Com-
tion Union (ITU), the United Nations’ innovators in attendance were con- mons: A transversal effort during
specialized agency for information nected with invited public- and the three days of the conference was

8 COMMUNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 10


designed to capture common core proposals focus on the development

principles and opportunities to build of AI-driven simulations of city en-
a platform enabling beneficial AI. “We are seeing vironments and bringing a human-
To provide AI to the masses, there the AI community centered approach to each vision. The
is a need to have usable and share- projects support linguistic diversity
able data in a common format that working together within cities; the enabling of block-
everyone can access. General data- to create an chain-based, citizen-centered deci-
sets and relevant information use- sion making; strategies to combat
ful to machine learning specialists infrastructure gender imbalance and violence, and
is often spread throughout multiple for responsible the use of AI to enhance the cultural
repositories—there is an opportu- heritage of each city to ensure that
nity to consolidate them to level the communication, there are as many different defini-
playing field. This, for example, can development, tions of a smart city as there are cities
be domain-specific, such as care, in the world. There is also a project to
treatment, and outcomes for health and trust.” establish a global network—the ‘In-
researchers, historical weather data, ternet of Cities’—to share the data,
satellite imagery, and landmass/ knowledge, and expertise required
ocean temperature figures for agri- to replicate successful smart cities
culture and climate prediction, or city around the world.
traffic, lighting, and crime statistics An additional project proposal looks With collaborative efforts such as
for city planners. at creating a global service platform— these, we are seeing the AI commu-
Data Commons would offer assem- with associated enabling infrastruc- nity working together to create an in-
blies of datasets and supporting us- ture and common capabilities—that frastructure for responsible commu-
age of AI tools, knowledge, and exper- would allow developers to establish nication, development, and trust. The
tise of AI practitioners to launch new and support immediate scaling of foundational work that began at the
AI projects, scale up fast, and con- new satellite data projects. first AI for Good Summit has allowed
tribute new and improved resources ˲˲ AI and Healthcare: As one of the us to start a dialogue, find a common
to the AI for Good community. Data fastest-growing economic sectors in frame of reference, and decide how
Commons would provide a founda- many countries, scalable technol- our steps would be smart and struc-
tion of the AI Commons, a global ini- ogy surrounding the convergence of tured. Our focus this year was to ac-
tiative proposed at the conclusion of health and AI is exciting. Fifteen proj- celerate progress, launching projects
the AI for Good Summit. AI Commons ect proposals are moving forward, in- that will show tangible results and
would help make access to AI capabil- cluding predictive projects surround- provide positive impact in key areas.
ities universal and provide the public ing vision loss and osteoarthritis, The cycle is set to continue. The
a platform to solve challenges with AI integration and analysis of medical 2019 summit will take stock of prog-
and drive inclusion. data, AI and healthcare policy, and ress and will continue the focus on
The AI Commons is expected to responses to disease outbreak as well identifying practical ways to identify
be announced in late Q3 with oppor- as other medical emergencies. There and implement AI for Good projects
tunities for all stakeholders to join was also discussion surrounding the This is a time of building infra-
and participate in its development creation of a new, open study plat- structure, guidelines, and kicking off
and deployment. form for stakeholders, supported by focused development of tangible tools
˲˲ AI-Powered Analysis of Satel- ITU and the World Health Organiza- to accelerate the beneficial. Using AI
lite Imagery: Satellites transmit the tion, that would serve as a repository for Good is the mantra that is gain-
equivalent of approximately two bil- of use cases of AI in healthcare to ing traction with more participation
lion one-megapixel photographs ev- identify data formats as well as in- and conversations that make sense,
ery day, and AI is the only thing that teroperability mechanisms required and the conversation is not going to
can let us see the whole world at once. to amplify their impact. stop. AI innovations constitute one of
Beyond recording these images, they ˲˲ Building Trust in AI: To build well- the platforms that can bring benefits
can create a global real-time database earned trust in the long term, Trustfac- for everyone, and is a platform that
of the world. Three project proposals tory.ai is being established as an incu- can be a public asset for the common
are focused on agriculture and use of bator to research, source, support, and good. There is a great need to extend
AI-powered satellite imagery analy- address key dimensions of trust in AI. AI to more people and more places in
sis to predict and prevent deforesta- The research collective is led by Cam- a responsible way. We believe giving
tion, pinpoint and track livestock, bridge University and the University the public a common platform can
and provide data analytics to enable of Padova—and stakeholders see this benefit everyone.
micro-insurance to smallholder fam- as a second leg of the infrastructure
ily farming—small farms that rely needed to expand AI usage globally. Amir Banifatemi is the Group Lead, AI and Frontier
Technologies for XPRIZE.
mainly on family labor that are seen ˲˲ AI and Smart Cities: With the
as the prime driver of agricultural goal to identify common reposito-
production in developing countries. ries of best practices, seven project © 2018 ACM 0001-0782/18/10 $15.00

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF THE ACM 9

ACM is the world’s largest computing society, offering benefits and resources that can advance your career and
enrich your knowledge. We dare to be the best we can be, believing what we do is a force for good, and in joining
together to shape the future of computing.


q Professional Membership: $99 USD q Student Membership: $19 USD
q Professional Membership plus q Student Membership plus ACM Digital Library: $42 USD
ACM Digital Library: $198 USD ($99 dues + $99 DL) q Student Membership plus Print CACM Magazine: $42 USD
q ACM Digital Library: $99 USD q Student Membership with ACM Digital Library plus
(must be an ACM member) Print CACM Magazine: $62 USD

q Join ACM-W: ACM-W supports, celebrates, and advocates internationally for the full engagement of women in
computing. Membership in ACM-W is open to all ACM members and is free of charge.
Priority Code: CAPP
Payment Information
Payment must accompany application. If paying by check
or money order, make payable to ACM, Inc., in U.S. dollars
Name or equivalent in foreign currency.

ACM Member # q AMEX q VISA/MasterCard q Check/money order

Mailing Address
Total Amount Due

Credit Card #
Exp. Date
ZIP/Postal Code/Country
q Please do not release my postal address to third parties

q Yes, please send me ACM Announcements via email Return completed application to:
q No, please do not send me ACM Announcements via email ACM General Post Office
P.O. Box 30777
Purposes of ACM New York, NY 10087-0777
ACM is dedicated to:
Prices include surface delivery charge. Expedited Air
1) Advancing the art, science, engineering, and Service, which is a partial air freight delivery service,
application of information technology is available outside North America. Contact ACM for
2) Fostering the open interchange of information more information.
to serve both professionals and the public Satisfaction Guaranteed!
3) Promoting the highest professional and ethics standards


1-800-342-6626 (US & Canada) Hours: 8:30AM - 4:30PM (US EST) acmhelp@acm.org
1-212-626-0500 (Global) Fax: 212-944-1318 acm.org/join/CAPP

Science | DOI:10.1145/3264625 Chris Edwards

Floating Voxels Provide

New Hope for 3D Displays
In search of holograms that can be viewed from any angle.

EW M OVI E S CE N E Shave had Even The Walt Disney Company, which points to an anticipated implementa-
such an effect on display- bought Lucasfilm and the distribution tion of having the 3D image seem to be
technology research and rights for the movie franchise in 2012, standing on an illuminated pedestal,
development as the droid is among those with engineers working similar to the game table on the Mil-
R2D2 projecting a three- on the idea. lennium Falcon that appears in a scene
dimensional (3D) image of Princess Two years ago, Daniel Joseph and later in Star Wars.
Leia pleading for help in 1977’s block- colleagues in entertainment giant Dis- The Disney system suffers from a
buster film Star Wars. Numerous en- ney’s Burbank, CA-based research and problem that is shared with similar
gineers have wondered just how they development operation filed for a pat- systems: the image is formed from an
might achieve that effect, of an image ent on a projector intended to display array of light sources fed through beam
you can see from any angle, in real life. floating 3D images. The U.S. patent splitters and mirrors some distance

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 11

behind the pedestal, which limits the his move away from holographic tech-
viewing angle to those looking toward nologies. In the first installment of the
the projection optics, and so cannot All volumetric movie series, protagonist Tony Stark
emulate the movies. displays to date share uses a 3D projector not just to visualize
Daniel Smalley, an assistant profes- the elements of his powered suit, but
sor of electrical and computer engineer- the same problem, also to create a virtual gauntlet around
ing at Brigham Young University, says, Smalley says. his hand.
“Like many in the holography field, I Smalley’s team overcame the need
felt that holograms would provide the “You don’t have to encapsulate their display by trap-
3D images of the future, but the annoy- the self-occlusion ping and moving a single dust-sized
ing issue is you have to be looking in the particle. The prototype uses an ul-
direction of the screen that generates to make objects traviolet laser taken from a Blu-ray
them. It’s counter to what you expect 3D that look realistic.” player to capture and move the piece
displays to do in the future.” of dust. A visible-light source tracks
Builders of volumetric displays that and illuminates it. Physicists have yet
can be viewed from any angle face their to develop a theory that fully explains
own challenge. “Fundamentally, you the process of such photophoretic
have the problem that photons will into the air and illuminate them. A team trapping, but it appears to rely on lo-
just keep traveling until they bounce led by John Howell, a professor of phys- cal heating from being struck by pho-
off something,” says V. Michael Bove, ics and optics based at the University of tons. Gas molecules hitting the hotter
principal research scientist and head Rochester, used cesium vapor to create surface acquire more kinetic energy
of the object-based media group at the the voxels in their experimental volumet- as they bounce off, pushing the par-
Massachusetts Institute of Technology. ric display; the cesium atoms glow where ticle away.
Systems such as the VX1 built by the light from two steerable lasers cross. Says Smalley, “On average it doesn’t
Australian company Voxon Photonics Yet in these displays, moving parts and work very well at all, but in the [statis-
use a fast-moving sheet to provide a re- poisonous particles need to be encapsu- tical] tails you see incredible behavior.
flective surface for photons. At a high- lated in a transparent dome or sphere. The particle just stays there. You can
enough speed, the sheet will seem to “What’s of increased interest is even blow on it gently. We had one par-
disappear, but bright lights bounced not have a display in the table but to ticle trapped in there for 15 hours. It
off it will persist to the viewer; the re- interact with it in a meaningful way. could have stayed for longer: we had to
sult is the illusion of a slightly translu- Volumetric displays do have this talk- switch the machine off.”
cent 3D object floating in space. Bove ing-head-in-a-jar character that works The particle’s composition seems
says the need to move the sheet at high against that. You have the sense that to be crucial. Smalley’s team settled
speed makes this an intrinsically noisy this imagery is bottled up,” Bove says. on black liquor—a by-product of the
option, and one likely to suffer from Smalley also sees interaction as key, paper-making process—after trying
mechanical wear. citing another Disney movie franchise, numerous candidates. “I do not be-
Another option is to disperse particles Iron Man, as additional inspiration for lieve we can say this is definitively the

ACM News

Hijacking the Cryptomine

The gold rush in cryptocurrencies The risk of being caught surreptitiously in the background; defeat cryptojacking.
has led cybercriminals to adopt cryptojacking is minimal; it is usually, until the browser session Legitimate uses of cryptojacking
new tactics. difficult to trace because of the is closed. Sometimes hackers are beginning to appear online. For
Cybersecurity provider anonymity of cryptocurrencies. Also, will launch a stealth “pop-under” instance, digital media outlet Salon
Symantec says the profitability of cryptojacking scripts do not damage window or a tiny one-pixel browser started a beta test early this year,
ransomware dropped in 2017 from computers or data, and nothing is to continue illicitly accessing a using Coinhive to mine the open
an average $1,017 in 2016 to $522 stolen (except processing power), so device’s processing power. source cryptocurrency Monero as an
per ransomware event. That’s why there is little incentive to follow Victims might be unaware they alternative to online advertising as
many cybercriminals have shifted up when an attack is discovered. have been cryptojacked. The effects a revenue stream. If a visitor has an
to using coin miners, software A common method of are mostly performance-related, ad blocker turned on when visiting
designed to mine cryptocurrencies. cryptojacking involves executing and include lags in computers’ Salon.com, they might see a prompt
Infecting the computing a JavaScript in a browser, stealing execution of commands, slower to either disable the ad blocker or
devices of others in order to amass resources from the user’s CPU, performance, and overheating. select a “suppress ads” option. The
the processing power needed to which are pooled with resources Most antivirus software latter choice allows Salon to put
mine cryptocurrencies is called from other cryptojacked devices and ad blockers can now detect readers’ unused computing power
cryptojacking. Symantec recently to mine cryptocurrencies. coin-mining software. Browser to use mining Monero while they
reported the detection of coin Browser-based cryptojacking extensions like No Coin or are visiting the site.
miners on endpoint computers doesn’t require a download, starts minerBlock, and JavaScript blockers — John Delaney is a freelance
had increased 8,500% in 2017. instantly, and works efficiently and like NoScript, can be installed to writer based in Queens, NY, USA.

12 COM MUNICATIO NS O F TH E ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0


best material. It seems unlikely that it ments are complex movement and dy-
is,” he says. namics, not super-high resolution,”
It is possible to produce freestand- “The general public Blundell argues.
ing volumetric images without inject- has for 40 years Smalley envisages applications
ing particles into the air. More than a where the user needs to inspect the
decade ago, Hidei Kimura, founder been seeing shape closely and move around it. The
and CEO of Japanese company Bur- cinematic depictions ability to produce mid-air streamers in
ton Inc., and Taro Uchiyama of Keio fluid-dynamics simulations and models
University found that when focused of physically of organs to help with planning medi-
on specific points, microsecond bursts impossible things, cal operations seem good examples.
of high-intensity infrared light could “A lot of 3D technologies can’t give you
cause air molecules to become glow- and when they do a strong spatial sense when you get up
ing plasma. Kimura envisaged the see what’s possible, close. With ours, you can,” he says.
technology being used to create levi- Bove says by looking closely at re-
tating signs above head height for use they’re disappointed.” quirements for target applications
in emergencies; the bursts would be and working with user-interface de-
intense enough to burn the hand of a signers, the developers of volumetric
user foolish enough to try to touch the displays can move from experiment to
glowing voxels. market more easily. “Can it be behind
Much shorter pulses could yield a age to a volume the size of a ping-pong a transparent barrier? Is it important
safer system. Yoichi Ochiai of the Uni- ball, and the results demonstrated so that it be viewable from any angle or
versity of Tsukuba and Kota Kumagai far are based on long-exposure images is 90 degrees OK? Is it acceptable for it
of the University of Utsunomiya in Ja- that took up to a minute to generate. to have moving parts?” he suggests as
pan showed at the ACM SIGGRAPH Says Barry Blundell, senior lecturer questions to be asked.
conference in 2015 the results of a pro- in computing at the University of Derby Developing volumetric technolo-
totype based on lasers that fire bursts in the U.K. and a researcher into volu- gies for specific applications may lead
no more than 100 femtoseconds long. metric displays since the late 1980s, to the problem of no individual market
According to Ochiai, users would “With the optical-trap display, I would being large enough to support research
simply get a tingling sensation from have to see images generated a lot and development, but such displays
touching the plasma voxels, though faster. The only way to do that is paral- look more technologically feasible,
users would need to be careful to not lelism; you’ve got to have more lasers Bove says. “The problem with the Leia
let their eyes get too close to the im- surrounding the display, and more par- display is that it needs all of the boxes
ages, as retinal damage is a distinct ticles. The problem could be that you to be ticked.”
possibility. Robert Stone, professor of need to have so much physical appara-
interactive multimedia systems at the tus that you lose the viewing freedom.”
Further Reading
University of Birmingham in the U.K., Smalley claims the technology ex-
says he has concerns over the eye form- ists to drive and illuminate a collection Smalley, D.E. et al
A Photophoretic-Trap Volumetric Display,
ing strong afterimages because of the of particles in the shape of the spatial
Nature, 553, pp486–490 (25 January 2018),
brightness of the plasma. light modulator, the same kind of de- doi:10.1038/nature25176
The plasma projector has the ad- vice as that used to research holograph-
Ochiai, Y., Kumagai, K., Hoshi, T., Rekimoto, J.,
vantage of being far more resistant to ic displays and optical computers. Bove Hasegawa, S., and Hayasaki, Y.
disturbance by moving hands than argues the laser and light-modulator Fairy Lights in Femtoseconds: Aerial and
the particle-based option. However, components needed for scaled-up dis- Volumetric Graphics Rendered by Focused
all volumetric displays to date have a plays are now relatively cheap. Femtosecond Laser Combined with
Computational Holographic Fields, ACM
common problem, Smalley says: “It Still, expectations may be set too high.
Transactions on Graphics, Volume 35, Issue
is like taking a bunch of fireflies and “The general public has for 40 years 2, (May 2016), doi:10.1145/2850414
organizing them into patterns. Every- been seeing cinematic depictions
Blundell, B.
thing looks like a ghost. You don’t have of physically impossible things, and On the Uncertain Future of the Volumetric
the self-occlusion to make objects that when they do see what’s possible, they 3D Display Paradigm, 3D Research, 8 (2)
look realistic. are disappointed,” says Bove. p11, doi:10.1007/s13319-017-0122-2
“We want to be able to take a point Smalley concedes, “At this stage, you Joseph, D.M., Smoot, L.S.,
and have it shine light in only one di- don’t have to be an expert to realize that Smithwick, Q.Y., and Ilardi, M.J.
rection. That would mean it begins to this isn’t the Princess Leia display you Retroreflector Display System for
look solid.” are looking for. But, if given the oppor- Generating Floating Image Effects, U.S.
Patent Application 2018/0024373 A1 (25
The lack of self-occlusion in the op- tunity to be developed further, I don’t January 2018)
tical-trap display is, for the moment, a think you would be disappointed.”
secondary issue. It is difficult to move Researchers may be trying too hard Chris Edwards is a Surrey, U.K.-based writer who reports
the single particle that flies around the to make fact out of fiction. “What some on electronics, IT, and synthetic biology

Brigham Young display any faster than of the people working on volumetrics
is possible today; that limits its cover- haven’t realized is that the key ele- © 2018 ACM 0001-0782/18/10 $15.00

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 13

Technology | DOI:10.1145/3264630 Samuel Greengard

Transient Electronics
Take Shape
Advances in materials science and chemistry are leading
to self-destructing circuits and transient electronics,
which could impact many fields.

N E O F T H E intriguing as- Mission: Possible
pects of the popular 1960s There are myriad possible uses for
television show “Mission self-destructing electronic circuits. For
Impossible” was the open- example, the technology would allow
ing sequence of every epi- farmers to place monitoring devices
sode, which featured a secret agent in a field and not have to worry about
listening to a recorded message about removing them later. Using different
an upcoming mission. At the end of the materials or combinations of materials
recording each week, the tape would that avoid toxic residue ranging from
sizzle, crackle, and disintegrate into a Tungsten to formulated polymers, the
heap of smoke and debris, ensuring no circuits would simply disintegrate at a
one else could access the top-secret in- certain point. The remaining material
formation it contained. would have little or no impact on the
Until recently, self-destructing elec- environment.
tronic systems remained within the The same technology also would let
realm of science fiction, but advances in doctors insert biomedical devices into
chemistry, engineering, and materials the human body to dispense medicine
science are finally allowing research- in a controlled way; in some cases, such
ers to construct circuits that break as with chemotherapy, such micro-
down on their own timetable. This targeting of cells could dramatically
includes systems that rely on conven- reduce side effects, and there would be
tional complementary oxide semicon- no need to surgically remove the device
ductor (CMOS) technology. The self-destructing audio tape of the at the end of treatment.
“Mission Impossible” television show
“The goal is to develop functional anticipated by decades the advent of Transient electronics could allow the
circuits that can operate for a period self-destructing electronics. military to deploy drones, robots, and
of time and then vaporize,” says Amit other electronic devices into the field
Lal, Robert M. Scharf 1977 Professor destructing circuits that could be used without the worry adversaries could re-
of Engineering in the Electrical and in smartphones, drones, and even in- cover them and benefit in any way.
Computer Engineering Department at side the human body. The environmental benefits of self-
Cornell University in Ithaca, NY, and While the technology is still in the destructing circuits are also obvious,
director of the university’s SonicMEMs early stages of development, it could considering tens of millions of tons of e-
lab. “It’s the Biblical ashes-to-ashes have a commercial impact within a few waste are generated every year, and toxic
concept applied to electronics.” years. For now, the biggest obstacles re- substances including mercury, lead,
The technology could reshape nu- volve around perfecting transient elec- cadmium and arsenic are not always
merous fields, including medicine, tronics and self-destructing circuits recycled, or completely destroyed dur-
agriculture, and the military. It could and scaling them for mass use. There’s ing incineration. In some cases, e-waste
also reduce environmental damage also a need to gain a deeper under- winds up in landfills, particularly in de-
caused by materials in semiconduc- standing of polymers and composite veloping nations. The resulting toxins
tors and electronics, which require re- materials, and to ensure these systems that leach into the soil, air, and water

cycling and too often wind up in land- fully vaporize without leaving traces of create health hazards that can result in
fills and water supplies. Already, Lal toxic chemicals. As Lal explains, “It’s neurological damage, reproductive dis-
and a team at Cornell have obtained not easy to design a circuit that works orders, and cancers.
a patent for water-soluble circuits perfectly and delivers a high level of New types of designs and encapsula-
that biodegrade without leaving toxic performance for a period of time, and tion layers will allow electronic systems
materials behind. Other researchers then make it vaporize in the desired formed with specialized materials to
at Northwestern University and the situation or at a precise moment, and operate in a stable, high-performance
University of Houston have built self- within a relatively short period of time.” manner for a prescribed period and

14 COMMUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0


then to degrade and disappear com-

pletely, at a molecular level, to biocom-
patible and environmentally compat-
ible end products. “The ability to reduce
even some electronic waste could be
technology Member
highly beneficial,” explains Ved Gund, a could reshape News
senior process engineer at Intel who col- numerous fields,
laborated with Lal on the development
of a destructible circuit while he was a as well as reducing USING BIG DATA
graduate student at Cornell. environmental When she had
her first class in
The common denominator among
all transient electronics is an abil- damage from programming
in college, “I
ity to make a device physically vanish materials in thought I was
the only person
through a controlled process, often
triggered by events based on external electronics. who had never
programmed before,” recalls
environmental cues. These could take Nuria Oliver, director of Data
the form of electronic signals, light, Science Research at
temperature, shock or pressure chang- multinational telecom company
Vodafone. Undeterred, she
es, and chemical processes (including wound up at the top of her class.
enzymes released by the human body). demic paper outlining how a partially That experience, she says,
It may mean programming different transient system with substrates built motivates her to this day.
Oliver is passionate about
functions into a device at different atop a thin and fragile electronic cir- inspiring more women to study
stages—essentially physically morph- cuit could be water soluble. The re- science, technology, engineering,
ing a system through an evolutionary search started Rogers and fellow sci- and math (STEM) topics in
process—or creating different devices entists down a path toward building school and pursuing careers in
research technology, as well as
within a device for a specific purpose. more sophisticated circuits and de- encouraging them to persevere
“You can achieve physical tran- vices using environmentally benign and not quit jobs in these fields.
sience in many different ways,” ex- end-products. Their focus has re- “Anyone—even if you have no
plains John Rogers, Louis Simpson volved primarily around military and experience—can do anything if
you apply yourself,” she says, and
and Kimberly Querrey Professor of Ma- medical applications, with the goal offers her experience as proof.
terials Science and Engineering, Bio- of developing circuits and other elec- Throughout her career,
medical Engineering, and Neurologi- tronics that self-destruct and leave no Oliver has been interested in
using artificial intelligence
cal Surgery at Northwestern University trace of their component materials. and machine learning to better
in Evanston, IL. This is important, he In 2017, Rogers and colleagues an- understand human behavior,
adds, because it allows transient elec- nounced more advanced ways to build with the goal of building
tronics and destructible circuits to be state-of-the-art silicon complementary technology that is meaningful in
peoples’ lives.
used in many different ways and in metal-oxide-semiconductor (CMOS) She received her undergraduate
many different environments, ranging foundries to produce high-performance, degree in electrical engineering
from harsh industrial conditions to in- water-soluble forms of electronics. and computer science from
the Universidad Politecnica
side the human body. of Madrid, Spain, in 1994.
Although the idea of producing Short Circuits After earning her Ph.D. in
transient electronics is nothing new, Researchers have continued to push Perceptual Intelligence from
the technology began to emerge over the boundaries of transient and self- the Massachusetts Institute
of Technology in 2000, Oliver
the last decade, and Rogers is one of destructive electronics and circuits. spent seven years with Microsoft
the pioneers in the field. In 2007, as For instance, Cunjiang Yu, Bill D. Cook Research in Redmond, WA, until
a member of the U.S. Defense Depart- Assistant Professor of Mechanical En- she was offered the opportunity
ment’s Defense Science Research gineering at the University of Houston to become the first female
scientific director at Telefonica
Council, he began collaborating in Texas, along with researchers in R&D, in Barcelona, Spain,
with the Defense Advanced Research China, have developed self-destruc- modeling human behavior from
Projects Agency (DARPA) on ways to tive electronics with copper, magne- mobile data.
In late 2016, Oliver was
produce electronics that can adopt a sium oxide, and indium gallium zinc also named chief data scientist
transient physical form. The thinking oxide supported on a polyanhydride for DataPop Alliance, an
at the time was simple, even if execut- substrate. Water vapor breaks down international non-profit
ing on the concept was extraordinarily the polymer substrate and eventually organization devoted to
leveraging big data to improve
difficult. “Ideally, you flip a switch or causes the electronic materials to dis- the world. Early last year, she
push a button remotely and the device solve. Yu’s research is significant be- joined Vodafone to lead its global
simply melts away, disintegrates, or cause it is the first known approach research agenda to analyze
mobile data to better understand
vanishes, rather than falling into the that directly utilizes the substrate as what people want and need from
hands of an adversary,” he explains. the mechanism triggering the dissolu- their mobile phones.
In 2009, Rogers published an aca- tion of the electronics. —John Delaney

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 15

The Cornell group, in conjunction “These projects require an interdisci-

with Honeywell Aerospace, has ex- plinary approach and experimentation
plored self-destructing technology by Rogers and his with a lot of different chemicals and
experimenting with a number of differ- research group materials,” Yu explains. “We are only
ent approaches, including systems that beginning to understand how to build
use liquids and signals to trigger the are designing these self-destructive electronics and
disintegration process. In one instance, a system to deliver engineer the desired systems.”
they created a circuit with microscopic Nevertheless, the field continues to
cavities of novel polymers contain- programmable advance and commercialization of the
ing sodium bifluoride and rubidium. drug doses to technology could take place within the
Exposing the shell to radio waves of a next few years. In the future, inexpen-
specific frequency triggers graphene- a specific part sive and disposable circuits could also
on-nitride micro-valves in the shell to of the body, then introduce new types of devices and sys-
open, allowing the alkali metals to oxi- tems used within the Internet of Things
dize and produce a thermal reaction naturally degrade (IoT). Low power requirements could
that causes an already thinned-out chip and disappear. support vast networks of connected cir-
to disintegrate and vaporize rapidly. cuits that could operate for years and
“The technique uses the metals in the pose no environmental hazard.
chips as an energy source. They are at- Says Rogers: “Transient electronics
tached to a special polymer that reacts are beginning to take shape in a tan-
to the heat,” Lal explains. surgeries. This involves using mixtures gible way. The technology will almost
The disintegration process is trig- of chemicals and polymers that cause certainly impact a wide range of areas
gered by a tiny block that measures disintegration and packing them into in the years to come.”
0.04 inches wide. After the electronics layers with electrodes that will trigger
disintegrate, the result is a fine powder the destruction process.
Further Reading
consisting of cesium and rubidium ox-
ides, sand-like particles from the sili- Materially There Gund, V., Ruyack, A., Camera, K.,
Ardanuc, S., Ober, C., and Lal, A. (2015).
con chip, and tiny flakes of carbon from Developing new types of circuits and Multi-modal graphene polymer interface
the graphene, along with the remaining electronics that self-destruct requires characterization platform for vaporizable
battery (the research team is also work- rethinking and redesigning semicon- electronics. 2015. 873-876. 10.1109/
ing on a way to make the battery vapor- ductors that have never been engineered MEMSYS.2015.7051098. https://www.
izable, too). “The project requires ongo- for anything other than maximum per- researchgate.net/publication/283633594_
ing research into polymers and how to formance over a desired lifespan, Gund characterization_platform_for_vaporizable_
optimize both mechanical and materi- says. Adding to the task: the design and electronics
als functions,” Intel’s Gund says. One engineering process can vary greatly, Gund, V., Ruyack, A., Camera, K., Ardanuc, S.,
area of particular interest is how to use depending on the desired performance Ober, C., and Lal, A. (2016).
flexible layers of a material substrate and results. A biomedical device may re- Transient Micropackets for Silicon Dioxide
to produce a circuit that operates like quire 10 weeks of high-performance op- and Polymer-Based Vaporizable Electronics.
1153-1156. 10.1109/MEMSYS.2016.7421840.
conventional silicon electronics, while eration before it is made to degrade and
using plastics and other materials that dissolve into the body, while a military publication/301709792_Transient_
can also be broken down or recycled us- device might be required to disintegrate micropackets_for_silicon_dioxide_and_
ing the vaporization process. in a matter of seconds. What’s more, de- polymer-based_vaporizable_electronics.
Meanwhile, Rogers and his research pending on the device and how it used, Chang, J., Fang, H., Bower, C.A., Song E.,
group have focused on engineering the trigger mechanism might vary. Yu, X., and Rogers, J.A.
a system that could wirelessly deliver Researchers continue to explore Materials and processing approaches for
foundry-compatible transient electronics.
programmable drug doses to a specific how different combinations of chemi- Proceedings of the National Academy of
part of the body, then naturally degrade cals and substances interact to pro- Sciences Jul 2017, 114 (28) E5522-E5529;
and disappear. This technology might duce a desired result, and how they DOI: 10.1073/pnas.1707849114. http://
be used to deliver medication post- can get to the point where there is www.pnas.org/content/114/28/E5522
surgery, for example. The challenge of little or no trace of the circuit or elec- Gao, Y., Zhang, Y., Wang, X., Sim, K.,
this approach, Rogers says, “is that we tronic component. So, far, most of the Liu, J., Chen, J., Feng, X., Xu, H., and Yu, C.
have to build a device that is very stable research has been conducted through Moisture-triggered physically transient
electronics. Science Advances, Sept. 1,
over a relevant time period but then is trial and error and testing different 2017: Vol. 3, no. 9, e1701222
ultimately completely unstable, in the combinations of materials together. In DOI: 10.1126/sciadv.1701222. http://
sense that it eventually vanishes with- the future, Yu says, machine learning advances.sciencemag.org/content/3/9/
out a trace.” The team is working to per- might also serve as a valuable tool for e1701222.full.
fect a silicon, magnesium, magnesium sorting through growing mountains
Samuel Greengard is an author and journalist based in
oxide, and silk circuit that dissolves in of data and discovering combinations West Linn, OR, USA.
the body in much the same way that that can be used for different types of
absorbable sutures vanish after minor circuits and in different situations. © 2018 ACM 0001-0782/18/10 $15.00

16 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0


Society | DOI:10.1145/3264627 Esther Shein

The Dangers of Automating

Social Programs
Is it possible to keep bias out of a social program
driven by one or more algorithms?

Applications of Robotic
anna Green Brown for an Caseworker Client
Process Automation (RPA) • Automating application INTAKE • Scheduling appointments for
example of a client who and cognitive technologies screening human services programs
fell through the cracks and across the life cycle of a • Automating verification • Addressing queries

• Predicting high-risk cases • Auto-filling application forms
lost social services benefits human services case • Automating eligibility
they may have been eligible for because
AI Technologies:
of a program driven by artificial intelli- Chatbot
gence (AI), and you will get an earful. RPA
There was the “highly educated and Machine learning

capable” client who had had heart fail-

ure and was on a heart and lung trans-
plant wait list. The questions he was SERVICE DELIVERY
presented in a Social Security benefits AND CASE
application “didn’t encapsulate his is- MANAGEMENT
sue” and his child subsequently did Client
• Remote diagnosis
not receive benefits. Service provider
“It’s almost impossible for an AI sys-
tem to anticipate issues related to the Caseworker
• Addressing queries
nuance of timing,’’ Green Brown says. • Automating client Caseworker
follow-up and Fraud detection
Then there’s the client who had to documentation
apply for a Medicaid recertification, • Automating redetermination
of eligibility
but misread a question and received a • Personalizing service delivery delinquency

denial a month later. “Suddenly, Med-

icaid has ended and you’re not getting which may not tell their whole story. hood of someone committing a future
oxygen delivered. This happens to old “The art of actual conversation is what crime, setting credit scores, and in fa-
people frequently,’’ she says. teases out information,’’ she says. A cial recognition systems. As automated
Another client died of cancer that human can tell something isn’t right systems relying on AI and machine
Green Brown says was preventable, but simply by observing a person for a learning become more prevalent, the
the woman did not know social service few minutes; determining why they trick, of course, is finding a way to en-
programs existed, did not have an edu- are uncomfortable, for example, and sure they are neutral in their decision-
cation, and did not speak English. “I whether it is because they have a hear- making. Experts have mixed views on
can’t say it was AI-related,” she notes, ing problem, or a cognitive or psycho- whether they can be.
“but she didn’t use a computer, so how logical issue. AI-based technologies can undoubt-
is she going to get access to services?” “The stakes are high when it comes edly play a positive role in helping hu-
Such cautionary tales illustrate what to trying to save time and money versus man services agencies cut costs, signif-
can happen when systems become trying to understand a person’s unique icantly reduce labor, and deliver faster
automated, the human element is re- circumstances,’’ Green Brown says. and better services. Yet taking the hu-
moved, and a person in need lacks a “Data is great at understanding who man element out of the equation can
support system to help them navigate the outliers are; it can show fraud and be dangerous, agrees the 2017 Deloitte

the murky waters of applying for gov- show a person isn’t necessarily getting report “AI-augmented human services:
ernment assistance programs like So- all benefits they need, but it doesn’t Using cognitive technologies to trans-
cial Security and Medicaid. necessarily mean it’s correct informa- form program delivery.”
There are so many factors that go tion, and it’s not always indicative of “AI can augment the work of case-
into an application or appeals process eligibility of benefits.” workers by automating paperwork,
for social services that many people There are well-documented ex- while machine learning can help case-
just give up, Green Brown says. They amples of bias in automated systems workers know which cases need urgent
can also lose benefits when a line of used to provide guidelines in sentenc- attention. But ultimately, humans are
questioning ends in the system, but ing criminals, predicting the likeli- the users of AI systems, and these sys-

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 17

tems should be designed with human Machines fall short in that they have
needs in mind,’’ the report states. That no “common sense,” so if a data error
means they first need to determine the is input, it will continue to apply that “Humans are better
biggest pain points for caseworkers, error, Etzioni says. Likewise, if there is than computers
and the individuals and families they a pattern in the data that is objection-
serve. Issues to factor in are what are able because the data is from the past at exploring those
the most complex processes; can they but is being used to create predictive grey areas around
be simplified; what activities take the models for the future, the machine will
most time and whether they can be not override it. the edges of problems.
streamlined, the report suggests. “It won’t say, ‘this behavior is racist or Computers are
Use of these systems is in the early sexist and we want to change that’; on the
stages, but we can expect to see a grow- contrary, the behavior of the algorithm is better at the
ing number of government agencies to amplify behaviors found in the data,’’ black-and-white
implementing AI systems that can au- he says. “Data codifies past biases.”
tomate social services to reduce costs Because machine learning systems decisions in
and speed up delivery of services, says seek a signal or pattern in the data, “we the middle.”
James Hendler, director of the Rensse- need to be very careful in the applica-
laer Institute for Data Exploration and tion of these systems,” Etzioni says. “If
Applications and one of the originators we are careful, there’s a great potential
of the Semantic Web. benefit as well.”
“There’s definitely a drive, as more To make AI and machine learning systems, since a lot of these models
people need social services, to bring systems work appropriately, many cog- make assumptions that don’t always
in any kind of computing automation nitive technologies need to be trained hold in practice. They felt it was im-
and obviously, AI and machine learn- and retrained, according to the De- portant to consider the possibility that
ing are offering some new opportuni- loitte report. “They improve via deep an algorithm could respond “I don’t
ties in that space,’’ Hendler says. learning methods as they interact with know” or “pass,” which led them to
One of the ways an AI system can be users. To make the most of their invest- think about the relationship between
beneficial is in instances in which some- ments in AI, agencies should adopt an a model and its surrounding system.
one seeking benefits needs to access agile approach [with software systems], “There is often an assumption in ma-
cross-agency information. For example, if continuously testing and training their chine learning that the data is a repre-
someone is trying to determine wheth- cognitive technologies.” sentative sample, or that we know exact-
er they can get their parents into a gov- David Madras, a Ph.D. student and ly what objective we want to optimize.”
ernment-funded senior living facility, machine learning researcher at the That has proven not to be the case in
there are myriad questions to answer. University of Toronto (U of T), believes many decision problems, he says.
“The potential of AI and machine learn- if an algorithm is not certain of some- Madras acknowledges the difficulty
ing is figuring out how to get people to the thing, rather than reach a conclusion, of knowing how to add fairness to (or
right places to answer their questions, it should have the option to indicate subtract unfairness from) an algo-
and it may require going to many places uncertainty and defer to a human. rithm. “Firstly, unfairness can creep
and piecing together information. AI can Madras and colleagues at U of T in at many points in the process, from
help you pull it together as one activity.” developed an algorithmic model that problem definition, to data collection,
One of the main, persistent prob- includes fairness. The definition of to optimization, to user interaction.”
lems these systems have, however, is fairness they used for their model is Also, he adds, “Nobody has a great
inherent bias, because data is input by based on “equalized odds,” which they single definition of ‘fairness.’ It’s a very
biased humans, experts say. found in a 2016 paper, “Equality of Op- complex, context-specific idea [that]
Just like “Murphy’s Law,” which portunity in Supervised Learning,” by doesn’t lend itself easily to one-size-
states that “anything that could go computer scientists from Google, the fits-all solutions.”
wrong, will,” Oren Etzioni, chief ex- University of Chicago, and the Univer- The definition they chose for their
ecutive officer of the Allen Institute sity of Texas, Austin. According to that model could just as easily be replaced
for Artificial Intelligence, says there’s paper, Madras explains, “the model’s by another, he notes.
a Murphy’s Law for AI: “It’s a law of false positive and false negative rates In terms of whether social services
unintended consequences, because should be equal for different groups systems can be unbiased when the al-
a system looks at a vast range of pos- (for example, divided by race). Intui- gorithm running them may have built-
sibilities and will find a very counter- tively, this means the types of mistakes in biases, Madras says that when mod-
intuitive solution to a problem.” should be the same for different types els learn from historical data, they will
“People struggle with their own bi- of people (there are mistakes that can pick up any natural biases, which will
ases, whether racist or sexist—or be- advantage someone, and mistakes that be a factor in their decision-making.
cause they’re just plain hungry,’’ he can disadvantage someone).” “It’s also very difficult to make an
says. “Research has shown that there The U of T researchers wanted to ex- algorithm unbiased when it is operat-
are [judicial] sentencing differences amine the unintended side effects of ing in a highly biased environment;
based on the time of day.” machine learning in decision-making especially when a model is learned

18 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0


from historical data, the tendency researchers stated. “However, what “The real danger is people over-
is to repeat those patterns in some we should be focusing on is design- trusting these ‘unbiased’ AI systems,”
sense,’’ Madras says. ing AI that delivers results that are in he says. “What I’m afraid of is most
Etzioni believes an AI system can line with peoples’ well-being. By ob- people don’t understand these issues
be bias-free even when bias is input, serving human reactions to various … and just will trust the system the way
although that is not an easy thing to outcomes, AI could learn through they trust other computer systems. If
achieve. An original algorithm tries to a technique called ‘cooperative in- they don’t know these systems have
maximize consistency with data, he verse reinforcement learning’ what these limitations, they won’t be look-
says, but that past data may not be the our preferences are, and then work ing for the alternatives that humans
only criteria. towards producing results consistent are good at.”
“If we can define a criterion and with those preferences.”
mathematically describe what it AI systems need to be held account-
Further Reading
means to be free of bias, we can give able, says Alexandra Chouldechova,
that to the machine,’’ he says. “The an assistant professor of statistics and Madras, D., Creager, E., Pitassi, T., and Zemel, R.
Learning Adversarially Fair and
challenge becomes describing formal- public policy at Carnegie Mellon Uni-
Transferable Representations, 17 Feb. 2018,
ly or mathematically what bias means, versity’s Heinz College of Information Cornell University Library, https://arxiv.org/
and secondly, you have to have some Systems and Public Policy. abs/1802.06309
adherence to the data. So there’s really “Systems fail to achieve their pur- Buolamwini, J. and Gebru, T.
a tension between consistency with the ported goals all the time,’’ Choul- Gender Shades: Intersectional Accuracy
data, which is clearly desirable, and be- dechova notes. “The questions are: Disparities in Commercial Gender
ing bias-free.” Why? Can it be fixed? Could it have Classification, Proceedings of Machine
Learning Research, 2018, Conference on
People are working so both consis- been prevented in the first place?
Fairness, Accountability and Transparency.
tency and being bias-free can be sup- “By being clear about a system’s in- http://proceedings.mlr.press/v81/
ported, he adds. tended purpose at the outset, transpar- buolamwini18a/buolamwini18a.pdf
For AI to augment the work of gov- ent about its development and deploy- Dovey Fishman, T., Eggers, W.D., and Kishnani, P.
ernment case workers and make social ment, and proactive in anticipating its AI-augmented human services: Using
programs more efficient is to couple impact, we can hopefully reach a place cognitive technologies to transform
the technical progress being made where there will be fewer adverse unin- program delivery, Deloitte Insights, 2017,
with educating people on how to use tended consequences.” https://www2.deloitte.com/insights/
these programs, Etzioni says. For the foreseeable future, Hen- intelligence-technologies-human-services-
“Part of the problem is when a hu- dler believes humans and computers programs.html
man just blindly adheres to the rec- working together will outperform ei- Zhao, J., Wang, T., Yatskar, M.,
ommendations of the system without ther one separately. For the partner- Ordonez, V., and Chang, K..
trying to make sense of them, and the ship to work, a human must be able Men Also Like Shopping: Reducing
system says, ‘It must be true,’ but if to understand the decision-making of Gender Bias Amplification using Corpus-
the machine’s analysis is one output the AI system, he says. level Constraints, University of Virginia.
Proceedings of the 2017 Conference on
and a sophisticated person analyzes “We currently teach people to take Empirical Methods in Natural Language
it, we find ourselves in the best of the data and feed it into AI systems to Processing, pages 2979–2989 Copenhagen,
both worlds.” get an ‘unbiased answer.’ That unbi- Denmark, Sept. 7–11, 2017. https://pdfs.
AI, he says, really should stand for ased answer is used to make predic- semanticscholar.org/566f/34fd344607693e
“augmented intelligence,” where tech- tions and help people find services,’’
nology plays a supporting role, he says. Hendler says. “The problem is, the 1569884054.1523294823
“Humans are better than com- data coming in has been chosen in
Tan, S., Caruana, R., Hooker, G., and Lou, Y.
puters at exploring those grey ar- various ways, and we don’t educate Auditing Black-Box Models Using
eas around the edges of problems,’’ computer or data scientists how to Transparent Model Distillation With
agrees Hendler. “Computers are bet- know the data in your database will Side Information, 17 Oct. 2017, Cornell
ter at the black-and-white decisions model the real world.” University Library,
in the middle.” This is certainly not a new prob-
The issue of transparency of algo- lem. Hendler recalls the famous case O’Neil, C.
rithms and bias was discussed at a of Stanislov Petrov, a Soviet lieuten- Weapons of Math Destruction. 2016.
Crown Random House.
November 2017 conference held by ant-colonel whose job was to monitor
the Paris-based Organization for Eco- his country’s satellite system. In 1983, Hardt, M., Price, E., and Srebro, N.
Equality of Opportunity
nomic Cooperation and Development the computers sounded an alarm in-
in Supervised Learning
(OECD). Although several beneficial so- dicating the U.S. had launched nu- October 11, 2016
cietal use-cases of AI were mentioned, clear missiles. Instead of launching a https://arxiv.org/pdf/1610.02413.pdf
researchers said the solution lies in ad- counterattack, Petrov felt something
dressing system bias from a policy per- was wrong and refused; it turned out Esther Shein is a freelance technology and business
writer based in the area of Boston, MA, USA.
spective as well as a design perspective. to be a computer malfunction. AI sci-
“Right now, AI is designed so as entists, says Hendler, should learn
to optimize a given objective,” the from Petrov. © 2018 ACM 0001-0782/18/10 $15.00

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 19

DOI:10.1145/3267352 Michael A. Cusumano

Technology Strategy
and Management
The Business of
Quantum Computing
Considering the similarities of quantum computing development
to the early years of conventional computing.

Laureate Richard
N 1 9 8 1 , N O BE L represent information equal to the es- computer needs multiple qubits to do
Feynman challenged the com- timated number of particles in the calculations, and at least 50 qubits to
puting community to build a known universe.21 To perform calcula- do anything useful.14 We might need
quantum computer. We have tions, qubits exploit superposition 4,000 to 8,000 entangled qubits to sur-
come a long way. In 2015, and “entanglement.” This refers to pass current encryption technology us-
McKinsey estimated there were 7,000 when two quantum systems (such as ing very large integers.3 Programming
researchers working on quantum com- an electron or a nucleus), once they in- the devices also requires specialized
puting, with a combined budget of $1.5 teract, become connected and retain a hardware design skills, not conven-
billion.20 In 2018, dozens of universi- specific correlation in their spin or en- tional software programing skills.3
ties, approximately 30 major compa- ergy states (which represent combina- Entangled qubits are difficult to
nies, and more than a dozen startups tions of 0 and 1), even if physically sep- use and scale because of another phe-
had notable R&D efforts.a Now seems arate. Entanglement makes it possible nomenon called “decoherence.” The
like a good time to review the business. for quantum bits to work together and specific correlations between quan-
How do quantum computers work? represent multiple combinations of tum states can dissipate over time,
Quantum computers are built around values simultaneously, rather than thus destroying the ability of qubits to
circuits called quantum bits or qubits. represent one combination at a time. explore multiple solutions simultane-
One qubit can represent not just 0 or 1 Once a calculation is finished, you ob- ously. A useful analogy is to think of
as in traditional digital computers, but serve the qubits directly as 0 or 1 values qubit outputs like smoke rings blown
0 or 1 or both simultaneously—a phe- to determine the solution, as with a from a cigar.14 The rings can represent
nomenon called “superposition.” A classical computer. information but disintegrate (lose
pair of qubits can represent four states, What are the technical hurdles? Qu- their “coherence”) quickly. Since en-
three qubits eight states, and so on. bits resemble hardwired logic gates tangled qubits have a small probabili-
N qubits can represent 2n bits of in- usually made of atomic particles and su- ty of taking on different values due to
formation, and even 300 qubits can perconductor materials chilled to near- external interactions, the computa-
absolute zero. A one-qubit system is not tions require another process to de-
a https://bit.ly/2OXEA5n so difficult to build, but a quantum tect and correct errors.

20 COM MUNICATIO NS O F TH E ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0


The D-Wave 2000Q chip, designed to run quantum computing problems, increases from 1,000 qubits to 2,000 qubits, allowing larger
problems to be run—increasing the number of qubits yields an exponential increase in the size of the feasible search space.

How many different ways are there to time. The answer that requires the low- more than 400 per year in 2016–2017.
build quantum computers? There are est energy represents the optimal solu- The U.S. leads with approximately 800
several competing technologies. D- tion.10 However, some critics note that total patents, three to four times the
Wave was founded in 1999 to accumu- D-Wave qubits do not all seem to work numbers from Japan and China. The
late patent rights in exchange for re- together or exhibit quantum entangle- company with the largest portfolio is
search grants.17 It has been funded ment, and may not operate faster than D-Wave, followed by IBM (which
mainly by venture capital, corporate conventional computers.4 started research in 1990) and then
investors such as Goldman Sachs, and Google and IBM, as well as startups Microsoft. IBM leads in annual pat-
more recently, Jeff Bezos and the CIA.13 such as Quantum Circuits and Righet- ent filings. At universities, the lead-
The company has focused on “adiabat- ti Computing, deploy a different logic- ers in patent applications are MIT,
ic quantum computing,” also known gate approach, using entangled elec- Harvard, Zhejiang (China), Yale, and
as “quantum annealing.” D-Wave used trons or nuclei.19 Xanadu, a Toronto Tsinghua (China).2
this approach to build a 28-qubit de- startup, uses photons.b Microsoft’s de- What are some applications where
vice in 2007 and has been marketing a sign relies on quasi-particles called quantum computers should excel?
2,000-qubit device since 2017. Each anyons. Arranged into “topological qu- Experts list mathematical problems that
D-Wave qubit is a separate lattice con- bits,” these resemble braided knots on require massive parallel computations
tained within a magnetic field of Jo- a string, with (theoretically) high levels such as in optimization and simula-
sephson Junctions (logic circuits made of stability and coherence. Microsoft tion, cryptography and secure commu-
of superconductor materials that ex- plans to build a device within five years nications, pattern matching and big-
ploit quantum tunneling effects) and and make it commercially available via data analysis, and artificial intelligence
couplers (which link the circuits and the cloud.1,16 and machine learning.

pass information). You program the Who leads in the patent race? Pat- D-Wave computers seem to generate
device by loading mathematical equa- ent-related publications have in- “good enough” solutions to complex
tions into the lattices. The processor creased from a handful in the 1990s to combinatorial optimization problems
then explores all possible solutions si- with many potential solutions. For ex-
multaneously, rather than one at a b https://bit.ly/2B04tP1 ample, in 2012, Harvard researchers

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 21

used a D-Wave computer to do com- still work closely with universities

plex simulations of protein molecule and national laboratories. There is
unfolding (useful in drug discov- Perhaps no consensus as to what is the best
ery).22 Since 2013, NASA and Google, the “killer app” technology or design. D-Wave led the
along with several universities, have first generation but its computers
been using D-Wave computers in will be quantum are technically limited and scientifi-
their joint Quantum AI Lab.7 The Lab encryption cally controversial. Although D-Wave
has explored Web search, speech rec- should survive as a niche player, IBM
ognition, planning and scheduling, and secure and Google seem more likely to domi-
and operations management. 9 Since communications. nate the next generation, with Micro-
2014, Northrup-Grumman has been soft and maybe a startup or two close
using D-Wave to simulate large-scale on their heels.12
software systems behavior (useful for
error detection).4 Volkswagen, BMW, References
1. Bisson, S. Inside Microsoft’s quantum computing
and Google are relying on D-Wave to world. InfoWorld (Oct. 17, 2017).
analyze the huge amounts of data tackle optimization and simulation 2. Brachman, S. U.S. leads world in quantum computing
patent filings with IBM leading the charge. IP
needed for self-driving cars. In 2017, problems. They cannot run Shor’s Watchdog (Dec. 4, 2017).
3. Bright, P. Microsoft makes play for next wave of
Volkswagen used a $15-million D-Wave algorithm, and so may not be not computing with quantum computing toolkit. Ars
computer accessed via the cloud to op- useful for cryptography or quantum Technica (Sept. 25, 2017).
4. Brooks, M. Quantum computers buyers’ guide: Buy one
timize the airport routes of 10,000 tax- communications. IBM, Google, and today. New Scientist (Oct. 15, 2014).
is in Beijing. The machine processed Microsoft, as well as several startups, 5. Campbell, F. Microsoft’s quantum computing
vaporware. Forbes.com (Dec. 18, 2017).
GPS data in seconds that would nor- are designing more general-purpose 6. Castellanos, S. Companies look to make quantum leap
mally take a computer 45 minutes. The devices, but these are still theoreti- with new technology. The Wall Street Journal (May
6, 2017).
programming took six months, how- cal, experimental, or small scale. 7. Choi, C. Google and NASA launch quantum computing
ever, and some experts doubt the re- For the business to progress faster, AI lab. MIT Technology Review (May 16, 2013).
8. Condon, S. Google takes steps to commercialize
sults, which have not been published more people need access to bigger quantum computing. ZDNet (July 17, 2017).
in a scientific journal.6,11 quantum computers so they can build 9. D-Wave Systems, Inc. D-Wave 2000Q system to be
installed at quantum artificial intelligence lab run
Perhaps the “killer app” will be better programming tools and test re- by Google, NASA, and Universities Space Research
quantum encryption and secure com- al-world applications. Toward this Association. Press Release (Mar. 13, 2017).
10. D-Wave Systems, Inc. Introduction to the D-Wave
munications. These applications uti- end, IBM has made small quantum quantum hardware; https://bit.ly/2FzstKS
lize an algorithm discovered in 1994 by computers available via the cloud and 11. Ewing, J. BMW and Volkswagen try to beat Google
and Apple at their own game. The New York Times
Peter Shor, formerly of Bell Labs and is heading toward bigger devices; us- (June 22, 2017).
now at MIT. Shor demonstrated how to 12. Grossman, L. Quantum leap. Time (Feb. 17, 2014).
ers have already run approximately 13. Guedim, Z. 11 Companies set for quantum leap in
use a quantum computer to factor very 300,000 experiments.12,15 Google has computing. EdgyLabs (Oct. 12, 2017).
14. Hardy, Q. A strange computer promises great speed.
large numbers. Entanglement also made its D-Wave computer available to The New York Times (Mar. 21, 2013).
makes it possible to have unbreakable researchers as a cloud service.8 Google is 15. Knight, W. Serious quantum computers are finally
here. What are we going to do with them? MIT
cryptographic keys across different lo- also designing bigger machines with a Technology Review (Feb. 21, 2018).
cations. Governments (the U.S. and different technology. Microsoft an- 16. Lee, C. How IBM’s new five qubit univeral quantum
computer works. Ars Technica (May 4, 2016).
China in particular) as well as compa- nounced in 2017 that it would offer up to 17. Linn, A. The future is quantum: Microsoft releases free
nies (AT&T, Alibaba, BT, Fujitsu, HP, 40 qubits via a simulator on the Azure preview of quantum development kit. (Dec. 11, 2017);
Huawei, Mitsubishi, NEC, Raytheon, cloud. Microsoft has also created a 18. MacCormack, A., Agrawal, A., and Henderson, R.
and Toshiba, among others) have been quantum programming language called D-Wave systems: Building a quantum computer.
Harvard Business School Case #9-604-073 (Apr.
pursuing these applications.c China Q# and integrated this with Visual 2004), Boston, MA.
seems especially advanced.18 Studio.3,d However, Microsoft has not yet 19. Matthews, O. How China is using quantum physics
to take over the world and stop hackers. Newsweek
Do quantum computers represent a built physical devices and the program- (Oct. 30, 2017).
new general-purpose computing “plat- ming language may be completely spe- 20. Metz, C. Yale professors race Google and IBM to the
first quantum computer. The New York Times
form?” No. Quantum computers are cific to its architecture.5 (Nov. 13, 2017).
special-purpose devices that exploit In short, quantum computing still 21. Palmer, J. Here, there, and everywhere: Quantum
technology is beginning to come into its own.
quantum phenomena for massively resembles conventional computing The Economist (May 20, 2018).
22. Veritasium. How does a quantum computer work?
parallel computations. They are not circa the late 1940s and early 1950s. (June 17, 2013); https://bit.ly/1ApDtjk
suited to everyday computing tasks We have laboratory devices and some 23. Wang, B. Dwave adiabatic quantum computer used by
Harvard to solve protein folding problems. Next Big
that require speed, precision, and commercial products and services, Future (Aug. 16, 2012).
ease of use at low cost. The compet- but mostly from one company. We
ing technologies also seem useful for have incompatible architectures still Michael A. Cusumano (cusumano@mit.edu) is a
different applications, and so multi- in the research stage, with different professor at the MIT Sloan School of Management and
founding director of the Tokyo Entrepreneurship and
ple types of quantum computers may strengths and weaknesses. All the Innovation Center at Tokyo University of Science.
persist, splitting potential applica- machines require specialized skills
tion ecosystems. D-Wave computers to build and program. Companies The author thanks Ganesh Vaidyanathan for his comments.

c https://bit.ly/2OXEA5n d https://bit.ly/2B4SMFg Copyright held by author.

22 COMM UNICATIO NS O F THE AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0


DOI:10.1145/3267354 Peter Swire

• Carl Landwehr, Column Editor

Privacy and Security

A Pedagogic
Cybersecurity Framework
A proposal for teaching the organizational, legal,
and international aspects of cybersecurity.

“ E A L ” C Y B E R S E C U R I T Y TO D AY
devotes enormous effort to
non-code vulnerabilities and
responses. The Cybersecu-
rity Workforce Frameworka
of the National Initiative for Cyberse-
curity Education lists 33 specialty ar-
eas for cybersecurity jobs. Ten of the
specialty areas primarily involve cod-
ing, but more than half primarily in-
volve non-code work (15 areas, in my
estimate) or are mixed (eight areas,
per my assessment).
This column proposes a Pedagogic
Cybersecurity Framework (PCF) for
categorizing and teaching the jumble
of non-code yet vital cybersecurity
topics. From my experience teach-
ing cybersecurity to computer sci-
ence and other majors at Georgia
Tech, the PCF clarifies how the var-
ied pieces in a multidisciplinary cy-
bersecurity course fit together. The
framework organizes the subjects
that have not been included in tra-
ditional cybersecurity courses, but
instead address cybersecurity man-
agement, policy, law, and interna-
tional affairs.
The PCF adds layers beyond the the “user layer.”b The framework pro- framework would benefit cybersecuri-
traditional seven layers in the Open posed here adds three layers—layer ty students, instructors, researchers,
Systems Interconnection model 8 is organizations, layer 9 is govern- and practitioners. Layers 8–10 clas-
(“OSI model” or “OSI stack”). Previ- ments, and layer 10 is international. sify vulnerabilities and mitigations

ous writers have acknowledged the This column explains how the new that are frequently studied by non-
possibility of a layer or layers beyond computer scientists, but are also
seven, most commonly calling layer 8 b Varying previous definitions of higher
critical for a holistic understand-
layers of the OSI Model are available at ing of the cybersecurity ecosystem by
a https://bit.ly/2McPRB3 https://en.wikipedia.org/wiki/Layer_8. computing professionals.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 23

Table 1. Vulnerabilities at each layer of the expanded OSI stack. The Abstraction Layers
of the OSI Model
As discussed in the column, for layers 8–10, “A” refers to vulnerabilities The PCF builds on the Open Systems
and risk mitigation arising within the organization or nation; “B” refers Interconnection model (OSI) stack fa-
to vulnerability and risk mitigation in relation with other actors at that miliar to most computer scientists. It
level; and “C” refers to other limits created by actors at that level.
treats the stack primarily as a concep-
Layer Vulnerability tual framework for organizing how we
1. Physical Cut the wire; stress equipment; wiretap understand computing systems, par-
ticularly in the security domain. The
2. Data link Add noise or delay (threatens availability)
OSI model describes abstraction layers
3. Network DNS and BGP attacks; false certificates that enable the student or practitioner
4. Transport Man in the middle to focus on where a problem may ex-
ist, such as the physical, network, or
5. Session Session splicing (Firesheep); MS SMB
application layer. While retaining the
6. Presentation Attacks on encryption; ASN-1 parser attack abstraction layers from the OSI model,
7. Application Malware; manual exploitation of vulnerabilities; SQL injection; buffer overflow the PCF does not emphasize the role
8. Organization A: Insider attacks; poor training or policies
of the OSI model as a standardizing
B: Sub-contractors with weak cybersecurity; lack of information sharing model. Instead, it broadens students’
C: Weak technical or organizational standards understanding by focusing attention
9. Government A: Laws prohibiting effective cybersecurity (for example, limits on encryption); on the critical domains that introduce
weak laws for IoT or other security well-documented and well-understood
B: Badly drafted cybercrime laws (for example, prohibiting security research)
C: Excessive government surveillance
risks from management, government,
and international affairs. I provide
10. International A: Nation-state cyberattacks
B: Lack of workable international agreements to limit cyberattacks supplemental materials online that
C: Supranational legal rules that weaken cybersecurity further discuss the relationship of the
(for example, some International Telecommunications Union proposals) PCF to the OSI model and expand other
points made in this column.c
As a conceptual framework for un-
derstanding computer systems, the
Table 2. The pedagogic cybersecurity framework. seven traditional layers apply intuitive-
ly to cybersecurity risks, as discussed
A: Risk Mitigation
by Glenn Surman in his 2002 article
Layer of the Within an
Expanded OSI Organization or B: Relations with C: Other Limits Protocol “Understanding Security Using the OSI
Stack Nation Other Actors from This Level Data Unit Model.”2 Surman concluded: “The most
8: Organization 8A: Internal 8B: Vulnerability 8C: Standards and Contracts critical thing you should take from this
policies or plans management in limits originating paper is that for every layer there are at-
of action to reduce contracts with from the private tacks being created, or attacks awaiting
risk within an other entities, sector (for
organization (for like vendors (for example, PCI DSS
activation as a result of poor defence.”
example, incident example, cyber- standard, led by Bob Blakley from Citicorp assisted with
response plans). insurance). the PCI Cyber these illustrations of vulnerabilities
Security Standards that exist at each of the seven layers,
and I have added vulnerabilities exist-
9: Government 9A: Laws that 9B: Laws that 9C: Government Laws ing at layers 8, 9, and 10.
govern what an govern how limits on its
individual or organizations own actions (for
As a way to introduce layers 8 through
organization can and individuals example, Fourth 10, each horizontal layer highlights im-
or must do (for interact (for Amendment, portant types of cybersecurity vulner-
example, HIPAA example, limits on illegal
abilities. At layer 8, organizations face
Security Rule). Computer Fraud searches).
and Abuse Act). a wide range of cyber-risks, and take
many actions to mitigate such risks. At
10: International 10A: Unilateral 10B: Formal 10C: Limits on Diplomacy
actions by one and informal nations that layer 9, governments enact and enforce
government relationship come from laws—good laws can reduce cyberse-
directed at one management with other nations curity risks, while bad laws can make
or more other other nations (for example, the
nations (for (for example, United Nations and
them worse. At layer 10, the interna-
example, U.S. the Budapest international law). tional realm, no one nation can impose
Cyber Command Convention’s its laws, but treaties or discussions with
launching a provisions about Russia and China, for instance, may im-
cyberattack on a cybercrime and
hostile nation). Mutual Legal prove cybersecurity. As shown in Table
c Supplementary materials on the framework
are available at https://bit.ly/2MJCrZq

24 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0


1, the vulnerabilities in these new layers applies to governments writing and

are further organized by institutional enforcing laws about cybersecurity.
form—whether the vulnerability arises I have often Layer 10 applies where there is no
within the organization (or nation), encountered government to issue laws. Study of
between organizations (or nations), or layer 10 thus includes both state
from other institutions at that layer. practitioners and non-state actors that have
In addition to categorizing vulner- (and researchers) transborder effects.
abilities, the PCF builds on another In the matrix, each of the three col-
aspect of the OSI model, the “protocol who believe “real” umns refines the sorts of institutions
data unit,” such as bits for the physi- cybersecurity making the decisions. For each layer,
cal layer, packets for the network layer, column A contains issues arising
and data for the application and other involves writing code. within the institution—the organiza-
top layers. These protocol data units tion or nation. Each “issue” identifies
“describe the rules that control hori- cyber vulnerabilities or mitigating
zontal communications,” within a sin- activities. Column B contains issues
gle layer of the OSI stack.d defined by relations with other actors
At layer 8, for organizations, I sug- The 3x3 Institutional Matrix at that level. Column C contains issues
gest the controlling rules come from Universities have traditionally studied where other limits arise from actors at
contracts. The much-cited law and eco- the three non-code layers in different the same layer of the stack.
nomics scholars Jensen and Meckling departments. In general, business This three-column approach be-
have defined corporations as a “nexus schools focus on managing compa- comes clearer as applied to layer 8, the
of contracts.”1 Contracts are the gover- nies and other organizations. Law organizational layer. Column A in-
nance structure for relations between schools are the experts in law. Inter- cludes cybersecurity activities within
corporations, such as data-use agree- national relations programs study a single organization. A company (or
ments between an organization and international affairs. These different other organization that faces cyber-
its contractors. Less intuitively for non- university departments are organized security attacks) takes numerous ac-
lawyers, contracts also govern arrange- based on the institutions they primar- tions to reduce cyber-risk. It develops
ments within a corporation, governing ily study: companies, laws, and trans- incident response plans and other
the roles and actions of the board of national institutions. internal policies, and trains its em-
directors, management, and employ- By contrast, my experience is that ployees. One way to conceptualize cell
ees. Contracts are thus the protocol computer scientists often group all 8A is to think of the responsibilities of
data unit for layer 8, providing the rules of these issues into the general term a CISO in managing cyber-risk within
within that layer. “policy.” Traditionally in computer the organization.
At layer 9, the controlling rules science, this soft realm of “policy” is Column B in layer 8 (cell 8B) con-
for government—the protocol data the generic term for everything not cerns the organization’s relations with
units—are laws. Governments enact expressed in machine language. But other actors. First, a company creates
and enforce laws, requiring actions public policy departments do not data-use agreements and other con-
from the organizations within the gov- intensively cover all aspects of man- tracts with vendors and other entities.
ernment’s jurisdiction. The interna- agement, law, and international re- Flawed management of these rela-
tional realm of layer 10 operates where lations, so the computer science use tions can expose a company to risk,
no binding law applies. Actors at layer of “policy” creates confusion for the such as if it hires a subcontractor to
10 interact through diplomacy (or lack other departments that increasingly manage systems or data and the con-
of diplomacy), such as negotiating a teach and research on cybersecurity. tractor does so badly. Another much-
cyber-related treaty, and sometimes The proposed framework matches the discussed aspect of cybersecurity is
through declared or undeclared war. typical departmental organization in information sharing between organi-
Put another way, the traditional universities, and provides a visual rep- zations, such as through an Informa-
seven layers concern protocols ex- resentation of the key dimensions for tion Sharing and Analysis Center.
pressed in machine language; layers what computer scientists have often The third column, cell 8C, concerns
8 to 10 concern protocols (contracts, simply called “policy.” other limits that originate in the pri-
laws, diplomacy) expressed in natural As an additional way to organize vate sector. The PCI DSS standard is a
language. The layers operate in a way the many non-code cybersecurity- well-known example, governing secu-
familiar from the OSI stack: organiza- concerns, the PCF employs a 3x3 ma- rity at the point of sale. This standard
tions at layer 8 select the applications trix that refines which institutions has a powerful effect on the cyberse-
at layer 7. Governments at layer 9 set are involved in each area of cyber- curity of millions of merchants. The
laws to govern organizations. Actions vulnerability or response. Table 2 contractual standard originates in the
at layer 10 affect the governments at portrays the matrix. In Figure 2, each private sector, led by the PCI Security
layer 9, and apply when no single gov- layer (row) is defined by the institu- Standards Council. If the standard is
ernment can set the law. tions that make decisions affecting designed and implemented well, then
cybersecurity. Layer 8 applies to orga- cybersecurity improves; if done badly,
d https://bit.ly/2x40Aoj nizations facing cyberattacks. Layer 9 cyber-risks and costs increase.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 25

Looking at layer 8 as a whole, the and management issues of how to de-

simple point is that overall cybersecu- sign and manage cybersecurity con-
rity significantly depends on how well The PCF provides a tracts: How should cybersecurity be
an organization handles risk within its parsimonious way to treated in outsourcing or insurance
organization (8A), its contracts and re- contracts? Cell 9A concerns legal and
lations with other actors (8B), and stan- identify and develop a political science issues of how laws
dards and norms that come from the response to a growing get drafted and implemented. Cell
private sector (8C). 10C calls on international relations
Governments, for purposes of the number of non-code expertise to discuss the role of supra-
PCF, create laws. Cell 9A contains cybersecurity risks. national institutions. Few individuals
laws that govern what an individual are expert in all of this literature. Re-
or organization can do. For instance, searchers can develop an issue list for
using U.S. examples for illustration, each cell, along with canonical read-
the HIPAA Security Rule sets require- ings to assign in general examinations.
ments for medical providers. As a dif- For cybersecurity practitioners, I
ferent example, consider legislation as the Budapest Convention’s provi- have often encountered practitioners
that would prohibit the use of strong sions about cybercrime and Mutual (and researchers) who believe “real”
encryption or require a backdoor. I Legal Assistance. More generally, cell cybersecurity involves writing code,
have opposed such legislation, but it 10B applies to the range of possible perhaps with some vague acknowl-
illustrates how a government law, ap- cooperation with other nations on cy- edgment of the need for “interdisci-
plying to each organization, can affect berattack or defense. plinary” study. The sheer volume of
cybersecurity risk. Finally, cell 10C applies to limits on issues identified in the 3x3 matrix
Cell 9B contains laws that govern nations that come from other nations. emphasizes the growing significance
how organizations and individuals For instance, some countries have of non-code issues—bad decisions in
interact. Some of the HIPAA require- proposed to set cybersecurity rules any part of the matrix can negatively
ments fit here, such as the business through the International Telecom- affect cybersecurity. As with the ex-
associate requirements of HIPAA that munications Union, associated with isting seven layers of the stack, orga-
govern contracts with outside parties. the United Nations. If such rules are nizations can identify their vulner-
An important example in cell 9B is the implemented, then supranational laws abilities by systematically examining
Computer Fraud and Abuse Act, the could govern cyber actions that have layers 8 to 10. Organizations can then
anti-hacking law that defines when it transborder effects. better identify and mobilize expertise
is criminal to access computer systems for these non-code cyber issues.
without authorization. Applying the Framework In sum, the PCF provides a parsi-
Whereas cells 9A and 9B primar- Adding layers 8, 9, and 10 to the OSI monious way to identify and develop
ily concern government laws affect- stack in the PCF brings important ad- a response to the growing number
ing the private sector, cell 9C applies vantages to the study and practice of of non-code cybersecurity risks. The
to government limits on govern- cybersecurity. I have personally expe- 3x3 matrix visually categorizes and
ment action. The limit on illegal rienced the framework’s usefulness in communicates the range of non-code
searches in the Fourth Amendment teaching cybersecurity at my own insti- cybersecurity issues. No longer can
is one example. More broadly, cell tution: my cybersecurity classes cover “real” cybersecurity refer only to tech-
9C concerns the controversial topic every topic mentioned in this column. nical measures. Instead, a large and
of government surveillance. Sur- The PCF provides students with invalu- growing amount of cyber-risk arises
veillance sometimes aids security, able context for how all the issues fit from problems at layers 8, 9, and 10.
such as when a criminal is detected, together, to ensure they understand Extending the stack to these 10 layers
and sometimes hurts security, such the “big picture.” The framework also results in an effective mental model
as when government actions create clarifies the scope of a cyber-curricu- for identifying and mitigating the full
backdoors or other vulnerabilities. lum. Some classes, for instance, focus range of these risks.
The international layer applies to ac- primarily on how a CISO or company
tions taken within one nation that are should manage a company’s risks References
1. Jensen, M.C. and Meckling, W.H. Theory of the firm:
intended to have cyber effects in other (layer 8). Others are mostly about in- Managerial behavior, agency costs and ownership
nations. Cell 10A concerns unilateral structure. Journal of Financial Economics 3, 4 (Oct.
ternational affairs (layer 10), perhaps 1976), 305-360.
actions by one government, such as with discussion of national cyberse- 2. Surman, G. Understanding security using the OSI
model. GSEC Practical Version 1.3 (Mar. 29, 2002);
the U.S. The government, for instance, curity laws (cell 9A). The PCF enables https://bit.ly/2BaJGrV.
may decide that U.S. Cyber Command program directors and students to con-
should launch a cyberattack on a hos- cisely describe the coverage of a cyber- Peter Swire (Peter.Swire@scheller.gatech.edu) is the
tile nation. security class or curriculum. Elizabeth & Tommy Holder Chair of Law and Ethics in the
Scheller College of Business and Associate Director for
Cell 10B involves relations with The 3x3 matrix clarifies a research Policy in the Institute for Information Security and Privacy
other nations, which is the main task agenda for those seeking to identify at Georgia Institute of Technology in Atlanta, GA, USA.
of diplomacy. There are formal trea- and mitigate non-code cyber prob-
ties that affect cybersecurity, such lems. For example, cell 8B raises legal Copyright held by author.

26 COM MUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0


DOI:10.1145/3267356 George V. Neville-Neil

Article development led by

Kode Vicious
The Obscene Coupling
Known as Spaghetti Code
Teach your junior programmers how to read code.
Dear KV,
Forgive me, for my ACM member-
ship has lapsed, and for my sins I
have been saddled with mentoring a
spaghetti coder.
I am working on a piece of new soft-
ware—greenfield for once—but with
stiff reliability requirements. My help-
er, a young, self-proclaimed “devop,”
aims to improve as a programmer, and,
unfortunately, this person got stuck
with me.
No matter how hard I constrain
the work I dole out, I just cannot stop
this helper from the obscene coupling
known as spaghetti code, all masquer-
ading under obsessive, perfect syntax.
We cannot even get into the hard reli-
ability aspects of the software, because
tangled messes that lint perfectly and
break opaquely just keep piling up.
After many approaches, each one
narrower in scope than the last, I mentality in implementation and Once upon a time, spaghetti code
have come down to doling out work open this person’s mind to engage was defined by the fact that it jumped
units that are constrained to writ- the actual problem at hand—what the all over the place without any rhyme
ing single, well-defined functions software does! or reason, but, as you say, you have
in a Python library, but even then I I do not want to botch this and pro- someone, who even when given a con-
am failing to keep this person from duce the next Darth Vader! strained contract such as single func-
needlessly chaining functions, si- Mr. Function Defines Form tions, is still able to make a plate of
lently mixing and transparently pasta of it.
passing data through multiple lay- Perhaps it is time to introduce the
ers of interfaces, and, most pain- Dear Function, idea of narrative to your Padawan.
fully, burying important error out- Well, at least you didn’t mention goto, Code, as I have pointed out countless
put in ways we all know too well as the root of much of the spaghetti code times, is a form of communication

spaghetti code. of my well-spent youth. Yes, KV was between the people who write and
Assuming this apprentice is will- once young, but because of program- maintain it and is only incidentally
ing and eager, how can one go about mers such as your ward, he has never executable on a machine, which we
breaking this fundamental coupling looked young or beautiful. call a computer. I cannot seem to say

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 27

can all be in one main() function. I

recommend that you find a few such
The concept programs—well written, well com-
of simple narrative mented, and that do one thing and do
it well. Then make your Padawan read
can be applied them and explain them to you.
to code. KV has extolled the virtues of read-
ing good code as a way of learning to
write good code, and for young read-
ers, short programs are best. Even
though you are working on greenfield
code—a rarity in our industry—there
this often enough, clearly, because must be some scripts or code lying
I say it a lot. Someday I will lose my about that you do not hate and that
voice, and the people I am screaming extol the virtues you wish to instill in
at will finally think they will get some this apprentice. The most important
Advertise with ACM! peace; but if that ever happens, I have part of any of these programs is that
a recorded version I can play through they do one thing, they do it clearly,
a megaphone. and it is obvious to even the most
Reach the innovators Communication is just a fancy inexperienced programmer what is
and thought leaders word for storytelling, something that going on. Find that code, explain its
humans have probably been doing beauty, and then make them extend
working at the since before we acquired language. and maintain it.
cutting edge Unless you are an accomplished sur- Since you both are working on the
realist, you tell a story by starting at same code base, you also have ample
of computing the beginning, then over the course opportunity for leadership by showing
and information of time expose the reader to more this person how you code. You must do
of the details, finally arriving at the this carefully or the junior programmer
technology through end where, hopefully, the reader ex- will think you are pulling rank, but,
ACM’s magazines, periences a satisfying bit of closure. with a bit of gentle show and tell, you
The goal of the writer (or coder) is to can get your Padawan to see what you
websites form in the mind of the reader the are driving at. This human interaction
and newsletters. same image the writer had. That is is often difficult for those of us who
the process of communication, and prefer to spend our days with seeming-
it does not matter if it is prose, pro- ly logical machines. Mentorship is the
◊◆◊◆◊ gram, or poetry—at the end of the ultimate test of leadership and com-
day, if the recipient of our message passion, and I really hope you do not
has no clue what we meant, then all wind up sliced in half on the deck of a
Request a media kit was for naught. planet-smashing space station.
with specifications Of course, as many brilliant writers KV
have proven over time, clear narrative
and pricing: is not entirely necessary, but let’s just
Related articles
stick with the clear narrative meta- on queue.acm.org
Ilia Rodriguez phor for code, rather than claiming
Human-KV Interaction
we should write an accounting system
+1 212-626-0686 based on Naked Lunch. I mean, I would
Kode Vicious
acmmediasales@acm.org enjoy it, but would it work? Only the
Reading, Writing, and Code
Mugwumps would know. Diomidis Spinellis
The concept of simple narrative https://queue.acm.org/detail.cfm?id=957782
can be applied to code in the follow- A Conversation with Steve Bourne, Eric
ing way. We are trying to write down Allman, and Bryan Cantrill
the steps that are required to do a https://queue.acm.org/detail.cfm?id=1413258
particular job with a machine in such
a way that when other readers come George V. Neville-Neil (kv@acm.org) is the proprietor of
Neville-Neil Consulting and co-chair of the ACM Queue
upon the narrative (code)—which is editorial board. He works on networking and operating
usually thrust upon them with a bug systems code for fun and profit, teaches courses on
various programming-related subjects, and encourages
list as long as a baby’s arm—they are your comments, quips, and code snips pertaining to his
able to pick up the story wherever they Communications column.

choose. For a short program, some-

thing less than 100 lines, the narrative Copyright held by author.

28 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0


DOI:10.1145/3183558 Jean-François Abramatic, Roberto Di Cosmo, and Stefano Zacchiroli

Building the Universal
Archive of Source Code
A global collaborative project for the benefit of all.

fabric that binds our personal
and social lives, embodying a
vast part of the technologi-
cal knowledge that powers
our industry and fuels innovation. Soft-
ware is a pillar of most scientific research
activities in all fields, from mathematics
to physics, from chemistry to biology,
from finance to social sciences. Soft-
ware is also an essential mediator for ac-
cessing any digital information.
In short, a rapidly increasing part of
our collective knowledge is embodied
in, or dependent on, software artifacts.
Our ability to design, use, understand,
adapt, and evolve systems and devices
on which our lives have come to depend
relies on our ability to understand,
adapt, and evolve the source code of
the software that controls them.
Software source code is a precious, Source code is spread around a variety attention to software safety, security,
unique form of knowledge. It can be of platforms and infrastructures that we reliability, and traceability. But un-
readily translated into a form execut- use to develop and/or distribute it, and like other scientific fields, we lack
able by a machine, and yet it is human software projects often migrate from large-scale research instruments for
readable: Harold Abelson wrote “Pro- one to another: there is no universal enabling massive analysis of all the
grams must be written for humans to catalog that tracks it all. available software source code.
read,”1 and source code is the preferred Software can be deleted, corrupted, As computer scientists and profes-
form for modification of software arti- or misplaced. What’s even more worry- sionals, it is our duty, responsibility,
facts by developers.3 Quite differently ing, in recent years we have seen major and privilege to build a shared infra-
from other forms of knowledge, we code forges shut down, endangering structure that answers these needs.
have grown accustomed to use version- hundreds of thousands of publicly Not just for our community, not just
control systems that trace source code available software projects at once.6 for the technical and scientific com-
development, and provide precious in- We clearly need a universal archive munity, but for society as a whole.
sight into its evolution. As Len Shustek of software source code. Software Heritagea is an initiative
puts it, “Source code provides a view The deep penetration of software launched at Inria—the French Institute

into the mind of the designer.”4 in all aspects of our world brings for Research in Computer Science and
And yet, we have not been taking along failures and risks whose po- Automation—precisely to take up this
good care of this precious form of tential impact is growing. Users now
knowledge. understand the need for an organized a See https://www.softwareheritage.org

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 29

mission. While a full article detailing and enables full deduplication (mas-
our approach is available online,2 we sively reducing storage costs), integrity
focus here on the challenges raised by We are at a unique checking, and tracking of reuse across
the three main goals: collecting, pre- turning point in all software projects at the file level.
serving, and sharing the source code But it also poses novel challenges
of all the software ever written. the history when it comes to efficiently indexing
of computer science and querying its contents.
There are various kinds of source code. and technology. Sharing
Some is current, actively developed, The raw material that Software Heritage
and technically easy to make available; collects must be properly organized
some other is legacy source code that to ease its fruition. On top of the infor-
must be painfully retrieved from offline mation captured by version-control
media. Some is open, and free for all to ness need to keep software closed fades systems, we need metadata describing
read and reuse; some is closed behind away, a focused search (that requires a the software and means to classify the
proprietary doors. Software Heritage’s costly and dedicated effort) can succeed millions of harvested projects, written
ambition is to collect it all. in recovering and liberating its source in one of the thousands of known pro-
For current, open source code, we code, growing our software commons. gramming languages.e We need to ex-
need an automated process to harvest all Finally, by providing a means to tract and reconcile existing information
software projects, with all the available safely keep closed source software un- from many different sources, encoded
development history, from the many der embargo, much like what happens in one of the many different software
places where development and distri- already with software escrow, we may ontologies, and complete it using either
bution take place, like forges and pack- succeed in collecting current and future automatic tools or crowdsourcing.
age repositories. Yes, we really mean closed source, and be ready to liberate it We must also support the many use
harvesting everything available, with no when time comes, dispensing altogeth- cases that it enables. Programmers
a priori filtering. Because the value of er with costly technical recovery efforts. may want to search for specific project
an active software project will only be versions or code snippets to reuse, and
known in the future, and because stor- Preservation then browse them online or download
ing all present and future source code In the extensive literature on digital history-full source code bundles. Com-
can be done at a reasonable cost. preservation, it is now well established panies may want to access an API to
The technical challenge is to build that long-term preservation requires build applications that use the archive.
crawlers for each code-hosting plat- full access to the source code of the Researchers may want to access the
form, as there is no common protocol tools used for the task. Software Heri- whole corpus to perform big data opera-
available, and to develop adapters for tage uses and develops exclusively free tions or train machine learning models.
all version-control systems and package and open source software tools for We must carefully assess which
formats. It is a significant undertaking, building its archive. functionalities are generic enough to
but once a standard platform is avail- Also, replication and diversifica- be incorporated in the archive, and
able each of these crawlers and adapters tion are best practices to mitigate the which are so specific that they are best
can be developed in parallel. threats—from technical failures to implemented externally by third par-
For legacy, open source code, we legal and economic decisions—that ties. And there are of course legal and
need a crowdsourcing platform to endanger any long-term preservation ethical issues to be dealt with when
empower the volunteers that are will- initiative. Hence, we want to foster a redistributing parts­ —or all—of the
ing to help recover their preferred geographically distributed network of contents of the archive.
software artifacts. Guidelines must be mirrors, implemented using a variety
offered to help properly reconstruct of storage technologies, in different ad- Current Status
from the raw material the interesting ministrative domains, controlled by a Software Heritage is an active project
history that lies behind it, like in the plurality of institutions, and located in that has already assembled the largest
beautiful work that has been done for different jurisdictions. existing collection of software source
the history of Unix.5 Finally, preserving software source code. At the time of writing the Software
Closed software contains precious code also requires preserving the de- Heritage Archive contains more than
knowledge that is more difficult to re- velopment history of source code, four billion unique source code files and
cover. For example, the Computer His- which carries precious insights into one billion individual commits, gath-
tory Museumb and Living Computersc the structure of programs and also ered from more than 80 million pub-
have shown, in the case of the mythi- tracks inter-project relationships. licly available source code repositories
cal Alto system,d that once the busi- Software Heritage’s unique approach (including a full and up-to-date mirror
is to store all available source code of GitHub) and packages (including a
b See http://www.computerhistory.org/ and its revisions into a single Merkle full and up-to-date mirror of Debian).
c See http://www.livingcomputers.org/
d See http://xeroxalto.computerhistory.org and
DAG (Directed Acyclic Graph), shared Three copies are currently maintained,
http://www.livingcomputers.org/Discover/ among all software projects. This
News/ContrAlto-A-Xerox-Alto-Emulator.aspx data structure facilitates distribution e See http://hopl.info/

30 COMM UNICATIO NS O F THE AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0


including one on a public cloud.

As a graph, the Merkle DAG under-
the expertise from many fields of our
discipline; and building on a commu- Calendar
pinning the archive consists of 10 billion
nodes and 100 billion edges; in terms of
resources, the compressed and fully de-
nity that shares the vision.
As an open initiative, Software Heri-
tage strives to act as a host and a cata-
of Events
duplicated archive requires some 200TB lyzer for this community, and we are October 14–17
of storage space. These figures grow now calling for contributors to join UIST ‘18: The 31th Annual ACM
Symposium on User Interface
constantly, as the archive is kept up to forces and tackle the issues highlight-
Software and Technology,
date by periodically crawling major code ed in this Viewpoint, and the many oth- Berlin, Germany,
hosting sites and software distributions, ers that will arise along the way. A few Co-Sponsored: ACM/SIG,
adding new software artifacts, but never of these issues include: Contact: Patrick Baudisch,
Email: patrickbaudisch@gmx.
removing anything. The contents of the ˲˲ For the collection phase, we need
archive can already be browsed online, help recovering important software
or navigated via a REST API.f from the past and building adaptors for October 15–19
the many hosting platforms and source CCS ‘18: 2018 ACM SIGSAC
Conference on Computer and
Next Steps code distribution formats. Communications Security,
We are at a unique turning point in ˲˲ For the preservation phase, we Toronto, ON, Canada
the history of computer science and need resources to host mirrors, as well Sponsored: ACM/SIG,
technology. Looking backward, we see as contributors willing to try different Contact: David J.F. Lie,
Email: lie@eecg.toronto.edu
many important pieces of historical technologies for storing and mirroring
software that are lost, misplaced, or be- the archive. October 16–20
hind barriers. On the other hand, many ˲˲ For the sharing phase, help is ICMI ‘18: International
of our founding fathers are still here. needed to organize the contents, to Conference on
Multimodal Interaction,
They have the knowledge and the will build efficient indexing and querying Boulder, CO, USA
to share what is necessary to rebuild the mechanisms, and to develop applica- Sponsored: ACM/SIG,
full history of our discipline—a unique tions for specific domains. Contact: Sidney D’Mello,
Email: sidney.dmello@gmail.
opportunity that no other field of sci- We—technologists, engineers, com
ence or technology has ever offered. scientists, and IT professionals—have
Looking to the future, we see soft- a noble mission and a grand challenge: October 22–26
ware development skyrocketing. It is let’s work together to deliver on it. CIKM 2018: The 27th ACM
International Conference on
urgent to build the missing infrastruc- Information and Knowledge
ture and put in place the good practices 1. Abelson, H., Sussman, J., and Sussman, J. The Management,
necessary to ensure our entire software Structure and Interpretation of Computer Programs. Torino, Italy,
Preface by A.J. Perlis, MIT Press, 1985. Co-Sponsored: ACM/SIG,
commons will be properly collected 2. Di Cosmo, R. and Zacchiroli, S. Software Heritage: Why Contact: Alfredo Cuzzocrea,
and preserved. Every year that goes by and How to Preserve Software Source Code. iPRES 2017.
Email: cuzzocrea@si.dimes.
3. Free Software Foundation, Inc. The GNU General
without acting significantly increases Public License, Version 3, §1, 2007. unical.it
the backlog. 4. Shustek, L.J. What should we collect to preserve the
history of software. IEEE Annals of the History of October 22–26
By launching Software Heritage, Computing, 2006.
MM ‘18: ACM Multimedia
Inria has done the initial effort, creat- 5. Spinellis, D. A repository of Unix history and evolution.
Empirical Software Engineering, 2017.
ing the archive infrastructure, estab- 6. Squire, M. The Lives and Deaths of Open Source Code Seoul, Republic of Korea,
lishing an agreement with UNESCO, Forges. OpenSym, 2017. Sponsored: ACM/SIG,
Contact: Kyoung Mu Lee,
and assembling an initial group of
Jean-François Abramatic (Jean-Francois.Abramatic@ Email: kyoungmu@snu.ac.kr
supportersg and committed sponsors, inria.fr) is research director emeritus at Inria, the
including Microsoft, Intel, Société French Institute for Research in Computer Science and
October 28–31
Générale, Huawei, Google, GitHub, CHI PLAY ‘18: The Annual
Roberto Di Cosmo (roberto@dicosmo.org) is director of Symposium on Computer-
Qwant, Nokia Bell Labs, DANS, FossID, Software Heritage at Inria, and professor of computer Human Interaction in Play,
UQAM, and the University of Bologna. science at IRIF, University Paris Diderot. Melbourne, VIC, Australia
Now we need to move forward, and Stefano Zacchiroli (zack@upsilon.cc) is associate
Sponsored: ACM/SIG,
grow Software Heritage into an inter- professor of computer science at IRIF, University Paris
Contact: Florian Mueller,
Diderot, and CTO of Software Heritage at Inria. Email: floyd@floydmueller.com
national common infrastructure.
Four ingredients are key to the suc- Copyright held by authors. October 28–November 2
cess of our mission: raising awareness MSWIM ‘18: 21th ACM Int’l
of the importance of source code as a Conference on Modeling,
Analysis and Simulation of
first-class citizen in our cultural heri- Wireless and Mobile Systems,
tage; gathering the resources needed Montreal, QC, Canada
to create the infrastructure; leveraging Sponsored: ACM/SIG,
Watch the authors discuss Contact: Azzedine Boukerche,
their work in this exclusive Email: boukerch@site.uottawa.ca
Communications video.
f See https://archive.softwareheritage.org/
g See https://www.softwareheritage.org/support/ building-the-universal-archive-of-
testimonials/ source-code

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 31

DOI:10.1145/3209580 Jordi Cabot, Javier Luis Cánovas Izquierdo, and Valerio Cosentino

Are CS Conferences (Too)
Closed Communities?
Assessing whether newcomers have a more difficult time
achieving paper acceptance at established conferences.

ences is a key factor, albeit
controversial,3,4 in the dis-
semination of ideas and ca-
reer promotion in many ar-
eas of computer science. Therefore, it
is a major goal for every CS researcher.
However, many researchers believe
publishing in a top conference is
something reserved for the established
members of the conference commu-
nity. For newcomers, this is a tough
nut to crack. Indeed, when talking with
fellow researchers the assumed unspo-
ken truth is always the same: If you are
not one of “them,” you have no chance
to get “in” on your own.
If this were true, it would imply that
senior researchers wishing to change
fields during their research career may
have a difficult time doing so. And the
impact would be even more dramatic change it)? Our goal in this Viewpoint Computer Software category, for which
for junior researchers: they could only is to shed some light on these issues. we were able to find available data in
access top venues by going together with the DBLP dataset, the well-known on-
their supervisor, limiting their options Looking at the Data line reference for computer science
to make a name for themselves—exactly To assess whether it is actually true bibliographic information. The choice
the opposite of what evaluation commit- that newcomers have a difficult time of CORE as ranking system is based on
tees typically require from candidates. getting their papers accepted, we have its widespread use.
Indeed, candidates are supposed to evaluated the number of newcomer We have analyzed the conferences us-
show their ability to propose and de- papers (research papers where all au- ing a seven-year window (that is, an au-
velop valid research lines independently thors are new to the conference, that is, thor is considered new to a conference if
of their supervisor, even better if it is in none of the authors has ever published he or she has not published in that confer-
a slightly different research field and a paper of any kind in that same con- ence in the last seven years). We only count

hence in a different community. ference) in 65 conferences. The list of full papers in the main research track
But is it true that conferences are selected conferences corresponds to (since getting short papers, posters,
closed communities? Or is it just a the list of international CS conferences demos, and so forth is typically easier
myth spread by those that tried and in the CORE ranking,a 2015 edition, but it barely counts toward promotion).
failed? And if so, how do we change Results show that newcomers’ pa-
this situation (and do we really need to a https://bit.ly/2MnAncz pers are indeed scarce. Most confer-

32 COMMUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0


ences (88%) show a percentage of new- research lines in a new field, senior
comer papers under 40%. This value is researchers moving to a new research
significantly lower in top conferences, Satellite events interest, industrial researchers trying
with a median value of 14%. As specific play a positive role to disseminate their results …) able to
examples, well-regarded conferences bring a completely fresh perspective
show the following values: ICSE (5%), in the community. to the community.
OOPSLA (13%), ICFP (11%), RE (6%). The main challenge in opening up
We may be tempted to quickly dis- conferences comes from the fact that
miss these numbers by attributing the we do not really know the reasons why
low percentage of newcomers papers these numbers are so low. Do some
to a lack of newcomer submissions. potential newcomers refrain from sub-
While it is true that CS communities ties. Most likely, some readers believe mitting in the first place? Do they get
are shrinking (at least based on ACM this is exactly how things should be rejected more often than established
tables for SIG memberships), which and that newcomers must first learn authors? If the latter, are they being
could imply that the “newcomers pool” the community’s particular “culture” fairly rejected because their papers do
is smaller, our analysis suggests that (in the widest sense of the word, in- not follow the right structure, process,
newcomer paper submissions rep- cluding its topics of interest, pre- or evaluation standards? Or is there
resent at least one-third of the total ferred research methods, social be- a positive (unconscious) bias toward
number of submissions.b havior, vocabulary, and even writing known community members during
Additionally, for each conference, style) either by simply attending the the review phase?
we have also calculated the number of conference or warming-up publish- Narrowing down a root cause—or
semi-newcomer papers. A semi-new- ing in satellite events, before being causes—requires much more confer-
comer is a researcher that has never able to get their papers accepted in ence data to be publicly disclosed for
published in the main track but that the main research track. analysis. We hope this is a direction
has published before in other tracks We dare to disagree and argue that we will follow as a community. In the
(for example, a demo or a poster). Data the situation is getting to a point in meantime, we would like to suggest a
indicates publishing a paper as a semi- which is worth discussing how to few ideas we think are worth pursuing
newcomer is also difficult but slightly change course. The overall presence and that, most likely, should be com-
easier than doing so as a complete of newcomers decreases over time.2 bined in order to tackle this multifac-
newcomer. If you want to be part of a Besides, increasing travel and eco- eted challenge:
given community, it seems to pay off nomical restrictions make it difficult ˲˲ Open the review process. More
to first participate in that community to follow the (so far) “easier” path to and more conferences are adopting
via lesser competitive tracks or collo- enter the community, for example, a double-blind review model to avoid
cated satellite events. And the good many outsider researchers will not get bias. Its usefulness to avoid author
news is that, unsurprisingly, newcom- funded to attend a satellite event, pre- identification seems to be confirmed6
ers have reasonable chances of suc- venting them from learning the ropes but it is probably still fairly easy to
cess to get papers accepted in those of that particular community. spot whether the authors are at least
satellite events. Our data indicates the While closed communities have members of the community so bias
percentage of newcomer papers in sat- indeed some positive aspects (for ex- is not completely out of the question.
ellite events is over 30% in most confer- ample, a particular focus, a heritage We could go even further and aim for
ences and it frequently goes up to 50% to build upon, sense of security, and triple-blind reviews or, alternatively,
and over. Clearly, satellite events play a so forth) we believe they are now be- open reviews (where reviewers sign
positive role in the growth of the com- coming too closed. In our opinion, the reviews and/or reviews are later
munity. The full data is available, in- a healthier number for conferences released publicly).
cluding all conferences values and the would be having at least 25% of new- ˲˲ Identify and promote research
corresponding boxplot distributions comer papers in each edition. This topics with a lower entry barrier for
based on the conference rankings.c would ensure a continuous influx of newcomers either because they are new
fresh ideas and new members in the topics, and therefore not many people
Opening Up Conferences community among other benefits in the community work on them, or
We believe the data confirms CS con- of open communities such as better because they require less advanced
ferencesd behave as closed communi- diversity and inclusiveness. While skills/infrastructure.
junior researchers co-authoring a ˲˲ Increasing acceptance rates to

b This calculation requires access to the set of

paper with their supervisor for the have more slots available. This has
papers submitted and rejected. Since this data first time (in fact, the most common been proposed as a solution to the ran-
is not publicly available, this analysis was only path to enter a top conference) could domness of the peer-review system.8
done on the four conferences for which one of be considered new members as well, We could even decide to reserve a few
the authors acted as PC-Chair. we argue that conferences must also slots for newcomer papers. Obviously,
c https://bit.ly/2nCoWzU
d At least in the subarea we have evaluated (com-
make the effort to open up to com- this goes against the traditional con-
puter software category) but we believe results plete outsiders (including junior re- ference publication model and could
can be generalized to other areas. searchers trying to start independent trigger cascade effects on the role of

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 33

conferences but there is already a part in previous research works. M. Biryu-

of the community that challenges the kov et al.1 study individual newcomer
idea that very low acceptance rates authors, B. Vasilescu et al.9 and J.L.
are indeed good for us. ICSE’17 con- Cánovas et al.2 calculate just a coarse-
ference went to the extreme of limit- grained newcomers value as part of a
ing the number of papers to be sub- larger set of general metrics. We hope
mitted by a single author (restriction to trigger additional research and, es-
ACM Conference
Conference dropped in 2018 since the community pecially, general discussions around
felt it strongly discouraged collabora-
tion). Given that newcomers typically
the trade-offs of closing/opening up
more of our research communities5
Now Available via
Available via submit far fewer papers, this could with this Viewpoint.
help prevent established researchers We are aware this is a challeng-
Print-on-Demand! filling so many slots. An interesting ing process due to the leadership role
experience nevertheless worth being many conferences play in our research
reevaluated in the future (even if with system. And we acknowledge opening
Did you know that you can different “parameters”). up a conference is, in fact, an act of
˲˲ Adopt more journal-like review generosity. Unless we avoid the zero-
now order many popular
systems. Introducing revision cycles sum game of the current publication
ACM conference proceedings in a conference could help newcom- model (with a somehow fixed number
via print-on-demand? ers to fix obvious but easy-to-correct of slots to keep acceptance rates low)
mistakes that would otherwise force any explicit action to increase new-
a paper rejection. Even better, a roll- comer participation implies decreas-
Institutions, libraries and ing deadline, allowing submissions all ing our own chances to get published.
individuals can choose year-round (VLDB-style) would avoid Still, we believe the newcomers’ prob-
paper acceptance to be decided on the lem cannot be swept under the carpet
from more than 100 titles basis of the paper itself and not related any longer if we want to ensure we
on a continually updated to the others in order to avoid over the keep a vibrant and growing commu-
limit acceptance rates. nity in our research area.
list through Amazon, Barnes ˲˲ Start mentoring programs where
& Noble, Baker & Taylor, young researchers can pre-submit References
1. Biryukov, M. and Dong, C. Analysis of computer science
Ingram and NACSCORP: their work and get some advice (typi- communities based on DBLP. Lecture Notes in Computer
cally from former PC members) be- Science (including Subser. Lect. Notes Artif. Intell. Lect.
CHI, KDD, Multimedia, fore the actual submission. While
Notes Bioinformatics), 6273 LNCS (2010), 228–235.
2. Cánovas Izquierdo, J.L., Cosentino, V., and Cabot, J.

SIGIR, SIGCOMM, SIGCSE, mentoring may have a limited success Analysis of co-authorship graphs of CORE-ranked
software conferences. Scientometrics 109, 3 (Dec.
in getting the newcomers’ papers in 2016), 1665–1693.
SIGMOD/PODS, immediately, it could have a positive 3. Franceschet, M. The role of conference publications in
CS. Commun. ACM 53, 12 (Dec. 2010), 129.
and many more. long-lasting effect in speeding up the 4. Freyne, J. et al. Relative status of journal and
conference publications in computer science.
newcomer learning. Commun. ACM 53, 11 (Nov. 2010), 124.
˲˲ Draw ideas from other domains 5. Gebert, D. and Boerner, S. The open and the closed
For available titles and where they may face similar problems. corporation as conflicting forms of organization. J.
Appl. Behav. Sci. 35, 3 (Sept. 1999), 341–359.
ordering info, visit: For instance, in the open source com- 6. Le Goues, C. et al. Effectiveness of anonymization in
double-blind review. Commun. ACM 61, 6 (June 2018),
munity, many projects struggle to at-
librarians.acm.org/pod tract new contributors and have come
7. Steinmacher, I. et al. A systematic literature review
on the barriers faced by newcomers to open source
up with proposals to attract more peo- software projects. Inf. Softw. Technol. 59 (2015), 67–85.
ple.7 Examples (adapted to our field) 8. Vardi, M.Y. Divination by program committee.
Commun. ACM 60, 9 (Aug. 2017), 7.
would be to have a dedicated portal for 9. Vasilescu, B. et al. How healthy are software
newcomers clearly explaining how pa- engineering conferences? Sci. Comput. Program. 89,
PART C (2014), 251–272.
pers in the conference are evaluated,
showing examples of good papers (in Jordi Cabot (jordi.cabot@icrea.cat) is an ICREA
terms of style and structure), listing typ- Research Professor at the Universitat Oberta de
Catalunya (UOC), an Internet-centered open university
ical mistakes first submitters do based based in Barcelona, Spain.
on the experience of PC members, and Javier Luis Cánovas Izquierdo (jcanovasi@uoc.edu) is
so forth. And, importantly, encourag- a Postdoctoral Research Fellow at the Universitat Oberta
de Catalunya.
ing them to keep trying if they are not
Valerio Cosentino (vcosentino@uoc.edu) was a
initially successful—they may not be Postdoctoral Research Fellow at the Universitat Oberta
aware senior researchers also get many de Catalunya. Since September 2017, he is a software
developer at Bitergia, an open source company devoted to
papers rejected. offer software development analytics, part of the CHAOSS
Despite the number of works ana- project of the Linux Foundation.
lyzing co-authorship graphs, newcom-
ers metrics have been mostly ignored Copyright held by authors

34 COMMUNICATIO NS O F TH E ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

As part
part of
of its
its mission,
mission, ACM
ACM brings
brings broad
broad recognition
recognition to
to outstanding
outstanding technical
part of its mission,
professional ACM brings
achievements broad
in recognition
computing and to outstanding
information technical
and professional achievements in computing and information technology.
and professional achievements in computing and information technology.
ACM welcomes nominations for those who deserve recognition for their accomplishments. Please refer to the ACM Awards
ACM welcomes nominations for those who deserve recognition for their accomplishments. Please refer to the ACM Awards
website at https://awards.acm.org
ACM welcomes for who
nominations for those guidelines
how to nominate,
for theirlists of the members Please
accomplishments. of the 2018 Award
refer to Committees,
the ACM Awards
website at https://awards.acm.org for guidelines on how to nominate, lists of the members of the 2018 Award Committees,
and listings
website of past award recipientsfor
at https://awards.acm.org and their citations.
guidelines on how to nominate, lists of the members of the 2018 Award Committees,
and listings of past award recipients and their citations.
and listings of past award recipients and their citations.
Nominations are due January 15, 2019 with the exceptions of the Doctoral Dissertation Award (due October 31, 2018)
Nominations are due January 15, 2019 with the exceptions of the Doctoral Dissertation Award (due October 31, 2018)
and the ACM are
Nominations – IEEE
2019 Memorial HPC Fellowship
with the exceptions of the(due May 1,
Doctoral 2019).
Dissertation Award (due October 31, 2018)
and the ACM – IEEE CS George Michael Memorial HPC Fellowship (due May 1, 2019).
and the ACM – IEEE CS George Michael Memorial HPC Fellowship (due May 1, 2019).
A.M. Turing Award: ACM’s most prestigious award recognizes contributions of a technical nature which are of lasting and major technical
A.M. Turing Award: ACM’s most prestigious award recognizes contributions of a technical nature which are of lasting and major technical
A.M. Turing to the computing
Award: ACM’s most community.
prestigiousThe award
award is accompanied
recognizes by a prize
contributions of a of $1,000,000
technical with
nature financial
which are ofsupport
and majorbytechnical
importance to the computing community. The award is accompanied by a prize of $1,000,000 with financial support provided by Google.
importance to the computing community. The award is accompanied by a prize of $1,000,000 with financial support provided by Google.
ACM Prize in Computing (previously known as the ACM-Infosys Foundation Award in the Computing Sciences): recognizes an early-
ACM Prize in Computing (previously known as the ACM-Infosys Foundation Award in the Computing Sciences): recognizes an early-
ACMmid-career fundamental,
Prize in Computing innovativeknown
(previously contribution
as the in computingFoundation
ACM-Infosys that, through its depth,
Award in theimpact and broad
Computing implications,
Sciences): exemplifies
recognizes the
an early-
to mid-career fundamental, innovative contribution in computing that, through its depth, impact and broad implications, exemplifies the
greatest achievements
to mid-career fundamental,in theinnovative
The award carries a prize of
in computing $250,000.
that, through Financial support
its depth, impactisandprovided
broadby Infosys Ltd.exemplifies the
greatest achievements in the discipline. The award carries a prize of $250,000. Financial support is provided by Infosys Ltd.
greatest achievements in the discipline. The award carries a prize of $250,000. Financial support is provided by Infosys Ltd.
Distinguished Service Award: recognizes outstanding service contributions to the computing community as a whole.
Distinguished Service Award: recognizes outstanding service contributions to the computing community as a whole.
Distinguished Service Award: recognizes outstanding service contributions to the computing community as a whole.
Doctoral Dissertation Award: presented annually to the author(s) of the best doctoral dissertation(s) in computer science and
Doctoral Dissertation Award: presented annually to the author(s) of the best doctoral dissertation(s) in computer science and
engineering, and is accompanied
Doctoral Dissertation by a prize
Award: presented of $20,000.
annually to theThe Honorable
author(s) Mention
of the Award isdissertation(s)
best doctoral accompaniedinby a prize totaling
computer science$10,000.
engineering, and is accompanied by a prize of $20,000. The Honorable Mention Award is accompanied by a prize totaling $10,000.
Winning dissertations
engineering, are published
and is accompanied by in the ACM
a prize Digital Library
of $20,000. and the Mention
The Honorable ACM Books Series.
Award is accompanied by a prize totaling $10,000.
Winning dissertations are published in the ACM Digital Library and the ACM Books Series.
Winning dissertations are published in the ACM Digital Library and the ACM Books Series.
ACM – IEEE CS George Michael Memorial HPC Fellowships: honors exceptional PhD students throughout the world whose research
ACM – IEEE CS George Michael Memorial HPC Fellowships: honors exceptional PhD students throughout the world whose research
on high-performance
CS George Michael computing
HPC Fellowships:networking,
honorsstorage, or large-scale
exceptional PhD studentsdata throughout
analysis using thethe mostwhose
world powerful
focus is on high-performance computing applications, networking, storage, or large-scale data analysis using the most powerful
computers that are currently available.
focus is on high-performance computing The Fellowshipsnetworking,
applications, includes a $5,000
or large-scale data analysis using the most powerful
computers that are currently available. The Fellowships includes a $5,000 honorarium.
computers that are currently available. The Fellowships includes a $5,000 honorarium.
Grace Murray Hopper Award: presented to the outstanding young computer professional of the year, selected on the basis of a
Grace Murray Hopper Award: presented to the outstanding young computer professional of the year, selected on the basis of a
single recent major
Grace Murray Hopper technical
service contribution. The candidate
to the outstanding young must have professional
computer been 35 years ofof
theage or less
year, at theon
selected time
of a
single recent major technical or service contribution. The candidate must have been 35 years of age or less at the time the qualifying
single recent wasmajor made. A prize
technical of $35,000
or service accompanies
contribution. The the award. must
candidate Financial
been 35isyears
of agebyor Microsoft.
less at the time the qualifying
contribution was made. A prize of $35,000 accompanies the award. Financial support is provided by Microsoft.
contribution was made. A prize of $35,000 accompanies the award. Financial support is provided by Microsoft.
Paris Kanellakis Theory and Practice Award: honors specific theoretical accomplishments that have had a significant and demonstrable
Paris Kanellakis Theory and Practice Award: honors specific theoretical accomplishments that have had a significant and demonstrable
effect on the practice
Paris Kanellakis Theory of computing.
and PracticeThis award
Award: is accompanied
honors by a prizeaccomplishments
specific theoretical of $10,000 and isthat endowed
have hadby contributions
a significant and from the Kanellakis
effect on the practice of computing. This award is accompanied by a prize of $10,000 and is endowed by contributions from the Kanellakis
family, and
effect on thefinancial
practicesupport by ACM’sThis
of computing. SIGACT,
is accompanied by a prizeand the ACMand
of $10,000 SIGisProject
endowed Fund,byand individual from
contributions contributions.
the Kanellakis
family, and financial support by ACM’s SIGACT, SIGDA, SIGMOD, SIGPLAN, and the ACM SIG Project Fund, and individual contributions.
family, and financial support by ACM’s SIGACT, SIGDA, SIGMOD, SIGPLAN, and the ACM SIG Project Fund, and individual contributions.
Karl V. Karlstrom Outstanding Educator Award: presented to an outstanding educator who is appointed to a recognized educational
Karl V. Karlstrom Outstanding Educator Award: presented to an outstanding educator who is appointed to a recognized educational
Karl V. Karlstrom institution,
Educatorfor advancing new teaching
Award: presented methodologies,
to an outstanding effecting
educator whonew curriculum
is appointed todevelopment or expansion
a recognized educational
baccalaureate institution, recognized for advancing new teaching methodologies, effecting new curriculum development or expansion
in computer science
baccalaureate and engineering,
institution, recognized for or advancing
making a significant
new teachingcontribution to ACM’seffecting
methodologies, educational
new mission.
curriculum Thedevelopment
Karlstrom Award is
or expansion
in computer science and engineering, or making a significant contribution to ACM’s educational mission. The Karlstrom Award is
accompanied by a prize
in computer science andof $10,000. Financial
engineering, or makingsupport is provided
a significant by Pearson
contribution toEducation.
ACM’s educational mission. The Karlstrom Award is
accompanied by a prize of $10,000. Financial support is provided by Pearson Education.
accompanied by a prize of $10,000. Financial support is provided by Pearson Education.
Eugene L. Lawler Award for Humanitarian Contributions within Computer Science and Informatics: recognizes an individual or a group
Eugene L. Lawler Award for Humanitarian Contributions within Computer Science and Informatics: recognizes an individual or a group
EugenehaveL. made
Lawlera Award
significant contribution through
for Humanitarian the use within
Contributions of computing
Computer technology;
Science andthe award is intentionally
Informatics: recognizes defined broadly.or
an individual This
a group
who have made a significant contribution through the use of computing technology; the award is intentionally defined broadly. This
who haveendowed award is accompanied
made a significant contribution by a prizethe
through of $5,000, and alternates
use of computing with thethe
technology; ACM Policy
award is Award.
intentionally defined broadly. This
biennial, endowed award is accompanied by a prize of $5,000, and alternates with the ACM Policy Award.
biennial, endowed award is accompanied by a prize of $5,000, and alternates with the ACM Policy Award.
ACM – AAAI Allen Newell Award: presented to individuals selected for career contributions that have breadth within computer science,
ACM – AAAI Allen Newell Award: presented to individuals selected for career contributions that have breadth within computer science,
ACMthat bridgeAllen
– AAAI computer
Award:and other disciplines.
presented The $10,000
to individuals selectedprize is provided
for career by ACMthat
contributions and have
and by individual contributions.
within computer science,
or that bridge computer science and other disciplines. The $10,000 prize is provided by ACM and AAAI, and by individual contributions.
or that bridge computer science and other disciplines. The $10,000 prize is provided by ACM and AAAI, and by individual contributions.
Outstanding Contribution to ACM Award: recognizes outstanding service contributions to the Association. Candidates are selected
Outstanding Contribution to ACM Award: recognizes outstanding service contributions to the Association. Candidates are selected
based on the value
Outstanding and degree
Contribution of service
to ACM Award: overall.
recognizes outstanding service contributions to the Association. Candidates are selected
based on the value and degree of service overall.
based on the value and degree of service overall.
ACM Policy Award: recognizes an individual or small group that had a significant positive impact on the formation or execution of public
ACM Policy Award: recognizes an individual or small group that had a significant positive impact on the formation or execution of public
policy affecting
ACM Policy Award:computing or the
recognizes ancomputing
individual or community.
small groupThe biennial
that award is accompanied
had a significant positive impactby aon$10,000 prize. The
the formation next awardofwill
or execution be
policy affecting computing or the computing community. The biennial award is accompanied by a $10,000 prize. The next award will be
the 2019
policy award.computing or the computing community. The biennial award is accompanied by a $10,000 prize. The next award will be
the 2019 award.
the 2019 award.
Software System Award: presented to an institution or individuals recognized for developing a software system that has had a lasting
Software System Award: presented to an institution or individuals recognized for developing a software system that has had a lasting
influence, reflected
Software System in contributions
Award: presentedto toconcepts, in commercial
an institution acceptance,
or individuals recognized or both. A prize ofa$35,000
for developing software accompanies
system that the has award with
had a lasting
influence, reflected in contributions to concepts, in commercial acceptance, or both. A prize of $35,000 accompanies the award with
reflected provided by IBM. to concepts, in commercial acceptance, or both. A prize of $35,000 accompanies the award with
in contributions
financial support provided by IBM.
financial support provided by IBM.
ACM Athena Lecturer Award: celebrates women researchers who have made fundamental contributions to computer science. The award
ACM Athena Lecturer Award: celebrates women researchers who have made fundamental contributions to computer science. The award
ACM Athena a $25,000
Award: celebrates women researchers who have made fundamental contributions to computer science. The award
includes a $25,000 honorarium.
includes a $25,000 honorarium.
For SIG-specific Awards, please visit https://awards.acm.org/sig-awards.
For SIG-specific Awards, please visit https://awards.acm.org/sig-awards.
For SIG-specific Awards, please visit https://awards.acm.org/sig-awards.
Vinton G. Cerf, ACM Awards Committee Co-Chair John R. White, ACM Awards Committee Co-Chair
Vinton G. Cerf, ACM Awards Committee Co-Chair John R. White, ACM Awards Committee Co-Chair
Insup G. Cerf, ACM Awards
SIG Governing Committee
Board Co-Chair Liaison
Awards Committee John R. White,
Rosemary ACM Awards
McGuinness, ACMCommittee Co-Chair Liaison
Awards Committee
Insup Lee, SIG Governing Board Awards Committee Liaison Rosemary McGuinness, ACM Awards Committee Liaison
Insup Lee, SIG Governing Board Awards Committee Liaison Rosemary McGuinness, ACM Awards Committee Liaison
DOI:10.1145/ 3233231
transparency to humans and post hoc

Article development led by
explanations as competing concepts.
Throughout, the feasibility and desir-
ability of different notions of inter-
In machine learning, the concept of pretability are discussed. The article
interpretability is both important and slippery. questions the oft-made assertions that
linear models are interpretable and
BY ZACHARY C. LIPTON that deep neural networks are not.
Until recently, humans had a mo-

The Mythos
nopoly on agency in society. If you ap-
plied for a job, loan, or bail, a human
decided your fate. If you went to the
hospital, a human would attempt to

of Model
categorize your malady and recom-
mend treatment. For consequential
decisions such as these, you might de-

mand an explanation from the deci-
sion-making agent.
If your loan application is denied,
for example, you might want to under-
stand the agent’s reasoning in a bid to
strengthen your next application. If
the decision was based on a flawed
premise, you might contest this prem-
ise in the hope of overturning the de-
cision. In the hospital, a doctor’s ex-
planation might educate you about
models boast
S U P E R V I S E D M AC H I N E - L E A R N I N G your condition.
remarkable predictive capabilities. But can you trust In societal contexts, the reasons for a
your model? Will it work in deployment? What else decision often matter. For example, in-
tentionally causing death (murder) vs.
can it tell you about the world? Models should be not unintentionally (manslaughter) are
only good, but also interpretable, yet the task of distinct crimes. Similarly, a hiring de-
cision being based (directly or indirect-
interpretation appears underspecified. The ly) on a protected characteristic such as
academic literature has provided diverse and race has a bearing on its legality. How-
sometimes non-overlapping motivations for ever, today’s predictive models are not
capable of reasoning at all.
interpretability and has offered myriad techniques Over the past 20 years, rapid prog-
for rendering interpretable models. Despite this ress in machine learning (ML) has led
to the deployment of automatic deci-
ambiguity, many authors proclaim their models to be sion processes. Most ML-based deci-
interpretable axiomatically, absent further argument. sion making in practical use works in
Problematically, it is not clear what common properties the following way: the ML algorithm
is trained to take some input and pre-
unite these techniques. dict the corresponding output. For ex-
This article seeks to refine the discourse on ample, given a set of attributes char-
acterizing a financial transaction, an
interpretability. First it examines the objectives of ML algorithm can predict the long-
previous papers addressing interpretability, finding term return on investment. Given im-
them to be diverse and occasionally discordant. ages from a CT scan, the algorithm
can assign a probability that the scan
Then, it explores model properties and techniques depicts a cancerous tumor. The ML al-
thought to confer interpretability, identifying gorithm takes in a large corpus of (in-

36 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

put, output) pairs, and outputs a model As ML penetrates critical areas such diverse, suggesting that interpretability
that can predict the output corre- as medicine, the criminal justice sys- is not a monolithic concept but several
sponding to a previously unseen in- tem, and financial markets, the inabil- distinct ideas that must be disentan-
put. Formally, researchers call this ity of humans to understand these gled before any progress can be made.
problem setting supervised learning. models seems problematic. Some sug- This article focuses on supervised
Then, to automate decisions fully, gest model interpretability as a remedy, learning rather than other ML para-
one feeds the model’s output into but in the academic literature, few au- digms such as reinforcement learning
some decision rule. For example, thors articulate precisely what inter- and interactive learning. This scope de-
spam filters programmatically dis- pretability means or precisely how rives from the current primacy of su-
card email messages predicted to be their proposed solution is useful. pervised learning in real-world applica-
spam with a level of confidence ex- Despite the lack of a definition, a tions and an interest in the common
ceeding some threshold. growing body of literature proposes claim that linear models are interpre-
Thus, ML-based systems do not purportedly interpretable algorithms. table while deep neural networks are
know why a given input should receive From this, you might conclude that ei- not.15 To gain conceptual clarity, con-

some label, only that certain inputs are ther: the definition of interpretability is sider these refining questions: What is
correlated with that label. For exam- universally agreed upon, but no one has interpretability? Why is it important?
ple, shown a dataset in which the only bothered to set it in writing; or the term Let’s address the second question
orange objects are basketballs, an im- interpretability is ill-defined, and, thus, first. Many authors have proposed in-
age classifier might learn to classify all claims regarding interpretability of var- terpretability as a means to engender
orange objects as basketballs. This ious models exhibit a quasi-scientific trust.9,24 This leads to a similarly vexing
model would achieve high accuracy character. An investigation of the litera- epistemological question: What is
even on held out images, despite fail- ture suggests the latter. Both the objec- trust? Does it refer to faith that a model
ing to grasp the difference that actually tives and methods put forth in the liter- will perform well? Does trust require a
makes a difference. ature investigating interpretability are low-level mechanistic understanding

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 37

of models? Or perhaps trust is a subjec- you could consider the model’s com-
tive concept? plexity: Is it simple enough to be ex-
Other authors suggest that an inter- amined all at once by a human?
pretable model is desirable because it Other work has investigated so-
might help uncover causal structure in
observational data.1 The legal notion of What is trust? called post hoc interpretations. These
interpretations might explain predic-
a right to explanation offers yet another
lens on interpretability. Finally, some-
Is it simply tions without elucidating the mecha-
nisms by which models work. Exam-
times the goal of interpretability might confidence ples include the verbal explanations
simply be to get more useful informa-
tion from the model.
that a model will produced by people or the saliency
maps used to analyze deep neural net-
While the discussed desiderata, or perform well? works. Thus, human decisions might
objectives of interpretability, are di- admit post hoc interpretability despite
verse, they typically speak to situations the black-box nature of human brains,
where standard ML problem formula- revealing a contradiction between two
tions, for example, maximizing accu- popular notions of interpretability.
racy on a set of hold-out data for which
the training data is perfectly represen- Desiderata of
tative, are imperfectly matched to the Interpretability Research
complex real-life tasks they are meant This section spells out the various
to solve. Consider medical research desiderata of interpretability research.
with longitudinal data. The real goal The demand for interpretability arises
may be to discover potentially causal when a mismatch occurs between the
associations that can guide interven- formal objectives of supervised learn-
tions, as with smoking and cancer.29 ing (test-set predictive performance)
The optimization objective for most and the real-world costs in a deploy-
supervised learning models, however, ment setting.
is simply to minimize error, a feat that Typically, evaluation metrics re-
might be achieved in a purely correla- quire only predictions and ground-
tive fashion. truth labels. When stakeholders ad-
Another example of such a mis- ditionally demand interpretability,
match is that available training data you might infer the existence of ob-
imperfectly represents the likely de- jectives that cannot be captured in
ployment environment. Real environ- this fashion. In other words, because
ments often have changing dynamics. most common evaluation metrics for
Imagine training a product recom- supervised learning require only pre-
mender for an online store, where new dictions, together with ground truth, to
products are periodically introduced, produce a score, the very desire for an
and customer preferences can change interpretation suggests that some-
over time. In more extreme cases, ac- times predictions alone and metrics
tions from an ML-based system may calculated on them do not suffice to
alter the environment, invalidating fu- characterize the model. You should
ture predictions. then ask, what are these other objec-
After addressing the desiderata of tives and under what circumstances
interpretability, this article considers are they sought?
which properties of models might Often, real-world objectives are dif-
render them interpretable. Some pa- ficult to encode as simple mathemati-
pers equate interpretability with un- cal functions. Otherwise, they might
derstandability or intelligibility,16 just be incorporated into the objective
(that is, you can grasp how the models function and the problem would be
work). In these papers, understand- considered solved. For example, an al-
able models are sometimes called gorithm for making hiring decisions
transparent, while incomprehensible should simultaneously optimize pro-
models are called black boxes. But ductivity, ethics, and legality. But how
what constitutes transparency? You would you go about writing a func-
might look to the algorithm itself: tion that measures ethics or legality?
Will it converge? Does it produce a The problem can also arise when you
unique solution? Or you might look to desire robustness to changes in the
its parameters: Do you understand dynamics between the training and
what each represents? Alternatively, deployment environments.

38 COM MUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0


Trust. Some authors suggest inter- variables. You might hope, however, never encountered during training.
pretability is a prerequisite for trust.9,23 that by interpreting supervised learn- However, these are mistakes a human
Again, what is trust? Is it simply confi- ing models, you could generate hy- would not make, and it would be pref-
dence that a model will perform well? potheses that scientists could then erable that models not make these
If so, a sufficiently accurate model test. For example, Liu et al.14 empha- mistakes, either. Already, supervised
should be demonstrably trustworthy, size regression trees and Bayesian learning models are regularly subject
and interpretability would serve no neural networks, suggesting these to such adversarial manipulation. Con-
purpose. Trust might also be defined models are interpretable and thus bet- sider the models used to generate cred-
subjectively. For example, a person ter able to provide clues about the it ratings; higher scores should signify
might feel more at ease with a well-un- causal relationships between physio- a higher probability that an individual
derstood model, even if this under- logic signals and affective states. The repays a loan. According to its own
standing serves no obvious purpose. task of inferring causal relationships technical report, FICO trains credit
Alternatively, when the training and from observational data has been ex- models using logistic regression,6 spe-
deployment objectives diverge, trust tensively studied.22 Causal inference cifically citing interpretability as a mo-
might denote confidence that the mod- methods, however, tend to rely on tivation for the choice of model. Fea-
el will perform well with respect to the strong assumptions and are not widely tures include dummy variables
real objectives and scenarios. used by practitioners, especially on representing binned values for average
For example, consider the growing large, complex datasets. age of accounts, debt ratio, the number
use of ML models to forecast crime Transferability. Typically, training of late payments, and the number of
rates for purposes of allocating police and test data are chosen by randomly accounts in good standing.
officers. The model may be trusted to partitioning examples from the same Several of these factors can be ma-
make accurate predictions but not to distribution. A model’s generalization nipulated at will by credit-seekers. For
account for racial biases in the training error is then judged by the gap between example, one’s debt ratio can be im-
data or for the model’s own effect in its performance on training and test proved simply by requesting periodic
perpetuating a cycle of incarceration data. Humans exhibit a far richer capac- increases to credit lines while keeping
by over-policing some neighborhoods. ity to generalize, however, transferring spending patterns constant.
Another sense in which an end user learned skills to unfamiliar situations. Similarly, simply applying for new
might be said to trust an ML model ML algorithms are already used in accounts when the probability of ac-
might be if they are comfortable with these situations, such as when the en- ceptance is reasonably high can in-
relinquishing control to it. Through vironment is nonstationary. Models crease the total number of accounts.
this lens, you might care not only about are also deployed in settings where Indeed, FICO and Experian both ac-
how often a model is right, but also for their use might alter the environment, knowledge that credit ratings can be
which examples it is right. If the model invalidating their future predictions. manipulated, even suggesting guides
tends to make mistakes on only those Along these lines, Caruana et al.3 de- for improving one’s credit rating.
kinds of inputs where humans also scribe a model trained to predict prob- These rating-improvement strategies
make mistakes, and thus is typically ac- ability of death from pneumonia that may fundamentally change one’s un-
curate whenever humans are accurate, assigned less risk to patients if they also derlying ability to pay a debt. The fact
then you might trust the model owing had asthma. Presumably, asthma was that individuals actively and success-
to the absence of any expected cost of predictive of a lower risk of death be- fully game the rating system may inval-
relinquishing control. If a model tends cause of the more aggressive treatment idate its predictive power.
to make mistakes for inputs that hu- these patients received. If the model Informativeness. Sometimes, deci-
mans classify accurately, however, were deployed to aid in triage, these pa- sion theory is applied to the outputs of
then there may always be an advantage tients might then receive less aggres- supervised models to take actions in
to maintaining human supervision of sive treatment, invalidating the model. the real world. In another common use
the algorithms. Even worse, there are situations, paradigm, however, the supervised
Causality. Although supervised such as machine learning for security, model is used instead to provide infor-
learning models are only optimized where the environment might be ac- mation to human decision-makers, a
directly to make associations, re- tively adversarial. Consider the recent- setting considered by Kim et al.11 and
searchers often use them in the hope ly discovered susceptibility of convo- Huysmans et al.8 While the machine-
of inferring properties of the natural lutional neural networks (CNNs). The learning objective might be to reduce
world. For example, a simple regres- CNNs were made to misclassify images error, the real-world purpose is to pro-
sion model might reveal a strong as- that were imperceptibly (to a human) vide useful information. The most ob-
sociation between thalidomide use perturbed.26 Of course, this is not over- vious way that a model conveys infor-
and birth defects, or between smoking fitting in the classical sense. The mod- mation is via its outputs. However, we
and lung cancer.29 els both achieve strong results on train- might hope that by probing the pat-
The associations learned by super- ing data and generalize well when used terns that the model has extracted, we
vised learning algorithms are not guar- to classify held out test data. The cru- can convey additional information to a
anteed to reflect causal relationships. cial distinction is that these images human decision maker.
There could always be unobserved have been altered in ways that, while An interpretation may prove infor-
causes responsible for both associated subtle to human observers, the models mative even without shedding light on

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 39

a model’s inner workings. For exam- in reasonable time step through every fragile with respect to feature selection
ple, a diagnosis model might provide calculation required to produce a pre- and preprocessing. For example, the
intuition to a human decision maker diction. This accords with the common coefficient corresponding to the asso-
by pointing to similar cases in support claim that sparse linear models, as ciation between flu risk and vaccina-
of a diagnostic decision. In some cas- produced by lasso regression,27 are tion might be positive or negative, de-
es, a supervised learning model is more interpretable than dense linear pending on whether the feature set
trained when the real task more close- models learned on the same inputs. includes indicators of old age, infancy,
ly resembles unsupervised learning. Ribeiro et al.23 also adopt this notion or immunodeficiency.
The real goal might be to explore the of interpretability, suggesting that an Algorithmic transparency. A final no-
underlying structure of the data, and interpretable model is one that “can tion of transparency might apply at the
the labeling objective serves only as be readily presented to the user with level of the learning algorithm itself. In
weak supervision. visual or textual artifacts.” the case of linear models, you may un-
Fair and ethical decision making. At The trade-offs between model size derstand the shape of the error surface.
present, politicians, journalists, and and computation to apply a single pre- You can prove that training will con-
researchers have expressed concern diction varies across models. For exam- verge to a unique solution, even for pre-
that interpretations must be produced ple, in some models, such as decision viously unseen datasets. This might
for assessing whether decisions pro- trees, the size of the model (total num- provide some confidence that the mod-
duced automatically by algorithms ber of nodes) may grow quite large el will behave in an online setting re-
conform to ethical standards.7 Recidi- compared to the time required to per- quiring programmatic retraining on
vism predictions are already used to form inference (length of pass from previously unseen data. On the other
determine whom to release and whom root to leaf). This suggests simulatabil- hand, modern deep learning methods
to detain, raising ethical concerns. ity may admit two subtypes: one based lack this sort of algorithmic transpar-
How can you be sure predictions do not on the size of the model and another ency. While the heuristic optimization
discriminate on the basis of race? Con- based on the computation required to procedures for neural networks are de-
ventional evaluation metrics such as perform inference. monstrably powerful, we do not under-
accuracy or AUC (area under the curve) Fixing a notion of simulatability, the stand how they work, and at present
offer little assurance that ML-based de- quantity denoted by reasonable is sub- cannot guarantee a priori they will
cisions will behave acceptably. Thus, jective. Clearly, however, given the lim- work on new problems. Note, however,
demands for fairness often lead to de- ited capacity of human cognition, this that humans exhibit none of these
mands for interpretable models. ambiguity might span only several or- forms of transparency.
ders of magnitude. In this light, nei- Post hoc interpretability represents a
The Transparency ther linear models, rule-based systems, distinct approach to extracting infor-
Notion of Interpretability nor decision trees are intrinsically in- mation from learned models. While
Let’s now consider the techniques and terpretable. Sufficiently high-dimen- post hoc interpretations often do not
model properties that are proposed sional models, unwieldy rule lists, and elucidate precisely how a model works,
to confer interpretability. These fall deep decision trees could all be consid- they may nonetheless confer useful in-
broadly into two categories. The first ered less transparent than compara- formation for practitioners and end us-
relates to transparency (that is, how tively compact neural networks. ers of machine learning. Some com-
does the model work?). The second Decomposability. A second notion of mon approaches to post hoc
consists of post hoc explanations (that transparency might be that each part interpretations include natural lan-
is, what else can the model tell me?) of the model—input, parameter, and guage explanations, visualizations of
Informally, transparency is the oppo- calculation—admits an intuitive expla- learned representations or models,
site of opacity or “black-boxness.” It con- nation. This accords with the property of and explanations by example (for ex-
notes some sense of understanding the intelligibility as described by Lou ample, a particular tumor is classified
mechanism by which the model works. et al.15 For example, each node in a as malignant because to the model it
Transparency is considered here at the decision tree might correspond to a looks a lot like certain other tumors).
level of the entire model (simulatabili- plain text description (for example, all To the extent that we might consider
ty), at the level of individual compo- patients with diastolic blood pressure humans to be interpretable, this is the
nents such as parameters (decompos- over 150). Similarly, the parameters of sort of interpretability that applies. For
ability), and at the level of the training a linear model could be described as all we know, the processes by which hu-
algorithm (algorithmic transparency). representing strengths of association mans make decisions and those by
Simulatability. In the strictest sense, between each feature and the label. which they explain them may be dis-
a model might be called transparent if Note this notion of interpretability tinct. One advantage of this concept of
a person can contemplate the entire requires that inputs themselves be in- interpretability is that opaque models
model at once. This definition suggests dividually interpretable, disqualifying can be interpreted after the fact, with-
an interpretable model is a simple some models with highly engineered out sacrificing predictive performance.
model. For example, for a model to be or anonymous features. While this no- Text explanations. Humans often
fully understood, a human should be tion is popular, it should not be accept- justify decisions verbally. Similarly,
able to take the input data together ed blindly. The weights of a linear mod- one model might be trained to gener-
with the parameters of the model and el might seem intuitive, but they can be ate predictions, and a separate model,

40 COM MUNICATIO NS O F TH E ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0


such as a recurrent neural network interpretations is to render visualiza-

language model, to generate an expla- tions in the hope of determining qual-
nation. Such an approach is taken in a itatively what a model has learned.
line of work by Krening et al.12 They One popular method is to visualize
propose a system in which one model
(a reinforcement learner) chooses ac- While post hoc high-dimensional distributed repre-
sentations with t-distributed stochas-
tions to optimize cumulative dis-
counted return. They train another
interpretations tic neighbor embedding (t-SNE),28 a
technique that renders 2D visualiza-
model to map a model’s state repre- often do not tions in which nearby data points are
sentation onto verbal explanations of
strategy. These explanations are
elucidate precisely likely to appear close together.
Mordvintsev et al.20 attempt to ex-
trained to maximize the likelihood of how a model works, plain what an image classification
previously observed ground-truth ex-
planations from human players and
they may confer network has learned by altering the
input through gradient descent to en-
may not faithfully describe the agent’s useful information hance the activations of certain nodes
decisions, however plausible they ap-
pear. A connection exists between this for practitioners selected from the hidden layers. An
inspection of the perturbed inputs
approach and recent work on neural
image captioning in which the repre-
and end users of can give clues to what the model has
learned. Likely because the model
sentations learned by a discriminative machine learning. was trained on a large corpus of ani-
CNN (trained for image classification) mal images, they observed that en-
are co-opted by a second model to hancing some nodes caused certain
generate captions. These captions dog faces to appear throughout the
might be regarded as interpretations input image.
that accompany classifications. In the computer vision community,
In work on recommender systems, similar approaches have been ex-
McAuley and Leskovec18 use text to ex- plored to investigate what informa-
plain the decisions of a latent factor tion is retained at various layers of a
model. Their method consists of simul- neural network. Mahendran and Ve-
taneously training a latent factor model daldi17 pass an image through a dis-
for rating prediction and a topic model criminative CNN to generate a repre-
for product reviews. During training sentation. They then demonstrate the
they alternate between decreasing the original image can be recovered with
squared error on rating prediction and high fidelity even from reasonably
increasing the likelihood of review text. high-level representations (level 6 of
The models are connected because an AlexNet) by performing gradient
they use normalized latent factors as descent on randomly initialized pix-
topic distributions. In other words, la- els. As before with text, discussions
tent factors are regularized such that of visualization focus on form factor
they are also good at explaining the and appeal, but we still lack a rigorous
topic distributions in review text. The standard of correctness.
authors then explain user-item com- Local explanations. While it may be
patibility by examining the top words difficult to describe succinctly the full
in the topics corresponding to match- mapping learned by a neural network,
ing components of their latent factors. some of the literature focuses instead
Note that the practice of interpreting on explaining what a neural network
topic models by presenting the top depends on locally. One popular ap-
words is itself a post hoc interpreta- proach for deep neural nets is to com-
tion technique that has invited scruti- pute a saliency map. Typically, they
ny.4 Moreover note we have only spoken take the gradient of the output corre-
to the form factor of an explanation sponding to the correct class with re-
(that it consists of natural language), spect to a given input vector. For imag-
but not what precisely constitutes cor- es, this gradient can be applied as a
rectness. So far, the literature has mask, highlighting regions of the in-
dodged the issue of correctness, some- put that, if changed, would most influ-
times punting the issue by embracing ence the output.25,30
a subjective view of the problem and Note that these explanations of what
asking people what they prefer. a model is focusing on may be mislead-
Visualization. Another common ing. The saliency map is a local explana-
approach to generating post hoc tion only. Once you move a single pixel,

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 41

you may get a very different saliency related work in Bayesian methods,
map. This contrasts with linear mod- investigating case-based reasoning
els, which model global relationships approaches for interpreting genera-
between inputs and outputs. tive models.
Another attempt at local explana-
tions is made by Ribeiro et al.23 In this An inspection Discussion
work, the authors explain the decisions
of any model in a local region near a
of the perturbed The concept of interpretability ap-
pears simultaneously important and
particular point by learning a separate inputs can give slippery. Earlier, this article analyzed
sparse linear model to explain the deci-
sions of the first. Strangely, although
clues to what both the motivations for interpretabil-
ity and some attempts by the research
the method’s appeal over saliency the model community to confer it. Now let’s con-
maps owes to its ability to provide ex-
planations for non-differentiable mod-
has learned. sider the implications of this analysis
and offer several takeaways.
els, it is more often used when the ˲˲ Linear models are not strictly more
model subject to interpretation is in interpretable than deep neural networks.
fact differentiable. In this case, what is Despite this claim’s enduring popular-
provided, besides a noisy estimate of ity, its truth value depends on which
the gradient, remains unclear. In this notion of interpretability is employed.
paper, the explanation is offered in With respect to algorithmic transpar-
terms of a set of superpixels. Whether ency, this claim seems uncontrover-
or not this is more informative than a sial, but given high-dimensional or
plain gradient may depend strongly on heavily engineered features, linear
how one chooses the superpixels. models lose simulatability or decom-
Moreover, absent a rigorously defined posability, respectively.
objective, who is to say which hyper- When choosing between linear and
parameters are correct? deep models, you must often make a
Explanation by example. One post tradeoff between algorithmic transpar-
hoc mechanism for explaining the de- ency and decomposability. This is be-
cisions of a model might be to report cause deep neural networks tend to op-
(in addition to predictions) which erate on raw or lightly processed
other examples are most similar with features. So, if nothing else, the features
respect to the model, a method sug- are intuitively meaningful, and post hoc
gested by Caruana et al.2 Training a reasoning is sensible. To get compara-
deep neural network or latent variable ble performance, however, linear mod-
model for a discriminative task pro- els often must operate on heavily hand-
vides access to not only predictions engineered features. Lipton et al.13
but also the learned representations. demonstrate such a case where linear
Then, for any example, in addition to models can approach the performance
generating a prediction, you can use of recurrent neural networks (RNNs)
the activations of the hidden layers to only at the cost of decomposability.
identify the k-nearest neighbors based For some kinds of post hoc interpre-
on the proximity in the space learned tation, deep neural networks exhibit a
by the model. This sort of explanation clear advantage. They learn rich repre-
by example has precedent in how hu- sentations that can be visualized, ver-
mans sometimes justify actions by balized, or used for clustering. Consid-
analogy. For example, doctors often ering the desiderata for interpretability,
refer to case studies to support a linear models appear to have a better
planned treatment protocol. track record for studying the natural
In the neural network literature, world, but there seems to be no theo-
Mikolov et al.19 use such an approach to retical reason why this must be so.
examine the learned representations Conceivably, post hoc interpretations
of words after training the word2vec could prove useful in similar scenarios.
model. Their model is trained for dis- ˲˲ Claims about interpretability must
criminative skip-gram prediction, to be qualified. As demonstrated here,
examine which relationships the mod- the term interpretability does not ref-
el has learned they enumerate nearest erence a monolithic concept. To be
neighbors of words based on distanc- meaningful, any assertion regarding
es calculated in the latent space. Kim interpretability should fix a specific
et al.10 and Doshi-Velez et al.5 have done definition. If the model satisfies a form

42 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0


of transparency, this can be shown models and environments. This capa- via intuitive interaction. Massachusetts Institute of
Technology, Cambridge, MA, 2015.
directly. For post hoc interpretabil- bility, however, may come at the cost of 12. Krening, S., Harrison, B., Feigh, K., Isbell, C., Riedl,
ity, work in this field should fix a clear allowing models to experiment in the M. and Thomaz, A. Learning from explanations using
sentiment and advice in RL. IEEE Trans. Cognitive and
objective and demonstrate evidence world, incurring real consequences. Developmental Systems 9, 1 (2017), 41–55.
that the offered form of interpretation Notably, reinforcement learners 13. Lipton, Z.C., Kale, D.C. and Wetzel, R. Modeling missing
data in clinical time series with RNNs. In Proceedings
achieves it. are able to learn causal relationships of Machine Learning for Healthcare, 2016.
˲˲ In some cases, transparency may be between their actions and real-world 14. Liu, C., Rani, P. and Sarkar, N. 2006. An empirical study
of machine-learning techniques for affect recognition
at odds with the broader objectives of impacts. Like supervised learning, in human-robot interaction. Pattern Analysis and
AI (artificial intelligence). Some argu- however, reinforcement learning re- Applications 9, 1 (2006), 58–69.
15. Lou, Y., Caruana, R. and Gehrke, J. Intelligible models
ments against black-box algorithms lies on a well-defined scalar objective. for classification and regression. In Proceedings of the
18th ACM SIGKDD Intern. Conf. Knowledge Discovery
appear to preclude any model that For problems such as fairness, where and Data Mining, 2012, 150–158.
could match or surpass human abili- we struggle to verbalize precise defi- 16. Lou, Y., Caruana, R., Gehrke, J. and Hooker, G. Accurate
intelligible models with pairwise interactions. In
ties on complex tasks. As a concrete nitions of success, a shift of the ML Proceedings of the 19th ACM SIGKDD Intern. Conf.
example, the short-term goal of build- paradigm is unlikely to eliminate the Knowledge Discovery and Data Mining, 2013, 623–631.
17. Mahendran, A. and Vedaldi, A. Understanding
ing trust with doctors by developing problems we face. deep image representations by inverting them. In
transparent models might clash with Proceedings of the IEEE Conf. Computer Vision and
Pattern Recognition, 2015, 1–9.
the longer-term goal of improving 18. McAuley, J. and Leskovec, J. Hidden factors and
Related articles
health care. Be careful when giving on queue.acm.org
hidden topics: Understanding rating dimensions with
review text. In Proceedings of the 7th ACM Conf.
up predictive power that the desire Recommender Systems, 2013, 165–172.
Accountability in
for transparency is justified and not 19. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S. and
Algorithmic Decision Making Dean, J. Distributed representations of words and
simply a concession to institutional Nicholas Diakopoulos phrases and their compositionality. In Proceedings of
biases against new methods. the 26th Intern. Conf. Neural Information Processing
https://queue.acm.org/detail.cfm?id=2886105 Systems 2, 2013, 3111–3119.
˲˲ Post hoc interpretations can poten- 20. Mordvintsev, A., Olah, C. and Tyka, M. Inceptionism:
Black Box Debugging
tially mislead. Beware of blindly em- Going deeper into neural networks. Google AI Blog;
James A. Whittaker and Herbert H. Thompson https://ai.googleblog.com/2015/06/inceptionism-
bracing post hoc notions of interpret- https://queue.acm.org/detail.cfm?id=966807 going- deeper-into-neural.html.
21. Mounk, Y. Is Harvard unfair to Asian-Americans?
ability, especially when optimized to Hazy: Making It Easier to Build New York Times (Nov. 24, 2014); http://www.nytimes.
placate subjective demands. In such and Maintain Big-Data Analytics com/2014/11/25/opinion/is-harvard-unfair-to-asian-
Arun Kumar, Feng Niu, and Christopher Ré americans.html.
cases, one might—deliberately or 22. Pearl, J. Causality. Cambridge University Press,
not—optimize an algorithm to pres- https://queue.acm.org/detail.cfm?id=2431055 Cambridge, MA, 2009.
23. Ribeiro, M.T., Singh, S. and Guestrin, C. ‘Why should
ent misleading but plausible expla- I trust you?’ Explaining the predictions of any
nations. As humans, we are known to 1. Athey, S. and Imbens, G.W. Machine-learning
classifier. In Proceedings of the 22nd SIGKDD Intern.
Conf. Knowledge Discovery and Data Mining, 2016,
engage in this behavior, as evidenced methodsm 2015; https://arxiv.org/abs/1504.01132v1.
2. Caruana, R., Kangarloo, H., Dionisio, J. D, Sinha, U. and
in hiring practices and college admis- Johnson, D. Case-based explanation of non-case-
24. Ridgeway, G., Madigan, D., Richardson, T. and O’Kane,
J. Interpretable boosted naïve Bayes classification.
sions. Several journalists and social based learning methods. In Proceedings of the Amer.
In Proceedings of the 4th Intern. Conf. Knowledge
Med. Info. Assoc. Symp., 1999, 12–215.
scientists have demonstrated that 3. Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M.
Discovery and Data Mining, 1998, 101–104.
25. Simonyan, K., Vedaldi, A., Zisserman, A. Deep
acceptance decisions attributed to and Elhadad, N. Intelligible models for healthcare:
inside convolutional networks: Visualising image
Predicting pneumonia risk and hospital 30-day
virtues such as leadership or origi- classification models and saliency maps, 2013; https://
readmission. In Proceedings of the 21st SIGKDD
arxiv. org/abs/1312.6034.
nality often disguise racial or gender Intern. Conf. Knowledge Discovery and Data Mining,
26. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J.,
2017, 1721–1730.
discrimination.21 In the rush to gain Erhan, D., Goodfellow, I. and Fergus, R. Intriguing
4. Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J.L., Blei,
properties of neural networks, 2013; https://arxiv.org/
acceptance for machine learning and D.M. 2009. Reading tea leaves: how humans interpret
topic models. In Proceedings of the 22nd Intern.
27. Tibshirani, R. 1996. Regression shrinkage and selection
to emulate human intelligence, we Conf. Neural Information Processing Systems, 2009,
via the lasso. J. Royal Statistical Society: Series B:
should all be careful not to reproduce 5. Doshi-Velez, F., Wallace, B. and Adams, R. Graph-
Statistical Methodology 58, 1 (1996), 267–288.
28. Van der Maaten, L. and Hinton, G. Visualizing data
pathological behavior at scale. sparse lDA: A topic model with structured sparsity.
using t-SNE. J. Machine Learning Research 9 (2008),
In Proceedings of the 29th Assoc. Advance. Artificial
Intelligence Conf., 2015, 2575–2581.
29. Wang, H.-X., Fratiglioni, L., Frisoni, G. B., Viitanen,
Future Work 6. Fair Isaac Corporation (FICO). Introduction to model
M. and Winblad, B. Smoking and the occurrence
builder scorecard, 2011; http://www.fico.com/en/
There are several promising directions latest- thinking/white-papers/introduction-to-model-
of Alzheimer’s disease: Cross-sectional and
longitudinal data in a population-based study. Amer. J.
for future work. First, for some prob- builder- scorecard.
Epidemiology 149, 7 (1999), 640–644.
7. Goodman, B. and Flaxman, S. European Union
lems, the discrepancy between real-life regulations on algorithmic decision-making and
30. Wang, Z., Freitas, N. and Lanctot, M. Dueling network
architectures for deep reinforcement learning. In
and machine-learning objectives could a ‘right to explanation,’ 2016; https://arxiv.org/
Proceedings of the 33rd Intern. Conf. Machine Learning
be mitigated by developing richer loss 8. Huysmans, J., Dejaeger, K., Mues, C., Vanthienen,
48, 2016, 1995–2003.

functions and performance metrics. J. and Baesens, B. An empirical evaluation of the

comprehensibility of decision table, tree- and rule- Zachary C. Lipton (Twitter @zacharylipton or GitHub @
Exemplars of this direction include re- based predictive models. J. Decision Support Systems zackchase) is an assistant professor at Carnegie Mellon
search on sparsity-inducing regulariz- 51, 1 (2011), 141–154. University in Pittsburgh, PA, USA. His work addresses
9. Kim, B. Interactive and interpretable machine- diverse application areas, including medical diagnosis,
ers and cost-sensitive learning. Second, learning models for human-machine collaboration. dialogue systems, and product recommendation.
this analysis can be expanded to other Ph.D. thesis. Massachusetts Institute of Technology, He is the founding editor of the Approximately Correct
Cambridge, MA, 2015. blog and the lead author of Deep Learning—The Straight
ML paradigms such as reinforcement 10. Kim, B., Rudin, C. and Shah, J.A. The Bayesian Dope, an open source interactive book teaching deep
case model: A generative approach for case-based
learning. Reinforcement learners can reasoning and prototype classification. In Proceedings
learning through Jupyter notebooks.
address some (but not all) of the ob- of the 27th Intern. Conf. Neural Information Processing
Systems, Vol. 2, 1952–1960, 2014.
jectives of interpretability research by 11. Kim, B., Glassman, E., Johnson, B. and Shah, J. iBCM: Copyright held by owner/author.
directly modeling interaction between Interactive Bayesian case model empowering humans Publication rights licensed to ACM. $15.00.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 43
DOI:10.1145/ 3233239

Article development led by

The best careers are not defined

by titles or résumé bullet points.

The Secret
Formula for
the Right
Next Role and the one after that, and the one
after that.
When you are looking at the options
for your next role, there are smarter
choices that you can make. Here are
the most important factors to consider
when picking your next opportunity.

Pick a Goal, Not a Title

CHANGING JOB S — E S PE C I ALLYthe higher up you get A title looks good on a résumé, and
might pump up your ego a little bit, but
in your career—is a complex process. There are so making your job title a serious factor in
many factors to consider, and often the factors that your job search is a big mistake.
Your title is so much less important
stand out most are the ones that matter the least: than the work you do and the skills you
fancy titles, exciting projects, tempting promises of develop while in a role. Those hiring you
future success … for your next role will know that. They
might see that you were a VP in your last
But those factors that seem so valuable in the job, but if you don’t have any results or
moment are just that—they are momentary. Your skills to show for it, you won’t stand out
among the many other candidates who
career isn’t just about this one next step you are taking. were also VPs in their last jobs.
Your career is a journey that will last a long time. If you want to be truly successful,
It is smarter to invest in your long-term success. then your career path should be about
acquiring skills and accomplishments,
Focus on factors that will increase your career capital not just upgrading to shinier and fan-
and make you a more valuable hire in your next role, cier titles.

44 COMM UNICATIO NS O F THE AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

First of all, different titles mean Yes, I had to learn some TPM skills, ger picture for your career. Where do
different things in different compa- but what was truly valuable about that you want to be in 10 years? What is your
nies. I have been everything from VP job was the access it gave me. Because ultimate career goal?
and CTO at successful startups to of what my team was focused on, I got This is different for everyone.
CEO of my own company, but after to be in meetings with top executives Think about where you want to end
years of having executive-level titles, I who were running 1,000-plus-person up, and work backward from there.
took a role without one. teams. I was presenting to VPs who What skills do you need in order to
If I had rejected that job opportu- had decision-making power for a get there? What steps will you need to
nity because the title was lower than huge organization. take along the way?
any that I had had in the previous 10 I had a huge scope. Instead of be- Focusing on the short-term win
years, I would have missed out on ing siloed in one department where I of getting a fancy title or bigger pay-
one of the biggest, most life-chang- was the boss, I was able to get on the check is a mistake. If a job is not ac-
ing growth opportunities that I have radar of key leaders throughout the tively putting you into the situations
ever had. organization. I was able to gain influ- you need in order to grow or make the
Moreover, in that role, instead of be- ence and visibility; I saw the priorities right contacts, then it is not really the
ing a software engineer I was in the job for the whole company (not just my right choice. It will delay you getting
category of technical program manager department), which allowed me to where you need to go.
(TPM). I had never been a TPM before, align myself with the most important When you are looking at an opportu-
and to be honest, it was not a role I work being done. nity, consider whether this role will help
identified with. No one would describe I got to learn, and I gained visibility. I you level up your career. Ask yourself the
me as organized, and I didn’t have the built my network and got to know many following questions:
background skills; I write code and lead ˲˲ What skills do I still need to build in

people in the organization as a whole.

engineering teams. Over time, I earned even bigger influ- order to make progress toward my goals?
Even though the title was a demo- ence and control. And my title had noth- ˲˲ What benefits will the job afford
tion, and it was a job family that didn’t ing to do with it. me that maybe are not visible in the
fit, I still took the position because of When you are looking at different job description?
what I could gain from it. job opportunities, think about the big- ˲˲ Who will I meet?

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 45

˲˲ What is this job setting me up for? the future), but if it’s not, then you extra invested in that new hire doing
˲˲ What
will I have gained from this are just stuck. well once he or she joined the team?
role in two years, and are those gains For example, I have a friend who re- Even a minimal investment will have
valuable to me? ally wanted to work on machine learn- a psychological impact on your po-
ing, so he joined a team doing that type tential coworkers. If they meet you or
Pick People, Not Projects of work. For the 18 months he didn’t interview you, they will have already
Another easy trap to fall into when get to do anything related to machine invested some amount of time in you
picking your next job is to focus too learning, and instead was stuck writ- and will be more inclined to want to
much on the projects you think you ing deployment scripts and updates to see that investment rewarded.
will get to work on. data loaders—work that was much less ˲˲ You will not be “brand new” on
Of course, we all want to work on interesting to him than the project he your first day. As humans, we are
things that are interesting and excit- was on previously. naturally resistant to change and to
ing or that could make us rich and Projects are never guaranteed, so new people whom we know nothing
famous. The truth is projects get can- ensure you understand the specifics about. If you show up on your first
celed all the time. They change and and exactly what work you will get the day having met no one yet, you are
become less exciting. The roles within chance to do. Also, instead of think- a stranger; your coworkers are more
them change, and you could end up ing just about the work, I recommend likely to see you as an “outsider” tak-
doing legwork that is not actually very thinking also about whom you will be ing up space. Even a short meeting in
interesting or exciting to you. working with. advance will prime them to see you as
In college, I got a job working in Basing your decision on the people familiar the next time you see them.
a lab. I was so happy because I was you will be working with is one of the Plus, you will have some baseline
envisioning myself working on excit- best ways to pick a job. If you must knowledge about the team that can
ing experiments and getting my work choose between an exciting project help you fit in more quickly, as op-
published in major journals. While or a great team, always go for the posed to starting to learn about the
those exciting projects did happen great team. team culture after you have joined.
in this lab, I never got to do them. I Some 99% of my happiness in a job
ended up running the same experi- has to do with who my manager and Be Smart When You
ment day after day, collecting the coworkers are. I bet it is the same for Choose Your Next Role
same data over and over again. This you. You spend so much time at work; When you are searching for the next
is often what research is—you need if you work full time, you probably step in your career, don’t just think
to make sure any results are statisti- spend as much (or more) time with about the surface-level benefits. Drill
cally significant, so you do the same your coworkers than you do with your down on your biggest goals and do a
thing repeatedly. friends or family. little thinking about whether or not
The projects the lab was working In some organizations, it is com- each job will help you get closer to
on were exciting, but my life in the mon to interview with the boss and at those goals.
lab was not. least one other member of the team, The best careers are not defined
It is so important to consider what though this does not always happen. by titles or résumé bullet points. The
your day-to-day life will be like in a You should always ask for the opportu- smarter you are about what you choose
role. What will you actually spend nity to meet more of the people you will next, the closer you will get to the
your time doing? Will it add value be working with. things you truly want from your life
to your career? What will you get the This has a few benefits: and your work.
chance to learn? ˲˲ You can meet with the people you
Remember, when you are new to a will work with every day. Not only will
Related articles
team, you have no career capital built you get a feel for what it will be like on queue.acm.org
up with this organization. Career capi- working with them, you can also ask
10 Ways to Be a Better Interviewer
tal is your currency at work; when you them for insight into other aspects of
Kate Matsudaira
provide a lot of concrete, visible value the role. Do they like working there? https://queue.acm.org/detail.cfm?id=3125635
to the team or the organization, you How much turnover is there on the
Avoiding Obsolescence
have more leverage to do the things team? How does collaboration work? Kode Vicious
you want, such as work on the most ex- Does leadership listen to input on deci- https://queue.acm.org/detail.cfm?id=1781175
citing projects or get more flexibility in sions? What are the things they would A Generation Lost in the Bazaar
your schedule. want to change about the team/com- Poul-Henning Kamp
When you are new, you have not pany/culture? Why do they work there https://queue.acm.org/detail.cfm?id=2349257
earned this leverage. That means if vs. anywhere else?
you are assigned to a boring role on ˲˲ Your coworkers will feel invested Kate Matsudaira (katemats.com) is an experienced
technology leader. She has worked at Microsoft and
an exciting project, you pretty much in your success if they are part of Amazon and successful startups before starting her own
just have to do it. Sometimes that the process of hiring you. Think company, Popforms, which was acquired by Safari Books.
can be OK (maybe you actually want- about it—if you met with a candi-
ed to learn this boring skill because date you liked and fought for him Copyright © 2018 held by owner/author.
it will help you get a job you want in or her to be hired, wouldn’t you be Publication rights licensed to ACM. $15.00

46 COMMUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

DOI:10.1145 / 3 2 3 3 2 3 5

Article development led by

The interactions between storage and

applications can be complex and subtle.

Mind Your
State for Your
State of Mind

APPLICATIONS HAVE HAD an interesting evolution as

they have moved into the distributed and scalable
world. Similarly, storage and its cousin databases have
changed side by side with applications. Many times, the
semantics, performance, and failure models of storage
and applications do a subtle dance as they change in
support of changing business require- its own discrete database with its own
ments and environmental challeng- transactions but used messaging to
es. Adding scale to the mix has really coordinate across boundaries. Soon,
stirred things up. This article looks at we were using microservices, each of
some of these issues and their impact which likely did not have its own data
on systems. but reached directly to a distributed
Before database transactions, there store shared across many separate ser-
were complexities in updating data, es- vices. This scaled better—if you got the
pecially if failures happened. This held implementation right.
true even though the systems were cen- Different types of distributed stores
tralized and avoided the complexities offer various average speeds, variation
presented by distribution. Database in responsiveness, capacity, availabil-
transactions dramatically simplified ity, and durability. Diverse application
the life of application developers. It was patterns use the stored data for dis-
great while it lasted … tinct purposes. They provide various
As solutions scaled beyond a single guarantees to their users based largely
database, life got ever more challeng- on their use of storage. These different
ing. First, we tried to make multiple da- guarantees from the app sometimes
tabases look like one database. Then, show variations in what the users see in
we were hooking multiple applications semantics, response time, durability,
together using service-oriented archi- and more. While these can be surpris-
tecture (SOA). In SOA, each service had ing, it may be OK. What matters is the

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 47

fulfillment of the business needs and rally leads to applications and app code There have been and continue to
clarity of expectations. encapsulating the data so the distrusted be significant changes to the style of
This article provides a partial taxon- outsider cannot just modify the data computation, to storage, and to how
omy of diverse storage solutions avail- with abandon. these application patterns are used to
able over a distributed cluster. Part of As the industry started running stuff access storage.
this is an exploration of the interac- at huge scale, it learned that busting a This is only a partial list of storage
tions among different features of a service into smaller microservices has a and compute models. It is not meant to
store. The article then considers how couple of big advantages: be complete.
distinct application patterns have ˲˲ Better engineering. Breaking your Challenges in modern microservice-
grown over time to leverage these services (that is, trust boundaries) into based applications. These days, mi-
stores and the business requirements smaller pieces allows better engineer- croservices power many scalable apps.
they meet. This may have surprising ing flexibility as small teams make Microservices are pools of identical or
implications. quicker changes. equivalent services running over a col-
˲˲ Better operability. Making these lection of servers. Incoming requests
The Evolution of State, Storage, smaller pieces stateless and restartable are load balanced across the pool.
And Computing … At Least So Far allows for more resilient operations as
This section starts by examining some of failures, rolling upgrades of versions,
the profound changes that have occurred and adjustments for varying demand When a request waits for a microser-
in both storage and computation. The are dynamically handled. vice, any one from the same pool will
focus then turns to a discussion of both Microservices became an essential do the job. Sometimes, systems imple-
durable state and session state and how part of the software engineering and op- ment affinitization, where a subsequent
they have evolved over time. Finally, there erations landscape. request is likely to go to the same spe-
is a brief reminder of how data is treated cific microservice. Still, the outcome
differently inside a classic database and must be correct if you land on any of the
outside as it moves across trust and trans- Careful Replacement Variations microservices.
actional boundaries. ˲˲ A write may trash the previous
Trends in storage and computing. value … write somewhere else first.
Changes in storage and computing ˲ ˲ A client crash may interrupt a Microservices help scalable systems
have put demands on how storage is sequence of writes … plan carefully. in two broad ways:
accessed and the expected behavior in ˲˲ Improved software engineering.
doing so. This is especially interesting Building systems consisting of small
as work is smeared over pools of small Computing’s use of storage has and independent microservices results
computation known as microservices. evolved. It has been quite a wild ride of in agility. Teams owning the microser-
Storage has evolved. It used to be application changes as their use of stor- vices must be accountable and have
that storage was only directly attached age has evolved: independence and ownership. When
to your computer. Then came shared ˲˲ Direct file I/O used careful replace- something needs changing, change it.
appliances such as storage area net- ment for recoverability. Careful replace- When something is broken, the owning
works (SANs). These are big, expensive ment is a technique that is at least as team is responsible.
devices with a lot of sophisticated soft- old as the 1960s. It involves thoughtful ˲˲ Improved operations. Health-medi-
ware and hardware to provide highly ordering of changes to durable storage ated deployment allows for slow rollout
available storage to a bunch of servers such that failures can be tolerated. of new versions into the running system.
attached to them. This led to storage ˲˲ Transactional changes were sup- By watching the system’s health, new
clusters of commodity servers con- ported for application developers, pro- versions can be rolled back. These roll-
tained in a network. viding a huge improvement. It meant ing upgrades to the microservices can
Computing has evolved. A few de- the app developer did not need to be be sensitive to fault zones so an inde-
cades ago, it was only a single pro- so careful when dealing with storage. pendent failure during a flaky upgrade
cess on a single server. Years went by It also allowed a grouping of changes is not too damaging. Simply having a lot
before people started worrying about to records so a bunch of records were of separate and equivalent microser-
communicating across multiple pro- atomically updated. This was a lot eas- vices means a failure of one or more of
cesses on a single server. Then the ier. SANs implemented the required them is automatically repaired.
world moved on with great excitement careful replacement for the hardware Durable state is not usually kept
to RPCs (remote procedure calls) storage, allowing bigger and better da- in microservices. Instead, it is kept in
across a tiny cluster of servers. At the tabases. Databases evolved to support back-end databases, key-value stores,
time, we didn’t think about trust since two-tier and N-tier applications using caches, or other things. The remainder
everyone was in the same trust zone. transactional updates. of this article looks at some of these.
We were all in the family! ˲˲ Key-value stores offered more scale Microservices cannot easily update
In the 2000s, the concept of services but less declarative functionality for the state across all of the microser-
or SOA began to emerge, sometimes un- processing the application’s data. Mul- vices in the pool. This is especially true
der different names.6 The basic aspect tirecord transactions were lost as scale when they are coming and going willy-
of a service is trust isolation. This natu- was gained. nilly. It is common to keep the latest

48 COMMUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0


state out of reach of the microservices Outside data always has some form of Then, an interesting development
and provide older versions of the state a unique identifier such as a URI (uni- in storage occurred. Some stores are
that are accessible in a scalable cache. form resource identifier) or a key. The fast but sometimes return stale values.
Sometimes, this leads to read-through identifier may be implicit within a ses- Others always return the latest value
requests by the scalable cache to du- sion or an environment. Outside data but occasionally stall when one of the
rable state that is not directly address- typically is manifest as a message, file, servers is slow. This section shows how
able to the calling microservice. or key-value pair. predictable answers result in unpre-
This is now becoming a tried and dictable latencies.10 Finally, it exam-
true pattern. Figure 1 is taken from a The Evolution of Durable ines the role immutable data can play
2007 paper by DeCandia et al. on Ama- State Semantics in supporting very large systems with
zon’s Dynamo.2 While the nomencla- Storage systems and databases have predictable answers and response
ture is slightly different, it shows three evolved through the decades and so times for some business functions.
tiers of microservices accessing a back- have the semantics of updating their Careful replacement of disk
end tier of different stores. state. This section begins in the bad old blocks. It used to be, back in the 1970s
Durable state and session state. days when I first started building sys- and 1980s, that a disk write might
Durable state is stuff that gets remem- tems. Back in the 1970s and 1980s, disk leave data unreadable. The write went
bered across requests and persists storage had to be carefully updated to through a number of state changes
across failures. This may be captured avoid trashing disk blocks. From there, from the old V1 version, to unreadable
as database data, file-system files, we move forward to atomic record up- garbage, to the new V2 version. When
key values, and more. Durable state dates and the challenges that arose be- the disk head was writing a block, the
is updated in a number of different fore transactions. When transactions magnetic representation of the bits in
ways, largely dependent on the kind came along a lot of things got a lot the block would be turned to mush on
of store holding it. It may be changed easier—if you were making a change at the way to being updated to the new
by single updates to a key value or one place and one time. Adding cross- version. A power failure would cause
file, or it may be changed by a trans- database and cross-time behavior led you to lose the old value (see Figure 2).
action or distributed transaction to the same challenges you had with When implementing a reliable ap-
implemented by a database or other more primitive storage systems. This plication, it’s essential that you do not
store. was helped by using messaging subsys- lose the old value of the data. For exam-
Session state is the stuff that gets re- tems to glue stuff together. ple, if you’re implementing the trans-
membered across requests in a session
but not across failures. Session state Figure 1. Example of Amazon’s Dynamo microservice architecture.
exists within the endpoints associated
with the session. Multioperation trans-
actions use a form of session state.7 Client Requests
Session state is hard to do when the
session is smeared across service in- Page
stances. If different microservices in ... Rendering
the pool process subsequent messages Components
in the transaction, session state is chal-
lenging to implement. It’s difficult to re- Request Routing
tain session state at the instance when
the next message to the pool may land
at a different service instance. Services
Data on the outside versus data on
the inside. The 2005 paper “Data on
the Outside Versus Data on the Inside”5 Request Routing
speaks about the fundamental differ-
ences between data kept in a locked
transactional store (for example, a rela- Services
tional database) and data kept in other
Data on the inside refers to locked
transactionally updated data. It lives
in one place (for example, a database)
and at one time, the transactional S3
point in time.
Data on the outside is unlocked
and immutable, although it may be Dynamo Instances Other Datastores
versioned with a sequence of versions
that are in their own right immutable.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 49

Figure 2. V1 is trashed before V2 is written. be updated. Typically, you would write action by transaction. This leads to
to record X, wait to know it’s perma- messaging semantics.
file 1 V1 nent, and then write to record Y. Messaging semantics. In transac-
So, could you untangle the mess if a tional messaging a transaction makes
crash happened? a bunch of changes to its data and then
Frequently, there was an application- expresses a desire to send a message.
file 1
dependent pattern that provided in- This desire is atomically recorded with
sight into the order you needed to write. the transaction. A transaction may
After a crash and restart: atomically consume an incoming mes-
file 1 V2 ˲˲ If record A was updated but record sage. That means the work of the trans-
B was not written, the application can action, including changes to the appli-
clean up the mess. cation data, occurs if, and only if, the
Figure 3. “Ping-Pong” technique delays ˲˲ If record B was updated but record incoming message is consumed.
overwrite of V1.
A was not written, the application could It is possible to support the seman-
not cope and could not recover. tics of exactly-once delivery. The desire
file 1 V1 An example of careful replacement to send is atomically committed with
for records is message queuing. If the the sending transaction. A committed
application writes and confirms the desire to send a message causes one
file 1 V1 V2 presence of a message in a queue (call or more transmissions. The system
it record A), and the work to process retries until the destination acknowl-
that message is idempotent, then the edges it has received the message in
application can cope with crashes its queue. The message must be pro-
file 1
based on careful replacement for re- cessed at the receiver at most once.
cords. Idempotent means it is correct This means it must be idempotently
if restarted.4,7 processed (see Figure 4).
file 1 V2 V2 Transactions and careful replace- There are challenges with at-
ment. Transactions bundle and solve most-once processing at the destina-
careful record replacement. Multiple ap- tion. To accomplish this, you need
action system for a database, it’s really plication records may be updated in a to remember the messages you have
bad to lose the most recently commit- single transaction, and they are all-or- processed so you don’t process them
ted transactions because the partially nothing. The database system ensures twice. But how do you remember the
full last block of your transaction log is the record updates are atomic. messages? You have to detect dupli-
being rewritten. One trick to avoid this ˲˲ Databases automatically handle cates. How long do you remember?
is to take turns writing to mirrored logs any challenges with careful storage re- Does the destination split? Does
on different disks. Only after know- placement. Users are not aware of the the destination move? If you mess
ing for sure that mirror A has the new funky failure behaviors that may oc- this up, will the application process
block do you write it to mirror B. After cur when systems crash or power fails. the message more than once? What
a crash, you rewrite the last block of the If present, databases also support if the message is being delivered to
log onto both mirrors to ensure a con- distributed transactions over a small a microservice-based application?
sistent answer. number of intimate database servers. Where is the knowledge of the set of
Another well-known technique, es- ˲˲ Work across time (that is, work- processed messages kept?
pecially for the tail of the log, is called flow) needs careful transactional re- Read your writes? Yes? No? It used
ping-pong.4 In this approach, the last placement. While the set of records in to be, back in the day, if you wrote
(and incomplete) block of the log is a transaction is atomically updated something, you could read it. Now,
left where it lies at the end of the log. with the help of the database, long- it’s not always that simple. Consider
The next version of that block, contain- running workflows3,4 are essential to the following:
ing the previous contents and more, is accomplish correct work over time. Linearizable stores offer read-
written to a later block. Only after the Failures, restarts, and new work can your-write behavior. In a linearizable
extended contents are durable on the advance the state of the application store each update creates a new ver-
later block will the new version over- transaction by transaction. Work sion of the value, and the store never
write the earlier version. In this fashion, across time leverages message pro- returns an old value or a different
there are no windows in which a power cessing. value. It always returns the latest in a
failure will lose the contents of the log ˲˲ Work across space (that is, across linear series of values.
(see Figure 3). boundaries) also needs careful transac-
Careful replacement for record tional replacement. Different systems,
writes. Updates to records in pre-data- applications, departments, and/or Linearizable stores will sometimes de-
base days didn’t have transactions. As- companies have separate trust bound- lay for a looooong time.
suming each record write was atomic, aries and typically do not do transac- To ensure they always give the cor-
you still couldn’t update two records tions across them. Work across space rect value, they will always update
and get any guarantees they would both necessitates work across time, trans- every replica.

50 COMMUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0


If a server is slow or dead and con- store a brand-new value for an identi- Slip-Slidin’ Away …
tains one of the replicas, it may take fier and, later on, delete it. Many ap- This section looks at a number of guar-
tens of seconds to decide what to do … plication patterns are based on immu- antees that are slipping away. Everyone
Meanwhile, the user waits. table items. wishes they had a computational mod-
Imagine a system where you are sim- el such as a von Neumann machine,12
ply recording stuff you have seen. Ev- which provides computation, storage,
Nonlinearizable stores do not offer erything you know is based on observa- and predictable linear behavior. Once
to read your writes. A nonlinearizeable tions. The past is never changed—sort distribution kicks in, however, that’s in-
store means there’s no guarantee that a of like an accountant’s ledger where deed only a wish.
write will update all the replicas. Some- nothing is updated. You can put a Single-process computation as John
times, a read may find an old value. unique ID on each artifact and look at it von Neumann conceived has evolved
Reading and writing a nonlinearizable later but never change it. This is an ex- to multiprocess- and multiserver-using
store has a very consistent response tremely common pattern. sessions and session state. These state-
time with much higher probability. When keeping immutable objects ful sessions supported composable
A read or write can skip over a sick or or values in a key/value store, you transactions that spanned multiple
dead server. Occasionally, this results never get a stale answer. There’s only records and multiple servers working
in an older value coming back from the one immutable value for the unique together. As the work started decom-
skipped server. But, hey, it’s fast—and key. That means a nonlinearizable posing into microservices, however, it
predictably so. store offers the one and only correct became hard to use transactions the
Imagine a key/value store where key- answer. All the store types give the way they had been used.
K has value V1 and the store keeps it correct answer, just with different To cope with scalable environments,
on servers S1, S2, and S3. You decide to characteristics for read and write la- data had to be busted up into key val-
update the value to V2. The store tries tencies (see Figure 6). Storing immu- ues. Scalable stores worked well for up-
to change the values on its three serv- table data means you never get a stale dating a single key at a time but not for
ers, but S2 does not answer because it version because there is not one. atomic transactions across keys. Most
is down. Therefore, the store decides
to write V2 onto S1, S3, and S4 so that Figure 4. Transaction messaging.
the new value is always written to three
servers. Later, when S2 comes up, a At Least
read might find the old value V1. This Transaction T1 Transaction T2
has the following trade-offs: Delivery
Writes To Data Writes To Data
˲˲ The write of three stores always
happens quickly. W X Y Z A B C D
˲˲ The store is not linearizable and
sometimes returns an old value.
This very useful technique underlies At Most
a number of scalable storage systems Once Processing
such as Dynamo2 and Cassandra.11
Cached data offers scalable read
throughput with great response time. Figure 5. Different types of storage offer different guarantees.
Key-value pairs live in many servers
and are updated by propagating new
Fast Predictable Fast Predictable Read
versions. Each read hits one of the Reads? Writes? Your Writes?
servers and returns one of the versions
Linearizable Store No No Yes
(see Figure 5).
Non-Linearizable Store Yes Yes No
Scalable Cache Yes w/Scale No No
Different Stores for Different Uses
OK to stall on reads?
OK to stall on writes?
OK to return stale versions? Figure 6. Immutable data allows “read-your-write-behavior.”
You can’t have everything!
Fast Predictable Fast Predictable Read
Reads? Writes? Your Writes?
Immutability: A solid rock to stand Linearizable Store No No Immutable
on. When you store immutable data,
Non-Linearizable Store Yes Yes Immutable
each lookup always returns the same
result.8 Immutable stores do not ever Scalable Cache Yes w/Scale No Immutable

exhibit update anomalies because you

never update them. All you can do is

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 51

of these scalable key-value stores en- tion. In many circumstances, rich and sion-1]), you can view the version as im-
sured linearizable, strongly consistent complex transactions could occur mutable data. Each version becomes an
updates to their single keys. Unfortu- over N-tier environments, even across immutable thing to be kept. Using the
nately, these linearizable stores would multiple back-end databases using extended [Key, Version], you can refer-
occasionally cause delays seen by users. distributed transactions. ence immutable data in the store.
This led to the construction of nonlin- Transactions, sessions, and microser- Version history may be linear, mean-
earizable stores with the big advantage vices. Microservices leave much to be de- ing one version supersedes the previous
that they have excellent response times sired when it comes to session state. Re- one. This is achieved by using a lineariz-
for reads and writes. In exchange, they quests are load balanced through a router, able store. Version history may be a di-
sometimes give a reader an old value. and one of many microservice instances is rected acyclic graph (DAG). This happens
Finally, this section points out that selected. Usually, later traffic is sent to the when writing to a nonlinearizable store.
some uses of data find the correct an- same instance but not always. You cannot Imagine you have a notepad on
swer important enough to use care- count on getting back to where you were. which to scribble stuff. But you really
ful replacement of the stored values. Without session state, you can- have multiple notepads. You scribble
These uses are not the best for nonlin- not easily create transactions cross- stuff on whichever notepad is closest
earizable stores. ing requests. Typically, microservice to you at the time. When you want to
Honestly, it ain’t like it used to be. environments support a transaction read the information, you look at the
Same process evolves to different within a single request but not across closest notepad even if it’s not the one
process. Applications and the database multiple requests. you wrote on most recently. Sometimes,
used to run in the same process. A li- Furthermore, if a microservice ac- you get two notepads next to each other,
brary call to the database code allowed cesses a scalable key-value store as it look at both, and write something in
access to the data. Sometimes, multiple processes a single request, the scalable both to consolidate the scribbles. This
applications were loaded together. key-value store will usually support only is the kind of behavior that comes from
Later, the database and applica- atomic updates to a single key. While it a nonlinearizable store. Updates do not
tions were split into different processes won’t break the data by failing in the mid- march forward in linear order.
connected by a session. The session dle of updating a key as older file systems Careful replacement and read your
described the session state and had in- did, programmers are on their own when writes. In careful replacement you need
formation about the user, transaction changing values tied to multiple keys. to be careful about the ordering of what
in flight, the application being run, and Keys, versions, and nonlinear his- you update. This is essential to handle
the cursor state and return values. tory. Each key is represented by some some failures, as discussed earlier. Pre-
Later still, the application and da- number, string, key, or URI. That key dictable behavior across trust boundar-
tabase moved to different servers. The can reference something that’s im- ies is needed when working with other
session state made that possible. mutable. For example, “The New York companies. It’s also essential when do-
Stateful sessions and transactions. Times, June 1, 2018, San Francisco Bay ing long-running workflows.
Stateful sessions were a natural out- Area edition” is immutable across space Careful replacement is predicated
come of shared processes. You knew and time. A key may also reference some- on read-your-writes behavior, which
who you were talking to and you could thing that changes over time—for exam- depends on a linearizable store. Lin-
remember stuff about the other guy. ple, “today’s New York Times.” earizable stores almost always have the
Stateful sessions worked well for When a key references something property of occasionally stalling when
classic SOA. When talking to a ser- that changes, it can be understood as waiting for a bum server.
vice, you expected a long session with referencing a sequence of versions, each
state on each side. Stateful sessions of which is immutable. By first binding Some Example Application Patterns
meant the application could do mul- the changing value of the key to a unique Let’s look at some application patterns
tiple interactions within a transac- version of the key (for example, [Key, Ver- and how they impact the management
of durable state (see Figure 7).
Figure 7. Applications patterns. Workflow over key-value with care-
ful replacement. This pattern demon-
strates how applications perform work-
workflow over key-value A traditional workflow application over a scalable collection
of key-value data. flow when the durable state is too large
transactional blobs-by-ref A centralized and transactional system managing
to fit in a single database.
very large collections of immutable blobs. An object is uniquely identified by its
e-commerce—shopping cart The familiar but still surprising world of e-commerce key. Work arrives from the outside via
shopping carts. human interaction or messaging. Work-
e-commerce—product Consider a very large ecommerce product catalog with flow can be captured in the values. New
catalog enormous numbers of product descriptions and huge traffic values replace old ones. The messages
reading the catalog.
are contained as data within the object.9
search Track a ginormous number of document (for example, the entire Scalable workflow applications can
Web) and organize searchable indices to locate documents by
words and phrases. Must scale to ever increasing read workload. be built over key-value stores. You must
have single-item linearizability (read
your writes, see Figure 8.) With a linear

52 COMM UNICATIO NS O F THE AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0


version history, one new version always exist in the version history DAG. Rela- the partition for the product identifier
supersedes the earlier one. A nonlinear tively simple shopping-cart semantics and then load-balanced across replicas.
history has a DAG version history. In this facilitate combining different versions Back-end processing of the feeds
case, the linearizable behavior of the of a single user’s shopping cart.2 and crawls, as well as the pub-sub dis-
store also implies that a stall within one E-commerce—Product catalog. tribution of updates to the caches, are
of the store servers will stall the write Product catalogs for large e-commerce throughput sensitive, not latency-sensi-
to the store. This is the “must be right” sites are processed offline and stuffed tive. Different replicas may be updated
even if it’s not “right now” case. into large scalable caches. Feeds from
The workflow implemented by careful partners and crawls of the Web are Figure 8. Linear vs. nonlinear histories.
replacement will be a mess if you can’t crunched to produce a sanitized
read the last value written. Hence, this and hopefully consistent collection of
usage pattern will stall and not be stale. product-catalog entries.
Transactional blobs-by-ref. This is a Each product in the catalog has a
pretty common application pattern. The unique identifier. Typically, the identifi-
application runs using transactions and er takes you to a partition of the catalog.
a relational database. It also stores big The partition has a bunch of replicas,
blobs such as documents, photos, PDFs, each containing many product descrip-
videos, music, and more. The blobs can tions (see Figure 9). One typical imple-
be large and numerous. Hence, these are mentation of a scalable product cache
a challenge to implement directly in the has partitions with replicas. In this
relational database. depiction, the columns are partitions
Each of these blobs is an immutable and the rows depict replicas. The back-
set of bits. To modify a blob (for exam- end processing produces new product Linear Directed Acyclic Graph
Version History Version History
ple, editing a photo), you always create a descriptions that are distributed with
new blob to replace the old one. The im- pub-sub. Incoming requests are sent to
mutable blobs typically have a univer-
sally unique identifier (UUID) as their Figure 9. Partitions with replicas.
key in a scalable key-value store.
Storing immutable blobs in a non-
linearizable database does not have any
problems with returning a stale version.
Since there’s only one immutable ver- Incoming Read Requests

sion, there are no stale versions.

Storing immutable data in a nonlin-
earizable store enjoys the best of both
a-e f-j k-o p-t u-z
worlds: it’s both right and right now.
E-commerce shopping cart. In e-com-
merce, each shopping cart is for a sepa-
rate customer. There’s no need or desire a-e f-j k-o p-t u-z
for cross-cart consistency. Each shopping
cart has a unique identity or key.
Customers are very unhappy if their
a-e f-j k-o p-t u-z
access to a shopping cart stalls. Large
e-commerce sites can measure the per-
centage of abandoned carts and cus-
tomer sessions when they get slow. Slow a-e f-j k-o p-t u-z
carts correspond to a large drop-off in
business. Product catalogs, reviews,
and more must be fast and responsive
or customers leave.
Shopping carts should be right now
even if they are not right. It is measur- Automatic Pub-sub Distribution
ably better for business and the cus-
tomer experience to return a stale or feeds from
otherwise incorrect answer if it can be
Backend Processing
done quickly. Users are asked to verify (Feed and Crawl)
the contents of the shopping cart before crawl
confirming the sale. the Web
In a nonlinearizable store, some-
times multiple old versions of the cart

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 53

Figure 10. Application pattern trade-offs. leading technique to support scalable

Finally, different applications de-
Application Predictable Predictable Reads your
pattern read latency? write latency? writes? Trade-offs mand different behaviors from durable
state. Do you want it right or do you
Careful replace- No No Yes Works across
ment with k/v multiple key/values want it right now? Human beings usu-
Tx’l blobs Yes Yes Immutable Non-linearizable ally want an answer right now rather than
for ref plus immutable right. Many application solutions based
E-commerce— Yes Yes No Sometimes give on object identity may be tolerant of
shopping cart stale results stale versions. Immutable objects can
E-commerce— Yes No No Scalable cache means provide the best of both worlds by being
product catalog that stale is ok both right and right now.
Search Yes No No Scalable cache Consider your application’s require-
plus search
ments carefully. If you are not careful,
you will have problems with your state
that you will definitely mind.
asynchronously, meaning it is not trated. This is aggravated by the need
surprising to read a new version of the to hear back from all the servers. If any
Related articles
description, retry, and then get an old server is a laggard, the response is de- on queue.acm.org
version from a cache replica that’s not layed. The mechanism for coping with
Non-volatile Storage
yet updated. this at Google is beautifully described in
Mihir Nanavati et al.
User lookups are very sensitive to la- the 2013 article “The Tail at Scale.”1 https://queue.acm.org/detail.cfm?id=2874238
tency. Just as shopping cart response In search, it is OK to get stale an-
Network Applications Are Interactive
times must be fast, product-catalog swers, but the latency for the response Antony Alappatt
lookups must be fast. It is common for must be short. There’s no notion of lin- https://queue.acm.org/detail.cfm?id=3145628
a client working to display the descrip- earizable reads nor of read-your-writes. Storage Systems: Not Just
tion of a product to wait for an answer, Search clearly needs to return answers a Bunch of Disks Anymore
time out, and retry to a different replica right now even if they are not right. Erik Riedel
if necessary to ensure the latency for the It’s about the application pattern. https://queue.acm.org/detail.cfm?id=864059
response is fast. Each application pattern shows dif-
Note the management of the short ferent characteristics and trade-offs, 1. Dean, J. and Barosso, L.A. The tail at scale. Commun.
latency depends on the fact that any ver- shown in Figure 10. ACM 56, 2 (Feb. 2013), 74–80.
2. DeCandia, G. et al. Dynamo: Amazon’s highly available
sion of the product-catalog description is key-value store. In Proceedings of the 21st ACM
OK. This is another example of the busi- Conclusion SIGOPS Symp. Operating System Principals, 2007,
ness needing an answer right now more State means different things. Session 3. Garcia-Molina, H. and Salem, K. Sagas. In Proceedings
than it needs the answer to be right. state captures stuff across requests but of the ACM SIGMOD Conf. Management of Data,
1987, 249–259; https://www.cs.cornell.edu/andru/
Search. Say you are building a search not across failures. Durable state re- cs711/2002fa/reading/sagas.pdf
system for the contents of the Web. Web members stuff across failures. 4. Gray, J. and Reuter, A. Transaction Processing: Concepts
and Techniques. Morgan Kaufmann, Burlington, MA,
crawlers feed search indexers. Each Increasingly, most scalable comput- 1992, 508–509.
document is given a unique ID. Search ing consists of microservices with state- 5. Helland, P. Data on the outside versus data on the
inside. In Proceedings of the Conf. Innovative Database
terms are identified for each document. less interfaces. Microservices need parti- Research, 2005.
The index terms are assigned to a shard. tioning, failures, and rolling upgrades, 6. Helland, P. Fiefdoms and emissaries, 2002; download.
Updates to the index are not super and this implies that stateful sessions fiefdoms_emissaries.ppt.
7. Helland, P. Idempotence is not a medical condition.
latency-sensitive. Mostly, changes ob- are problematic. Microservices may call acmqueue 10, 4 (2012), 56–65.
served by crawling the Web are not laten- other microservices to read data or get 8. Helland, P. Immutability changes everything.
acmqueue 13, 9 (2016); https://queue.acm.org/detail.
cy-sensitive. Other than time-sensitive stuff done. cfm?id=2884038.
news feeds, the changes need not be im- Transactions across stateless calls are 9. Helland, P. Life beyond distributed transactions.
acmqueue 14, 5 (2016); https://queue.acm.org/detail.
mediately visible. When a random docu- usually not supported in microservice cfm?id=3025012.
ment is produced at some remote loca- solutions. Microservices and their load- 10. Helland, P. Standing on distributed shoulders of giants.
acmqueue 14, 2 (2016); https://queue.acm.org/detail.
tion in the world, it might take a while to balanced service pools make server-side cfm?id=2953944.
be seen. session state difficult, which, in turn, 11. Lakshman, A. and Malik, P. Cassandra: A decentralized
structured storage system. ACM SIGOPS Operating
Search results are, however, sensitive makes it difficult to have transactions Systems Review 44, 2 (2010), 35–40.
to latency. In general, a search request across calls and objects. Without trans- 12. von Neumann, J. First draft of a report on the EDVAC.
IEEE Annals of the History of Computing 15, 4 (1993),
from a user is fed into servers that ask all actions, coordinated changes across 27–75.
of the shards for matching results. This objects in durable state need to use the
looks a lot like the product catalog de- careful replacement technique in which Pat Helland has been implementing transaction systems,
databases, application platforms, distributed systems,
picted in Figure 9, but the user requests updates are ordered, confirmed, and fault-tolerant systems, and messaging systems since 1978.
hit all the shards, not just one of them. idempotent. This is challenging to pro- He currently works at Salesforce.
It’s very important that searches gram but is a natural consequence of mi- Copyright held by owner/author.
get quick results, or users will get frus- croservices, which have emerged as the Publication rights licensed to ACM. $15.00.

54 COM MUNICATIO NS O F TH E ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

Montreal, QC
ACM/IEEE IoTDI is the premier venue for all topics
related to the Internet of Things. This conference
April 15-18, 2019 is an interdisciplinary forum to discuss challenges,
technologies and emerging directions in system
design and implementation that pertain to IoT.

Selected papers will be invited to the

ACM Transactions on the Internet of Things (TIOT)

analytic foundations & theory fog computing

4th ACM/IEEE Conference on Internet of Things Design and Implementation

edge computing wearables

security industrial iot (IIoT)
evaluation testbeds
novel protocols machine learning
cloud backends reliability & robustness
smart deployment experiences

buildings cities health transportation


General Co-Chairs
Klara Nahrstedt, University of Illinois at Urbana-Champaign, USA
Olaf Landsiedel, Chalmers University, Sweden

Program Co-Chairs
Gian Pietro Picco, University of Trento, Italy
Prashant Shenoy, University of Massachussetts, Amherst, USA

Important Dates
Abstract Registration: Oct. 10, 2018
Paper Submission: Oct. 17, 2018
Author Rebuttal: Dec. 10, 2018
Notification Date: Jan. 15, 2019

CPS-IoT Week
contributed articles
DOI:10.1145/ 3271625
The triumph of these achievements
What just happened in artificial intelligence has led some to describe the automa-
tion of these tasks as having reached
and how it is being misunderstood. human-level intelligence. This percep-
tion, originally hinted at in academic
BY ADNAN DARWICHE circles, has gained momentum more
broadly and is leading to some impli-

cations. For example, some coverage
of AI in public arenas, particularly
comments made by several notable fig-
ures, has led to mixing this excitement

with fear of what AI might bring us all
in the future (doomsday scenarios).b
Moreover, a trend is emerging in which
machine learning research is being

or Animal-Like
streamlined into neural network re-
search, under its newly acquired label
“deep learning.” This perception has
also caused some to question the wis-

dom of continuing to invest in other
machine learning approaches or even
other mainstream areas of AI (such as
knowledge representation, symbolic
reasoning, and planning).
This turn of events in the history of
AI has created a dilemma for research-
ers in the broader AI community. On
the one hand, one cannot but be im-
pressed with, and enjoy, what we have
been able to accomplish with neural
“The vision systems of the eagle and the snake networks. On the other hand, main-
stream scientific intuition stands in
outperform everything that we can make in the way of accepting that a method
the laboratory, but snakes and eagles cannot b Stephen Hawking said: “The development of
build an eyeglass or a telescope or a microscope.” full artificial intelligence could spell the end
of the human race;” and Elon Musk said AI is:
— Judea Pearla “ … potentially more dangerous than nukes.”

key insights
of neural networks in
T H E REC ENT SUC C E S S E S ˽˽ The recent successes of deep learning
have revealed something very interesting
applications like speech recognition, vision, and about the structure of our world, yet this
autonomous navigation has led to great excitement by seems to be the least pursued and talked
about topic today.
members of the artificial intelligence (AI) community, ˽˽ In AI, the key question today is not
as well as by the general public. Over a relatively short whether we should use model-based or
function-based approaches but how to
time, by the science clock, we managed to automate integrate and fuse them so we can realize

their collective benefits.

some tasks that have defied us for decades, using one
˽˽ We need a new generation of AI
of the more classical techniques due to AI research. researchers who are well versed in and
appreciate classical AI, machine learning,
a Lecture by Judea Pearl, The Mathematics of Causal Inference, with Reflections on Machine Learning and computer science more broadly while
and the Logic of Science; https://www.youtube.com/watch?v=zHjdd--W6o4 also being informed about AI history.

56 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 57
contributed articles

that does not require explicit model- reasoning is required to compute the
ing or sophisticated reasoning is suf- function outputs from its inputs. The
ficient for reproducing human-level main tool of this approach is the neural
intelligence. This dilemma is further network. Many college students have
amplified by the observation that re-
cent developments did not culminate In my own quest exercised a version of it in a physics
or chemistry lab, where they fit simple
in a clearly characterized and profound
scientific discovery (such as a new
to fully appreciate functions to data collected from vari-
ous experiments, as in Figure 2. The
theory of the mind) that would nor- the progress main difference here is we are now em-
mally mandate massive updates to the
AI curricula. Scholars from outside AI
enabled by deep ploying functions with multiple inputs
and outputs; the structure of these
and computer science often sense this learning, I came functions can be quite complex; and
dilemma, as they complain they are
not receiving an intellectually satisfy-
to the conclusion the problems being tackled are ones
we tend to associate with perception or
ing answer to the question: “What just that recent cognition, as opposed to, say, estimat-
happened in AI?”
The answer lies in a careful assess- developments tell ing the relationship between volume
and pressure in a sealed container.d
ment of what we managed to achieve us more about the The main observation in AI recently
with deep learning and in identifying
and appreciating the key scientific out- problems tackled is that the function-based approach
can be quite effective at certain AI
comes of recent developments in this
area of research. This has unfortunate-
and the structure tasks, more so than the model-based
approach or at least earlier attempts at
ly been lacking to a great extent. My of our world than using this approach. This has surprised
aim here is to trigger such a discussion,
encouraged by the positive and curious
about neural not only mainstream AI researchers,
who mainly practice the model-based
feedback I have been receiving on the networks per se. approach, but also machine learning
thoughts expressed in this article. researchers who practice various ap-
proaches, of which the function-based
Background approach is but one.e This has had
To lay the ground for the discussion, I many implications, some positive and
first mark two distinct approaches for some giving grounds for concern.
tackling problems that have been of On the positive side is the increas-
interest to AI. I call the first one “mod- ing number of tasks and applications
el-based” and the second “function- now within reach, using a tool that can
based.” Consider the object-recogni- be very familiar to someone with only
tion and -localization task in Figure 1. a broad engineering background, par-
To solve it, the model-based approach ticularly one accustomed to estimat-
requires one to represent knowledge ing functions and using them to make
about dogs and hats, among other predictions. What is of concern, how-
things, and involves reasoning with ever, is the current imbalance between
such knowledge. The main tools of exploiting, enjoying, and cheering
the approach today are logic and prob- this tool on the one hand and thinking
ability (mathematical modeling more about it on the other. This thinking is
generally) and can be thought of as not only important for realizing the full
the “represent-and-reason”c approach potential of the tool but also for scien-
originally envisioned by the founders tifically characterizing its potential
of AI. It is also the approach normally
expected, at some level, by informed
d This is also called the “curve-fitting” ap-
members of the scientific community. proach. While the term “curve” highlights the
The function-based approach, on the efficient evaluation of a function and captures
other hand, formulates this task as a the spirit of the function-based approach, it
function-fitting problem, with func- underplays the complex and rich structure of
functions encoded by today’s (deep) neural
tion inputs coming directly from the networks, which can have millions if not bil-
image pixels and outputs correspond- lions of parameters.
ing to the high-level recognitions we e Machine learning includes the function-based
seek. The function must have a form approach but has a wide enough span that it
that can be evaluated efficiently so no overlaps with the model-based approach; for
example, one can learn the parameters and
structure of a model but may still need non-
c This term might be likened to what has been trivial reasoning to obtain answers from the
called “good old-fashioned AI.” learned model.

58 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

reach. The lack of such characteriza- of labeled data; the increased com- a class of practical applications that
tion is a culprit of current misconcep- putational power we now have at correspond to functions that, we now
tions about AI progress and where it our hands; and the increasingly so- know, are simple enough to allow
may lead us in the future. phisticated statistical and optimiza- compact representations that can be
tion techniques for fitting functions evaluated efficiently (again, without
What Just Happened in AI? (including new activation functions the need for reasoning), and whose
In my own quest to fully appreciate the and new/deeper network structures). estimation is within reach of current
progress enabled by deep learning, The second is that we have identified thresholds for gathering data, com-
I came to the conclusion that recent
developments tell us more about the Figure 1. Object recognition and localization in an image (ImageNet).
problems tackled and the structure of
our world than about neural networks
per se. These networks are param-
eterized functions that are expressive
enough to capture any relationship
between inputs and outputs and have
a form that can be evaluated efficiently.
This has been known for decades and
described at length in textbooks. What
caused the current turn of events?
To shed some light on this question,
let me state again what we have discov-
ered recently. That is, some seemingly
complex abilities that are typically as-
sociated with perception or cognition
can be captured and reproduced to
a reasonable extent by simply fitting
functions to data, without having to ex-
plicitly model the environment or sym-
bolically reason about it. While this
is a remarkable finding, it highlights
problems and thresholds more than it
highlights technology, a point I explain
Every behavior, intelligent or not,
can be captured by a function that
maps inputs (environmental sensing) Figure 2. Fitting a simple function to data.
to outputs (thoughts or actions). How-
ever, the size of this function can be
quite large for certain tasks, assuming
the function can be evaluated efficient- 10

ly. In fact, the function may have an un-

bounded size in general, as it may have
to map from life histories. The two key 9
questions then are the following: For
tasks of interest, are the correspond-
ing functions simple enough to admit 8
a compact representation that allows

mapping inputs to outputs efficiently,

as in neural networks (without the 7
need for reasoning)? And, if the answer
is yes, are we currently able to estimate
these functions from input-output 6
pairs (labeled data)?
What has happened in AI recently
are three developments that bear di- 5
rectly on these questions: The first is
our improved ability to fit functions 0 2 4 6 8 10
to data, which has been enabled by
the availability of massive amounts

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 59
contributed articles

putational speed, and estimation fort, if not general agreement, with the speed up automation.h To address these
techniques. This includes recogniz- remarks I made. I did get a few “I beg to concerns, the focus should be shifted
ing and localizing objects in some differ” responses though, all centering toward policy and regulatory consider-
classes of images and certain tasks on recent advancements relating to op- ations for dealing with the new level of
that pertain to natural language and timizing functions, which are key to the automation our society is embarking
speech. The third development, successful training of neural networks on, instead of fearing AI.
which goes largely unnoticed, is (such as results on stochastic gradient
that we gradually changed our ob- descent, dropouts, and new activation On Objectives and Success
jectives and measures for success functions). The objections stemmed Let me now address the third reason for
in ways that reduced the technical from not having named them as break- the current turn of events, which relates
challenges considerably, at least as throughs (in AI). My answer: They all to the change in objectives and how we
entertained by early AI researchers, fall under the enabler I outlined earlier: measure success as a broad AI com-
while maintaining our ability to cap- “increasingly sophisticated statistical munity. This reason is quite substantial
italize on the obtained results com- and optimization techniques for fitting yet goes largely unnoticed, especially by
mercially, a point I discuss further functions.” Follow up question: Does younger researchers. I am referring here
later in the section on objectives and it matter that they are statistical and to the gradual but sustained shift over AI
success. optimization techniques, as opposed history from trying to develop technolo-
Interestingly, none of these develop- to classical AI techniques? Answer: It gies that were meant to be intelligent and
ments amounts to a major technical does not matter as far as acknowledg- part of integrated AI systems to develop-
breakthrough in AI per se (such as the ing and appreciating scientific inquiry ing technologies that perform well and
establishment of probability as a foun- and progress, but it does matter as far are integrated with consumer products;
dation of commonsense reasoning in as explaining what just happened and, this distinction can be likened to what
the late 1980s and the introduction of more important, forecasting what may has been called “Strong AI” vs. “Weak AI.”
neural networks more than 50 years happen next. This shift was paralleled by a sharp-
ago).f Yet the combination of these fac- Consider an educated individual sit- ening of performance metrics and by
tors created a milestone in AI history, as ting next to you, the AI researcher, on progress against these metrics, partic-
it had a profound impact on real-world a plane; I get that a lot. They figure out ularly by deep learning, leading to an
applications and the successful deploy- you do AI research and ask: What are the increased deployment of AI systems.
ment of various AI techniques that have developments that enabled the current However, these metrics and corre-
been in the works for a very long time, progress in AI? You recount the func- sponding progress did not necessarily
particularly neural networks.g tion-based story and lay out the three en- align with improving intelligence, or
ablers. They will likely be impressed and furthering our understanding of intelli-
‘I Beg to Differ’ also intellectually satisfied. However, if gence as sought by early AI researchers.i
I shared these remarks in various con- the answer is, “We just discovered a new One must thus be careful not to draw
texts during the course of preparing this theory of the mind,” you will likely not certain conclusions based on current
article. The audiences ranged from AI be surprised if they also end up worry- progress, which would be justified only
and computer science to law and pub- ing about a Skynet coming soon to mess if one were to make progress against
lic-policy researchers with an interest up our lives. Public perceptions about AI earlier objectives. This caution particu-
in AI. What I found striking is the great progress and its future are very impor- larly refers to current perceptions that
interest in this discussion and the com- tant. The current misperceptions and as- we may have made considerable prog-
sociated fears are being nurtured by the ress toward achieving “full AI.”
f Research on neural networks has gone through absence of scientific, precise, and bold Consider machine translation, which
many turns since their early traces in the 1940s.
perspectives on what just happened, received significant attention in the early
Nils Nilsson of Stanford University told me he
does not think the pessimistic predictions of leaving much to the imagination. days of AI. The represent-and-reason ap-
the 1969 book Perceptrons: An Introduction to This is not to suggest that only a proach aimed to comprehend text before
Computational Geometry by Marvin Minsky and new theory of the mind or an advance translating it and is considered to have
Seymour Papert was the real reason for the de- of such scale would justify some of the failed on this task, with function-based ap-
cline in neural network research back then, as
legitimate concerns surrounding AI. In proaches being the state of the art today.
is widely believed. Instead, it was the inability
to train multiple layers of weights that Nilsson fact, even limited AI technologies can In the early days of AI, success was mea-
also wrestled with at SRI during that time “but lead to autonomous systems that may sured by how far a system’s accuracy was
couldn’t get anywhere,” as he explained to me. pose all kinds of risks. However, these
g A perspective relayed to me by an anonymous re- concerns are not new to our industrial-
viewer is that science advances because instru- h See also the first report of the One Hundred
ments improve and that recent developments
ized society; recall safety concerns when Year Study on Artificial Intelligence (AI100) for
in neural networks could be viewed as improve- the autopilot was introduced into the a complementary perspective; https://ai100.
ments to our machine learning instruments. aerospace industry and job-loss con- stanford.edu/
The analogy given here was to genomics and the cerns when ATMs were introduced into i An anonymous reviewer said that throughout
development of high-throughput sequencing, the banking industry. The headline here AI there are metrics for evaluating task per-
which was not the result of a scientific break- formance but not for evaluating the fit among
through but rather of intense engineering ef-
should therefore be “automation” more an agent, its goals, and its environment. Such
forts, yet such efforts have indeed revealed a vast than “AI,” as the latter is just a tech- global metrics may be needed to assess and
amount about the human genome. nology that happened to improve and improve the intelligence of AI systems.

60 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

from 100% compared to humans, and Perhaps one of the broadest applica-
successful translation was predicated tions of these systems today is in user
on the ability to comprehend text. Gov- interfaces (such as automated tech-
ernment intelligence was a main driv- nical support and the commanding
ing application; a failure to translate
correctly can potentially lead to a politi- Some seemingly of software systems, as in phone and
navigation systems in vehicles). These
cal crisis. Today, the main application
of machine translation is to webpages
complex abilities systems fail often; try to say something
that is not very prototypical or not to
and social-media content, leading to a that are typically hide your accent if you have one. But
new mode of operation and a different
measure of success. In the new context,
associated with when these systems fail, they send
the user back to a human operator or
there is no explicit need for a transla- perception or force the user to command the soft-
tion system to comprehend text, only
to perform well based on the adopted
cognition can ware through classical means; some
users even adjust their speech to get
metrics. From a consumer’s viewpoint, be captured and the systems to work. Again, while the
success is effectively measured in terms
of how far a system’s accuracy is from reproduced to a performance of these systems has im-
proved, according to the adopted met-
0%. If I am looking at a page written in reasonable extent rics, they are today embedded in new
French, a language I do not speak, I am
happy with any translation that gives me by simply fitting contexts and governed by new modes
of operation that can tolerate lack of
a sense of what the page is saying. In fact,
the machine-translation community
functions to data. robustness or intelligence. Moreover,
as in text, improving their performance
rightfully calls this “gist translation.” It against current metrics is not neces-
can work impressively well on prototypi- sarily directed toward, nor requires
cal sentences that appear often in the addressing, the challenge of compre-
data (such as in social media) but can hending speech.l
fail badly on novel text (such as poetry). Moving to vision applications, it
It is still very valuable yet corresponds to has been noted that some object-rec-
a task that is significantly different from ognition systems, based on neural net-
what was tackled by early AI researchers. works, surpass human performance in
We did indeed make significant progress recognizing certain objects in images.
recently with function-based translation, But reports also indicate how making
thanks to deep learning. But this prog- simple changes to images may some-
ress has not been directed toward the times hinder the ability of neural net-
classical challenge of comprehending works to recognize objects correctly.
text, which aimed to acquire knowledge Some transformations or deformations
from text to enable reasoning about its to objects in images, which preserve
content,j instead of just translating it.k the human ability to recognize them,
Similar observations can be made can also hinder the ability of networks
about speech-recognition systems. to recognize them. While this does not
measure up to the expectations of early
j There are other views as to what “comprehen- AI researchers or even contemporary vi-
sion” might mean, as in, say, what might be
sion researchers, as far as robustness
revealed about language from the internal en-
codings of learned translation functions. and intelligence is concerned, we still
k With regard to the observation that the repre- manage to benefit from these technolo-
sent-and-reason approach is considered to have gies in a number of applications. This
failed on machine translation, Stuart Russell of includes recognizing faces during au-
the University of California, Berkeley, pointed
tofocus in smart cameras (people do
out to me that this is probably a correct descrip-
tion of an incorrect diagnosis, as not enough ef- not normally deform their faces but if
fort was directed toward pursuing an adequate they do, bad luck, an unfocused image);
represent-and-reason approach, particularly looking up images that contain cats in
one that is trainable, since language has too online search (it is ok if you end up get-
many quirks to be captured by hand. This ob-
servation is part of a broader perspective I sub-
ting a dog instead); and localizing sur-
scribe to calling for revisiting represent-and-rea- rounding vehicles in an image taken by
son approaches while augmenting them with
advances in machine learning. This task would, l An anonymous reviewer suggested that tran-
however, require a new generation of research- scription is perhaps the main application of
ers well versed in both approaches; see the sec- speech systems today, with substantial prog-
tion in this article on the power of success for ress made toward the preferred metric of
hints as to what might stand in the way of having “word error rate.” The same observation ap-
this breed of researchers. plies to this class of applications.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 61
contributed articles

the camera of a self-driving car (the vul- and regulatory levels.m The second
nerability of these systems to mistakes is that while function-based systems
remains controversial in both its scope have been an enabling and positive
and how to deal with it at the policy and development, we do need to be acute-
regulatory levels).
The significance of these observa- We succeeded in ly aware of the reasons behind their
success to better understand the im-
tions stems from their bearing on our
ability to forecast the future and deci-
these applications plications. A key finding here is that
some tasks in perception and cogni-
sions as to what research to invest in. by having tion can be emulated to a reasonable
In particular, does the success in ad-
dressing these selected tasks, which
circumvented extent without having to understand
or formalize these tasks as originally
are driven by circumscribed com- certain technical believed and sought, as in some text,
mercial applications, justify the worry
about doomsday scenarios? Does it
challenges instead speech, and vision applications. That
is, we succeeded in these applica-
justify claims that AI-based systems of having solved tions by having circumvented certain
can now comprehend language or
speech or do vision at the levels that them directly. technical challenges instead of hav-
ing solved them directly.n This ob-
humans do? Does it justify this cur- servation is not meant to discount
rent imbalance of attitudes toward current success but to highlight its
various machine learning and AI ap- nature and lay the grounds for this
proaches? If you work for a company question: How far can we go with this
that has an interest in such an appli- direction? I revisit this issue later in
cation, then the answer is perhaps, the article.
and justifiably, yes. But, if you are con-
cerned with scientific inquiry and un- Human-Level or Animal-Level?
derstanding intelligence more broad- Let me now get to the thoughts that
ly, then the answer is hopefully no. triggered the title of this article in
In summary, what has just hap- the first place. I believe human-level
pened in AI is nothing close to a break- intelligence is not required for the
through that justifies worrying about tasks currently conquered by neural
doomsday scenarios. What just hap- networks, as such tasks barely rise
pened is the successful employment to the level of abilities possessed by
of AI technology in some widespread many animals. Judea Pearl cited ea-
applications, aided greatly by devel- gles and snakes as having vision sys-
opments in related fields, and by new tems that surpass what we can build
modes of operation that can tolerate today. Cats have navigation abilities
lack of robustness or intelligence. that are far superior to any of those
Put another way—and in response to in existing automatous-navigation
headlines I see today, like “AI Has Ar- systems, including self-driving cars.
rived” and “I Didn’t See AI Coming”— Dogs can recognize and react to hu-
AI has not yet arrived according to the
early objective of capturing intelli-
m Eric Horvitz of Microsoft Research brought
gent behavior. What really has arrived
up the idea of subjecting certain AI systems to
are numerous applications that can trials as is done to approve drugs. The proper
benefit from improved AI techniques labeling of certain AI systems should also be
that still fall short of AI ambitions but considered, also as is done with drugs. For
are good enough to be capitalized on example, it has been suggested that the term
“self-driving car” is perhaps responsible for
commercially. This by itself is posi-
the misuse of this AI-based technology by
tive, until we confuse it with some- some drivers who expect more from the tech-
thing else. nology than is currently warranted.
Let me close this section by n For example, one can now use learned func-
stressing two points: The first is tions to recognize cats in images without
having to describe or model what a cat is, as
to reemphasize an earlier observa- originally thought and sought, by simply fitting
tion that while current AI technol- a function based on labeled data of the form:
ogy is still quite limited, the impact (image, cat), (image, not cat). While this ap-
it may have on automation, and proach works better than modeling a cat (for
hence society, may be substantial now), it does not entail success in “learning”
what a cat is, to the point where one can rec-
(such as in jobs and safety). This ognize, say, deformed images of cats or in-
in turn calls for profound treat- fer aspects of cats that are not relayed in the
ments at the technological, policy, training dataset.

62 COM MUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

man speech, and African grey parrots functions, even though we may not ity into this consequential question:
can generate sounds that mimic hu- be there today, given current thresh- How effective will function-based
man speech to impressive levels. Yet olds. But it begs the question: If it approaches be when applied to new
none of these animals has the cogni- is a matter of thresholds, and given and broader applications than those
tive abilities and intelligence typically current successes, why not focus all already targeted, particularly those
attributed to humans. our attention on moving thresholds that mandate more stringent mea-
One of the reactions I received to further? While there is merit to this sures of success? The question has
such remarks was: “I don’t know of any proposal, which seems to have been two parts: The first concerns the class
animal that can play Go!” This was in adopted by key industries, it does of cognitive tasks whose correspond-
reference to the AlphaGo system, which face challenges that stem from both ing functions are simple enough to al-
set a milestone in 2016 by beating the academic and policy considerations. low compact representations that can
world champion in the game. Indeed, I address academic considerations be evaluated efficiently (as in neural
we do not know of animals that can play next while leaving policy consider- networks) and whose estimation is
a game as complex as Go. But first recall ations to a later section. within reach of current thresholds—
the difference between performance From an academic viewpoint, the or thresholds we expect to attain in,
and intelligence: A calculator outper- history of AI tells us to be quite cau- say, 10 to 20 years. The second al-
forms humans at arithmetic without tious, as we have seen similar phe- ludes to the fact that these functions
possessing human or even animal cog- nomena before. Those of us who have are only approximations of cognitive
nitive abilities. Moreover, contrary to been around long enough can recall tasks; that is, they do not always get it
what seems to be widely believed, Al- the era of expert systems in the 1980s. right. How suitable or acceptable will
phaGo is not a neural network since At that time, we discovered ways to such approximations be when tar-
its architecture is based on a collection build functions using rules that were geting cognitive tasks that mandate
of AI techniques that have been in the devised through “knowledge engi- measures of success that are tighter
works for at least 50 years.o This includes neering” sessions, as they were then than those required by the currently
the minimax technique for two-player called. The functions created through targeted applications?
games, stochastic search, learning from this process, called “expert systems”
self-play, use of evaluation functions and “knowledge-based systems,” were The Power of Success
to cut off minimax search trees, and claimed to achieve performance that Before I comment on policy consid-
reinforcement learning, in addition to surpassed human experts in some erations, let me highlight a relevant
two neural networks. While a Go player cases, particularly in medical diagno- phenomenon that recurs in the his-
can be viewed as a function that maps a sis.q The term “knowledge is power” tory of science, with AI no exception.
board configuration (input) to an action was used and symbolized a jubilant I call it the “bullied-by-success” phe-
(output), the AlphaGo player was not state of affairs, resembling what “deep nomenon, in reference to the sub-
built by learning a single function from learning” has come to symbolize to- duing of a research community into
input-output pairs; only some of its day.r The period following this era mainly pursing what is currently suc-
components were built that way.p The came to be known as the “AI Winter,” cessful, at the expense of pursuing
issue here is not only about assigning as we could finally delimit the class of enough what may be more successful
credit but about whether a competitive applications that yielded to such sys- or needed in the future.
Go function can be small enough to be tems, and that class fell well short of Going back to AI history, some of
represented and estimated under cur- AI ambitions. the perspectives promoted during
rent data-gathering, storage, and com- While the current derivative for the expert-systems era can be safely
putational thresholds. It would be progress on neural networks has been characterized today as having been
quite interesting if this was the case, impressive, it has not been sustained scientifically absurd. Yet, due to the
but we do not yet know the answer. I long enough to allow sufficient visibil- perceived success of expert systems
should also note that AlphaGo is a then, these perspectives had a domi-
great example of what one can achieve q One academic outcome of the expert system nating effect on the course of scientific
today by integrating model-based and era was the introduction of a dedicated mas- dialogue and direction, leading to a
ter’s degree at Stanford University called the
function-based approaches. bullied-by-success community.s I saw a
“Master’s in AI” that was separate from the
master’s in computer science and had sig- similar phenomenon during the tran-
Pushing Thresholds nificantly looser course requirements. It sition from logic-based approaches
One cannot of course preclude the was a two-year program, with the second to probability-based approaches for
possibility of constructing a competi- year dedicated to building an expert system. commonsense reasoning in the late
I was a member of the very last class that
tive Go function or similarly complex graduated from the program before it was
1980s. Popular arguments then, like
terminated and recall that one of its justifi- “People don’t reason probabilistically,”
o Oren Etzioni of the Allen Institute for Artificial cations was that classical computer science
Intelligence laid out this argument during a techniques can be harmful to the “heuris- s A colleague could not but joke that the broad
talk at UCLA in March 2016 called Myths and tic” thinking needed to effectively build ex- machine learning community is being bullied
Facts about the Future of AI. pert systems. today by the success of its deep learning sub-
p AlphaZero, the successor to AlphaGo, used one r The phrase “knowledge is power” is appar- community, just as the broader AI community
neural network instead of two and data generat- ently due to English philosopher Sir Francis has been bullied by the success of its machine
ed through self-play, setting another milestone. Bacon (1561–1626). learning sub-community.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 63
contributed articles

which I believe carries merit, were genuine academic interests instead formation that goes beyond what can
completely silenced when proba- of just yielding to current fashions.t be extracted from data. To elaborate
bilistic approaches started solving on these points, I first need to explain
commonsense reasoning problems Policy Considerations why a function may not qualify as a
that had defied logical approaches Let me now address some policy con- model, a question I received during a
for more than a decade. The bullied- cerns with regard to focusing all our discussion on the subject.
by-success community then made attention on functions instead of Consider an engineered system
even more far-reaching choices in also on models. A major concern that allows us to blow air into a bal-
this case, as symbolic logic almost here relates to interpretability and loon that then raises a lever that is
disappeared from the AI curricula. explainability. If a medical-diagnosis positioned on top of the balloon.
Departments that were viewed as system recommends surgery, we The input to this system is the
world centers for representing and would need to know why. If a self- amount of air we blow (X), while the
reasoning with symbolic logic bare- driving car kills someone, we would output is the position of the lever
ly offered any logic courses as a re- also need to know why. If a voice (Y). We can learn a function that
sult. Now we are paying the price. command unintentionally shuts captures the behavior of the system
As one example: Not realizing that down a power-generation system, it by collecting X-Y pairs and then esti-
probabilistic reasoning attributes would need to be explained as well. mating the function Y = f (X). While
numbers to Boolean propositions in Answering “Why?” questions is cen- this function may be all we need for
the first place, and that logic was at tral to assigning blame and respon- certain applications, it would not
the heart of probabilistic reasoning sibility and lies at the heart of legal qualify as a model, as it does not
except in its simplest form, we have systems. It is also now recognized capture the system mechanism.
now come to the conclusion that we that opacity, or lack of explainabili- Modeling that mechanism is essen-
need to attribute probabilities to ty, is “one of the biggest obstacles tial for certain explanations (Why is
more complex Boolean propositions to widespread adoption of artificial the change in the lever position not
and even to first-order sentences. The intelligence.”u a linear function of the amount of
resulting frameworks are referred to Models are more interpretable air blown?) and for causal reasoning
as “first-order probabilistic models” than functions.v Moreover, models more generally (What if the balloon
or “relational probabilistic models,” offer a wider class of explanations is pinched?). One may try to address
and there is a great need for skill in than functions, including explana- these issues by adding more inputs
symbolic logic to advance these for- tions of novel situations and expla- to the function but may also blow up
malisms. The only problem is that nations that can form a basis for the function size, among other dif-
this skill has almost vanished from “understanding” and “control.” This ficulties; more on this next.
within the AI community. is due to models having access to in- In his The Book of Why: The New Sci-
The blame for this phenomenon ence of Cause and Effect, Judea Pearl
cannot be assigned to any particular t I made these remarks over a dinner table that explained further the differences be-
party. It is natural for the successful included a young machine learning researcher, tween a (causal) model and a function,
to be overjoyed and sometimes also whose reaction was: “I feel much better now.” He even though he did not use the term
was apparently subjected to this phenomenon “function” explicitly. In Chapter 1, he
inflate that success. It is expected that
by support-vector-machine (SVM) researchers
industry will exploit such success in during his Ph.D. work when SVMs were at their
wrote: “There is only one way a thinking
ways that may redefine the employ- peak and considered “it” at the time. Another entity (computer or human) can work
ment market and influence the aca- young vision researcher, pressed on whether out what would happen in multiple
demic interests of graduate students. deep learning is able to address the ambitions of scenarios, including some that it has
vision research, said, “The reality is that you can-
It is also understandable that the rest never experienced before. It must pos-
not publish a vision paper today in a top confer-
of the academic community may play ence if it does not contain a deep learning com- sess, consult, and manipulate a mental
along for the sake of its survival: win a ponent, which is kind of depressing.” causal model of that reality.” He then
grant, get a paper in, attract a student. u See Castellanos, S. and Norton, S. Inside gave an example of a navigation system
While each of these behaviors seems Darpa’s push to make artificial intelligence based on either reasoning with a map
explain itself. The Wall Street Journal (Aug.
rational locally, their combination (model) or consulting a GPS system that
10, 2017); http://on.wsj.com/2vmZKlM; DAR-
can be harmful to scientific inquiry PA’s program on “explainable artificial intel- gives only a list of left-right turns for ar-
and hence irrational globally. Beyond ligence”; https://www.darpa.mil/program/ riving at a destination (function). The
raising awareness about this recur- explainable-artificial-intelligence; and the rest of the discussion focused on what
ring phenomenon, decision makers E.U. general data protection regulation on “ex- can be done with the model but not the
plainability”; https://www.privacy-regulation.
at the governmental and academic eu/en/r71.htm
function. Pearl’s argument particularly
levels bear a particular responsibility v I am referring here to learned and large func- focused on how a model can handle
for mitigating its negative effects. Se- tions of the kind that stand behind some of the novel scenarios (such as encountering
nior members of the academic com- current successes (such as neural networks roadblocks that invalidate the function
with thousands or millions of parameters). recommendations) while pointing to
munity also bear the responsibility
This excludes simple or well-understood
of putting current developments in learned functions and functions synthesized
the combinatorial impossibility of en-
historical perspective, to empower from models, as they can be interpretable or coding such contingencies in the func-
junior researchers in pursuing their explainable by design. tion, as it must have a bounded size.

64 COMMUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

There is today growing work on model. However, to do this based on a

explaining functions, where the vo- learned function, the function would
cabulary of explanations is restricted need to be trained in the presence of
to the function inputs. For example, smokers or other smoke-producing
in medical diagnosis, an explanation
may point to important inputs (such Human-level agents while defining smoke as an
input to the function and assuring
as age, weight, and heart attack histo-
ry) when explaining why the function
intelligence is that smoke mediates the relationship
between fire and alarm, a task that re-
is recommending surgery. The func- not required quires external manipulation.
tion may have many more additional
inputs, so the role of an explanation
for the tasks As Pearl told me, model-based
explanations are also important be-
is to deem them irrelevant. In vision currently cause they give us a sense of “under-
applications, such explanations may
point to a specific part of the image
conquered by standing” or “being in control” of a
phenomenon. For example, knowing
that has led to recognizing an object; neural networks, that a certain diet prevents heart dis-
again, the role of an explanation is to
deem some pixels irrelevant to the as such tasks ease does not satisfy our desire for
understanding unless we know why.
recognition. These explanations are barely rise to Knowing that the diet works by lower-
practically useful, but due to their
limited vocabulary and the limited in- the level of abilities ing the cholesterol level in the blood
partially satisfies this desire because it
formation they can access, they could
face challenges when encountering
possessed opens up new possibilities of control.
For instance, it drives us to explore
novel situations. Moreover, they may by many animals. cholesterol-lowering drugs, which
not be sufficient when one is seeking may be more effective than diet. Such
explanations for the purpose of un- control possibilities are implicit in
derstanding or control. models but cannot be inferred from
Consider a function that predicts a learned, black-box function, as it
the sound of an alarm based on many has no access to the necessary infor-
inputs, including fire. An input- mation (such as that cholesterol level
based explanation may point to fire mediates the relationship between
as a culprit of the alarm sound. Such diet and heart disease).
an explanation relies effectively on A number of researchers contacted
comparing this scenario to similar me about the first draft of this sec-
scenarios in the data, in which the tion, which was focused entirely on
sound of the alarm was heard soon explanations, to turn my attention to
after fire was detected; these scenar- additional policy considerations that
ios are summarized by the function seem to require models. Like expla-
parameters. While this may explain nations, they all fell under the label
why the function reached a certain “reasoning about AI systems” but
conclusion, it does not explain why this time to ensure that the devel-
the conclusion (alarm sound) may be oped systems would satisfy certain
true in the physical world.w Nor does properties. At the top of these prop-
it explain how fire triggers the alarm; erties were safety and fairness, par-
is it, say, through smoke or through ticularly as they relate to AI systems
heat? The importance of these dis- that are driven only by data. These
tinctions surfaces when novel situ- considerations constitute further ex-
ations arise that have not been seen amples where models may be need-
before. For example, if the alarm is ed, not only to explain or compen-
triggered by smoke, then inviting a sate for the lack of enough data, but
smoker into our living room might to further ensure we are able to build
trigger an alarm even in the absence the right AI systems and reason about
of fire. In this case, pointing to fire as them rigorously.
an explanation of the sound would be
problematic. Humans arrive at such A Theory of Cognitive Functions
conclusions without ever seeing a One reaction I received concerning
smoker, which can also be achieved my model-based vs. function-based
through reasoning on an appropriate perspective was during a workshop
dedicated to deep learning at the Si-
w The function imitates data instead of reason- mons Institute for the Theory of Com-
ing about a model of the physical world. puting in March 2017. The workshop

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 65
contributed articles

title was “Representation Learning,” a as mapping audio signals to words

term used with increasing frequency and mapping words to some mean-
by deep learning researchers. If you ing). What is needed is a catalogue
have followed presentations on deep of cognitive functions and a study of
learning, you will notice that a critical
component of getting these systems to If I had my way, their representational complexity—
the size and nature of architectures
work amounts to finding the correct ar-
chitecture of the neural network. More-
I would rename needed to represent them—in ad-
dition to a study of their learnabil-
over, the architectures vary depending the field of ity and approximability. For Boolean
on the task, and some of their compo-
nents are sometimes portrayed as do-
deep learning functions, we have a deep theory of
this kind. In particular, researchers
ing something that can be described as “learning have cataloged various functions in
at an intuitive level. For example, in
language, one uses an encoder-decod-
approximations of terms of the space needed to repre-
sent them in different forms (such
er architecture in which the encoder cognitive functions.” as CNFs, DNFs, and OBDDs). What
transforms a sentence in the source we need is something similar for
language into an internal encoding, real-valued functions that are meant
and the decoder then generates a sen- to capture cognitive behaviors. In a
tence in the target language. sense, we already have some leads
The reaction here was that deep into such a theory; for example, re-
learning is not learning a function searchers seem to know what archi-
(black box) but a representation since tectures, or “function classes,” can be
the architecture is not arbitrary but more effective for certain object-
driven by the given task.x I see this dif- recognition tasks. This needs to be
ferently. Architecting the structure of a formalized and put on solid theoreti-
neural network is “function engineer- cal ground.z Such a theory would also
ing” not “representation learning,” par- include results on the learnability
ticularly since the structure is penalized of function classes using estimation
and rewarded by virtue of its conformity techniques employed by the deep
with input-output pairs. The outcome learning community, particularly
of function engineering amounts to re- “gradient descent.” Interestingly,
stricting the class of functions that can such results were presented at the
be learned using parameter estimation Representation Learning workshop
techniques. This process is akin to re- I referenced earlier in a talk called
stricting the class of distributions that “Failures of Deep Learning” in which
can be learned after one fixes the topol- very simple functions were presented
ogy of a probabilistic graphical model. that defeat current estimation tech-
The practice of representation learning niques. Even more interestingly,
is then an exercise in identifying the some have dismissed the importance
classes of functions that are suitable for of such results in side discussions
certain tasks.y on the grounds that the identified
In this context, I think what is functions are not of practical signifi-
needed most is a theory of cogni- cance; read “these are not cognitive
tive functions. A cognitive function functions” or “we have come a long
captures a relationship that is typi- way by learning approximations to
cally associated with cognition (such functions.” In fact, if I had my way, I
would rename the field of deep learn-
ing as “learning approximations of
x There are other broader interpretations of the
term “representation learning.” cognitive functions.”
y An anonymous reviewer suggested today’s The term “cognitive functions” sur-
practice of building deep neural networks prised some colleagues who told me
can be viewed as the application of a new pro- that “perception functions” may be
gramming paradigm called “differentiable
programming.” In this view, networks are
more suitable, given that the current
carefully structured by a programmer using successes of deep learning have been
various differentiable program modules (such
as convolutional layers, pooling layers, LSTM z The properties of learned functions may carry
layers, residual blocks, and embedding lay- quite a bit of insight about the structure of our
ers). The compiler then differentiates and world; for example, linguists are called upon
structures them for GPU execution. The key is to study this phenomenon and unveil what
to structure the program so the gradients are learned translation functions may be reveal-
guided to do the right thing. ing about the structure of language.

66 COMMUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

mostly in instinct-based perception profound scientific contributions.aa tinuing to share its contents verbally
(such as computer vision and language On the other hand, I am reminded how in various contexts and revising ac-
processing). I agree with this obser- times of achievements can potentially cordingly. The decision to eventually
vation, except nothing at this stage slow scientific progress by shifting aca- release a first draft in July 2017 was
prohibits functions from providing demic interests, resources, and brain triggered by two events: a discussion
reasonable approximations to more power too significantly toward exploit- of these thoughts at a workshop orga-
high-level cognitive tasks. In fact, Go ing what was just discovered, at the nized by the UCLA School of Law and
functions have been constructed us- expense of understanding the discov- other discussions with colleagues
ing neural networks, even though they eries and preparing for the moment outside of AI, including architecture,
are not yet competitive with hybrid when their practical applications have programming languages, networks,
systems (such as AlphaGo). Admit- been delimited or exhausted. and theory. These discussions re-
tedly, it is also possible that we might There are many dimensions to vealed a substantial interest in the
later realize that functions (of practical such preparation. For the deep learn- subject and led me to conclude that
size) cannot provide reasonable ap- ing community, perhaps the most sig- the most important objective I should
proximations to a wide enough class nificant is a transition from the “look be seeking is “starting a discussion.”
of cognitive functions despite prog- what else we can do” mode to a “look I may have erred in certain parts, I
ress on pushing computational and what else you can do” mode. This is may have failed to give due credit, and
data thresholds. The association with not only an invitation to reach out to I may have missed parts of the evolv-
perception would then be more estab- and empower the broader AI commu- ing scene. I just hope the thoughts I
lished in that case. Time will tell. nity; it is also a challenge since such a share here will start that discussion,
transition is not only a function of at- and the collective wisdom of the com-
Conclusion titude but also an ability to character- munity will correct what I may have
This article was motivated by concerns ize progress in ways that enable people gotten wrong.
I and others have had on how current from outside the community to under-
progress in AI is being framed and stand and capitalize on it. The broader Acknowledgments
perceived. Without a scholarly discus- AI community is also both invited and I benefited greatly from the feedback
sion of the causes and effects of recent challenged to identify fundamental I received from anonymous review-
achievements, and without a proper ways in which functions can be turned ers and from colleagues who are
perspective on the obtained results, into a boon for building and learning too many to enumerate but whose
one stands to hinder further progress models. Given where we stand today, input and discussions were critical
by perhaps misguiding the young gen- the question is not whether it is func- to shaping the thoughts expressed
eration of researchers or misallocating tions or models but how to profoundly here. However, I must specifically
resources at the academic, industrial, integrate and fuse functions with mod- acknowledge Judea Pearl for inspir-
and governmental levels. One also els.ab This aim requires genuine cross- ing the article and for helping with
stands to misinform a public that has fertilization and the training of a new various arguments; Stuart Russell
developed a keen interest in AI and its generation of researchers who are well- for providing very thoughtful and
implications. The current negative dis- versed in and appreciative of various AI constructive feedback; Guy Van den
cussions by the general public on the methods, and who are better informed Broeck for keeping me interested in
AI singularity, also called “super intel- about the history of AI. the project every time I almost gave
ligence,” is partly due to the lack of ac- I conclude with this reflection: up; and Arthur Choi for being a gen-
curate framings and characterizations I wrote the first draft of this article erous and honest companion to the
of recent progress. With almost every- in November 2016. A number of col- thinking that went into it. Finally, I
one being either overexcited or over- leagues provided positive feedback wish to thank Nils Nilsson for telling
whelmed by the new developments, then, with one warning about a nega- me that he wished he had written the
substantial scholarly discussions and tive tone. I put the draft on hold for article and for kindly inviting me to
reflections have gone missing. some months as a result while con- share his feedback with others. This
I had the privilege of starting my is an ultimate reward.
research career in AI around the mid-
aa Judea Pearl’s seminal work on probabilistic
to-late 1980s during one of the major Adnan Darwiche (darwiche@cs.ucla.edu) is a professor
approaches to commonsense reasoning is one in and chairman of the Computer Science Department at
crises in the field, a period marked example outcome of the crisis. the University of California, Los Angeles, CA, USA.
by inability instead of ability. I was ab An anonymous reviewer brought to my atten-
dismayed then, as I sat in classes at tion works on the analyses of human cogni- Copyright held by author.
tion, particularly Daniel Kahneman’s book
Stanford University, witnessing how Thinking Fast and Slow. The reviewer said
AI researchers were being significant- “fast” naturally maps onto function-based
ly challenged by some of the simpler and “slow” onto model-based, and there is a
tasks performed routinely by humans. strong argument in the literature on cogni-
I now realize how such crises can be tive science that people must at least com- Watch the author discuss
bine them both. The reviewer further pointed his work in this exclusive
enabling for scientific discovery, as out that there are a variety of cognitive ar- Communications video.
they fuel academic thinking, empower chitectures that embody specific hypotheses human-level-intelligence-or-
researchers, and create grounds for about such hybrids. animal-like-abilities

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 67
contributed articles
DOI:10.1145/ 3230627
One might think surviving such
Verified software secures the Unmanned an attack is not a big deal, certainly
that military aircraft would be robust
Little Bird autonomous helicopter against against cyber attacks. In reality, a
mid-flight cyber attacks. “red team” of professional penetra-
tion testers hired by the Defense Ad-
Cyber Military Systems (HACMS)

program had in 2013 compromised
the baseline version of the ULB, de-
signed for safety rather than secu-
rity, to the point where it could have

crashed it or diverted to any location
of its choice. In this light, risking
an in-flight attack with a human on
board indicates that something had

Software in
changed dramatically.
This article explains that change
and the technology that enabled it.
Specifically, it is about technology de-

the Real World

veloped under the HACMS program,
aiming to ensure the safe operation of
critical real-world systems in a hostile
cyber environment—multiple autono-
mous vehicles in this case. The tech-
nology is based on formally verified
software, or software with machine-
checked mathematical proofs it be-
haves according to its specification.
While this article is not about the for-
mal methods themselves, it explains
how the verified artifacts can be used
to secure practical systems. The most
a helicopter took off from a Boeing
I N FE B R UAR Y 2017,
impressive outcome of HACMS is ar-
facility in Mesa, AZ, on a routine mission around guably that the technology could be
nearby hills. It flew its course fully autonomously, retrofitted onto existing real-world
systems, dramatically improving their
and the safety pilot, required by the Federal Aviation cyber resilience, a process called “seis-
Administration, did not touch any controls during mic security retrofit” in analogy to,
say, the seismic retrofit of buildings.
the flight. This was not the first autonomous flight Moreover, most of the re-engineering
of the AH-6, dubbed the Unmanned Little Bird
(ULB);3 it had been doing them for years. This time, key insights
however, the aircraft was subjected to mid-flight ˽˽ Formal proof based on micro-kernel-
enforced software architecture can scale
cyber attacks. The central mission computer was to real systems at low cost.
attacked by rogue camera software, as well as by a ˽˽ Mixed assurance levels and security
virus delivered through a compromised USB stick that levels within one system are possible and
desirable; not all code has to be assured
had been inserted during maintenance. The attack to the highest level.

compromised some subsystems but could not affect ˽˽ High assurance can be retrofitted to
suitable existing systems with only
the safe operation of the aircraft. moderate redesign and refactoring.

68 COMMUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

Boeing Little Bird in unmanned flight test.

was done by Boeing engineers, not by tecture, enforce the desired security Formal Verification
formal verification researchers. property, and our verified component Mathematical correctness proofs of pro-
By far, not all the software on the framework, CAmkES. The CAmkES grams go back to at least the 1960s,14
HACMS vehicles was built on the basis framework integrates with architec- but for a long time, their real-world
of mathematical models and reason- ture analysis tools from Rockwell Col- benefit to software development was
ing; the field of formal verification is lins and the University of Minnesota, limited in scale and depth. However, a
not yet ready for such scale. However, along with trusted high-assurance number of impressive breakthroughs
HACMS demonstrated that significant software components using domain- have been seen in recent years in the
improvement is feasible by applying specific languages from Galois Inc. formal code-level verification of real-
formal techniques strategically to the The HACMS achievements are life systems, from the verified C com-
most critical parts of the overall sys- based on the software engineer’s trusty piler CompCert28 to the verified seL4
tem. The HACMS approach works for old friend—modularization. What is microkernel,22,23,33 verified conference
systems in which the desired secu- new is that formal methods provide system CoCon,21 verified ML compiler

rity property can be achieved through proof that interfaces are observed and CakeML,25 verified interactive theorem
purely architecture-level enforcement. module internals are encapsulated. provers Milawa,9 and Candle,24 veri-
Its foundation is our verified microker- This guaranteed enforcement of mod- fied crash-resistant file system FSCQ,5
nel, seL4, discussed later, which guar- ularization allows engineers, like those verified distributed system IronFleet,19
antees isolation between subsystems at Boeing, who are not formal-method and verified concurrent kernel frame-
except for well-defined communica- experts, to construct new or even ret- work CertiKOS,17 as well as significant
tion channels that are subject to the rofit existing systems, as discussed mathematical theorems, including the
system’s security policy. This isolation later, and achieve high resilience, even Four Colour Theorem,15 mechanized
is leveraged by system-level compo- though the tools do not yet provide an proof of the Kepler Conjecture,18 and
nent architectures that, through archi- overall proof of system security. Odd Order Theorem.16 None of these

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 69
contributed articles

Figure 1. Isolation and controlled communi- hold—through other means of verifi- other publicly available software in hu-
cation with seL4. cation like testing. Moreover, in many man history in terms not only of lines
cases systems consist of a combina- of proof but strength of properties
Untrusted VM Untrusted Trusted tion of verified and non-verified code, proved. At the heart of this verification
and in them, formal verification acts story sits the proof of “functional cor-
Guest apps Native apps Native apps
as a lens, focusing review, testing, and rectness” of the kernel’s C implemen-
Guest OS debugging on the system’s critical non- tation,23 guaranteeing every behavior
verified code. of the kernel is predicted by its formal
abstract specification; see the online
Hardware seL4 appendix (dl.acm.org/citation.cfm?
We begin with the foundation for build- doid=3230627&picked=formats) for an
ing provably trustworthy systems—the idea of how these proofs look. Following
are toy systems. For instance, Comp- operating system (OS) kernel, the sys- this guarantee, we added further proofs
Cert is a commercial product, the seL4 tem’s most critical part and enabler we explain after first introducing the
microkernel is used in aerospace, au- of cost-effective trustworthiness of the main kernel mechanisms.
tonomous aviation, and as an Internet entire system. seL4 API. The seL4 kernel provides a
of Things platform, and the CoCon The seL4 microkernel provides a minimal set of mechanisms for imple-
system has been used in multiple full- formally verified minimal set of mecha- menting secure systems: threads, ca-
scale scientific conferences. nisms for implementing secure sys- pability management, virtual address
These verification projects required tems. Unlike standard separation ker- spaces, inter-process communication
significant effort, and for verification nels31 they are purposefully general and (IPC), signaling, and interrupt delivery.
to be practical for widespread use, the so can be combined for implementing a The kernel maintains its state in
effort needs to decrease. Here, we dem- range of security policies for a range of “kernel objects.” For example, for each
onstrate how strategically combining system requirements. thread in a system there is a “thread
formal and informal techniques, par- One of the main design goals of object” that stores information about
tially automating the formal ones, and seL4 (see the sidebar “Proof Effort”) scheduling, execution, and access con-
carefully architecting the software to is to enforce strong isolation between trol. User-space programs can refer to
maximize the benefits of isolated com- mutually distrusting components that kernel objects only indirectly through
ponents, allowed us to dramatically in- may run on top of it. The mechanisms “capabilities”10 that combine a refer-
crease the assurance of systems whose support its use as a hypervisor to, say, ence to an object with a set of access
overall size and complexity is orders- host entire Linux operating systems rights to this object. For example, a
of-magnitude greater than that of the while keeping them isolated from se- thread cannot start or stop another
systems mentioned earlier. curity-critical components that might thread unless it has a capability to the
Note we primarily use formal veri- run alongside, as outlined in Figure 1. corresponding thread object.
fication to provide proofs about cor- In particular, this functionality allows Threads communicate and syn-
rectness of code that a system’s safety system designers to deploy legacy chronize by sending messages through
or security relies on. But it has other components that may have latent vul- IPC “endpoint” objects. One thread
benefits as well. For example, code nerabilities alongside highly trustwor- with a send capability to an appropri-
correctness proofs make assumptions thy components. ate endpoint can message another
about the context in which the code is The seL4 kernel is unique among thread that has a receive capability to
run (such as behavior of hardware and general-purpose microkernels. Not that endpoint. “Notification” objects
configuration of software). Since for- only does it deliver the best perfor- provide synchronization through sets
mal verification makes these assump- mance in its class, 20 its 10,000 lines of binary semaphores. Virtual address
tions explicit, developer effort can of C code have been subjected to translation is managed by kernel ob-
focus on ensuring the assumptions more formal verification than any jects that represent page directories,

Figure 2. Kernel objects for an example seL4-based system with two threads communicating via an endpoint.

CSpace CSpace
Thread ObjectA Thread Object
VSpace EP B
CNodeA1 CNodeB1








... VSpace

70 COMMUNICATIO NS O F TH E ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

page tables, and frame objects, or thin

abstractions over the corresponding
entities of the processor architecture.
Each thread has a designated “VSpace”
Proof Effort
seL4 design and code development took two person-years. Adding up all seL4-
capability that points to the root of the specific proofs over the years comes to a total of 18 person-years for 8,700 lines
thread’s address-translation object of C code. In comparison, L4Ka::Pistachio, another microkernel in the L4 family,
tree. Capabilities themselves are man- comparable in size to seL4, took six person-years to develop and provides no
aged by the kernel and stored in ker- significant level of assurance. This means there is only a factor 3.3 between verified
software and traditionally engineered software. According to the estimation
nel objects called “CNodes” arranged method by Colbert and Boehm,8 a traditional Common Criteria EAL7 certification
in a graph structure that maps object for 8,700 lines of C code would take more than 45.9 person-years. That means
references to access rights, analo- formal binary-level implementation verification is already more than a factor of
2.3 less costly than the highest certification level of Common Criteria yet provides
gous to page tables mapping virtual to significantly stronger assurance.
physical addresses. Each thread has a In comparison, the HACMS approach described here uses only these existing proofs
distinguished capability identifying for each new system, including the proofs generated from tools. The overall proof effort
a root CNode. We call the set of capa- for a system that fits this approach is thus reduced to person-weeks instead of years,
and testing can be significantly reduced to only validating proof assumptions.
bilities reachable from this root the
thread’s “CSpace.” Capabilities can
be transmitted over endpoints with to modify data in the system (including chitectures—ARMv6, ARMv7, ARMv7a,
the grant operation and can be shared by any system calls it might perform) ARMv8, RISC-V, Intel x86, and Intel
via shared CSpaces. Figure 2 outlines the access control policy does not ex- x64—and its machine-checked proof33
these kernel objects on an example. plicitly allow it to modify. For instance, is current on the ARMv7 architecture
Security proofs. With its generality, in Figure 2, the only authority compo- for the whole verification stack, as well
seL4’s kernel API is necessarily low-level nent A has over another component is as on ARMv7a with hypervisor exten-
and admits highly dynamic system ar- the send right to the endpoint from sions for functional correctness.
chitectures. Direct reasoning about which component B receives. This
this API can thus be a challenge. means the maximum state change A Security by Architecture
The higher-level concept of access can effect in the system is in A itself The previous section summarized the
control policies abstracts away from and in B’s thread state and message seL4 kernel software engineers can
individual kernel objects and capabili- buffer. It cannot modify any other parts use as a strong foundation for prov-
ties, capturing instead the access-con- of the system. ably trustworthy systems. The kernel
trol configuration of a system via a set of The dual of integrity is confidenti- forms the bottom layer of the trusted
abstract “subjects” (think components) ality, which states that a component computing base (TCB) of such sys-
and the authorities each has over the cannot read another component’s tems. The TCB is the part of the soft-
others (such as to read data and send data without permission,29 proved the ware that needs to work correctly for
a message). In the example in Figure 2, stronger property of intransitive non- the security property of interest to
the system would have components A interference for seL4; that is, given a hold. Real systems have a much larger
and B with authority over the endpoint. suitably configured system (with stron- TCB than just the microkernel they
Sewell et al.36 proved for such suit- ger restrictions than for integrity), no run on, and more of the software stack
able access control policies that seL4 component is able to learn information would need to be formally verified to
enforces two main security properties: about another component or its execu- gain the same level of assurance as for
authority confinement and integrity. tion without explicit permission. The the kernel. However, there are classes
Authority confinement states that proof expresses this property in terms of systems for which this is not neces-
the access control policy is a static (un- of an information-flow policy that can sary, for which the kernel-level isola-
changing) safe approximation of the be extracted from the access-control tion theorems are already enough to
concrete capabilities and kernel ob- policy used in the integrity proof. Infor- enforce specific system-level security
jects in the system for any future state mation will flow only when explicitly properties. This section includes an
of execution. This property implies that allowed by the policy. The proof cov- example of such a system.
no matter how the system develops, no ers explicit information flows, as well The systems for which this works
component will ever gain more author- as potential in-kernel covert storage are those in which component archi-
ity than the access control policy pre- channels, but timing channels are out- tectures alone already enforce the criti-
dicts. In Figure 2, the policy for compo- side its scope and must be addressed cal property, potentially together with a
nent B does not contain write access to through different means.6 few small, trusted components. Our ex-
component A, and B will thus never be Further proofs about seL4 include ample is the mission-control software
able to gain this access in the future. the extension of functional correct- of a quadcopter that was the research-
The property thus implies that reason- ness, and thus the security theorems, demonstration vehicle in the HACMS
ing at the policy level is a safe approxi- to the binary level for the ARMv7 ar- program mentioned earlier.
mation over reasoning about the con- chitecture35 and a sound worst-case Figure 3 outlines the quadcopter’s
crete access-control state of the system. execution time profile for the kernel2,34 main hardware components. It is in-
Integrity states that no matter what necessary for real-time systems. The tentionally more complex than needed
a component does, it will never be able seL4 kernel is available for multiple ar- for a quadcopter, as it is meant to be

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 71
contributed articles

representative of the ULB, and is, at hicle’s flight behavior. The operating behavior of that component, keys can-
this level of abstraction, the same as assumption is that the camera is un- not be leaked, as no other component
the ULB architecture. trusted and potentially compromised, has access to them; the link between
The figure includes two main com- or malicious, that its drivers and the Linux and the crypto component in Fig-
puters: a mission computer that com- legacy payload software are poten- ure 4 is for message passing only and
municates with the ground-control tially compromised, and any outside does not give access to memory. Only
station and manages mission-payload communication is likewise potentially authenticated messages can reach the
software (such as for controlling a compromised. For the purpose of this CAN bus, as the crypto component is
camera); and a flight computer with example, we assume a correct and the only connection to the driver. Un-
the task of flying the vehicle, reading strong cryptography implementation, trusted payload software and WiFi are,
sensor data, and controlling motors. or the key cannot be guessed, and that as part of the Linux VM, encapsulated
The computers communicate via an basic radio jamming and denial-of-ser- by component isolation and can com-
internal network, a controller area net- vice by overwhelming the ground sta- municate to the rest of the system only
work, or CAN bus, on the quadcopter, tion radio link are out of scope. via the trusted crypto component.
a dedicated Ethernet on the ULB. On Figure 4 outlines how we design It is easy to imagine that this kind
the quadcopter, the mission computer the quadcopter architecture to achieve of architecture analysis could be auto-
also has an insecure WiFi link, giving these properties. We use a virtual ma- mated to a high degree through model
us the opportunity to demonstrate fur- chine (VM) running Linux as a contain- checking and higher-level mechanized
ther security techniques. ment vessel for legacy payload soft- reasoning tools. As observed in MILS
The subsystem under consider- ware, camera drivers, and WiFi link. systems,1 component boundaries in
ation in this example is the mission We isolate the cryptography control an architecture are not just a conve-
computer. Four main properties must module in its own component, with nient decomposition tool for modu-
be enforced: only correctly authenti- connections to the CAN bus compo- larity and code management but,
cated commands from the ground sta- nent, to the ground station link, and with enforced isolation, provide ef-
tion are sent to the flight computer; to the Linux VM for sending image- fective boundaries for formal reason-
cryptographic keys are not leaked; no recognition data back to the ground ing about the behavior of the system.
additional messages are sent to the station. The purpose of the crypto However, the entire argument hinges
flight computer; and untrusted pay- component is to forward (only) autho- on the fact that component boundar-
load software cannot influence the ve- rized messages to the flight computer ies in the architecture are correctly en-
via the CAN interface stack and send forced at runtime in the final, binary
Figure 3. Autonomous-air-vehicle architecture. back diagnostic data to the ground sta- implementation of the system.
tion. The radio-link component sends The mechanisms of the seL4 kernel
and receives raw messages that are en- discussed earlier can achieve this en-
Ground Mission
Station Link Computer crypted, decrypted, and authenticated, forcement, but the level of abstraction
Camera respectively, by the crypto component. of the mechanisms is in stark contrast
Establishing the desired system to the boxes and arrows of an architec-

properties is now reduced purely to the ture diagram; even the more abstract
isolation properties and information- access-control policy still contains
Sensors Motors flow behavior of the architecture, and far more detail than the architecture
to the behavior of the single trusted diagram. A running system of this size
crypto component. Assuming correct contains tens of thousands of kernel
objects and capabilities that are cre-
Figure 4. Simplified quadcopter mission-computer architecture. ated programmatically, and errors in
configuration could lead to security
violations. We next discuss how we not
only automate the configuration and
Radio Data
Linux VM construction of such code but also how
we can automatically prove that archi-
WiFi tecture boundaries are enforced.
Crypto Verified Componentization
The same way reasoning about secu-
rity becomes easier with the formal ab-
CAN bus
Untrusted stractions of security policies, abstrac-
Driver Trusted tion also helps in building systems.
The CAmkES component platform,27
which runs on seL4 abstracts over the
seL4 low-level kernel mechanisms, provides
communication primitives, as well as
support for decomposing a system into

72 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

functional units, as in Figure 5. Using Figure 5. CAmkES workflow.

this platform, systems architects can
design and build seL4-based systems CAmkES
in terms of high-level components that
communicate with each other and with A B +
hardware devices through connectors
like remote procedure calls (RPCs),
dataports, and events.
Generated code. Internally, CAmkES
A B + proof glue
implements these abstractions using Thread
Thread CSpace CSpace code
seL4’s low-level kernel objects. Each Object Object
CNode EP CNode
component comprises (at least) one + proof
thread, a CSpace, and a VSpace. RPC


connectors use endpoint objects, and


CAmkES generates glue code to mar- VSpace VSpace
shal and unmarshal messages and
send them over IPC endpoints. Like-
wise, a dataport connector is imple-
mented through shared memory,
initialized system + proof
shared frame objects present in the ad-
dress spaces of two components, and
optionally restricting the direction of
communication. Finally, an event con- Figure 6. RPC-generated code.
nector is implemented using seL4’s
notification mechanism. A B
CAmkES also generates, in the cap- g() {
DL language,26 a low-level specification f(); ...
of the system’s initial configuration of handwritten }
kernel objects and capabilities. This
capDL specification is the input for the f() { g_stub() {
generic seL4 initializer that runs as the //glue: //glue:
// marshalling // seL4_Recv(ep,...)
first task after boot and performs the // unmarshalling
generated // seL4_Send(ep,...) seL4
necessary seL4 operations to instanti- // seL4_Recv(ep,...) // g_invoke()
ate and initialize the system.4 // unmarshalling // marshalling
} // seL4_Send(ep,...)
In summary, a component platform }
provides free code. The component ar-
chitecture describes a set of boxes and
arrows, and the implementation task is
reduced to simply filling in the boxes; tomate large parts of system construc- lent to calling g. The lemma the system
the platform generates the rest while tion without expanding the trusted generates ensures the invocation of the
enforcing the architecture. computing base. generated RPC glue code f behaves as
With a traditional component plat- Developers rarely look at the output a direct invocation of g, as if it were co-
form, the enforcement process would of code generators, focusing instead on located with the caller.
mean the generated code increases the the functionality and business logic of To be useful, the proofs the system
trusted computing base of the system, their systems. In the same way, we in- generates must be composable with
as it has the ability to influence the tend the glue code proofs to be artifacts (almost) arbitrary user-provided proofs,
functionality of the components. How- that do not need to be examined, mean- both of the function g and of the contexts
ever, CAmkES also generates proofs. ing developers can focus on proving the where g and f are used. To enable this
Automated proofs. While generat- correctness of their handwritten code. composability, the specification of the
ing glue code, CAmkES produces for- Mirroring the way a header generated by connectors is parameterized through
mal proofs in Isabelle/HOL, following CAmkES gives the developer an API for user-provided specifications of remote
a translation-validation approach,30 the generated code, the top-level gener- functions. In this way, proof engineers
demonstrating that the generated glue ated lemma statements produce a proof can reason about their architecture,
code obeys a high-level specification API. The lemmas describe the expected providing specifications and proofs for
and the generated capDL specification behavior of the connectors. In the ex- their components, and rely on specifica-
is a correct refinement of the CAmkES ample of RPC glue code outlined in Fig- tions for the generated code.
description.12 We have also proved that ure 6, the generated function f provides To date, we have demonstrated this
the generic seL4 initializer correctly a way to invoke a remote function g in process end-to-end using a specific
sets up the system in the desired ini- another component. To preserve the CAmkES RPC connector.12,13 Extending
tial configuration. In doing so, we au- abstraction, calling f must be equiva- the proof generator to support other

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 73
contributed articles

connectors, allowing construction of izer satisfies the one described in the mission-computer functionality. The
more diverse verified systems, should given specification.4 This proof holds system was built and re-engineered by
be simpler to achieve, because other for a precise model of the initializer Boeing engineers, using the methods,
connector patterns (data ports and but not yet at the implementation lev- tools, and components provided by the
events) are significantly less complex el. Compared to the depth of the rest HACMS partners.
than RPC. of the proof chain, this limitation may Step 1. Virtualization. The first step
Next to communication code, appear weak, but it is already more for- was to take the system as is and run it
CAmkES produces the initial access mal proof than would be required for in a VM on top of a secure hypervisor
control configuration that is designed the highest level (EAL7) of a Common (see Figure 7). In the seismic-retrofit
to enforce architecture boundaries. To Criteria security evaluation. metaphor, doing so corresponds to
prove the two system descriptions— situating the system on a more flex-
capDL and CAmkES—correspond, we Seismic Security Retrofit ible foundation. A VM on top of seL4
consider the CAmkES description as In practice, there are few opportuni- in this system consists of one CAmkES
an abstraction of the capDL descrip- ties to engineer a system from scratch component that includes a virtual ma-
tion. We use the established frame- for security, so the ability to retrofit chine monitor (VMM) and the guest
work36 mentioned earlier to infer for security is crucial for engineer- operating system, in this case Linux.
authority of one object over another ing secure systems. Our seL4-based The kernel provides abstractions of
object from a capDL description to lift framework supports an iterative pro- the virtualization hardware, while the
reasoning to a policy level. Addition- cess we call “seismic security retrofit,” VMM manages these abstractions for
ally, we have defined rules for inferring as a regular structural architect might the VM. The seL4 kernel constrains not
authority between components in a retrofit an existing building for great- only the guest but also the VMM, so the
CAmkES description. The produced er resilience against earthquakes. VMM implementation does not need
proof ensures the capDL objects, when We illustrate the process by walking to be trusted to enforce isolation. Fail-
represented as an authority graph through an example that incremental- ure of the VMM will lead to failure of
with objects grouped per component, ly adapts the existing software archi- the guest but not to failure of the com-
have the same intergroup edges as the tecture of an autonomous air vehicle, plete system.
equivalent graph between CAmkES moving it from a traditional testing Depending on system configura-
components.12 Intuitively, this corre- approach to a high-assurance sys- tion, the VM may have access to hard-
spondence between the edges means tem with theorems backed by formal ware devices through para-virtualized
an architecture analysis of the policy methods. While this example is based drivers, pass-through drivers, or both.
inferred by the CAmkES description on work done for a real vehicle—the In the case of pass-through drivers,
will hold for the policy inferred by the ULB—it is simplified for presentation developers can make use of a system
generated capDL description, which in and does not include all details. MMU or IOMMU to prevent hardware
turn is proved to satisfy authority con- The original vehicle architecture devices and drivers in the guest from
finement, integrity, and confidential- is the same as the architecture out- breaching isolation boundaries. Note
ity, as mentioned earlier. lined in Figure 3. Its functionality is that simply running a system in a VM
Finally, to prove correct initializa- split over two separate computers: a adds no additional security or reliabil-
tion, CAmkES leverages the generic flight computer that controls the ac- ity benefits. Instead, the reason for this
initializer that will run as the first user tual flying and the mission computer first step is to enable step 2.
task following boot time. In seL4, this that performs high-level tasks (such as Step 2. Multiple VMs. The second
first (and unique) user task has access ground-station communication and step in a seismic retrofit strengthens
to all available memory, using it to cre- camera-based navigation). The origi- existing walls. In software, the devel-
ate objects and capabilities accord- nal version of the mission computer oper can improve security and reli-
ing to the detailed capDL description was a monolithic software application ability by splitting the original system
it takes as input. We proved that the running on Linux. The rest of the ex- into multiple subsystem partitions,
state following execution of the initial- ample concentrates on a retrofit of this each consisting of a VM running the

Figure 7. All functionality in a single VM. Figure 8. Functionality split into multiple VMs. Figure 9. Functionality in native components.

Untrusted Trusted Untrusted Untrusted

Crypto Comms
Mission Manager and Camera (Native) (Native)
Ground Station Link and Camera
Network and Ground
and WiFi and
Stack Station Network Mission
WiFi and WiFi
Link Stack Manager
Camera (Native)
Net Net (Native) (Linux VM) (Linux VM) Net (Linux VM)
(Linux VM)
(Linux VM)
Virtual Machine Monitor (VMM) VMM VMM (Native) VMM

seL4 seL4 seL4

74 COM MUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

code of only part of the original system. benefit of transforming a component

Each VM/VMM combination runs in into native code is a reduced footprint
a separate CAmkES component that and better performance, removing the
introduces isolation between the dif- guest operating system and removing
ferent subsystems, keeping mutually
distrusting ones from affecting each We intend the execution and communication
overhead of the VMM.
other, and, later, allowing different as-
surance levels to coexist.
the glue code proofs Using a native component also in-
creases the potential for applying for-
In general, the partitions follow to be artifacts mal verification and other techniques
the existing software architecture, al-
though a redesign may be necessary
that do not need for improving the assurance and
trustworthiness of the component. Ex-
where the software architecture is in- to be examined, amples range from full functional veri-
adequate for effective isolation.
The partitions will in general need
meaning fication of handwritten code to cogen-
eration of code and proofs, application
to communicate with each other, so in developers can of model checking, using type-safe pro-
this step we also add appropriate com-
munication channels between them. focus on proving gramming languages, and static analy-
sis or traditional thorough testing of a
For security, it is critically important the correctness smaller codebase.
that these interfaces are narrow, lim-
iting the communication between of their handwritten Due to the isolation provided by
seL4 and the componentized archi-
partitions to only what is absolutely
necessary to maximize the benefits of
code. tecture, it becomes possible for com-
ponents of mixed assurance levels to
isolation. Moreover, interface proto- coexist in the system without decreas-
cols should be efficient, keeping the re- ing the overall assurance to that of the
quired number of messages or amount lowest-assurance component or in-
of data copying minimal. Critically, creasing the verification burden of the
seL4’s ability to enable controlled and lowest-assurance components to that
limited sharing of memory between of the highest-assurance ones.
partitions allows a developer to mini- In our example, we target the VM for
mize the amount of data copying. mission manager and ground-station
Besides the VMs that represent sub- link, implementing the communica-
systems of the original system, we also tions, cryptography, and mission-man-
extract and implement components ager functionality as native compo-
for any shared resources (such as the nents. We leave the camera and WiFi
network interface). to run in a VM as an untrusted legacy
We can iterate the entire step 2 until component (see Figure 9). This split
we have achieved the desired granular- was a trade-off between the effort to
ity of partitions. The right granularity reimplement the subsystems and the
is a trade-off between the strength of benefit gained by making them native
isolation on the one hand and the in- from both a performance and an assur-
creased overhead and cost of commu- ance perspective.
nication between partitions, as well as Step 4. Overall assurance. With all
re-engineering cost, on the other. parts in place, the final step is to analyze
In our example we end up with three the assurance of the overall system based
partitions: a VM that implements the on the assurance provided by the archi-
ground-station communication func- tecture and by individual components.
tionality running on Linux; another In HACMS, the communication,
VM that implements camera-based cryptography, and mission manager
navigation functionality (also running functionality were implemented in a
on Linux); and a native component that provably type-safe, domain-specific
provides shared access to the network, language called Ivory,11 with fixed
as in Figure 8. heap-memory allocation. Without fur-
Step 3. Native components. Once ther verification, Ivory does not give us
the system has been decomposed into high assurance of functional correct-
separate VM partitions, some or all ness but does give us assurance about
of the individual partitions can be re- robustness and crash-safety. Given
implemented as native components component isolation, we reason that
rather than as VMs. The aim is to sig- these assurances are preserved in the
nificantly reduce the attack surface for presence of untrusted components
the same functionality. An additional (such as the camera VM).

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 75
contributed articles

The networking component is im- tion boundaries according to its archi-

plemented in standard C code con- tecture description, and that it produc-
sisting of custom code for the plat- es correct RPC communication code.
form and pre-existing library code. Its The connection with a high-level secu-
assurance level corresponds to that
obtained through careful implemen- The camera VM rity analysis of the system remains in-
formal, and the communication code
tation of known code. Robustness
could be increased without much cost
is the weakest part theorems do not cover all communica-
tion primitives the platform provides.
through such techniques as driver of the system, While more work would be required to
synthesis32 and type-safe languages,
as with Ivory. However, in the overall
since it runs automatically arrive at an end-to-end
system-level theorem, it is clear at this
security analysis of the system, any a stock Linux stage that one is feasible.
compromise of the network com-
ponent would be able to inject or
system and is The main aim of the reported work
is to dramatically reduce verifica-
modify only network packets. Since expected to have tion effort for specific system classes.
the traffic is encrypted, such an attack
would not compromise the guaran- vulnerabilities. While the purely architecture-based
approach described here can be driven
tee that only authorized commands a good deal further than in the ULB ex-
reach the flight computer. ample, it is clearly limited by the fact
The camera VM is the weakest part it can express only properties that are
of the system, since it runs a stock enforced by the component architec-
Linux system and is expected to have ture of the system. If that architecture
vulnerabilities. However, as the VM is changes at runtime or if the properties
isolated, if attackers were to compro- of interest critically depend on the be-
mise the VM, they would not be able to havior of too many or too-large trusted
escape to other components. The worst components, returns will diminish.
an attacker could do is send incorrect The first step to loosen these limi-
data to the mission-manager compo- tations would be a library of pre-ver-
nent. As in the quadcopter, the mission ified high-assurance components for
manager validates data it receives from use as trusted building blocks in such
the camera VM. This is the part of the architectures. This library could in-
system on the ULB that demonstrated clude security patterns (such as input
containment of a compromise in the sanitizers, output filters, down-grad-
in-flight attack mentioned at the begin- ers, and runtime monitors) potential-
ning of the article. This was a white-box ly generated from higher-level speci-
attack, where the Red Team had access fications but also such infrastructure
to all code and documentation, as well components as reusable crypto mod-
as to all external communication, and ules, key storage, file systems, net-
was intentionally given root access to work stacks, and high-assurance driv-
the camera VM, simulating a success- ers. If the security property depends
ful attack against legacy software. Suc- on more than one such component,
cessfully containing the attack and it would become necessary to reason
being able to defend against this very about the trustworthiness of their in-
powerful Read Team scenario served teraction and composition. The main
to validate the strength of our secu- technical challenges here are concur-
rity claims and uncover any missed rency reasoning, protocols, and in-
assumptions, interface issues, or oth- formation-flow reasoning in the pres-
er security issues the research team ence of trusted components. Despite
might have failed to recognize. these limitations, this work demon-
strates that the rapid development
Limitations and Future Work of real high-assurance seL4-based
This article has given an overview of a systems is now a reality that can be
method for achieving very high levels achieved for a cost that is lower than
of assurance for systems in which secu- traditional testing.
rity property can be enforced through
their component architecture. We have Acknowledgments
proved theorems for the kernel level We are grateful to Kathleen Fisher,
and its correct configuration, as well as John Launchbury, and Raymond Rich-
theorems that ensure the component ards for their support as program man-
platform correctly configures protec- agers in HACMS, in particular Kathleen

76 COMM UNICATIO NS O F THE AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

Fisher for having the vision to start the International Conference (Noordwijk, the Netherlands, Proceedings of the First ACM Asia-Pacific Workshop
May 12–14). Curran, Red Hook, NY, 2008. on Systems (New Delhi, India, Aug. 30–Sept. 3). ACM
program. John Launchbury coined 9. Davis, J. and Myreen, M.O. The reflective Milawa Press, New York, 2010, 31–35.
the term “seismic security retrofit.” theorem prover is sound (down to the machine code 27. Kuz, I., Liu, Y., Gorton, I., and Heiser, G. CAmkES:
that runs it). Journal of Automated Reasoning 55, 2 A component model for secure microkernel-based
We thank Lee Pike for feedback on an (Aug. 2015), 117–183. embedded systems. Journal of Systems and Software
earlier draft. We would also like to ac- 10. Dennis, J.B. and Van Horn, E.C. Programming (Special Edition on Component-Based Software
semantics for multi-programmed computations. Engineering of Trustworthy Embedded Systems) 80, 5
knowledge our HACMS project part- Commun. ACM 9, 3 (Mar. 1966), 143–155. (May 2007), 687–699.
ners from Rockwell Collins, the Univer- 11. Elliott, T., Pike, L., Winwood, S., Hickey, P., Bielman, 28. Leroy, X. Formal verification of a realistic compiler.
J., Sharp, J., Seidel, E., and Launchbury, J. Guilt-free Commun. ACM 52, 7 (July 2009), 107–115.
sity of Minnesota, Galois, and Boeing. Ivory. In Proceedings of the ACM SIGPLAN Haskell 29. Murray, T., Matichuk, D., Brassil, M., Gammie, P.,
While we concentrated on the oper- Symposium (Vancouver, Canada, Sept. 3–4). ACM Bourke, T., Seefried, S., Lewis, C., Gao, X., and Klein, G.
Press, New York, 189–200. seL4: From general-purpose to a proof of information
ating system aspects of the HACMS 12. Fernandez, M. Formal Verification of a Component flow enforcement. In Proceedings of the 2013 IEEE
Platform. Ph.D. thesis. School of Computer Science & Symposium on Security and Privacy (San Francisco,
project here, the rapid construction CA, May 19–22). IEEE Press, Los Alamitos, CA, 2013,
Engineering, University of New South Wales, Sydney,
of high-assurance systems includes Australia, July 2016. 415–429.
13. Fernandez, M., Andronick, J., Klein, G., and Kuz, 30. Pnueli, A., Siegel, M., and Singerman, E. Translation
many further components, including I. Automated verification of RPC stub code. In validation. In Proceedings of the Fourth International
a trusted build, as well as architecture Proceedings of the 20th International Symposium on Conference on Tools and Algorithms for Construction
Formal Methods (Oslo, Norway, June 22–26). Springer, and Analysis of Systems (Lisbon, Portugal, Mar. 28–
and security-analysis tools. This mate- Heidelberg, Germany, 2015, 273–290. Apr. 4). Springer, Berlin, Germany, 1998, 151–166.
rial is based on research sponsored by 14. Floyd, R.W. Assigning meanings to programs. 31. Rushby, J. Design and verification of secure systems.
Mathematical Aspects of Computer Science 19, In Proceedings of the Eighth Symposium on Operating
the U.S. Air Force Research Laboratory (1967), 19–32. System Principles (Pacific Grove, CA, Dec. 14–16).
and the Defense Advanced Research 15. Gonthier, G. A Computer-Checked Proof of the Four- ACM Press, New York, 1981, 12–21.
Colour Theorem. Microsoft Research, Cambridge, U.K, 32. Ryzhyk, L., Chubb, P., Kuz, I., Le Sueur, E., and Heiser,
Projects Agency under agreement 2005; https://www.microsoft.com/en-us/research/wp- G. Automatic device driver synthesis with Termite. In
number FA8750-12-9-0179. The U.S. content/uploads/2016/02/gonthier-4colproof.pdf Proceedings of the 22nd ACM Symposium on Operating
16. Gonthier, G., Asperti, A., Avigad, J., Bertot, Y., Cohen, Systems Principles (Big Sky, MT, Oct. 11–14). ACM
government is authorized to repro- C., Garillot, F., Le Roux, S., Mahboubi, A., O’Connor, Press, New York, 2009, 73–86.
duce and distribute reprints for gov- R., Biha S.O., Pasca, I., Rideau, L., Solovyev, A., Tassi, 33. seL4 microkernel code and proofs; https://github.
E., and Théry, L. A machine-checked proof of the com/seL4/
ernmental purposes notwithstanding Odd Order Theorem. In Proceedings of the Fourth 34. Sewell, T., Kam, F., and Heiser, G. Complete, high-
any copyright notation thereon. The International Conference on Interactive Theorem assurance determination of loop bounds and infeasible
Proving, Volume 7998 of LNCS (Rennes, France, July paths for WCET analysis. In Proceedings of the 22nd
views and conclusions contained here- 22–26). Springer, Heidelberg, Germany, 2013, 163–179. IEEE Real Time and Embedded Technology and
Applications Symposium (Vienna, Austria, Apr. 11–14).
in are those of the authors and should 17. Gu, R., Shao, Z., Chen, H., Wu, X.(N.)., Kim, J., Sjöberg, V.,
IEEE Press, 2016.
and Costanzo, C. CertiKOS: An extensible architecture
not be interpreted as necessarily repre- for building certified concurrent OS kernels. In 35. Sewell, T., Myreen, M., and Klein, G. Translation
validation for a verified OS kernel. In Proceedings
senting the official policies or endorse- Proceedings of the 12th USENIX Symposium on
of the 34th Annual ACM SIGPLAN Conference on
Operating Systems Design and Implementation
ments, either expressed or implied, (Savannah, GA, Nov. 2–4). ACM Press, New York, 2016. Programming Language Design and Implementation
18. Hales, T.C., Adams, M., Bauer, G., Dang, D.T., Harrison, (Seattle, WA, June 16–22). ACM Press, New York,
of the Air Force Research Laboratory, 2013, 471–481.
J., Le Hoang, T., Kaliszyk, C., Magron, V., McLaughlin, S.,
Defense Advanced Research Projects Nguyen, T.T., Nguyen, T.Q., Nipkow, T., Obua, S., Pleso, 36. Sewell, T., Winwood, S., Gammie, P., Murray, T.,
J., Rute, J., Solovyev, A., Ta, A.H.T., Tran, T.N., Trieu, T.T., Andronick, J., and Klein, G. seL4 enforces integrity.
Agency, or U.S. government. In Proceedings of the International Conference
Urban, J., Vu, K.K., and Zumkeller, R. A formal proof
of the Kepler Conjecture. Forum of Mathematics, Pi, on Interactive Theorem Proving (Nijmegen, the
Volume 5. Cambridge University Press, 2017. Netherlands, Aug. 22–25). Springer, Heidelberg,
19. Hawblitzel, C., Howell, J., Kapritsos, M., Lorch, J.R., Germany, 2011, 325–340.
1. Alves-Foss, J., Oman, P.W., Taylor, C., and Harrison, S.
The MILS architecture for high-assurance embedded Parno, B., Roberts, M.L., Setty, S.T.V., and Zill, B.
systems. International Journal of Embedded Systems IronFleet: Proving practical distributed systems
Gerwin Klein (gerwin.klein@data61.csiro.au) is a Chief
2, 3-4 (2006), 239–247. correct. In Proceedings of the 25th ACM Symposium on
Research Scientist at Data61, CSIRO, and Conjoint
2. Blackham, B., Shi, Y., Chattopadhyay, S., Operating Systems Principles (Monterey, CA, Oct. 5–7).
Professor at UNSW, Sydney, Australia.
Roychoudhury, A., and Heiser, G. Timing analysis of a ACM Press, New York, 2015, 1–17.
protected operating system kernel. In Proceedings of 20. Heiser, G. and Elphinstone, K. L4 microkernels: The June Andronick (june.andronick@data61.csiro.au) is a
the 32nd IEEE Real-Time Systems Symposium (Vienna, lessons from 20 years of research and deployment. Principal Research Scientist at Data61, CSIRO, Conjoint
Austria, Nov. 29–Dec. 2). IEEE Computer Society ACM Transactions on Computer Systems 34, 1 (Apr. Associate Professor at UNSW, Sydney, Australia, and
Press, 2011, 339–348. 2016), 1:1–1:29. the leader of the Trustworthy Systems group at Data61,
3. Boeing. Unmanned Little Bird H-6U; http://www. 21. Kanav, S., Lammich, P., and Popescu, A. A known for the formal verification of the seL4 operating
boeing.com/defense/unmanned-little-bird-h-6u/ conference management system with verified system microkernel.
4. Boyton, A., Andronick, J., Bannister, C., Fernandez, document confidentiality. In Proceedings of the
M., Gao, X., Greenaway, D., Klein, G., Lewis, C., and 26th International Conference on Computer Aided Matthew Fernandez (matthew.fernandez@gmail.com)
Sewell, T. Formally verified system initialisation. In Verification (Vienna, Austria, July 18–22). ACM Press, participated in this project while he was a Ph.D. student
Proceedings of the 15th International Conference New York, 2014, 167–183. at UNSW, Sydney, Australia, and is today a researcher at
on Formal Engineering Methods (Queenstown, New 22. Klein, G., Andronick, J., Elphinstone, K., Murray, T., Intel Labs, USA.
Zealand, Oct. 29–Nov. 1). Springer, Heidelberg, Sewell, T., Kolanski, R., and Heiser, G. Comprehensive
Ihor Kuz (ihor.kuz@data61.csiro.au) is a Principal
Germany, 2013 70–85. formal verification of an OS microkernel. ACM Research Engineer at Data61, CSIRO , and also a Conjoint
5. Chen, H., Ziegler, D., Chajed, T., Chlipala, A., Frans Transactions on Computer Systems 32, 1 (Feb. 2014), Associate Professor at UNSW, Sydney, Australia.
Kaashoek, M., and Zeldovich, N. Using Crash 2:1–2:70.
Hoare logic for certifying the FSCQ file system. In 23. Klein, G., Elphinstone, K., Heiser, G., Andronick, J., Toby Murray (toby.murray@unimelb.edu.au) is a lecturer
Proceedings of the 25th ACM Symposium on Operating Cock, D., Derrin, P., Elkaduwe, D., Engelhardt, K., at the University of Melbourne, Australia, and a Senior
Systems Principles (Monterey, CA, Oct. 5–7). ACM Kolanski, R., Norrish, M., Sewell, T., Tuch, H., and Research Scientist at Data61, CSIRO.
Press, New York, 2015, 18–37. Winwood, S. seL4: Formal verification of an OS kernel.
6. Cock, D., Ge, Q., Murray, T., and Heiser, G. The last mile: In Proceedings of the 22nd ACM Symposium on Gernot Heiser (gernot@unsw.edu.au) is a Scientia
An empirical study of some timing channels on seL4. Operating Systems Principles (Big Sky, MT, Oct. 11–14). Professor and John Lions Chair of Computer Science at
In Proceedings of the ACM SIGSAC Conference on ACM Press, New York, 2009, 207–220. UNSW, Sydney, Australia, a Chief Research Scientist at
Computer and Communications Security (Scottsdale, 24. Kumar, R., Arthan, R., Myreen, M.O., and Owens, S. Data61, CSIRO, and a fellow of the ACM, the IEEE, and
AZ, Nov. 3–7). ACM Press, New York, 2014, 570–581. Self-formalisation of higher-order logic: Semantics, the Australian Academy of Technology and Engineering.
7. Cock, D., Klein, G., and Sewell, T. Secure microkernels, soundness, and a verified implementation. Journal of
state monads and scalable refinement. In Automated Reasoning 56, 3 (Apr. 2016), 221–259.
Proceedings of the 21st International Conference on 25. Kumar, R., Myreen, M., Norrish, M., and Owens,
Theorem Proving in Higher Order Logics (Montreal, S. CakeML: A verified implementation of ML. In
Canada, Aug. 18–21). Springer, Heidelberg, Germany, Proceedings of the 41st ACM SIGPLAN-SIGACT
2008, 167–182. Symposium on Principles of Programming Languages
8. Colbert, E. and Boehm, B. Cost estimation for (San Diego, CA, Jan. 22–24). ACM Press, New York,
secure software & systems. In Proceedings of 2014, 179–191.
the International Society of Parametric Analysts / 26. Kuz, I., Klein, G., Lewis, C., and Walker, A. capDL: A Copyright held by authors.
Society of Cost Estimating and Analysis 2008 Joint language for describing capability-based systems. In Publication rights licensed to ACM. $15.00

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 77
contributed articles
DOI:10.1145/ 3183583
outlined bold goals for HIT adoption
New York State healthcare providers increased as a key facet of each of their health-
care reform efforts, promising sig-
their use of the technology but delivered nificant benefits for healthcare pro-
only mixed results for their patients. viders and patients alike.20 Clinical
HIT systems, including electronic
BY QUANG “NEO” BUI, SEAN HANSEN, health records (EHRs), health infor-
MANLU LIU, AND QIANG (JOHN) TU mation exchanges (HIEs), comput-
erized provider order entry (CPOE),

The Productivity
and telemedicine technologies, are
seen as critical remedies to the com-
plexity and inefficiency that have long
plagued the U.S. healthcare industry.a

Paradox in
In 2009, the U.S. allocated more
than $30 billion, aiming to reduce
healthcare costs and increase quality

of care through adoption and use of
HIT systems.1 In that same year, the
Office of the National Coordinator for

Health Information Technology (ONC)
was established as part of the Health
Information Technology for Economic

and Clinical Health (HITECH) Act of
2009 to drive HIT adoption and co-
ordinate development of critical HIT
infrastructure. The ONC oversees a
range of programs (such as regional
extension centers, HIEs, privacy and
security policies, workforce develop-
ment, and curriculum development).
The HITECH Act introduced the prin-
ciple of “meaningful use” of HIT, a set
“HEALTH INFORMATION TECHNOLOGY connects doctors of guidelines for the substantive adop-
tion and application of HIT, including
and patients to more complete and accurate health
records … This technology is critical to improving a HIT reflects a range of technologies that can be
applied to the delivery and administration of
patient care, enabling coordination between providers healthcare service. In the present study, we fo-
and patients, reducing the risk of dangerous drug cus primarily on clinical HIT systems, empha-
sizing EHR and HIE systems, as they have been
interactions, and helping patients access prevention the leading areas of emphasis in the ongoing
wave of HIT adoption in the U.S.
and disease management services.”
— President Barack Obama, Presidential Proclamation key insights
on National Health Information Technology Week,
˽˽ No conclusive evidence has shown HIT
September 12, 2011 contribution to health outcomes among
New York State healthcare providers.

˽˽ Evidence indicates a HIT productivity

Health information technology (HIT)—the application paradox among healthcare providers
of information technologies to enable and enhance that mirrors the earlier experience of the
manufacturing sector.
the delivery of healthcare services—has been a central ˽˽ To address the paradox, a collective
point of focus for U.S. healthcare policy since 2007. approach is needed involving
multiple stakeholders and focusing
Both Presidents George W. Bush and Barack Obama on patient outcomes.

78 COMM UNICATIO NS O F THE AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

corresponding incentives and penal- benefits of HIT use (such as better clin- Evidence from New York
ties to motivate increased use.3 ical decision making and improved To explore the effect of HIT adoption
Despite aggressive investment and communications), other research sug- on health outcomes, we consider the

governmental support, evidence of gests the observable effects are limited evidence from the State of New York.
HIT’s contribution to health outcomes or even negative, marked by the risk of As the country’s fourth most popu-
remains mixed.7 A 2014 report from disrupted workflows, degradation of lous state and a national leader in
the U.S. Government Accountability physician-patient relationships, and HIT investment and adoption, New
Office (GAO) suggested that meaning- reduced clinical insight.25 In light of York offers a valuable context for as-
ful use requirements have had a mod- these findings, many researchers and sessing the effect of growing use of
est effect, and a comprehensive strate- public-policy observers have called for clinical HIT. Since 2007, New York
gy is needed to achieve better quality of additional studies to provide credible has invested more than $840 millionb
care through HIT.14 In addition, while evidence of improved health outcomes
several studies highlight perceived through expanded use of HIT.26 b https://www.health.ny.gov/technology/

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 79
contributed articles

Figure 1. Adoption of EHR functionalities by hospitals in New York State. 2015 for more than 180 hospitals across
the state. We tested a structural model
in which higher HIT investments
would lead to increased adoption and
36 use of EHR systems and HIEs that
Number of EHR Functionalities

in turn would result in better health

30 outcomes.d We tested the model us-
ing partial least squares software; for
details, see the online appendix “Re-
18 search Methodology”; dl.acm.org/
12 formats. In addition to our quantita-
tive analyses, we conducted a series of
6 semi-structured interviews with more
than 20 healthcare professionals
from 2013 to 2016 to explore their ex-
Each Column Represents One Hospital in the State of New York
perience around adoption and use of
HIT systems. Respondents included
Electronic Clinical Documentation Results Viewing Computerized Provider Order Entry
multiple classes of clinicians (such as
Decision Support Bar Coding Identification Other Functionalities
private practitioners, hospital physi-
cians, and nurse-practitioners), man-
agers, and IT professionals. The inter-
views were transcribed and coded in
Figure 2. HIE new participation rate and HIT investments from state grants in New York NVivo software to identify common
State, 2007–2017. patterns and themes.4
In general, we observed that in
35 160 New York State, 2014–2015, substan-
New HIE Participants
HIT Investment from State Grants (in millions)

tial HIT investments led to the wide-

30 140 spread acquisition and use of EHR
Number of New HIE Participants

HIT Investment

systems, implementation of clinical
25 decision-support functionality, and
100 significant participation in HIEs.
20 Specifically, New York healthcare
80 providers implemented most EHR
functionalities classified as “basic”
(see Figure 1). On average, New York
40 hospitals implemented 5.48 out of
six basic EHR functions (such as
5 20 electronic document viewing, results
viewing, CPOE, and decision sup-
0 0
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 port); and hospitals differ only by the
Year degree of implementation around
other advanced EHR functionalities
(such as barcode identification, tele-
in health information infrastructure. dination among healthcare providers health, mobile device connections).
In that time, a variety of initiatives across boundaries.c Additionally, the number of new hos-
within the state have sought to fos- To understand HIT effects among pitals joining local HIEs corresponds
ter information exchange, improve New York healthcare providers, we to the surge in the state’s public
quality and outcomes of care, reduce conducted a mixed-methods study us- funding for HIT investment in 2008,
healthcare costs, and engage con- ing both quantitative and qualitative significantly augmented in 2015 and
stituents in their care.22 Specifically, approaches. Our quantitative analy- 2016 (see Figure 2).e As of 2018, over
the state has focused on establishing ses used publicly available data from 80% of New York healthcare-provider
governance and policies that increase New York HIEs, New York State web-
participation in regional HIEs and sites, and databases made available d Details of our research methodology is
encourage EHR system adoption by by the not-for-profit American Hospi- provided in the online appendix “Re-
hospitals and individual providers. tal Association and the U.S. Centers search Methodology”; dl.acm.org/citation.
These efforts align with federal HIT for Medicare and Medicaid Services. cfm?doid=3183583&picked=formats
e These local HIEs received public grants from
meaningful-use initiatives aimed The dataset covered the period 2014– New York State to increase information shar-
at creating better management of ing among hospitals; https://www.health.
medical records and seamless coor- c https://www.healthit.gov/ ny.gov/technology/financial_investment.htm

80 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

organizations—162 out of 197—had rather than enhance the quality of are generated have an awful lot of
joined HIEs and regularly exchange care providers render. Prominent words but communicate very little.”
medical records data electronically. concerns include the perception — Physician, Family Practice
While the majority of New York hos- that HIT adoption results in extra “The highlighted efficiency from
pitals have implemented and used EHR workload, ineffective communica- reducing duplicate lab tests and cut-
and HIEs in their practice, the evidence tion, poor information quality, and ting costs is just not there yet. I am not
is inconclusive with respect to how ineffectiveness addressing opera- really sure that an EHR will provide
these initiatives have affected quality of tional needs. The following illustra- the savings that are talked about.”
care and broad health outcomes across tive statements highlight the con- — Physician, Internal Medicine
the state. We found no evidence of a re- cerns shared by our respondents: “I have charting at home. I ended up
lationship between HIT use and such “This whole business about elec- having to get a laptop through my work
critical health outcomes as improved tronic medical records helping with budget to bring home so that I wasn’t
interpersonal care, customer satisfac- communication I think is a total falla- sitting at the office until ... I would
tion, customer loyalty, patient mortal- cy. I think it really hinders communica- see my last person around 4:20, and I
ity, and reduced ER waiting times (see tion, unless you freehand-type or you would be there until 6:30 doing chart-
Figure 3). These results are in line with dictate, which defeats the main pur- ing because of being slow with the
previous studies suggesting unclear pose of electronic medical records.” system and be more attentive to the
evidence of HIT effects.15 — Physician, Pediatrics patient than I was to the computer.”
While HIE participation and EHR “I hear complaints from patients say- — Nurse Practitioner
use levels reveal no significant rela- ing, ‘They’re looking at the computer In summary, our mixed-methods
tionships with most outcome mea- and not at me.’” — Physician, Pediatrics analyses suggest strong evidence of
sures, we were surprised to find EHR “This is my issue with all electron- increased adoption and use of EHR
use also does have a significant ad- ic medical records: The notes that and HIE among New York healthcare
verse relationship with patient re-
admission rates and complication Figure 3. Effects of HIT investment on hospital performance.
rates. To further explore this coun-
terintuitive result, we looked at the Interpersonal
social-capital index in each county Care
where the hospitals operate. The so-
cial-capital index27 reflects the socio- Overall
economic growth of a community.f
The post-analyses suggest areas with EMR Exchange –0.227*** HIE Participation
0.017* Capabilities Level
low social capital often see higher Loyalty
readmission rates and complication HIT
rates. This low score is due to such
factors as rural market, low social ER Waiting
0.172*** Time
support, and low educational rate. EHR 0.269*** 0.145*
Functionalities EHR Use
One possible explanation for our
counterintuitive finding is that hos- Rates
pitals in areas with low social capital * p-value < 0.05; ** p-value < 0.01; *** p-value < 0.001
encounter inherent difficulties that 0.036**
Statistically significant relationship Complication
in turn increase patient readmission Rates
Statistically insignificant relationship
and complication rates regardless of
their use of HIT. We encourage future
research into this relationship.
Augmenting our quantitative Explaining the IT productivity paradox in HIT contexts.
analysis, our conversations with
healthcare providers suggest mixed
Causes Description
feelings and skepticism toward the
HIT mismeasurement Most HIT measures focus on efficiency rather than effective-
expected values of HIT. In particu- ness. Recent efforts like “meaningful use” level 2 are useful
lar, many clinicians were concerned but far from satisfactory.
that HIT initiatives were too often Delay delivering HIT benefits HITs are complex systems that require an average of two
not motivated by patient-oriented to four years to deliver significant benefits to healthcare
objectives and might undermine providers.
Redistribution of HIT benefits HIT gains are offset by unintended consequences in health-
care processes and procedures, including extra work and
f The social capital index was developed by
lack of human-doctor interaction.
the Northeast Regional Center for Rural
Development (http://aese.psu.edu/nercrd) Mismanagement of HIT systems Healthcare managers are not adequately trained to deal with
and uses an array of individual and commu- the complexity of HIT systems.
nity factors to measure the socioeconomic
growth of a community.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 81
contributed articles

providers but cast doubt on the Brynjolfsson5 said the productivity most exclusively on measures of input,
claim of substantial HIT effects on paradox could derive from the fact that or use of a certified system for records
health outcomes. “IT really is not productive at the firm capture, reporting, and data exchange.
level” or that managers have not been Despite incentivizing inputs, the ulti-
Assessing HIT able to apply IT resources effectively. mate objective of the meaningful-use
The challenge of finding evidence of “One of the health plans locally guidelines is substantive improvement
practical benefits accruing from IT in- made an attempt at doing reporting in health outcomes (such as quality of
vestment is not unique to healthcare. [on provider efficiency]. They based it care and fewer medical errors). This
Indeed, the IT productivity paradox,5 totally on cost. So [when they looked at disconnect suggests we may need bet-
an apparent disconnect between in- the report] one of the physicians that ter measures to capture the contribu-
vestment in IT resources and discern- was in the top had died six months tion of HIT investment to those ulti-
ible impact on organizational per- before. He looked very efficient mate objectives.29
formance, has been widely observed from a cost perspective. He hadn’t With respect to the question of a
with earlier waves of IT adoption in generated any cost to the system.” temporal lag, a number of studies have
manufacturing and other industrial — Director, Medical Society suggested this is a critical issue in the
sectors. In a seminal disposition on the In the years since the initial explo- healthcare context. For example, Me-
phenomenon, Erik Brynjolfsson5 sum- rations of the productivity paradox, non et al.21 found it takes, on aver-
marized a number of concerns that the apparent disconnect between IT age, from two to four years for HIT
emerged in the 1980s and early 1990s investment and organizational out- systems to improve health outcomes
around a lack of productivity gains comes has been largely resolved; that in a given healthcare-provider orga-
corresponding to rapid adoption of IT is, researchers have concluded that nization. Many providers lack the
resources. Several analysts had noted the first two explanations—mismea- necessary IT skills to quickly get
significant growth in technological surement and lagged effects—were acquainted with new HIT tools and
investment and innovation across the primary drivers of the paradoxical procedures, making implementation
developed economies had coincided observations6 and that IT investment more challenging. Given the fact that
with disappointing gains—or even de- is indeed correlated with significant the uptick of HIT investment com-
clines—in productivity.11,23 It appears improvement in various measures of menced only in 2009, it may take many
that just as in the manufacturing sec- value at firm, industry, and country more years for HIT influence to ripple
tor, HIT is struggling to produce cred- levels, but such gains might take years across healthcare providers.
ible improvements in key measures to materialize.12,13,28 However, the idio- The possibility of redistributive ef-
of performance. The IT productivity syncratic characteristics of the health- fects also warrants consideration in the
paradox has once again surfaced in the care sector (such as institutional het- HIT context. As the comments of our
healthcare industry. erogeneity, combination of public and study respondents underscore, many
In his exploration of the phenome- private influences, and comparatively healthcare providers fear the efficiency
non, Brynjolfsson5 suggested four pos- late adoption of IT innovations) un- in reporting and data analysis HIT en-
sible explanations: mismeasurement, derscore important differences with genders for insurance firms and regula-
temporal lags, redistribution, and the sectors explored previously. Con- tors comes at the expense of decreased
mismanagement. Mismeasurement re- sequently, a thorough consideration efficiency for clinicians who actually
fers to the idea that we lack appropriate of diverse possible factors is war- deliver clinical care. Indeed, this shift-
measures for productivity in a service- ranted.17,19 Indeed, the four proposed ing of efficiencies and burdens can be
based economy, with most traditional, explanations associated with the IT seen in one of the most common orga-
manufacturing-oriented measures of productivity paradox suggest critical nizational responses to HIT adoption:
productivity failing to account for in- clues for considering the inconclusive dedicated “scribes” to capture data dur-
direct benefits (such as quality and effects of contemporary HIT invest- ing a clinical encounter. The question
customer satisfaction). The issue of a ment (see the table here). of whether efficiency gains in one facet
temporal lag centers on the possibility As our analysis highlights, the idio- of the healthcare system are partially
that gains from IT investment could syncratic nature of the healthcare do- outweighed by efficiency or process
take years to develop as organizations main introduces a range of relatively losses elsewhere in the system thus re-
change their ways of working and the novel outcome measures for HIT in- quires additional analysis.
skills of their personnel. Less opti- vestment, including quality of care, Finally, the mismanagement of IT
mistically, redistribution suggests readmission rates, complication rates, resources may well play a role in the
the dearth of productivity improve- and diagnostic accuracy. While these mixed results of HIT adoption. Con-
ments could be the result of new IT are well-established measures of ef- cerns expressed to us by healthcare
resources merely shifting productiv- fectiveness for health services, their providers regarding the usefulness of
ity gains (or losses) from some mar- appropriateness for evaluation of the HIT resources suggest the possibility
ket participants to others. That is, IT efficiency and effectiveness of HIT re- of missteps in the design, implementa-
may indeed create productivity gains mains to be seen. Interestingly, the tion, and/or ongoing use of these sys-
for some players, but such gains are concept of “meaningful use” that has tems. These concerns lead to negative
counterbalanced by losses for other driven adoption of much HIT since the perceptions of HIT that likely result in
individuals or organizations. Finally, passage of the HITECH Act focuses al- misuse and jeopardize overall perfor-

82 COMMUNICATIO NS O F TH E ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

mance. Yet such concerns from multi- Efforts by the healthcare commu-
ple stakeholders are hardly captured in nity. Resolution of the apparent HIT
HIT development, and IT staff is inex- productivity paradox will require more
perienced in helping and adjusting the than the isolated efforts of healthcare
new systems to local needs. In our in-
terviews, several healthcare providers We found no providers, calling for a community ef-
fort. To this end, we suggest a stronger
expressed their struggles in managing
new systems due to their limited time
evidence of a leadership role for HIE-facilitating en-
tities, including regional health infor-
and personal technology anxiety. relationship mation organizations (RHIOs). As the
While each of the proposed mech-
anisms for paradoxical outcomes has
between HIT use ONC acknowledges, RHIOs are central
to data exchange across healthcare
some applicability in the healthcare and such critical institutions.30 Given the challenges in
context, the rich vein of research that
grew out of the productivity paradox
health outcomes the healthcare industry, we propose
that RHIOs should be more than mere
also offers some critical caveats for as improved data clearinghouses but formalized
assessing the practical effect of IT
investment and use.12,13,28 First, sig- interpersonal institutions that significantly improve
HIT use, especially in two major roles:
nificant variation exists across firms care, customer Encourage learning and adaptation
and industries with respect to the ef-
fect of IT investment on organization satisfaction, mechanisms in HIT practices. As with
many enterprise IT systems, HIT plat-
performance.9 Second, this varia-
tion and the existence of temporal
customer loyalty, forms are frequently complex and rig-
id, requiring significant resources and
lags are tied to the fact that perfor- patient mortality, enterprise-level effort to implement
mance gains are often associated not
merely with the adoption of new IT
and reduced ER effectively. For such complex projects
to yield tangible results, it takes time
resources but with the concomitant waiting times. for users to adapt to new routines and
redesign of business processes and practices, patients to get accustomed
investment in complementary assets to new processes and functionality,
and skills.6,28 Finally, the healthcare and in-house IT staff to discern what
literature reveals that measures of system modifications would make
productivity or business value remain the new system better fit with local
ambiguous and highly contingent on needs. RHIOs can serve as a platform
firm or industry conditions. Apply- through which different parties can
ing these lessons in the context of share resources, help others learn,
HIT, the evidence points to the need and contribute back to the broader
for more research to understand the community. In addition to creating a
complex nature of the healthcare in- mechanism for the development and
dustry and its business processes, exchange of a shared knowledgebase,
along with interdependence among these organizations represent a bridge
healthcare stakeholders in HIT devel- between different types of hospitals:
opment, adoption, and use. large/small, public/private, urban/ru-
ral. Managers can consider practices
Beyond the Paradox proposed in RHIO-based discourses to
Based on our analyses of the effects foster learning and adaptation in HIT
of clinical HIT adoption, we find that adoption (such as using collaborative
a number of viable mechanisms are teams to explore HIT functionalities,
available for achieving enhanced rewards to enforce positive behaviors,
health outcomes as a result of expand- and centers of excellence around HIT
ed HIT use, moving from meaningful best practices).
use to meaningful results. The U.S. Put users at the center of the HIT ex-
healthcare sector is an interdepen- perience. Commonly found in our in-
dent system. Leveraging and extend- terviews and in the HIT literature is
ing past insights from research on the the concern that HIT policies have
productivity paradox and IT business pushed healthcare providers toward a
value in general, we find it would ben- techno-centric perspective in which
efit from a collective approach that HIT is pursued “for IT’s sake” and HIT
brings together such diverse entities systems are designed without sub-
as hospitals, insurance companies, stantive input from prospective us-
regulators, and HIT vendors to seek ers.10 It is critical not to lose sight of the
systemic improvements. most important HIT stakeholders—

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 83
contributed articles

the patients whose wellness is directly lingering questions about potential

affected by HIT use and the healthcare mismeasurement in assessing the
providers who guide the patients long-term impact of HIT. As we have
through the treatment process. Health- noted in reference to the observed ad-
care providers thus need to encourage
both policymakers and technology de- Many clinicians verse effect of EHR use on patient re-
admission and complication rates in
velopers to emphasize inclusion of pa-
tients and healthcare providers in the
were concerned the New York State context, a range of
factors (such as urban/rural setting,
design processes of HIT systems. To en- that HIT initiatives social capital within a region, and
able a coherent and seamless experi-
ences across HIT systems, RHIOs can
were too often academic vs. non-academic hospital
adoption) can influence the contribu-
act as a forum for users’ experiences to not motivated by tion of HIT use on health outcomes.
be heard, providers’ suggestions to be
noted, and community members’ opin-
patient-oriented Clarifying the most relevant factors
would thus aid the healthcare field
ions to be constructively formed. Such objectives and in untangling the causal dynam-
a collaborative approach is essential to
HIT success, because, despite the exis- might undermine ics around HIT adoption and use.
In addition, another important im-
tence of competitive forces among rather than enhance provement regarding HIT evaluation
healthcare providers, patient wellness
should be regarded as the ultimate the quality of care would be increased use of evidence-
based and clinical HIT research.24
goal for all parties.
Academic research. Although the
providers render. Using rich data generated through
clinical HIT systems, future studies
literature on HIT evaluation is ex- could examine how HIT as “infor-
panding rapidly, there has not been matic intervention” can significantly
a parallel increase in academic un- improve patients’ health outcomes.
derstanding of how HIT contributes Other initiatives (such as the Preci-
to patient outcomes or how it can be sion Medicine Initiative launched in
used to improve health and health- 2016 by the U.S. National Institutes of
care. The related research should be Health) also underscore the need for
adapted to meet the needs of clini- more evidence-based HIT research in
cians, healthcare administrators, the future.g
and health policymakers. We thus Learn how to realize value from HIT.
suggest the following actions for aca- Early studies of HIT adoption and
demic researchers: use focused largely on determining
Develop enhanced measurements whether a particular HIT functional-
for clinical HIT impact. As noted, ity created value and to what extent.
the healthcare system today lacks With increasing adoption of EHRs
adequate outcome-oriented mea- and other forms of HIT, it is no lon-
surements of the efficiency and ef- ger sufficient for researchers to ask
fectiveness of HIT. For example, the whether HIT creates value in terms
U.S. Department of Health and Hu- of health outcomes.16 As researchers,
man Services released final criteria we need to help healthcare provid-
of “meaningful use” in 2010, aiming ers and policymakers learn how to
to improve quality and efficiency of realize value from HIT. That is, while
care by encouraging clinicians and HIT is being adopted, researchers
hospitals to use EHRs. However, as should focus on exploring the causal
of 2017, the existing measurement mechanisms underlying its use to
of “meaningful use” focused exclu- deliver health value to patients. Such
sively on input metrics. Accordingly, theory-building research could help
researchers are well positioned to clarify the antecedents of the produc-
develop appropriate means of out- tive application of HIT resources. In
come measurement to connect HIT particular, such research could le-
investment with productivity and verage recent research shifting from
clinical relevance. One important im- consideration of simple IT use to ef-
provement that can be made in HIT
evaluation is increased measurement g The Precision Medicine Initiative was
of context, implementation, and launched in 2016 by the U.S. National In-
stitutes of Health as a national, large-scale
context-sensitivity of effectiveness.18 research participation group for the testing
Exploring contextual and/or organi- and study of evidence-based interventions;
zational factors would help address https://allofus.nih.gov/

84 COMMUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

contributed articles

fective, enhanced, or idiosyncratic attention since 2007. Yet concrete Managed Care 17, 12 (Dec. 2011), 117–124.
17. Jones, S.S., Heaton, P.S., Rudin, R.S., and Schneider,
use of IT resources.2,8 Insight from and credible evidence that HIT im- E.C. Unraveling the IT Productivity Paradox: Lessons
the research could inform the afore- proves health outcomes remains in- for Health Care. The New England Journal of Medicine
366, 24 (June 14, 2012), 2243–2245.
mentioned efforts among healthcare conclusive. Our investigation of New 18. Jones, S.S., Rudin, R.S., Perry, T., and Shekelle, P.G.
system participants to identify and York State healthcare providers fur- Health information technology: An updated systematic
review with a focus on meaningful use. Annals of
disseminate best practices and foster ther indicates the healthcare industry Internal Medicine 160, 1 (Jan. 2014), 48–54.
more productive use patterns. may be experiencing an ongoing HIT 19. Lapointe, L. The IT productivity paradox in health: A
stakeholder’s perspective. International Journal of
Efforts by policymakers. Policymak- productivity paradox, mirroring earli- Medical Informatics 80, 2 (Feb. 2011), 102–115.
ers play a significant role in each of er patterns in manufacturing and oth- 20. Leidner, D.E., Preston, D., and Chen, D. An
examination of the antecedents and consequences
the measures we have proposed, as in er industrial sectors. While potential of organizational IT innovation in hospitals. Journal
of Strategic Information Systems 19, 3 (Sept. 2010),
community building through RHIOs HIT contribution to health outcomes 154–170.
and advancing outcome-oriented mea- remains an open question, we suggest 21. Menon, N.M., Yaylacicegi, U., and Cezar, A. Differential
effects of the two types of information systems:
sures of HIT use. While they should a collective approach is needed to ad- A hospital-based study. Journal of Management
work with academic researchers and dress the many issues raised by the Information Systems 26, 1 (July 2009), 297–316.
22. New York eHealth Collaborative. State HIE
the industry to identify more relevant HIT productivity paradox and hope Cooperative Agreement Program Strategic Plan. New
metrics for healthcare providers, it our research invites further inquiry York, 2009; https://www.healthit.gov/topic/onc-hitech-
is equally important they maintain a into this important issue. 23. Panko, R.R. Is office productivity stagnant? MIS
holistic view of the healthcare value Quarterly 15, 2 (June 1991), 191–203.
24. Payne, P.R.O., Lussier, Y., Foraker, R.E., and Embi, P.J.
chain. Instead of focusing on policies References
Rethinking the role and impact of health information
1. Adler-Milstein, J., Bates, D.W., and Jha, A.K. A survey
that incentivize only EHR adoption of health information exchange organizations in the
technology: Informatics as an interventional
discipline. BMC Medical Informatics and Decision
or HIE participation, policymakers United States: Implications for meaningful use. Annals
Making 16, 40 (Mar. 29, 2016), 1–7.
of Internal Medicine 154, 10 (May 2011), 666–671.
should also consider how to promote 25. Rosenbaum, L. Transitional chaos or enduring harm?
2. Bagayogo, F.F., Lapointe, L., and Bassellier, G.
The EHR and the disruption of medicine. The New
experimentation both within and Enhanced use of IT: A new perspective on post-
England Journal of Medicine 373, 17 (Oct. 22, 2015),
adoption. Journal of the Association for Information
across geographic boundaries. This 1585–1588.
Systems 15, 7 (July 2014). 361–387.
26. Rudin, R.S., Motala, A., Goldzweig, C.L., and Shekelle,
might include more flexible use-style 3. Blumenthal, D. and Tavenner, M. The ‘meaningful use’
P.G. Usage and effect of health information exchange:
regulation for electronic health records. The New
A systematic review. Annals of Internal Medicine 161,
incentive programs that reward not England Journal of Medicine 363, 6 (Aug. 5, 2010),
11 (Dec. 2014), 803–812.
only hospital-by-hospital efforts but 4. Boyatzis, R.E. Transforming Qualitative Information:
27. Rupasingha, A., Goetz, S.J., and Freshwater, D.
The production of social capital in U.S. counties.
also cross-hospital, cross-state, and Thematic Analysis and Code Development. Sage
The Journal of Socio-Economics 35, 1
Publications, Thousand Oaks, CA, 1998.
cross-boundary initiatives. It is dif- 5. Brynjolfsson, E. The productivity paradox of
(Feb. 2006), 83–101.
28. Schryen, G. Revisiting IS business value research:
ficult today to promote technologies information technology. Commun. ACM 36, 12 (Dec.
What we already know, what we still need to know,
1993), 66–77.
that provide value across geographi- 6. Brynjolfsson, E. and Hitt, L.M. Beyond the productivity
and how we can get there. European Journal of
Information Systems 22, 2 (Mar. 2013), 139–169.
cal locations (such as telemedicine) or paradox. Commun. ACM 41, 8 (Aug. 1998), 49–55.
29. Sharma, L., Chandrasekaran, A., Boyer, K.K., and
7. Buntin, M.B., Burke, M.F., Hoaglin, M.C., and
across institutional boundaries (such Blumenthal, D. The benefits of health information
McDermott, C.M. The impact of health information
technology bundles on hospital performance:
as healthcare supply-chain systems). technology: A review of the recent literature shows
An econometric study. Journal of Operations
predominantly positive results. Health Affairs 30, 3
In order to promote innovation and (2011), 464–471.
Management 41 (Jan. 2016), 25–41.
30. Vest, J.R. and Gamm, L.D. Health information
collaboration, policymakers might 8. Burton-Jones, A. and Grange, C. From use to effective
exchange: Persistent challenges and new strategies.
use: A representation theory perspective. Information
thus want to consider measures that Journal of the American Medical Informatics
Systems Research 24, 3 (Mar. 2012), 632–658.
Association 17, 3 (May 2010), 288–294.
target multiple parties in a health- 9. Chari, M.D., Devaraj, S., and David, P. The impact of
information technology investments and diversification
care value chain rather than a limited strategies on firm performance. Management Science
Quang “Neo” Bui (qnbui@saunders.rit.edu) is an assistant
number of dominant players. This 54, 1 (Jan. 2008), 224–234.
professor of management information systems in the
10. Cho, K.W., Bae, S.-K., Ryu, J.-H., Kim, K.N., An, C.-H.,
Saunders College of Business of the Rochester Institute
would include support for public-pri- and Chae, Y.M. Performance evaluation of public
of Technology, Rochester, NY, USA.
hospital information systems by the information
vate partnerships that bring together system success model. Healthcare Informatics Sean Hansen (shansen@saunders.rit.edu) is an associate
healthcare providers, payer organiza- Research 21, 1 (Jan. 2015), 43–48. professor of management information systems in the
11. David, P.A. The dynamo and the computer: An Saunders College of Business of the Rochester Institute
tions, and HIT providers or initiatives historical perspective on the modern productivity of Technology, Rochester, NY, USA.
that include large-scale participation paradox. The American Economic Review 80, 2 (May
1990), 355–361. Manlu Liu (manluliu@saunders.rit.edu) is an associate
groups (such as the Precision Medi- 12. Dedrick, J., Gurbaxani, V., and Kraemer, K.L. professor of management information systems and
cine Initiative). Such efforts could le- Information technology and economic performance: accounting in the Saunders College of Business of the
A critical review of the empirical evidence. ACM Rochester Institute of Technology, Rochester, NY, USA.
verage emergent technologies (such Computing Surveys 35, 1 (Mar. 2003), 1–28.
Qiang (John) Tu (jtu@saunders.rit.edu) is a professor
as big data analytics platforms, mo- 13. Devaraj, S. and Kohli, R. Information technology
of management information systems and the Senior
payoff in the healthcare industry: A longitudinal study.
bile health apps, and social media) to Journal of Management Information Systems 16, 4
Associate Dean in the Saunders College of Business of the
Rochester Institute of Technology, Rochester, NY, USA.
quickly assess the efficacy of a diverse (Apr. 2000), 41–67.
14. Government Accountability Office. Electronic Health
set of HIT projects and channel re- Record Programs: Participation Has Increased,
sources toward the ones that show the but Action Is Needed to Achieve Goals, Including
Improved Quality of Care. Washington, D.C., 2014;
greatest promise for bridging the gap https://www.gao.gov/assets/670/661399.pdf
between HIT use and health outcomes 15. Harrison, M.I., Koppel, R., and Bar-Lev, S. Unintended
consequences of information technologies in health
across populations. care: An interactive sociotechnical analysis. Journal
of the American Medical Informatics Association 14, 5
(Sept. 2007), 542–549.
Conclusion 16. Jha, A.K., Burke, M.F., DesRoches, C., Joshi, M.S.,
Kralovec, P.D., Campbell, E.G., and Buntin, M.B.
IT use in the healthcare industry has Progress toward meaningful use: Hospitals’ adoption
experienced tremendous growth and of electronic health records. American Journal of © 2018 ACM 0001-0782/18/10 $15.00

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 85
review articles
DOI:10.1145/ 3183582
or the amount of pollution an ecosys-
The future of computing research tem can bear, limits are less obvious in
computing. Many believe the only limit
relies on addressing an array worth considering is human ingenu-
of limitations on a planetary scale. ity, and that we can surpass any and all
other limits if we, as a global communi-
BY BONNIE NARDI, BILL TOMLINSON, ty, pool our creative resources. But we
DONALD J. PATTERSON, JAY CHEN, DANIEL PARGMAN, collectively face new global conditions
In this article we explore the relation-

ship between these potential futures
and computing research. What hidden
assumptions about the future are em-
bedded in most computing research?

within Limits
What possible or even probable futures
are we ignoring? What work should we
be doing to respond to fundamental
planetary limits, and to the ecological
and energy constraints that global soci-
ety faces over the coming years and de-
cades? Confronting such limits is likely
to present challenges that we—human-
ity—have never before faced.
Given that computing underlies vir-
tually all the infrastructure of global so-
COMPU TING R ESE ARC H E RS AND practitioners are often ciety—in commerce, communication,
transportation, agriculture, manufac-
seen as inventing the future. As such, we are implicitly turing, education, science, healthcare,
also in the business of predicting the future. We plot and governance—computing has an
trajectories for the future in the problems we select, enormous role to play in responding to
global limits and in shaping a society
the assumptions we make about technology and that meaningfully adapts to them. We
societal trends, and the ways we evaluate research. contend that the root of much of com-
puting research has been driven pre-
However, a great deal of computing research focuses dominantly by growth-oriented visions
on one particular type of future, one very much like
the present, only more so. This vision of the future key insights
assumes that current trajectories of ever-increasing ˽˽ Most computing work is premised on
industrial civilization’s default worldview
production and consumption will continue. This focus in which ongoing economic growth is
both achievable and desirable.
is perhaps not surprising, since computing machinery ˽˽ This growth-focused worldview, however,
as we know it has existed for only 80 years, in a period is at odds with findings from many other
scientific fields, which see growth as
of remarkable industrial and technological expansion. deeply problematic for ecological and
social reasons.
But humanity is rapidly approaching, or has already
˽˽ We proposed that the computing field
exceeded, a variety of planet-scale limits related to the transition toward “computing within
limits,” exploring ways that new forms
global climate system, fossil fuels, raw materials, and of computing supported well-being while
biocapacity.28,32,38 enabling human civilizations to live within
global ecological and material limits.
It is understandable that in computing we would not ˽˽ Computing underlies virtually all the
focus on limits. While planetary limits are obvious in infrastructure of global society, and will
therefore be critical in shaping a society
areas such as extractive capacity in mining or fishing, that meaningfully adapts to global limits.

86 COMMUNICATIO NS O F TH E AC M | O C TO BER 201 8 | VO L . 61 | NO. 1 0

of society’s future.26,34,39 If we broaden capacity. LIMITS sees ecological and draws attention to “planetary bound-
our view to a more diverse set of pos- environmental issues as a “predica- aries that must not be transgressed.”
sible futures, including non growth-re- ment”— that is, a situation for which Each of these topics will be discussed
liant futures, the societal challenges of there are not likely to be clear-cut “so- in greater detail.
ecological and energy limits can shape lutions” but rather a constellation of Here, we present background lit-
concrete technical challenges in com- complex issues that requires broad erature in ecological economics and
puting research and practice. new assumptions and approaches. We archaeology that has informed LIMITS
In order to consider these futures, seek to engage this predicament by research, and then review computing
we have been building a community of adopting a new framing for comput- research in sustainable human com-
scholars from computer science and ing research. We question the focus on puter interaction, crisis informatics,
engineering, information science, so- ongoing economic growth that lies at and information and communication
cial science, ecology, agriculture, and the heart of industrial civilization and technology for development (ICTD).
earth sciences to explore what we call propose a shift from emphasis on stan- Although LIMITS researchers come
“computing within limits” or “LIMITS” dards of living and material productiv- from many subfields of computing
for short. The LIMITS research com- ity to an emphasis on long-term well- including networking and software
munity integrates three topics: current being. LIMITS research looks ahead engineering, research in these three
and near-future ecological, material, to future scenarios cognizant of work areas in particular is closely related to
and energy limits; the ways new forms such as that of Rockström et al.28 that LIMITS with potential for deeper fu-
of computing may help support well-
being while living within these limits;
and the impact these limits are likely to
have on the field of computing. LIMITS
is concerned with the material impacts
of computation itself, but, more broad-
ly and more importantly, it engages a
deeper, transformative shift in com-
puting research and practice to one
that would use computing to contrib-
ute to the overall process of transition-

ing to a future in which the well-being

of humans and other species is the pri-
mary objective.
The LIMITS perspective is related
to Green IT,17 sharing an interest in
improvements in efficiency and other
traditionally “green” research topics.
However, LIMITS research questions
Green IT’s implicit assumption that
we can “engineer around” the finite-
ness of the Earth’s resources and waste

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 87
review articles

ture connections. We then briefly sum- eventually reach a “stationary state”). A

marize the three annual workshops on steady-state economy would maintain
LIMITS that began in 2015. Finally, we material throughput at a rate that is
discuss several key principles that have largely stable across time and that re-
arisen from LIMITS work to guide fu-
ture research. We see work in this area Computing has an mains within ecological limits.7 At the
same time, Daly notes that culture and
as a subfield that is an important alter-
native to traditional growth-oriented
enormous role to society need not be static: “Not only is
quality free to evolve, but its develop-
computing research. play in responding ment is positively encouraged in cer-

to global limits and tain directions. If we use ‘growth’ to
mean quantitative change, and ‘devel-
Since the beginning of computing, all in shaping a society opment’ to refer to qualitative change,
research and development has taken
place against a backdrop of exponen-
that meaningfully then we may say that a steady-state
economy develops but does not grow,
tial growth of, for example, transistors adapts to them. just as the planet Earth, of which the
per integrated circuit (Moore’s Law), human economy is a subsystem, devel-
disk storage density (Kryder’s Law), ops but does not grow.” Daly suggests
bandwidth capacity (Nielsen’s Law), that a single-minded focus on grow-
and fiber-optic capacity (Keck’s Law). ing the economy comes at the eventual
These developments have led to the cost of decreasing human well-being
establishment of a “cornucopian para- and quality of life. Such growth results
digm”23 where the design of new ser- in, for example, charging for things
vices stimulates demand, which drives that used to be free, the health conse-
growth of increased infrastructure quences of polluting the environment,
capacity, which then cycles back to en- and decreasing long-term possibilities
able the design of new services in a self- to produce food or earn a livelihood.
perpetuating cycle. The idea that expo- Looking at societal trends through
nential growth of computing capacity the lens of human history, archae-
and an ever-expanding infrastructure ologist Joseph Tainter’s book The
for computing will continue into the Collapse of Complex Societies argues
future is usually taken for granted. We that civilizations eventually collapse,
draw from research in ecological eco- declining over a period of decades or
nomics and the historical record in ar- centuries.33 Analyzing extensive his-
cheology to question this assumption. torical and archaeological materials,
This research suggests that other Tainter presented collapse as a pro-
futures are not just possible but prob- cess that arises from increasing so-
able. While most economists sidestep cietal complexity, which, over time,
questions of finite resources,6 econo- creates burdens for systems that they
mists in the subfield of ecological eventually cannot sustain.
economics have grappled with these Decline will result in less mate-
questions for decades. How can we rial abundance as we push the limits
maintain or increase well-being while of the Earth’s resources necessary for
staying within ecological limits? How economic activity. But it is not neces-
can we promote well-being and not ex- sary for our society to end in abject
ceed the assimilative and regenerative collapse. The societies that Tainter
capacities of the Earth’s biochemical studied—the Maya, the Mesopota-
life-support systems? We have already mians, the Minoans, the Inca, the Ro-
exceeded many such limits through, for mans, the Egyptians, and others—did
example, overfishing, deforestation, not possess the resources of science,
soil depletion, falling water tables, ris- history, and technology that we have
ing temperatures, and emitting CO2 amassed in the last 500 years. These
and other greenhouse gases at rates resources have the potential to be use-
that dangerously increase their con- fully deployed to fashion a transition
centrations in the atmosphere.28,32,38 from the current, unsustainable sys-
Ecological economist Herman Daly tem to a new system based on today’s
has proposed that we abandon the realities. We optimistically assume
idea of striving for economic growth that with advances in science and prog-
in favor of a steady-state economy (in ress in philosophies of human rights,
line with classical economist Adam we have a good chance of transforma-
Smith’s idea that the economy would tive change to a system more like the

88 COM MUNICATIO NS O F TH E ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

review articles

steady-state economy Herman Daly of scarcity.” This work helped lay the constitutes an important subfield of
envisions. The implication of the work groundwork, along with papers from human-computer interaction.19 There
in ecological economics and archaeol- other subfields of computing24,37 for are some key differences between cri-
ogy is that we should endeavor to build LIMITS research. sis informatics and LIMITS, although
computer systems that aim at increas- LIMITS has drawn heavily from col- we think that in the future the two may
ing well-being and quality of life while lapse informatics but shifts emphasis increasingly mutually inform one an-
contributing to staying within ecologi- to planetary limits rather than societal other. At present, crisis informatics
cal limits. Foregrounding human well- decline. LIMITS focuses on exposing research generally assumes an external
being is supported by the ACM Code of basic processes of resource use and entity that enacts a rescue when a disas-
Ethics and Professional Conduct, the waste management in complex human ter, such as a flood or earthquake, oc-
first imperative of which states: “As an systems. The metrics used to assess sus- curs. Events are conceived as localized,
ACM member I will contribute to so- tainability must shift correspondingly. describing a space into which the sur-
ciety and human well-being.” (https:// As examples, Pargman and Raghavan’s rounding society can pour resources
www.acm.org/aboutacm/acm-code-of- “Rethinking Sustainability in Comput- to alleviate the resulting disorder and
ethics-and-professional-conduct) ing: From Buzzword to Non-negotiable disruption. These scenarios accurately
We turn now to a review of comput- Limits”20 and Raghavan and Pargman’s describe an important subset of possi-
ing literature that has been founda- “Means and Ends in Human-Comput- ble issues confronting human civiliza-
tional for the development of comput- er Interaction: Sustainability through tions. LIMITS, however, assumes long
ing within LIMITS perspectives. Disintermediation,”25 offer major con- time frames and a global spatial scale.
tributions, arguing that “sustainabil- There is no external entity to provide
SCHI: Sustainable ity” must be grounded in rigorous met- relief. LIMITS emphasizes phenomena
Human-Computer Interaction rics arising from planetary limits, and such as climate change, soil erosion,
The Sustainable Human-Computer that the complexity of societal systems water pollution, civic instability, mass
Interaction community is about a might be reduced, easing resource use migration, reduced infrastructure, and
decade old, and a number of LIMITS and waste production. The forthcom- an economy that requires continuous
researchers have roots in this area. Eli ing edited collection Digital Technology growth.4,5,14,20,21,24,30,36
Blevis’s “Sustainable Interaction De- and Sustainability: Engaging the Para- Potentially there is a strong link be-
sign”3 is a primary source, offering a ru- dox10 incorporates influences from tween LIMITS and crisis informatics.
bric to identify how interaction designs LIMITS research. Several of the papers Some crisis informatics researchers
lead to material effects, as well as sev- mentioned here as well as Preist et al.23 are beginning to examine long-term
eral principles for engaging in sustain- have won best paper awards, signaling processes underlying crises, suggest-
able interaction design. Early papers interest in the issues. ing that when looked at more broadly,
that sparked interest among LIMITS “crises” are often more than acute
researchers were Jeff Wong’s “Prepare Crisis Informatics events of short duration, with roots in
for Descent: Interaction Design in Our We are often asked if computing within underlying processes that may have
New Future”40 and Silberman and Tom- LIMITS is the same as crisis informat- been developing over decades.1 This
linson’s “Precarious Infrastructure and ics. Crisis informatics is concerned understanding provides a bridge for
Postapocalyptic Computing.”31 Several with technology-based studies of di- future development and crossfertiliza-
high-profile CHI papers drew attention saster planning and response, and tion between the two subdisciplines.
to the challenges of sustainability and
the shortcomings of SHCI work in fail-
ing to address questions of physical,
material, and energy limits. DiSalvo et
al.’s “Mapping the Landscape of Sus-
tainable HCI”8 sought to provide struc-
ture to the array of papers in SHCI,
and identified gaps in the areas being
studied, such as the need to focus on
collectives and broader contexts, not
just individuals, the importance of en-
gaging with policy issues, and stronger
connections to sustainability work in
fields outside of computing.
From this context, Tomlinson et
al.’s “Collapse Informatics”35 was the

first full treatment of LIMITS topics in

the SHCI community. This paper ex-
plored “the study, design, and develop-
ment of sociotechnical systems in the
abundant present for use in a future

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 89
review articles

as highly relevant to global futures, not

just as problems that will be solved
through economic growth.

Computing Within
Limits Workshops
LIMITS ideas have been developed
through three workshops (2015–2017)
convened by the LIMITS community
(the latter two in cooperation with
ACM). The first two were held at the
University of California, Irvine, and
the third at Westmont College in Santa
Barbara, with funding from the two
universities as well as from Facebook
and Google. Participants came from
institutions in Abu Dhabi, Canada,
Hong Kong, Pakistan, Spain, Sweden,
Switzerland, the U.K., and the U.S.,
consistent with the global nature of
LIMITS concerns and research. The
ICTD: Information and equal distribution of wealth and the 2018 workshop was held in Toronto,
Communication Technology consequent problem of poverty in the co-located with the Fifth International
for Development Global South, the Global North must Conference on Information and Com-
ICTD is a relatively young field that shrink its resource footprint enough munication Technology for Sustain-
has explored the potential of comput- that countries in the Global South are ability (ICT4S). Sparked by discussions
ing for improving the socioeconomic afforded some space for necessary eco- at the workshops, LIMITS participants
situation of the poor. While comput- nomic growth. However, everyone— have co-authored several papers pub-
ing within LIMITS typically focuses North and South—must operate within lished in mainstream conferences and
on the future, Tomlinson et al.35 note some absolute global limits. The ethi- a research grant. The LIMITS work-
that our imagined “future” LIMITS cal argument for improving the quality shop papers are available at comput-
scenarios may already exist today in of life of the poor is easy to make, but ingwithinlimits.org
the conditions in which poor commu- reducing the Global North’s consump-
nities live around the world. However, tive (and exploitative) practices to af- Three Key Principles
few studies within the ICTD literature ford the Global South opportunities to We propose three principles that can
consider global ecological, material, grow, especially in the face of mount- help frame computing research and
and energy limits. Most research is ing resource and climate pressures, practice in a way that is consistent with
situated in resource-constrained con- remains an enormous challenge, and the ideas described in this paper and
texts and assumes the constraints will one computing should be cognizant of. the literature we have surveyed.
be relaxed in the future after sufficient Despite differing perspectives, LIM- Question growth. The industrial-
economic growth has occurred.12,15 ITS and ICTD have much in common ized world’s current economic system,
The only paper so far that explicitly and potential for integration and col- capitalism, is predicated on growth.
makes the link between LIMITS and laboration.4 For example, LIMITS work Economic growth has brought more
ICTD in an ICTD venue is Tomlinson has studied the use of digital technol- than an order of magnitude rise in per
et al.’s DEV paper, “Toward alternative ogy to design habitations in refugee capita income from $3 a day in 1800 to
decentralized infrastructures.”36 The camps,29 problems of networking in $100 in the early 2000s for most of Eu-
vacuum regarding the implications of rural populations in Zambia and Gua- rope and North America.16 However,
phenomena such as climate change in temala30 and infrastructure in condi- despite such unprecedented prosper-
the ICTD literature could be filled by a tions of scarcity in Haiti.21 While these ity, global income inequality is increas-
LIMITS perspective. are classic ICTD topics, the authors in ing. Wealth is accumulating in the
There is, however, a tension be- each case considered ecological, mate- hands of fewer and fewer astoundingly
tween economic development in poor rial, and energy limits in their analyses, rich persons.22 Poverty is widespread.
countries—the focus of ICTD—and unlike typical ICTD studies. The papers Such social dysfunction, along with
sustainability. As Herman Daly points engage models of scarcity, examining the burdens on ecosystems produced
out, the total resource footprint of the the cases as possible future global LIM- by economic activity,28,32,38 suggest we

Global North and the Global South ITS scenarios. Drought, flooding, envi- must rethink the growth paradigm.
combined together must stay within ronmental disasters, infrastructure The ubiquity and power of computing
the boundaries of a global steady state disruption, mass migration, and per- make it well positioned to act as an
economy that is sustainable in the long manent settlement in refugee camps agent of change to influence proposals
run. To ameliorate the problem of un- in low-resource environments are seen for transformative economic systems

90 COMMUNICATIO NS O F TH E ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

review articles

and methods of governance. While dis- become more numerous and more
cussion of specific proposals is beyond powerful as outcomes of global envi-
the scope of this article, we point to the ronmental changes. Our track record
work of, for example, Daniel O’Neill,18 of being prepared for dealing with un-
Peter Frase,9 and Tim Jackson13 as
thoughtful responses to current prob- We encourage predictable catastrophic events is not
encouraging. We would benefit from
lems that might inform the ways we
practice computing.
those working seriously considering LIMITS-related
scenarios rather than blithely denying
Daly’s notion of promoting devel- in computing to their possibility or treating their fore-
opment rather than economic growth
suggests a sound mechanism for mov-
build systems and shocks as isolated incidents. Engaging
with these difficult scenarios before
ing civilization forward, deploying our envision worlds that they occur, rather than only in their af-
creativity and capacity for innovation
in LIMITS-compliant ways. An econo-
are neither reliant termath, will help us evaluate our level
of preparedness and perhaps prevent
my that demands endless growth en- on nor contributing certain undesirable future scenarios
tails a cycle of consequences that must
be interrupted if we are to address mas- to runaway growth. from happening.21
To speak of LIMITS-scenarios only
sive problems such as climate change in the future tense, however, is mis-
and resource depletion.20 Exploring leading. These events are here now, as
relations between computing and the several climate-related catastrophes in
economy will be an important direc- the U.S. and Europe have shown, even
tion for future development of the during the writing of this article. Sci-
computing community and a consider- ence fiction author William Gibson
able challenge. famously said, “The future is already
Currently, the implicit organizing here—it’s just not evenly distributed.”
framework for a great deal of comput- We see this future currently on display
ing work puts a focus on increasing the in places such as Flint, Michigan where
proximate financial value of compa- toxic wastes have poisoned the water
nies. Even when particular products, supply. It is thus possible to frame LIM-
from a narrow perspective, are seeking ITS scenarios (including, for example,
to make people’s lives better through heat waves, drought, rising sea levels,
new technology, these products are and floods) not in terms of random ir-
typically embedded in a rapid churn regularities or threats that might afflict
of objects and services that foster run- us in the future, but in terms of an in-
away consumption.23,27 By shifting the creasing incidence of phenomena aris-
explicit focus, first and foremost, to ing from intensive economic activity.
the pursuit of long-term well-being, A concrete research strategy is to de-
we may finally escape the growth para- velop case studies of current changes
digm and build systems that more ef- that may model futures of relative scar-
fectively lead to sustainable improve- city. For example, a study of the con-
ments in the quality of life for humans tinuing impact of the 2010 earthquake
and other species. in Haiti found that the regrowth of in-
To make this principle actionable, frastructures was occurring in a more
we encourage researchers and practi- distributed fashion than would be typi-
tioners to consider whether their work cal for countries with more resources.21
is a) reliant on growth, b) seeking to Distribution networks for clean water,
make growth happen, c) contributing electricity, Internet, and gasoline were
to growth. We encourage those work- severely damaged in the earthquake.
ing in computing to build systems Corporate and government responses
and envision worlds that are neither were hampered by political and finan-
reliant on nor contributing to runaway cial obstacles. In many cases, survivors
growth. A number of existing LIMITS themselves began to rebuild the infra-
relevant papers have addressed this structures in a bottom up manner. For
principle.24,31,35 example, large private water tanks were
Consider models of scarcity. Clever installed on local properties. Wealthier
technological fixes may help us de- residences allowed adjacent poorer
fer catastrophes for some time, but households to tap into power lines via
not indefinitely, and especially not if jerry-rigged extension cords without
events such as wildfires, hundred-year paying for the service—a generous if
storms, and Category 5 hurricanes somewhat precarious arrangement.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 91
review articles

Such a re-arrangement certainly went sure. Mitigating the Jevons paradox

against existing building codes, but requires creative approaches that may
recognized the low cost of alleviating include substitution of goods by servic-
some resource deprivation in exchange es and dematerialization, for example,
for neighborhood stability.
A case study such as this can be gen- LIMITS research, by virtualization.11 Such changes have
the potential to entail a drop in abso-
erative by revealing opportunities for
developed and less developed regions
once applied, should lute consumption, although so far,
most approaches have tended to focus
to transfer technologies and schemes reduce energy on increasing efficiency, which may or
of sociotechnical organization that
present a different set of economic in-
expenditures may not result in absolute reductions.13
However, there is scope for significant
centives for actors. Being aware of the and material change; for example, the energy costs
wide diversity of current and future
potential contexts in which humans
consumption. of a virtual meeting that transmits
data to a large number of remote par-
may find themselves, more than a few This reduction is ticipants is tiny compared to the en-
of which are characterized by scarcity,
may help computing researchers and difficult to assess, ergy cost of a single airplane trip for a
single participant. The energy needed
practitioners design technology that but not something for data transmission is decreasing at a
promotes global well-being.
Several other LIMITS-relevant pa- we can sidestep. fast pace, unlike the energy costs of air
travel. Aslan et. al.2 estimate that data
pers have focused on aspects of this transmission costs decrease by 50% ev-
principle, including work found in ery two years.
Refs.4,14,29,30,40 Accounting for resource use must
Reduce energy and material consump- be done thoughtfully, with long-term
tion. Sticking to the dominant narrative goals in mind, in view of the big pic-
of growth is riskier than just making a ture. There is justification for spend-
bad guess. It is dangerous because it ing resources during a time of relative
creates a possibility that we will reach a abundance to prepare for a future of
point at which resources have precipi- scarcity.12 Not all investments need to
tously dwindled and we may not have pay off immediately. There is a place
enough remaining resources to make for experimenting when we don’t know
the necessary corrections to avert cat- for sure if savings will be accrued. But
astrophic outcomes. Therefore, it is such experimentation should fail fast,
important to acknowledge that com- and have a plausible hope of saving re-
puting uses energy and material re- sources. In this regard, we need to be
sources. If, as we have argued, these re- cognizant of the power of capital mar-
sources are declining, a threshold that kets in deciding what is a success and
LIMITS research should meet is that it what is a failure. While markets are
is worth the resources it consumes. Put very good at optimizing the delivery of
another way, LIMITS research, once the goods and services that they incen-
applied, should reduce energy expen- tivize, they tend not to be organized in
ditures and material consumption. such a way that promotes long-term
This reduction is difficult to assess, but returns or incorporates the costs of the
not something we can sidestep. externalities that push limits. Structur-
More broadly speaking, attempts to al changes such as cap-and trade mar-
limit resource usage in any human sys- kets, taxes, fees, rationing, and quotas
tem are notoriously challenging. Most are needed, in concert with technologi-
of us are well aware of the problems of cal changes, to address these issues.
CO2 emissions, but less aware of more Another key approach involves
subtle dynamics such as the Jevons finding energy savings through disin-
paradox, that is, that more efficient termediation, that is, the process of le-
technologies often encourage greater veraging technology to supplant “mid-
use of a resource, reducing or eliminat- dleman” actors in resource chains.25
ing savings. A more efficient gas engine Traditionally, in the absence of infor-
may reduce fuel consumption by half, mation technology, such middlemen
but stimulate more than twice as much provided value and extracted costs
driving (as well as more cars). A more by creating markets and distribution
efficient cryptocurrency mining chip centers for goods. For example, sys-
effectively increases electricity con- tems to directly connect small-scale
sumption through competitive pres- worker/producer owned facilities

92 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

review articles

with consumers could be of value in References

challenge for artificial intelligence. Commun. ACM 55,
4 (Apr. 2012), 86–97.
a new economy. Such simplification 1. Anderson, J. et al. Far far away in Far Rockaway:
27. Remy, C. and Huang, E. Addressing the obsolescence
Responses to risks and impacts during Hurricane
is responsive to Tainter’s argument Sandy through first-person social media narratives.
of end-user devices. In ICT Innovations for
Sustainability. Springer, New York, 2014.
that increasing complexity leads to in- In Proceedings of the 13th International Conference
28. Rockström, J. et al. A safe operating space for
on Information Systems for Crisis Response and
creasing burdens for systems which at Management, (2016).
humanity. Nature 461 (2009), 472–475.
29. Sabie, S., Chen, J., Abouzied, A., Hashim, F., Kahlon,
some point they cannot bear.33 Tech- 2. Aslan, J, Mayers, K., Koomey, J. and France, C.
H. and Easterbrook, S. Shelter dynamics in refugee
Electricity intensity of Internet data transmission:
nologies that provide services while and IDP camps: Customization, permanency, and
Untangling the estimates. J. Industrial Ecology, (2017).
opportunities. In Proceedings of the 3rd Workshop on
reducing complexity at the same time, 3. Blevis, E. Sustainable interaction design: Invention
Computing Within Limits, (2017). ACM, 11–20.
& disposal, renewal and reuse. In Proceedings of the
square conceptually with what we 30. Schmitt, P. and Belding, E. Navigating connectivity in
SIGCHI Conference on Human Factors in Computing
reduced infrastructure environments. In Proceedings
know from the historical and archaeo- Systems, (2007), ACM, 503–512.
of the 2nd Workshop on Computing Within Limits. ACM.
4. Chen, J. Computing within limits and ICTD. In
31. Silberman, M.S. and Tomlinson, B. Precarious
logical record about the relationship Proceedings of the 1st Workshop on Computing within
infrastructure and postapocalyptic computing. In
Limits, (2015). ACM.
between increasing societal complex- 5. Chen, J. A strategy for limits-aware computing. In
Proceedings of CHI 2010 Workshop on Examining
Appropriation, Re-use, and Maintenance for
ity and eventual societal decline. This Proceedings of the 2nd Workshop on Computing within
Limits, (2016). ACM.
and other efforts at disintermedia- 6. Costanza, R. Ecological Economics: The Science and
32. Steffen, W. et al. Planetary boundaries: Guiding human
development on a changing planet. Science 347
tion3,36 could help reduce energy and Management of Sustainability. Columbia University
(2015), 6223.
Press, 1992.
material consumption. 7. Daly, H. Steady State Economy. Island Press, 1977.
33. Tainter, J. The Collapse of Complex Societies.
Cambridge University Press, Cambridge, U.K., 1990.
8. DiSalvo, C, Sengers, P. and Brynjarsdóttir, H. Mapping
34. Thrun, S. et al. Stanley: The robot that won the DARPA
the landscape of sustainable HCI. In Proceedings
Conclusion of the SIGCHI Conference on Human Factors in
Grand Challenge. J. Field Robotics 23, 9 (2006),
While we do not know for certain what Computing Systems, (2010). ACM, 1975–1984.
35. Tomlinson, B., Six Silberman, M., Patterson, D., Pan,
9. Frase, P. Four Futures. Verso, London, U.K., 2016.
the future holds, scientists from dis- 10. Hazas, M. and Nathan, L. (Eds.). Digital Technology and
Y., and Blevis, E. Collapse informatics: augmenting
the sustainability & ICT4D discourse in HCI. In
ciplines such as climate science and Sustainability: Engaging the Paradox. Routledge, New
Proceedings of the 2012 SIGCHI Conference
York, 2018.
ecology have made evidence-based on Human Factors in Computing Systems. ACM,
11. Hilty, L.M. and Aebischer, B. (Eds.). ICT Innovations
predictions about directions the future for Sustainability. Vol. 310. Springer International
36. Tomlinson, B., Nardi, B., Patterson, D., Raturi, A.,
Publishing, 2015.
will likely take if current trends con- Richardson, D., Saphores, J-D. and Stokols, D.
12. Houston, L. and Jackson, S. Caring for the next
Toward alternative decentralized infrastructures.
tinue. However, what many comput- billion mobile handsets: Opening proprietary closures
In Proceedings of the 2015 Annual Symposium on
through the work of repair. In Proceedings of the
Computing for Development. ACM, 33–40.
ing researchers and practitioners do in 8th International Conference on Information and
37. Vardi, M. The financial meltdown and computing.
Communication Technologies and Development,
practice is to assume there is only one (2016). ACM.
Commun. ACM 52, 9 (Sept. 2009), 5.
38. Vitousek, P., Mooney, H., Lubchenco, J. and Melillo, J.
possible likely future—that current tra- 13. Jackson, T. Prosperity without Growth. Routledge,
Human domination of Earth’s ecosystems. Science 277
London, U.K., 2017.
jectories of increased growth and con- 14. Jang, E., Johnson, M., Burnell, E. and Heimerl, K.
(1997), 494–499.
39. Weiser, M. The computer for the 21st century.
sumption will continue. The burden of Unplanned obsolescence: Hardware and software
Scientific American 265, 3 (1991), 94–104.
after collapse. In Proceedings of the 3rd Workshop on
our message in this article is that sci- Computing within Limits, (2017). ACM.
40. Wong, J. Prepare for descent: Interaction design
in our new future. In Proceedings of the 2009
ence is telling us the kinds of growth we 15. Masinde, M., Bagula, A. and Muthama, N. The role
CHI Workshop on Defining the Role of HCI in the
of ICTs in downscaling and up-scaling integrated
have recently experienced are unsus- weather forecasts for farmers in sub-Saharan Africa.
Challenges of Sustainability.
tainable. Consequently, we believe the In Proceedings of the 5th International Conference on
Information and Communication Technologies and Bonnie Nardi (nardi@ics.uci.edu) is a professor in the
field of computing should be paying Development, (2012). ACM, 122–129. Department of Informatics at University of California,
serious attention to futures in which we 16. McCloskey, D. Bourgeois Dignity. University of Chicago Irvine, USA.
Press, Chicago, 2010.
encounter planetary limits. 17. Murugesan, S. Harnessing green IT: Principles and Bill Tomlinson (bill.tomlinson@vuw.ac.nz) is a professor
LIMITS thinking emphasizes incen- practices. IT Professional 10, 1 (2008), 24–33. in the Department of Informatics at the University of
18. O’Neill, D. Measuring progress in the degrowth California, Irvine, USA, and an adjunct professor in the
tivizing long-term returns. It seeks to transition to a steady state economy. Ecological School of Information Management, Victoria University of
align its efforts with the scientific dis- Economics 84 (2011), 1–11. Wellington, New Zealand.
19. Palen, L., Starbird, K., Vieweg, S. and Hughes, A.
ciplines documenting global transfor- Twitter-based information distribution during the 2009 Donald J. Patterson (dpatterson@westmont.edu) is
Red River Valley flood threat. Bulletin of the American a professor in the Department of Math and Computer
mations through climate change and Society for Information Science and Technology 36, 5 Science at Westmont College, Santa Barbara, CA, USA.
numerous other global effects. LIMITS (2010), 13–17. Jay Chen (jay.chen@nyu.edu) is an assistant professor in
20. Pargman, D. and Raghavan, B. Rethinking
seeks to explore ways that computing sustainability in computing: From buzzword to non-
the Department of Computer Science at NYU Abu Dhabi,
may support long-term well-being. We negotiable limits. In Proceedings of the 8th Nordic
Conference on Human-Computer Interaction, (2014). Daniel Pargman (pargman@kth.se) is an associate
see significant cause for concern in ACM, 638–647. professor in the Department of Media Technology and
many science-based projections of the 21. Patterson, D. 2015. Haitian resiliency: A case study in Interaction Design at KTH Royal Institute of Technology,
intermittent infrastructure. In Proceedings of the 1st Stockholm, Sweden.
future, and we want to enable our work Workshop on Computing Within Limits. ACM, 111–117.
Barath Raghavan (barath.raghavan@usc.edu) is an
to be relevant and useful with respect 22. Piketty, T. Capital in the 21st Century. Belknap Press,
assistant professor of computer science at the University
Cambridge, MA, 2014.
to these potential realities. 23. Preist, C., Schien, D. and Blevis, E. Understanding and of Southern California, Los Angeles, USA.
mitigating the effects of device and cloud service Birgit Penzenstadler (Birgit.Penzenstadler@csulb.edu)
design decisions on the environmental footprint of is an assistant professor in the Department of Computer
Acknowledgments digit al infrastructure. In Proceedings of the 2016 CHI Engineering and Computer Science at California State
The authors thank several anonymous Conference on Human Factors in Computing Systems. University, Long Beach, USA.
ACM, 1324–1337).
reviewers for their cogent comments, 24. Raghavan, B, and Ma, J. Networking in the long
as well as the entire LIMITS commu- emergency. Proceedings of the 2nd ACM SIGCOMM
Workshop on Green Networking, (2011). ACM, 37–42.
nity for helping shape the ideas in 25. Raghavan, B. and Pargman, D. Means and ends
in human-computer interaction: Sustainability
this article. This material is based in through disintermediation. In Proceedings of the
part on work supported by the NSF un- 2017 SIGCHI Conference on Human Factors in
Computing Systems. ACM, 786–796.
der Grants No. CCF-1442749 and IIS- 26. Ramchurn, S., Vytelingum, P., Rogers, A. and Jennings,
0644415. N. Putting the ‘smarts’ into the smart grid: A grand © 2018 ACM 0001-0782/18/10 $15.00

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 93
research highlights
P. 95 P. 96
Technical Fundamental Concepts
A Control Theorist’s of Reactive Control for
View on Reactive Autonomous Drones
Control for By Luca Mottola and Kamin Whitehouse
Autonomous Drones
By John Baillieul

P. 105 P. 106
Technical Enabling Highly Scalable
Perspective Remote Memory Access
The Future of MPI
By Marc Snir
Programming with
MPI-3 One Sided
By Robert Gerstenberger, Maciej Besta, and Torsten Hoefler

94 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

DOI:10.1145 / 3 2 6 441 1

Technical Perspective
To view the accompanying paper,
visit doi.acm.org/10.1145/3264417 rh

A Control Theorist’s View

on Reactive Control
for Autonomous Drones
By John Baillieul

1990s, at about the time

I N T H E L AT E control infrastructure, but only when Throughout flight experiments, the
as an upsurge of interest among the- their algorithms decide it is needed. parameters of the logistic-based de-
orists in real-time control in which For Mottola and Whitehouse, reactive cision rule are tuned with the aim of
feedback loops were closed through control is distinguished from the more minimizing false positive and—more
rate-limited communication chan- common approach to motion control importantly—false negative assess-
nels, the Bluetooth communication that they refer to as “time-triggered” ments of the significance of sensor
standard was introduced to enable control. The meaning of the terminol- reading differences. Although the
“local area networks of things.” Vari- ogy is a bit different from the way it is concept of “act-only-when-necessary”
ous research groups, including my used in most current work on mobile is simple and intuitive, the fact there
own, became interested in imple- robot control where the term “reactive are multiple sensors and actuators
menting feedback control using Blue- control” is used to distinguish fast, low- means there are very complex data de-
tooth channels in order to evaluate level, sensor-driven loops from slower pendencies that must be accounted
the design principles that we and “deliberative” control that involves for in real-time execution.
others had developed for commu- path planning or goal seeking naviga- How well does it work? The authors
nication-limited real-time systems. tion. The deliberative parts of motion deserve a great deal of credit for metic-
With device networks taking on ever control involve high-level decisions and ulous testing. They have logged more
increasing importance, our Bluetooth choices of ways to achieve an overall ob- than 260 hours of flight testing and
work was part of an emergent area jective—say, obtaining food in the case experimental benchmarking on three
within control theory that was aimed of animals or finding areas of high con- different flight vehicles—a quadcop-
at systems using existing infrastruc- centration of a chemical species for an tor, a hexacoptor and a challenging
ture rather than systems of sensors, extremum-seeking robot. Reactive con- tricoptor. They also report work with
actuators, and data links that were co- trol in the robotic literature normally three different off-the-shelf autopilot
optimized to work together to meet involves processing real-time streams implementations. The applications
performance objectives. of sensory data to guide low-level motor to which reactive flight control is best
The main challenge of using infra- response to follow a preplanned path or suited are those where setpoints do not
structure that was designed for purpos- a path created in the deliberative layer. change dramatically over the path; for
es other than real-time applications There is always more urgency in execut- example, hovering and following rela-
was that none of the infrastructure-op- ing the reactive layer of a control imple- tively straight paths as opposed to, say,
timized computation and communica- mentation, but a balance of reactive and aerial acrobatics. Nevertheless, the ex-
tion protocols are well suited to clos- deliberative is essential for achieving ro- periments show convincingly that the
ing feedback loops of control systems. bot autonomy. approach can handle challenging situ-
The work of Mottola and Whitehouse Reactive control in the following ations, particularly in outdoor flights
is somewhat along these lines—with paper involves a protocol for deter- where wind gusts provide significant
the infrastructure in this case being mining when sensor readings call for disturbances to which the control
the control logic and feedback control the autopilot’s control to function. system must react. A thought that oc-
algorithms that are found on popular Whereas classical feedback control curred to me after reading the paper is
UAV autopilot platforms such as Ardu- corrects for deviations from a setpoint that animal movements are guided by
pilot, Pixhawk, the Qualcomm Snap- or desired trajectory at every tick of a neurological circuits that must contin-
dragon, and the now discontinued system clock, reactive control in their ually refocus attention on the most rel-
OpenPilot. Several such autopilots are paper takes control action only when evant features in the environment. The
target platforms for the software de- a sensor input at a clock reading dif- current work may open a promising
scribed in the following paper. fers “significantly” from the previous new thrust toward understanding such
The authors introduce the notion reading. One of the contributions of aspects of biological motor control.
of “reactive control” in which an auto- this work is an algorithmic approach
pilot’s control logic is run only inter- to deciding when sensor-reading dif- John Baillieul is Distinguished Professor of Engineering
at Boston University. He is past editor-in-chief of the IEEE
mittently based on whether readings ferences are “significant.” The au- Transactions on Automatic Control and also past editor-in-
from sensors indicate a need to react to thors use a probabilistic logistic re- chief of the SIAM Journal of Control and Optimization.

something in the environment. Thus, gression approach to decide when

they employ the off-the-shelf existing a sensor reading requires reaction. Copyright held by author.

O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 95
research highlights
DOI:10.1145/ 32 6 441 7

Fundamental Concepts
of Reactive Control for
Autonomous Drones
By Luca Mottola and Kamin Whitehouse

Abstract Figure 1. Hardware and software components in modern drone

Autonomous drones represent a new breed of mobile platforms. Users configure high-level mission parameters at the
­computing system. Compared to smartphones and connected ground-control station (GCS), whereas the autopilot software
cars that only opportunistically sense or communicate, implements the low-level motion control aboard the drone.
drones allow motion control to become part of the applica-
tion logic. The efficiency of their movements is largely dic-
tated by the low-level control enabling their autonomous
operation based on high-level inputs. Existing imple- Ground-control station
mentations of such low-level control operate in a time-
triggered fashion. In contrast, we conceive a notion of
reactive control that allows drones to execute the low-level
control logic only upon recognizing the need to, based on
the influence of the environment onto the drone opera-
tion. As a result, reactive control can dynamically adapt
the control rate. This brings fundamental benefits, includ-
ing more accurate motion control, extended lifetime, and drone’s lifetime is often a result of how streamlined is the
better quality of service in end-user applications. Based autopilot operation.5, 24
on 260+ hours of real-world experiments using three aer- Unsurprisingly, most existing autopilots employ
ial drones, three different control logic, and three hard- Proportional-Integral-Derivative (PID)2 designs. Processing
ware platforms, we demonstrate, for example, up to 41% is thus time-triggered: every T time units, sensors are probed,
improvements in motion accuracy and up to 22% improve- control decisions are computed, and commands are sent
ments in flight time. to the actuators. Such a deterministic operation simplifies
implementations and allows designers to directly rely on a
vast body of existing literature.2
1. INTRODUCTION Reactive control. Based on a handful of key observations,
Robot vehicle platforms, often called “drones,” offer excit- a fundamental leap of abstraction, and an unconventional
ing new opportunities for mobile computing. While many use of recent advances in programming languages, we con-
mobile systems, such as smartphones and connected cars, ceive a notion of reactive control that allows autopilots
simply respond to device mobility, drones allow computer to significantly improve a drone’s performance in both
systems to actively control device location. Such a feature motion accuracy and energy consumption. Rather than
enables interactions with the physical world to happen periodically triggering the control logic, we only run the
in new ways and with new-found scale, efficiency, or control logic upon recognizing the need to. Depending on
precision.4, 8, 18 the influence of the environment onto the drone opera-
Autopilots. Figure 1 schematically illustrates the hard- tion, for example, due to wind gusts or pressure gradients,
ware and software components in modern drone platforms. control may run more or less frequently, regardless of the
Key to their operation is the autopilot software implement- the fixed rate of a corresponding time-triggered implemen-
ing the low-level motion control. The control loop processes tation. As a result, reactive control dynamically adapts the
high-level commands coming from a Ground-Control control rate.
Station (GCS) as well as various sensor inputs, such as, accel- Reactive control yields several advantages, includ-
erations and Global Positioning System (GPS) coordinates, ing more timely and adaptive control decisions leading
to operate actuators such as electrical motors that set the 3D to improved motion accuracy and energy efficiency. As it
orientation of the drone.
Together with the mechanical design, the autopilot soft-
The original version of this paper is entitled “Reactive
ware is crucial to determine a drone’s performance along
Control of Autonomous Drones” and was published in
a number of essential metrics. For example, the low-level
Proceedings of the 14th ACM International Conference on
control directly influences the quality of the shots when
Mobile Systems, Applications, and Services, Singapore,
using drones for imagery applications.17, 18 Further, it is
June 2016.
partly responsible for the overall energy efficiency, as a

96 COMM UNICATIO NS O F THE ACM | O C TO BER 201 8 | VO L . 61 | NO. 1 0

exclusively works in software, reactive control also requires time left from the execution of fast loop is given to an
no hardware modifications. We provide concrete evidence application-level scheduler that distributes it among non-
of these benefits across different aerial drone applications, critical tasks that may not always execute, such as logging.
based on 260+ hours of test flights in three increasingly The scheduler operates in a best-effort manner based on
demanding environments, using a combination of three ­programmer-provided priorities. Many autopilots share
aerial drones, three autopilot software, and three embed- similar designs.7
ded hardware platforms. Our results indicate, for example, Initially, fast loop blocks waiting for a new value from
that reactive control obtains up to 41% improvements in the Inertial Measurement Unit (IMU). This provides an
the accuracy of motion, and up to a 22% extension of flight indication of the forces the drone is subject to, obtained
times. by combining the readings of accelerometers, gyroscopes,
The remainder of the paper unfolds as follows. Section 2 magnetometers, and barometer. Once a new value is avail-
provides the necessary background, elaborates on the able, IMU information is combined with GPS readings to
fundamental intuitions behind reactive control, and out- determine how the motors should operate to minimize the
lines the issues that are to be solved to make it happen. error between the desired and actual pitch, roll, and yaw,
Section 3 describes the specific techniques we employ shown in Figure 3. Multiple PID controllers inside fast loop
to address these issues. Section 4 reports on the perfor- are used to this end.
mance of reactive control compared with traditional In Ardupilot as well as the vast majority of autopilots,
time-triggered implementations, whereas Section 5 stud- the control rate is statically set to strike a reasonable trade-
ies the impact of reactive control in a paradigmatic end- off between motion accuracy and resource consumption,
user application. We conclude the paper in Section 6 by based on a few “rules of thumbs.” 6, 25 For example, Ardupilot
discussing our current work towards obtaining official runs at a fixed 400Hz on the hardware we describe next.
certifications to fly drones running reactive control over This rate is not necessarily the maximum the hardware
public ground. supports. The 400Hz of Ardupilot, for example, are thought
to leave enough room—on average—to the scheduler. In
2. BUILDING UP TO REACTIVE CONTROL short bursts, control may run much faster than 400Hz, as
Reactive control relies on concepts and techniques germane long as some processing time is eventually allocated to the
to statistics, embedded software, programming languages, scheduler.
control, and low-power hardware. In the following, we try Hardware. Autopilots typically run on resource-constrained
and smooth the waters for the readers by walking them embedded hardware, for reasons of size and cost. A primary
through the characteristics of target platforms, the key example is the Pixhawk family of autopilot boards (goo.
observations leading to reactive control, and the issues that gl/wU4fmk), which feature a Cortex M4 core at 168MHz
are to be solved to concretely realize it. and a full sensor array for navigation, including a 16-bit gyro-
scope, a 14-bit accelerometer/magnetometer, a 16-bit 3-axis
2.1. Autopilots accelerometer/gyroscope, and a 24-bit barometer. Most often,
Drones can be regarded as a cruder form of modern robot- at least a sonar and a GPS are added to provide positioning
ics.9 The high-level inputs coming from the GCS may be a and altitude information, respectively.
waypoint or a trajectory. Autopilots implement the low-level Interestingly, the sensors on Pixhawk have similar
control in charge of translating these inputs into commands capabilities as those on modern mobile phones. In fact,
for the drone actuators. many argue that without the push to improve sensors due
Ardupilot (goo.gl/x2CHyM) is an example autopilot to the rise of mobile phones, drone technology would have
implementation, providing reliable low-level control for not emerged.9 Such sensors support energy-efficient high-
aerial drones and ground robots. The project boasts a large frequency sampling and often provide interrupt-driven
on-line community and is at the basis of many commercial modes to generate a value upon verifying certain conditions.
products. The ST LSM303D mounted on the Pixhawk, for example, can
Software. Figure 2 shows the execution of Ardupilot’s be programmed to generate an Serial Peripheral Interface
low-level control loop, split in two parts. The fast loop (SPI) interrupt based on three thresholds. This is useful,
only includes critical motion control functionality. The

Figure 3. Control based on raw, pitch, and yaw.

Figure 2. Ardupilot’s low-level control loop. The time for a single

iteration of the loop is split between fast loop, which only includes
critical motion control functionality, and an application-level
scheduler that runs non-critical tasks.

Setup Fast loop Scheduler


O C TO B E R 2 0 1 8 | VO L. 6 1 | N O. 1 0 | C OM M U N IC AT ION S OF T HE ACM 97
research highlights

for example, in human tracking applications for function- aspects by employing a form of auto-tuning of the
ality such as fall detection.15 conditions leading to running the control logic.
2) An indication for running the control logic may
2.2. Intuition originate from different sensors, at different rates,
Through our continuous work with drones as mobile com- and asynchronously with respect to each other.
puting platforms,16, 19 we eventually noticed that the auto- A problem is thus how to handle the possible inter-
pilots’ PID controllers are mostly tuned so that it is the leavings. Moreover, not running the control loop for
Proportional component to dictate the actual controller too long may negatively affect the drone’s stability,
operation. The Derivative component can be kept to a mini- possibly preventing to reclaim the correct behavior.
mum though a careful distribution of weights,6, 11 whereas We tackle these issues by only changing the execu-
precise sensor calibration may spare the Integral compo- tion of the control logic over time, rather than the
nent almost completely.6, 11, 22 logic itself.
As a result of this observation, we concluded that a sim- 3) Reactive control must run on resource-constrained
ple relation exists between current inputs from the navi- embedded hardware. When implementing reactive
gation sensors and the corresponding actuator settings. control, however, the code quickly turns into a “call-
With little impact from the time-dependent Derivative and back hell”10 as the operation becomes inherently
Integral components, and with the Proportional compo- event-driven. We experimentally find that, using
nent dominating, small variations in the current sensor standard languages and compilers, this negatively
inputs likely correspond to small variations in the actuator affects the execution speed, thus limiting the gains.7
settings. As an extreme case, as long as the sensor inputs We design and implement a custom realization of
do not change, the actuator settings should remain almost Reactive Programming (RP) techniques3 to tackle this
unaltered. In such a case, at least in principle, one may not problem.
run the control logic and simply retain the previous actua-
tor settings. The context where we are to address these issues shapes
Reactive control builds upon this intuition. We constantly the challenge in unseen ways. For example, aerial drone
monitor the navigation sensors to understand when the con- demonstrations exist showing motion control in tasks such
trol logic does need to run as a function of the instantaneous as throwing and catching balls,21 flying in formation,23 and
environment conditions. These manifest as changes in the carrying large payloads.14 In these settings, the low-level con-
inputs of navigation sensors. If these are sufficiently signifi- trol does not operate aboard the drone. At 100Hz or more, a
cant to warrant a change in the physical drone behavior to be powerful computer receives accurate localization data from
compensated, reactive control executes the control logic to high-end motion capture systems, runs sophisticated con-
compute new actuator settings. Otherwise, reactive control trol algorithms based on drone-specific mechanical models
retains the existing configuration. expressed through differential equations, and sends actua-
As we explain next, reactive control abstracts the problem tor commands to the drones. Differently, we aim at improv-
of recognizing such significant changes in a way that makes ing the performance of mainstream low-level control on
it computationally tractable with little processing resources. embedded hardware, targeting mobile sensing applications
Moreover, because of the aforementioned characteristics of that operate in the wild.
sensor hardware on autopilot boards, monitoring the sensor On the surface, reactive control may also resemble the
readings at the maximu