d.

note
Revising User Interfaces Through
Change Tracking, Annotations, and Alternatives
Björn Hartmann
University of California, Berkeley
EECS, Computer Science Division

Sean Follmer, Antonio Ricciardi,

Timothy Cardenas, Scott Klemmer
Stanford University HCI Group
 Revision tools for
word processing:
• Comments
• Change tracking
• Change
visualization
d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 2
 t m e nu
i f f e r en
4 d
y s t e ms
s
u s i n g!
-conf
n g m enu
e v i s e : wro
R
h l i gh ted
ig
item h

Revision tools
for user interface
designs?

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 3
What do UI designers use today?

Email survey of 10 UI interaction designers

Ad hoc choices of digital, physical media:

• Printing out screens and sketching on them
• Posting digital screenshots to a wiki
• Using a bug tracking database and revision control

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 4
What do UI designers use today?

Designers prefer expressing changes to the

current design through sketching
Designers prefer to capture history over
time through digital media

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 5
Research Questions
How can revision techniques of
commenting, change tracking, and change
visualization be applied to the revision of
user interfaces?
Are there techniques unique to UI design?
How do interactive revision tools change
designers’ practices?

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 6
Desing history alone is insufficient

Editable Graphical Histories, Kurlander,

IEEE Visual Languages’88
Co-existence of paper and digital
information for office work
Outpost Design Histories,
Klemmer, CHI’02
Capture & retrieval for web site design

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 7
Related Work

Capturing Design History

Editable Graphical Histories, Kurlander,
IEEE Visual Languages’88
Co-existence of paper and digital
information for office work
Outpost Design Histories,
Klemmer, CHI’02
Capture & retrieval for web site design

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 8
Sketch-based informal design tools

DENIM, Lin, CHI’00

Sketch-based prototyping of web site
information architecture
Topiary, Li, UIST’04
Prototyping of mobile, location-aware
applications

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 9
Sketch-based informal design tools

DENIM, Lin, CHI’00

Sketch-based prototyping of web site
information architecture
Topiary, Li, UIST’04
Prototyping of mobile, location-aware
applications

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 10
Paper annotations change digital models

Annotation Tools
Paper Augmented Digital Documents
Guimbretiere, UIST’03
Digital pen marks on paper change
underlying document
ModelCraft, Song, UIST’06
Digital pen marks on 3D objects change
CAD model

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 11
Paper annotations change digital models

Annotation Tools
Paper Augmented Digital Documents
Guimbretiere, UIST’03
Digital pen marks on paper change
underlying document
ModelCraft, Song, UIST’06
Digital pen marks on 3D objects change
CAD model

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 12
Inspiration
Interaction design specifies both
appearance and behavior.
What tools do exist for revising each?

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 13
Revision on top of Images

Microsoft & Pixar, from Buxton, Sketching User Experiences.

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 14
Source code comparison

Diff tools show two versions side-by-side

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 15
Guidelines for Interactive UI Revision

Goals:
• Enable rapid expression of changes,
at different levels of representation
• Communicate clearly what changed (and how)

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 16
Guidelines for Interactive UI Revision

Goals:
• Enable rapid expression of changes,
at different levels of representation
• Communicate clearly what changed (and how)
Approach:
• Use sketching as input
• Develop visual revision language
• Revision of both design and source

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 17
d.note: revision techniques for UI prototypes

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 18
d.note: revision techniques for UI prototypes

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 19
d.tools as a testbed

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 20
d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 21
Revision Language

Adding States Adding Transitions Adding Graphics

Removing States Removing Transitions Adding Comments

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 22
d.note: revision techniques for UI prototypes

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 23
d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 24
d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 25
State Alternatives

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 26
State Alternatives

Alternatives share:
• Incoming transitions
Alternatives define:
• Screen graphics
• Outgoing transitions

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 27
Gesture Set For Stylus Input

Gestures are recognized by Wobbrock’s $1

recognizer with multiple templates for each
gesture.

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 28
How does expressing revisions interactively
compare with sketching on static images?

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 29
Study 1: Method
Within-subjects design, n=12
Each participant revises two prototypes:
one with d.note,
one by sketching on a static image
Static condition assigned first,
prototype counterbalanced

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 30
Apparatus

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 31
Apparatus

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 32
Method: Tasks

Digital Camera Navigation Prototype

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 33
Method: Tasks

Keychain Photo Viewer Prototype

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 34
Results: 24 annotated diagrams (12 interactive, 12 static)

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 35
Results: Example detail of static image

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 36
Results: d.note example

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 37
Results: d.note example

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 38
Results: Diagram Analysis

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 39
Results
With d.note, participants:
• wrote less
(2.3 vs 9.8 mean text comments, p<0.01)

• deleted more
(2.8 vs 0.3 mean deletions, p<0.01)

• proposed fewer global, conceptual changes

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 40
Perceived advantages of interactive revision

Ability to test proposed changes

(7 of 12 participants)
Ability to make functional changes (6)
Less cluttered than sketching alone (2)
Notation easier to interpret (1)

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 41
Perceived disadvantages
Commenting is more difficult
(3 participants)
Tool has steeper learning curve (2)
Danger of getting stuck on details (2)

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 42
Why did participants write less?

It was easier to write/sketch with

production software than with our tool.
Formal notation reduced the need for
explanations.

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 43
Why did they delete more?

Immediate testing enabled discovery of

errors and ambiguities.

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 44
Why did they propose few high-level changes?

Participants focused on single-state

revisions because the tool gave them more
leverage to do so.

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 45
How well can others interpret
interactively created revisions?

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 46
Study 2: Interpretation
New set of participants (n=8) rates the
output of the last study
Each participant rates 4 diagrams:
camera revised with d.note
camera revised without d.note
keychain revised with d.note
keychain revised without d.note

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 47
Method: Task

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 48
Results: Same Number of Clear Revisions
mean number of revisions listed

interactive revision
no significant difference - t(30)=0.25, p=0.15
error bars show standard error

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 49
Results: Interactive Yielded Fewer Unclear Revisions
mean number of revisions listed

interactive revision
significant difference - t(30)=1.90, p<0.05
error bars show standard error

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 50
Perceived advantages of interactively created diagrams

Changes are concrete and specific

(4 of 8 participants)
Revisions contain proposed solutions (3)

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 51
Perceived Disadvantages
Visual clutter in regions of dense changes
(5 participants)
Hard to glean motivation for changes (4)

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 52
Future Work
Reduce visual complexity in
revised diagrams
Encourage capture of rationale

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 53
A Design Space of UI Revision Tools

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 54
Contributions
An operationalization of common revision
operations for user interfaces.

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 55
Contributions
An operationalization of common revision
operations for user interfaces.
New revision operations
unique to user interfaces:
testing proposed changes immediately
specifying alternatives to existing functionality

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 56
Contributions
Interactive revision offer benefits:
immediate testing and formal notation lead to
increased clarity of revisions

For a tradeoff:
formal notation discouraged broad, conceptual changes;
capturing motivation for changes

Future work:
encourage revisions at multiple levels of abstraction

d.note: Revising User Interfaces Through Change Tracking, Annotations, and Alternatives 57
bjoern@eecs.berkeley.edu
bid.berkeley.edu