Vous êtes sur la page 1sur 39

Eyes-Free User Interaction

T. V. Raman Charles L. Chen


Google Google
http://emacspeak.sf.net/raman www.clcworld.net

May 21, 2009


Overview
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Android
Dialer
Contacts
TTS
Gestures
MusicBrowser
Overlays
Conclusion

Eyes-Free Interaction Google-IO May 2009 – 2 / 39


Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Android Programming
Android: Engineer’s Dream!
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ Smart phones — computer in your pocket.


■ Open platform — liberates innovation.
■ New I/O — minimize user interaction.
Bend technology to your will!

Eyes-Free Interaction Google-IO May 2009 – 4 / 39


Innovative User Interfaces
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Create innovative end-user solutions that:


■ Fit into the user’s mode of working,
■ Avoid one-size fits all solutions,
■ Design context-aware solutions,
■ Tailor solutions to user’s needs.

Eyes-Free Interaction Google-IO May 2009 – 5 / 39


Eyes-Free Interaction
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Not just for blind users!

■ Eyes-busy environments.
■ One-handed use.
■ Minimize explicit user interaction.

Rethink all assumptions about the user.

Eyes-Free Interaction Google-IO May 2009 – 6 / 39


Devices That Sense
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Mobile phones make UI design exciting!

■ Can capture user intent — keyboard, . . ..


■ Can see — camera.
■ Can hear and speak — speech I/O.
■ Can sense and stimulate touch — touch screen and vibration.
■ Can sense motion — accelerometer, compass.
■ Can sense position — GPS, cell towers.
..
■ ..
Our Android has many eyes and ears!

Eyes-Free Interaction Google-IO May 2009 – 7 / 39


Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Touch Screen Interaction, Eyes-Free!


Using A Touch Screen
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

These are atomic tasks:


■ Identifying the item to touch.
■ Touching item to activate it.
■ Eyes-free use requires help with the former.
■ Activation should produce appropriate feedback.

Eyes-Free Interaction Google-IO May 2009 – 9 / 39


On-Screen Touchpads
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free use of an on-screen touchpad:

■ Need to see the buttons to know where to touch.


■ Because: buttons use absolute positioning.
■ Conclusion: (wrong) — you need to see to use a touch screen!
■ Solution: Relax constraint of absolute positioning.

What if we used relative positioning instead?

Eyes-Free Interaction Google-IO May 2009 – 10 / 39


Phone Keypad
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-free phone dialer:

■ Keypad centered where you touch — relative positioning.


■ Users already know layout of a phone keypad.
■ Can stroke from the center for a desired digit.

Enables eyes-free one-handed dialing!

Eyes-Free Interaction Google-IO May 2009 – 11 / 39


Dialer Feedback
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Enables one-handed, eyes-free use:

■ Buttons vibrate when you move over them.


■ Augmented by auditory feedback.
■ Pressing buttons produces spoken and tactile feedback.
■ Shake to erase input.

Synchronized auditory and haptic feedback is key.

Eyes-Free Interaction Google-IO May 2009 – 12 / 39


Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Text Input Via Touch Screen


Contact Manager
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

No one dials phone numbers any more!

■ Browsing contacts using touch screen.


■ Finding contacts.
■ Managing contacts.

Eyes-Free Interaction Google-IO May 2009 – 14 / 39


Entering Text Input
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Extending stroke dialer for textual input:

■ Arrange letters in 4 concentric circles.


■ Strokes access distinct circles.
■ Choose desired letter by tracing chosen circle.

Any letter is at most 3 steps away!

Eyes-Free Interaction Google-IO May 2009 – 15 / 39


Keypad Layout
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Gesture: Pick circle, and trace to desired character.

A I Q Letter Gesture
A Stroke up to the left
⊙ Y
B Stroke A, trace right
U M E E Stroke down to the right

Eyes-Free Interaction Google-IO May 2009 – 16 / 39


Managing Contacts
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Contacts live in the cloud.


■ Contacts managed over the network.
■ Updated automatically on the device.
■ No more editing contacts with two fingers!

Eyes-Free Interaction Google-IO May 2009 – 17 / 39


Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Text To Speech On Android


TTS Library
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ Create speech-enabled apps.


■ Developer compiles against the TTS library stub.
■ Text To Speech is an Android service.
■ Class TTS manages communication with the service.
■ Enables multiple apps to use TTS without code duplication.
■ TTS library can be updated without a need to recompile clients.

Eyes-Free Interaction Google-IO May 2009 – 19 / 39


TTS Method Summary
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ speak(String text, int queueMode, String[] params)


■ stop()
■ isSpeaking()
■ Also has methods for synthesizing to a file, specifying the language, etc.

Eyes-Free Interaction Google-IO May 2009 – 20 / 39


TTS Object Properties
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ Can automatically prompt the user to install the TTS Service.


■ When absent, fail silently (methods become no-ops)

See: http://eyes-free.googlecode.com/

Eyes-Free Interaction Google-IO May 2009 – 21 / 39


Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Gesture Library For Stroke Input


Gesture Overlay
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ Overlay watches for touch events


■ Returns identified gesture
■ Exposes same UI as the stroke dialer
■ Implemented as a custom transparent View with callbacks

Eyes-Free Interaction Google-IO May 2009 – 23 / 39


Gesture Overlay Implementation
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ Implement Interface GestureListener ,


■ Create a new TouchGestureControlOverlay with it,
■ Set a FrameLayout as the base View,
■ Add the View with the content as a child to the FrameLayout,
■ Add the TouchGestureControlOverlay as a child

Can easily enable/disable the TouchGestureControlOverlay

Eyes-Free Interaction Google-IO May 2009 – 24 / 39


Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

A Simple Music File Browser


Music File Browser
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Barebones version:
■ Uses a simple ListView
■ Based on a FileBrowser tutorial on Anddev.org
■ Browse directories and mp3 files on SDCard.
■ Play music by clicking on the file.
■ Directory path is the first entry.
■ Click this to cycle through files in current level.
■ Tactile feedback when scrolling through list items.

Eyes-Free Interaction Google-IO May 2009 – 26 / 39


Adding Spoken Output
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ Create object TTS in method onCreate.


■ Result: Runs method TTS.onInit as part of setup.
■ Can add application-specific code to method TTS.onInit.
■ Result: Application comes up speaking.

Eyes-Free Interaction Google-IO May 2009 – 27 / 39


Music Player Talks
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

private InitListener i = new InitListener() {


public void onInit(int version) {
tts.speak("Music File Browser started.", 0, null);
}
};

Eyes-Free Interaction Google-IO May 2009 – 28 / 39


Speaking List Items
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ Add TTS call to speak list items.


■ Can use event handler that produces tactile feedback.

Eyes-Free Interaction Google-IO May 2009 – 29 / 39


Speaking List Items
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Augment event handler with a TTS call.

public boolean onTrackballEvent(MotionEvent event) {


String filename = directoryEntries.get(selectedId);
tts.speak(filename.substring(1), 0, null);
}

Self-voicing functionality added with just 13 additional lines of code!

Eyes-Free Interaction Google-IO May 2009 – 30 / 39


Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Adding Gesture Input To Music


Browser
Advantages Of An Overlay
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ GestureOverlay is a transparent overlay.


■ Can be layered on an existing View.
■ Does not interfere with visual appearance.

Eyes-Free Interaction Google-IO May 2009 – 32 / 39


Gestures For Music Browser
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ Add GestureListener to handle user input.


■ Gesture Tap: play/pause.
■ Gesture Right: Browse to next track.

Add application-specific code in method onGestureFinish.

Eyes-Free Interaction Google-IO May 2009 – 33 / 39


Steps In Adding Gesture Input
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

private GestureListener g = new GestureListener() {


public void onGestureChange(Gesture arg0) {}
public void onGestureFinish(Gesture arg0) {
if (arg0 == Gesture.CENTER) {
// play/pause music
} else if (arg0 == Gesture.RIGHT) {
playNext();
}
}
public void onGestureStart(Gesture arg0) {}
};

Eyes-Free Interaction Google-IO May 2009 – 34 / 39


Adding Overlay To Music Player
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

myFrame = new FrameLayout(this);


myFrame.addView(myList);
myG = new TouchGestureControlOverlay(this,
gestureListener);
overlayActive = false;
setContentView(myFrame);

Eyes-Free Interaction Google-IO May 2009 – 35 / 39


Toggling Gesture Support
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

if (overlayActive) {
myFrame.removeView(myG);
overlayActive = false;
tts.speak("Gestures disabled.", 0, null);
} else {
myFrame.addView(myG);
overlayActive = true;
tts.speak("Gestures activated.", 0, null);
}

Gesture support enabled by adding about 40 lines of code.

Eyes-Free Interaction Google-IO May 2009 – 36 / 39


Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Conclusion
Summary
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

■ Programmable smart phones are an engineer’s dream!


■ I/O peripherals open up new UI avenues.
■ Provide desired information with minimal user interaction.
Profound impact on how we work and play!

Eyes-Free Interaction Google-IO May 2009 – 38 / 39


Watch Computing Take Off!
Android Dialer Contacts TTS Gestures MusicBrowser Overlays Conclusion

Eyes-Free Interaction Google-IO May 2009 – 39 / 39