Integration of Mobile Apps into the Vehicle HMI

Terminal Mode Deep Dive

Presenter: Matthias Benesch, Daimler AG Jrg Brakensiek, Nokia Corporation

Overview
Introduction VNC Protocol Demonstration Part I Device Attestation Audio Service Negotiation Demonstration Part II Summary

Jrg Brakensiek, Nokia Research Center Palo Alto |

June 2, 2010

Introduction Guiding Principles

Technology properties Existing Non-proprietary Well known Stable Do extend only if necessary Be Application & OS Agnostic Support legacy application (no change needed) Allow for optimization to automotive Can be implemented on existing platforms

Introduction Terminal Mode Stack

Terminal Mode RTP & BT
(Audio)

VNC
(Display & Control)

Device Attestation

UPnP
(Service Discovery & Initiation)

Internet Protocol Abstraction Layer BT WLAN USB 2.0

VNC Protocol Overview

HU Display CE Display
VNC Server CE Display User Input VNC Client

Consumer Electronics Device

RFB Protocol Display Control Automotive Head Unit

Copy CE display to Head-unit display Receive user input events from the Head-Unit

VNC Protocol Framebuffer: Concept

Memory Remove Header FB

Memory Scaling FB

Add Header

RFB Protocol

VNC Protocol Framebuffer: Protocol

Framebuffer (FB) Update

Header

ARGB 888 RGB 565

A 8

R 8

G 8 R 5 G 6

B 8 B 5

Position of update in source FB

X / Y position Width / Height Encoding

VNC Protocol Framebuffer: Messages

Setup Phase

Set FB Size, CE Pixel Format Set Head-Unit Pixel Format Request FB Update Provide FB Update

Operation Phase

VNC Protocol Framebuffer: Updates

Full Update Incremental Update Pro: Reduce fragmentation and data transfer size

Con: Additional Processing for incremental analysis

Phone Model N97 N97 MTU Size 1500 8260 FB Size [kByte] 900 900 Home Screen Full Update 4 FPS 7 FPS 3 FPS 5 FPS Camera Viewer Full Update Change 22 % 24 % Inc. Update 12 FPS 16 FPS

VNC Protocol Framebuffer: Optimization

8 A 8 R 8 G 8 B Color Conversion

R 5 Pro: Reduce Data transfer size

G 6

B 5

Con: Color conversion processing at phone and head-unit

Phone Color FB Size Home Screen Camera Viewer Model Format [kByte] Full Update Full Update Change Inc. Update N97 N97 RGB888 RGB565 900 450 7 FPS 12 FPS 5 FPS 10 FPS 24 % 22 % 16 FPS 20 FPS

VNC Protocol Framebuffer: Optimization

Wait for Change FB Request FB Update FB Request FB Update FB Changed

Pro:

Reduced CPU load

Con: External trigger from Display Driver / UI framework

Phone Color No Wait Model Format Incremental CPU Load N97 N97 RGB888 RGB565 70 FPS 70 FPS 100 % 100 % Wait for FB Update Change 18 % 15 % Incremental 7 FPS 7 FPS CPU Load 17 % 12 %

Application: Performance Monitor

VNC Protocol Framebuffer: Optimization

ARGBARGBARGBARGBARGB

3 RGB 0 RGB Pro: Reduced CPU load

Run-Length Encoding

Con: Run-Length Encoding processing at phone and head-unit

Phone Model N97 N97 Update Full Increment RAW Encoding Update 7 FPS 7 FPS CPU Load 70 % 17 % RLE Encoding Average RL 6 pixel/run 6 pixel/run Update 19 FPS 7 FPS CPU Load 85 % 10 %

Application: Performance Monitor

VNC Protocol Framebuffer: Extension

Crop

Client Display Config.

Memory FB Scale

Scaling support Display resolution & size Distance to driver

+ -

Back Option

Virtual FB

VNC Protocol Framebuffer: Extension

Application

Device Status Key lock Device lock Dim display Night mode Voice input Driver Distraction Framebuffer rotation Framebuffer orientation

VNC Protocol Input Events: Protocol

Key Events Unique key identifier Key press & Key release Pointer Events Event position ( x / y ) Button mask (left, middle, right) Phone does not distinguish between local and remote events Keyboard events defined in /usr/include/keysymdef.h

VNC Protocol Input Events: Identifier

Specific Device Events ITU Key 0 (0, ' ) ITU Key 1 (1, '.', ,') ITU Key 2 (2, A, B, C) ITU Key 3 (3, D, E, F) Phone call start Phone call end Soft key Left Soft key Middle Soft key Right Application key Multimedia Play Multimedia Pause Multimedia Stop Multimedia Forward Key Identifier 0x3000 0100 0x3000 0101 0x3000 0102 0x3000 0103 0x3000 0200 0x3000 0201 0x3000 0202 0x3000 0203 0x3000 0204 0x3000 0205 0x3000 0400 0x3000 0401 0x3000 0402 0x3000 0403 Specific Head-Unit Events Shift Right Shift Left Shift Up Shift Up-Right Shift Up-Left Shift Down Shift Down-Right Shift Down-Left Push Rotate clock wise Rotate anti-clock wise Head-Unit Key 0 Head-Unit Key 1 Head-Unit Key 255 Key Identifier 0x3000 0001 0x3000 0002 0x3000 0004 0x3000 0005 0x3000 0006 0x3000 0008 0x3000 0009 0x3000 000A 0x3000 0020 0x3000 0050 0x3000 0060 0x3000 0300 0x3000 0301 0x3000 03FF

VNC Protocol Input Events: Extension

Touch Head-Unit

Application

Device Events Knob Events

Request Event Mapping

Event Mapping

Device Events

Non-Touch Head-Unit + Knob

VNC Protocol Input Events: Extension

City:

City:

Remove Keyboard

Show Keyboard Event List

White List: { e, u, o, a, s, f, v, m }

City: Hel

City:

VNC Protocol Driver Distraction

Ok to show, while driving

Not ok to show, while driving

VNC Protocol Driver Distraction

Problem Not all content is save to show on the head-unit display while driving. Head-unit cannot reason about the content on its display from the pixel data. Solution Provide Context Information together with the pixel data.
Context Information Structure X-position of rectangle Y-position of rectangle Width of rectangle Height of rectangle Context Information Encoding Unique application identifier Trust Level for application category Trust Level for content category Application category Content category Driver Distraction Rules followed

VNC Protocol Driver Distraction

Application Category

Content Category Text Video Image Vector Graphics 3D Graphics User Interface Car Mode Media Audio Voice Audio Misc. content

Trust Level No Trust User Configuration Self-Registered Application Registered Application Application certificate Driver Distraction Rules Minimum font size required. No video is shown. No automatic scrolling text Maximum feedback time allowed

Unknown Application UI Framework Phone call application Media applications Messaging applications Navigation Browser Productivity UI less applications System

VNC Protocol Driver Distraction

Ok to show, while driving

Not ok to show, while driving

Application not allowed!

Go back to Home Screen

Audio Input and Output Overview

Bluetooth A2DP & HFP RTP over UDP/IP

BT Audio Support
BT A2DP for application audio BT HFP for phone audio

RTP Audio Streaming

For application and phone audio (bi-directional) Audio can be as well voice commands or text-to-speech

Device Attestation Threats

Navigation
Fake TM SW

Application not allowed!

SMS

Context Information Filter Device

Navigation

Navigation
Fake Device

Device Attestation Trust model

Device Manufacturer (DM) Certifies Head-unit Manufacturer (HM)
SKHM/PKHM SKDM/PKDM CertDM = Sig(SKHM, PKDM)

Certifies
CertD = Sig(SKDM, PKD) SKD/PKD

Trusts

Device Attestation Protocol Head-unit (H)

Trust root PKHM

Device (D)

Device Attestation Attestation Protocol

Random Nonce, Attested software id Measure requested software component Extend with IP address and Port number Create attestation signature (Quote)

CertDM, CertD, Quote, IP, Port Verify CertDM with PKHM ( PKDM) Verify CertD with PKDM ( PKD) Verify Quote with PKD Save IP and port

Summary
Terminal Mode consists of a set of basic protocols, enabling user-interface replications VNC for display synchronization and event input RTP and BT for audio transport UPnP for service negotiation Driver distraction is taken care of in various elements Context Information Individual Launch and Termination Device Attestation Full specification for download at www.nokia.com/terminalmode

27

Jrg Brakensiek, Nokia Research Center Palo Alto |

June 2, 2010

What does Terminal Mode not Address?

Providing IP connectivity to the Head-Unit (IP tethering) Video and high-end Gaming Use cases for rear-seat entertainment systems Provide the CE device access to car data Provide the head-unit access to CE device data Certification process for automotive applications