CoreMotion and the Gyroscope

(from Jenga to Augmented Reality)

VTM: iPhone Developers Conference. October 17, 2010.
Jeffrey Powers, Co-Founder, Occipital @jrpowers
Plus: How to build your own
spacetime portal.
Who am I?
Jeff Powers
University of Michigan Bachelors
in EE:Signals, Masters in
CS:Artificial Intelligence
Co-founder, Occipital
(computer vision / AR)
@jrpowers
! Founded Occipital in 2008. Occipital is about mobile
computer vision and augmented reality.
! 2008: TechStars in Boulder
! 2009: RedLaser
! 2010: Sold RedLaser, but remained independent.
!"#$%& #" # (#))
*+$ ",-. -/"%$0-%(
1223
" Today: 360 Panorama
" Realtime panorama capture
" Computer vision based.
This is actually the pano I took during the talk.
452
What well cover.
" Demos: What can CoreMotion do?
" The Hardware
" The chips.
" How accurate are they?
" Coding with CoreMotion.
" The APIs.
" Acclerometer
" Gyro
" Getting started.
" Pitfalls you probably arent aware of.
" Performance analysis and tips.
" Building your own Spacetime Portal
" Coremotion, AVFoundation, OpenGL ES 2.0
Warm up Demos
Lets see CoreMotion in action
67$+.8+9% &%:+ *$+: -;,+/% <
#//+=/8%:%/">
?-$." &%:+ (#. 452 ;#/+$#:#@ A7$+BC++."%&
9#/+$#:#.
D%:+ 8+:9#$%& 452 (E A7$+ +/ +$ +F
G-A H C=-)" "+ .,+( &%:+.@
I#8A=70%$ (+=)& C% 9$+=&
Hardware
The sensors that enable CoreMotion
CoreMotion Hardware Sensors
! Accelerometer
! Gyroscope
! Compass (sort of)
! Compass is used, but only to avoid drift.
! You cant determine compass bearing with CoreMotion
alone.
! This means CoreMotion is NOT for:
! GPS positioning
! Compass bearing
CoreMotion Sensors (on iPhone 4 )
!"#$%&&
JK,-. -. )+8#"%& C%,-/& #/& #C+0% ",%
L< 8,-9 M (,-8, -. C%)+( ",% 8#:%$#>N
1O1::
!+=$8%@ P,-9(+$Q.
CoreMotion Sensors (on iPhone 4 )
())*+*,"#*-*,
4O4::
./,"&)"$* JR%(SN
<O<::
!+=$8%@ P,-9(+$Q.
Coding with CoreMotion
From beginner to creating your own spacetime portal
! The obligatory Teapot.
! Get the WWDC 2010 CoreMotion
Teapot Sample
http://developer.apple.com/
videos/wwdc/2010/
(Download Sample Code)
Coding with CoreMotion
Getting Started
The APIs: Raw accelerometer access
" CMMotionManager The central access point for motion
sensing.
" CMAccelerometerData Raw accelerometer readings.
" CMGyroData Raw gyro readings.
" CMDeviceMotion Fusion of all sensors.
" .attitude (CMAttitude)
" .rotationRate (CMRotationRate)
" .gravity (CMAcceleration)
" .userAcceleration (CMAcceleration)
Accessing the Accelerometer with
CoreMotion
1. Create the MotionManager:
!"#$"%&'%'()* , --.&&"#$"%&'%'()* '//"01 $%$#12
Create only one of these!
2. Check sensor availability:
-!"#$"%&'%'()* $3400/)*"!)#)*45'$/'6/)1
Sensor Availability
" Hardware support:
" Accelerometer:
" iPhone 2G/3G/3GS/4
" iPad
" iPod touch 4G
" Gyroscope:
" iPhone 4
" iPod touch 4G
" DeviceMotion (Gyro + Accelerometer)
" Same as Gyroscope
" Prohibiting install without proper hardware:
" UIRequiredDeviceCapabilities
" '00)/)*"!)#)*
" (7*"30"8)
Accessing the Accelerometer with
CoreMotion
3. Set update frequency (in seconds):
-!"#$"%&'%'()* 3)#400)/)*"!)#)*98:'#);%#)*5'/<=>=?1
4. Start CoreMotion running: (two different options)
-!"#$"%&'%'()* 3#'*#400)/)*"!)#)*98:'#)312
"# -!"#$"%&'%'()*
3#'*#400)/)*"!)#)*98:'#)3@"AB)B)< C 1
Accessing the Accelerometer with
CoreMotion
5. Access the raw data:
.&400)/)*"!)#)*D'#' $ %&&'()%*% +
,-.*/.01%0%2'3 %&&'('3.-'*'3)%*%45
.&400)/)*'#$"% %&&'('3%*/.0 +
%&&'()%*%6%&&'('3%*/.05
Accessing the Accelerometer with
CoreMotion
1. Create the MotionManager:
!"#$"%&'%'()* , --.&&"#$"%&'%'()* '//"01
$%$#12
Create only one of these!
2. Check sensor availability:
-!"#$"%&'%'()* $3400/)*"!)#)*45'$/'6/)1
Accessing the Accelerometer with
CoreMotion
/* CMAccelerometerData *
* Contains a single accelerometer measurement. */
@interface CMAccelerometerData : CMLogItem
{ @privateid _internal; }
/* acceleration *
The acceleration measured by the accelerometer.*/
@property(readonly, nonatomic) CMAcceleration acceleration;
@end
/* CMAcceleration *
* A structure containing 3-axis acceleration data. *
* Fields: *
x: X-axis acceleration in G's. *
y: Y-axis acceleration in G's. *
z: Z-axis acceleration in G's. */
typedef struct {
double x; double y; double z;} CMAcceleration;
Accessing the Accelerometer with
CoreMotion
Hmm this:
typedef struct {
double x; double y; double z;}
CMAcceleration;
sounds a lot like UIAccelerometers UIAcceleration class:
@interface UIAcceleration : NSObject
{ @private NSTimeInterval timestamp;
UIAccelerationValue x, y, z;}
(Minus the timestamp.)
So, what about UIAccelerometer?
! Bad news if you were a UIAccelerometer fan:
!+=$8%@ L99)% -T! <>U G%*%$%/8% V-C$#$7
But UIAccelerometer had a
timestamp, wasnt it better?
" No.
" Every piece of sensor data in CoreMotion is a CMLogItem
@interface CMAccelerometerData : CMLogItem
" And guess what CMLogItem has
7/0*'38%&' 91:.2;*'- < =>"?@'&* A=>9.B/02C =>9.DE/02F
G7D3/H%*'/B I/0*'30%(:.2;*'-5J
K$ $ */-'L*%-D< M/-' %* NO/&O *O' /*'- /L H%(/B6 $ $K
E8*"8)*#7F*)':"%/7G %"%'#"!$0H IJ@$!);%#)*5'/ #$!)3#'!82
7'0B
PITFALL
! CMLogItems may come back nil!
! accelData, gyroData, deviceMotion
if([motionManager isAccelerometerActive]) {
CMAccelerometerData * aData = [motionManager accelerometerData];
// do something with acceleration
aData.acceleration // NOT SAFE!!
}
CMAccelerometerData * aData = [motionManager accelerometerData];
if(aData != NULL){
// do something with acceleration
aData.acceleration // SAFE
}
Accessing the Accelerometer with
CoreMotion
Recap
1. Create one (and only one) CMMotionManager
2. Check sensor availability
3. Set update interval
4. startAccelerometerUpdates
5. Poll x,y,z values periodically (or get them pushed to
you)
CoreMotion Axis Conventions
From iOS SDK Documentation
Accelerometer reports acceleration mainly due to gravity, but also
due to user motion.
CMRotationMatrix rotationMatrixFromGravity(float x, float y, float z)
{
// The Z axis of our rotated frame is opposite gravityvec3f_t zAxis =
vec3f_normalize(vec3f_init(-x, -y, -z));
// The Y axis of our rotated frame is an arbitrary vector perpendicular to gravity
// Note that this convention will have problems as zAxis.x approaches +/-1 since
the magnitude of
// [0, zAxis.z, -zAxis.y] will approach 0
//vec3f_t yAxis = vec3f_normalize(vec3f_init(0, -zAxis.z, zAxis.y));
// For Augmented Reality (AR), we want a different convention. Specifically
// we would prefer to ambiguity to occur when zAxis.z approaches +/-1
vec3f_t yAxis = vec3f_normalize(vec3f_init(zAxis.y, -zAxis.x, 0));
// The X axis is just the cross product of Y and Zvec3f_t
xAxis = vec3f_crossProduct(yAxis, zAxis);
CMRotationMatrix mat = { xAxis.x, yAxis.x, zAxis.x,
xAxis.y, yAxis.y, zAxis.y,
xAxis.z, yAxis.z, zAxis.z };
return mat;
}
From CMTeapot, tweaked for AR scenarios.
The APIs
" CMMotionManager The central access point for motion
sensing.
" CMAccelerometerData Raw accelerometer readings.
" CMGyroData Raw gyro readings.
" CMDeviceMotion Fusion of all sensors.
" .attitude (CMAttitude)
" .rotationRate (CMRotationRate)
" .gravity (CMAcceleration)
" .userAcceleration (CMAcceleration)
Accessing the Gyroscope with CoreMotion
[Same basic setup as Accelerometer]
1. [Same] Create one (and only one) CMMotionManager
2. [Same, but Gyro] Check sensor availability
3. [Same, but Gyro] Set update interval
4. startGyroUpdates
5. Poll gyro values periodically (or get them pushed to
you)
Accessing the Gyroscope with CoreMotion
CMGyroData
Typo is actually from the iOS headers!
/* CMAccelerometerData
* Contains a single accelerometer
measurement. */
@interface CMGyroData : CMLogItem
{ @private id _internal; }
/* * rotationRate: The rotation rate as
measured by the gyro. */@property(readonly,
nonatomic) CMRotationRate rotationRate;
@end
Accessing the Gyroscope with CoreMotion
CMRotationRate
/* * CMRotationRate * * Discussion: * A structure containing
3-axis rotation rate data. * * Fields: * x: * X-axis
rotation rate in radians/second. The sign follows the right hand *
rule (i.e. if the right hand is wrapped around the X axis such that
the * tip of the thumb points toward positive X, a positive
rotation is one * toward the tips of the other 4 fingers). *
y: * Y-axis rotation rate in radians/second. The sign follows
the right hand * rule (i.e. if the right hand is wrapped
around the Y axis such that the * tip of the thumb points
toward positive Y, a positive rotation is one * toward the
tips of the other 4 fingers). *z: *Z-axis rotation rate in radians/
second. The sign follows the right hand * rule (i.e. if the
right hand is wrapped around the Z axis such that the * tip
of the thumb points toward positive Z, a positive rotation is one
* toward the tips of the other 4 fingers). */
typedef struct {double x;double y;double z;} CMRotationRate;
Radians per second around each axis.
CoreMotion Axis Conventions
G+))
;-"8,
W#(
#""-"=&% X JW#( Y ;-"8, Y G+))N
K
The APIs
" CMMotionManager The central access point for motion
sensing.
" CMAccelerometerData Raw accelerometer readings.
" CMGyroData Raw gyro readings.
" CMDeviceMotion Fusion of all sensors.
" .attitude (CMAttitude)
" .rotationRate (CMRotationRate)
" .gravity (CMAcceleration)
" .userAcceleration (CMAcceleration)
Accessing DeviceMotion with CoreMotion
[Setup is just like Accel/Gyro]
1. [Same] Create one (and only one) CMMotionManager
2. [Same, but deviceMotion] Check availability
3. [Same, but deviceMotion] Set update interval
4. startDeviceMotionUpdates
5. Poll deviceMotion values periodically (or get them
pushed to you)
/* * CMDeviceMotion: A CMDeviceMotion object contains basic information
about the device's *motion. */
@interface CMDeviceMotion : CMLogItem
{ @private id _internal; }
/* attitude */
@property(readonly, nonatomic) CMAttitude *attitude;
/* rotationRate */
@property(readonly, nonatomic) CMRotationRate rotationRate;
/* gravity: Returns the gravity vector expressed in the device's
reference frame. Note *that the total acceleration of the device is
equal to gravity plus *userAcceleration. */
@property(readonly, nonatomic) CMAcceleration gravity;
/* userAcceleration: Returns the acceleration that the user is giving
to the device. */
@property(readonly, nonatomic) CMAcceleration userAcceleration;
@end
Accessing DeviceMotion with CoreMotion
CMDeviceMotion
@interface CMAttitude : NSObject <NSCopying, NSCoding>
{ @private id _internal; }
@property(readonly, nonatomic) double roll;
@property(readonly, nonatomic) double pitch;
@property(readonly, nonatomic) double yaw;
/* rotationMatrix */
@property(readonly, nonatomic) CMRotationMatrix rotationMatrix;
/* quaternion */
@property(readonly, nonatomic) CMQuaternion quaternion);
/* multiplyByInverseOfAttitude: * Multiplies attitude by the
inverse of the specified attitude. This gives *the attitude
change from the specified attitude. */
- (void)multiplyByInverseOfAttitude:(CMAttitude *)attitude;
@end
Accessing DeviceMotion with CoreMotion
CMAttitude
Attitude
" attitude = (yaw * pitch * roll)
T
motionOri =
matrix3d::matrix3d_from_z(yaw) *
matrix3d::matrix3d_from_x(pitch) *
matrix3d::matrix3d_from_y(roll);
return motionOri.inverse();

" Attitude is the rotation from the devices current position
back to its original position. This matrix can be used as-is to
keep a 3D object stationary.
" Rotation matrices above are just rotations about the specified
axes. See http://en.wikipedia.org/wiki/Rotation_matrix for
their definitions.
Z.%*=) "++) (,%/ )%#$/-/A &%0-8%I+"-+/@
67$+.8+9% #99 J*$%%N *$+: #8$+..#-$
"+ .%% 7#([9-"8,[$+)) #/A)%.
So, shouldnt I just ALWAYS use
deviceMotion?
! CoreMotion isnt free to run.
! Even if you set a low update rate, it will use a good
amount of CPU.
Performance Considerations
!"#$% '() )*$+,
-.. /0 1. /0
!"#$% 233 !"#$% 233
!"#$%"&'(') +,- ./- +,- 0/-
1%%"2"3'4"5"3 ,/- 0,- 6+- ,-
1%%"27893' ,0- 0/- ,/- ,-
0%1 2*3&4
D%0-8%I+"-+/ %#". U\] !"#$ "+"#) P;Z #$%&#'($)) +* =9&#"% $#"%>
.""1 2*3&4
H"^. /+" C%8#=.% +* ",% 67$+ -".%)*> G%#&-/A ",% A7$+ -/ #&&-"-+/ "+
",% #88%)%$+:%"%$ &+%. /+" .)+( #/7",-/A &+(/ /+"-8%#C)7>
Building your own spacetime
portal
Bonus: No particle physics PhD required
5",-%+
_#)0% !+*"(#$%
6"7,)*4 5,"#" 3%++$%$*, 8", 5",-%+9
What it takes to build real-world
Portal with CoreMotion
! OpenGL Sample project, plus:
! CoreMotion framework (attitude)
! CoreGraphics framework reference (Images)
! AVFoundation, CoreVideo framework reference (Camera)
! 360-degree images (for display in AR)
! Matrix and Vector classes from Teapot Sample
! AVFoundation captureOutput session feeding to OpenGL
(Ben Newhouse briefly covered this yesterday)
! Modified OpenGL ES 2.0 shaders to draw the hole
!9#8%"-:% ;+$"#) D%:+