Kinect Open Source Programming Secrets : Hacking the

Kinect With OpenNI, NITE, and Java

Table Of Contents:
Acknowledgment
Introduction
Chapter 1 Getting Started
The Kinect Sensor Hardware
Kinect Development Software
The 800-Pound Gorilla Enters
Programming the Kinect in Java
Kinect Software Installation
Cleaning Up First
Downloading the Packages
Installing and Configuring the Software
Fixing the XML Configuration Files
Testing OpenNI and NITE
Documentation and Other Sources of Help
OpenNI, NITE, and Kinect Documentation
Java Programming Documentation
OpenNI Concepts in Java
Generating Data and Metadata
Listeners
Capabilities
The Simplest Java OpenNI Example
Chapter 2 Kinect Imaging

xiii
xv
1
1
2
4
4
5
5
6
7
7
9
11
11
12
13
14
14
15
16
19

(18)
(1)
(3)
(1)
(1)
(6)
(1)
(1)
(1)
(2)
(2)
(2)
(1)
(1)
(3)
(1)
(1)
(1)
(3)
(28)

An Overview of the OpenNIView

Application Versions
Version 1 Grayscale Depth Map and XML
Updating the Panel
From Depth Values to Image Pixels
Rendering the Depth Map
Version 2 Grayscale Depth Map Without
XML
Version 3 Camera Image Map
Updating the Image
Drawing the Image
Version 4 Infrared Map
Updating the Image
Creating a Grayscale Image
Version 5 A Colored Depth Map
Creating a Color Model
Creating the Depth Map
Rendering the Image
Version 6 Combined Depth and Image
Maps
Creating a Color Model
Initializing the Context
Updating the Panel
Rendering the Images
Chapter 3 A Point Cloud for Depths
An Overview of the PointCloud Application
What Is Java 3D?
The HelloUniverse Scene Graph
Java 3D Documentation and Examples
Creating the 3D Scene for PointCloud
Lighting the Scene

19 (2)
21
23
25
27
28

(7)
(2)
(2)
(1)
(1)

29
31
33
34
35
36
37
38
39
40
41

(5)
(2)
(1)
(3)
(1)
(1)
(4)
(1)
(1)
(1)
(6)

42
43
44
45
47
47
50
51
53
54
55

(1)
(1)
(1)
(2)
(22)
(3)
(4)
(2)
(1)
(6)
(2)

Setting the Scene's Background

Positioning the Point Cloud
Viewer Positioning
Viewer Movement
Point Clouds in Java 3D
Initializing the PointArray
Updating the Points
Obtaining the Depth Map
Chapter 4 Tracking Users in 2D
An Overview of the GorillasTracker
Application
Configuring the Tracker
Updating the Scene
Drawing the Scene
Managing Skeletons
Configuring the User Generator
Listening for Users
Using Other Listeners
Updating the Skeletons
A More Complex Skeleton
Drawing the Skeletons
Drawing a Skeleton
Drawing a Head Image
Displaying User Status Information
Chapter 5 Viewing Users in 3D
An Overview of the UsersViewer3D
Application
Configuring the Tracker
Managing the Skeletons
Configuring the UserGenerator
Setting Up the Observers

57
57
58
59
60
60
64
66
69
69

(1)
(1)
(1)
(1)
(6)
(4)
(2)
(3)
(24)
(3)

72
73
75
76
77
79
82
83
86
87
88
90
91
93
93

(4)
(2)
(1)
(6)
(2)
(3)
(1)
(3)
(1)
(6)
(2)
(1)
(2)
(40)
(3)

96 (2)
98 (7)
98 (2)
100(4)

Updating the Skeletons

Bringing a Skeleton to Life
Creating a Skeleton Scene Graph
Building Joints
Building Limbs
Updating the Skeleton
Modifying a Skeleton's Visibility
Deleting the Skeleton
Creating a 3D Joint
Creating the Joint Subgraph
Updating the Joint's Position
Smoothing a Position
Handling Joint Visibility
Creating a 3D Limb
Creating the Limb Subgraph
Updating the Limb
Adjusting Limb Position and Length
Changing the Limb's Orientation
An Interpolation and Cross-Product
Problem
Chapter 6 The Tilt Motor, LED, and
Accelerometer
Controlling the Kinect's Motor, LED, and
Accelerometer
A Quick Introduction to USB
Becoming a USB Detective
Installing USB Support for Java
Installing libusb-win32
Installing a libusb-win32 Driver for the
Kinect Motor
Installing and Testing LibusbJava

104(1)
105(9)
105(4)
109(2)
111(2)
113(1)
113(1)
114(1)
114(7)
114(4)
118(2)
120(1)
121(1)
121(12)
122(3)
125(2)
127(2)
129(2)
131(2)
133(22)
133(1)
134(2)
136(1)
137(5)
138(1)
138(2)
140(1)

Using a Modified LibusbJava

Motor Protocol Discovery
Setting the Tilt Motor, LED, and
Accelerometer
Setting the LED Status Light
Tilting the Kinect
Reading the Motor's State
What Is the Motor's Current Status?
Reading the Tilt Angle
Reading the Accelerometer
Reading the Kinect's Speed
Detecting Tilting Limits
Testing the MotorCommunicator
Chapter 7 NITE Hand Gestures
An Overview of the GestureDetect
Application
Focus Detection
Gesture Tracking
Detecting Gestures
Initializing OpenNI
Processing Hand Events
Processing Gesture Focus Events
Initializing NITE
Processing Session Events
Detecting Hand Points
The Wave Detector
The Push Detector
The Swipe Detector
The Circle Detector
The Steady Detector
Filter Objects

141(1)
142(1)
143(5)
146(1)
147(1)
148(4)
149(1)
150(1)
151(1)
152(1)
152(1)
152(3)
155(24)
155(1)
156(1)
157(1)
158(9)
159(1)
160(2)
162(2)
164(1)
165(2)
167(11)
170(2)
172(1)
173(2)
175(2)
177(1)
178(1)

Chapter 8 NITE HandsTracker

An Overview of the HandsTracker
Application
Configuring NITE
Configuring the Kinect
Tracking the Session
Detecting Hand Points
Updates from the Kinect
Painting the Panel
Storing a Hand Trail
Adding a Point
Drawing the Trail
Chapter 9 Kinect Breakout
An Overview of the Kinect Breakout
Application
Creating the Breakout Panel
Listening for Keys
Configuring the Kinect
Moving the Paddle
The Game Loop
Updating the Game
Painting the Game
A Sprite
The Sprite Constructor
Collision Detection
Updating a Sprite
Drawing a Sprite
The Paddle
Moving the Paddle
Using the Paddle's Direction
Managing the Bricks

179(16)
179(2)
181(1)
181(2)
183(2)
185(2)
187(1)
188(3)
191(4)
191(1)
192(3)
195(28)
195(1)
196(14)
199(1)
199(5)
204(1)
205(2)
207(2)
209(1)
210(3)
210(2)
212(1)
212(1)
213(1)
213(3)
215(1)
216(1)
216(3)

Collision Detection
Drawing the Bricks
Defining a Brick
The Ball
Initializing the Ball
Updating the Ball
Chapter 10 Gesture GUIs
An Overview of the TestGestureGUIs
Application
Component States
TestGestureGUIs Application Classes
Setting Up the Kinect
Changing Between Coordinate Spaces
The Gesture GUI Manager
Calculating the Component's Rectangles
Updating the GGUI Components
A GGUI Component Class
Component Information
Rendering the GGUI Component
The Button GGUI Component
The Dial GGUI Component
Updating the Dial
Drawing the Dial
The Slider GGUI Component
Initializing the Slider
Updating the Slider
Drawing the Slider
Chapter 11 Kinect Capture
An Overview of the CapturePics Application
Displaying Pictures
Snapping a Picture Again and Again and...

218(1)
218(1)
219(1)
219(4)
219(1)
220(3)
223(28)
223(3)
223(2)
225(1)
226(4)
230(1)
231(3)
232(1)
233(1)
234(6)
237(1)
238(2)
240(1)
241(5)
242(3)
245(1)
246(5)
248(1)
248(2)
250(1)
251(10)
252(1)
253(4)
253(2)

Terminating the Application

Painting the Panel
Capturing the Kinect's Camera
Obtaining an Image
Closing Down
Chapter 12 Motion Detection Using OpenCV
Detection Tools
OpenCV
JavaCV
A Couple of OpenCV/JavaCV Examples
Motion and Change Detection in OpenCV
The CVMotionDetector Test Rig
Initializing CVMotionDetector
Detecting Movement
A Moment Spent on Moments
From Points to Pixels
Moments in OpenCV
The GUI-Based Motion Detector
The Webcam Display Loop
Reporting on the Center of Gravity
Rendering Motion Detection
Modifying the OpenCV Motion Detection
Drawbacks to This Approach to Movement
Detection
More OpenCV Examples
Chapter 13 FAAST-Style Body Gestures
Extending GorillasTracker
An Overview of the New GorillasTracker
Creating the Gesture Detectors
Updating the Skeleton and Detectors
The Observers

255(1)
256(1)
257(4)
258(1)
259(2)
261(26)
261(6)
261(1)
262(1)
263(4)
267(9)
268(2)
270(1)
271(1)
272(2)
274(1)
275(1)
276(6)
277(2)
279(1)
280(2)
282(2)
284(1)
285(2)
287(22)
288(3)
291(3)
292(1)
292(1)
293(1)

A Skeletal Reminder
Detecting Basic Gestures
Checking for Gestures
Calculating Skeleton Lengths
Checking a Gesture
Turning to the Left
Moving the Right Hand Up and Down
Complex Gestures
Finding a Complex Gesture
Index

294(2)
296(9)
297(1)
298(1)
299(3)
302(1)
303(2)
305(4)
306(3)
309