Kinect Open Source Programming Secrets : Hacking the Kinect with OpenNI, NITE, and Java

Book excerpt: Program Kinect to do awesome things using a unique selection of open source software! The Kinect motion-sensing device for the Xbox 360 and Windows became the world's fastest-selling consumer electronics device when it was released (8 million sold in its first 60 days) and won prestigious awards, such as "Gaming Gadget of the Year." Now Kinect Open Source Programming Secrets lets YOU harness the Kinect's powerful sensing capabilities for gaming, science, multimedia projects, and a mind-boggling array of other applications on platforms running Windows, Mac OS, and Linux. Dr. Andrew Davison, a user interface programming expert, delivers exclusive coverage of how to program the Kinect sensor with the Java wrappers for OpenNI and NITE, which are APIs created by PrimeSense, the primary developers of the Kinect's technology. Beginning with the basics--depth imaging, 3D point clouds, skeletal tracking, and hand gestures--the book examines many other topics, including Kinect gaming, FAAST-style gestures that aren't part of standard NITE, motion detection using OpenCV, how to create gesture-driven GUIs, accessing the Kinect's motor and accelerometer, and other tips and techniques. Inside: Free open source APIs to let you develop amazing Kinect hacks for commercial or private use Full coverage of depth detection, camera, and infrared imaging point clouds; Kinect gaming; 3D programming; gesture-based GUIs, and more Online access to detailed code examples on the author's web site, plus bonus chapters on speech recognition, beamforming, and other exotica From the Author Why Buy This Book? I can suggest four reasons for buying this book: It offers a unique choice of Kinect programming tools. It explains the official Java wrappers for those tools. It covers topics not found elsewhere. It provides depth, but with brevity. Unique Programming Tools This is the only book on programming the Kinect using the OpenNI library, NITE, and Java (as of April 2012, when this book went to press). Official Java Wrappers This is the only book that explains the official Java wrappers for OpenNI and NITE (again, as of April 2012). By “official,” I mean that these bindings were developed by PrimeSense. Obvious advantages of Java include object-orientation, cross-platform support, availability for free, and many people (including you, probably) knowing how to program with it. Most important, programming in Java gives you access to a massive number of libraries—for graphics, networking, and beyond—that can be linked to the Kinect without much effort. For example, I’ll demonstrate how to use the Java 3D graphics library and the Java binding for the OpenCV computer vision package. The main drawback of using the PrimeSense Java wrappers is their lack of documentation. As I explain in Chapter 1, I had to decompile the libraries’ JAR files, and work out the correspondences between the Java source and the somewhat better documented C++ OpenNI/NITE APIs. (This is why including Secrets in the book’s title isn’t too excessive.) A Wide Range of Topics This book covers programming topics not found elsewhere. I start off with the basics, of course, with chapters on depth, infrared, and RGB imaging, point clouds, skeletal user tracking, hand tracking, and gesture support. Moving beyond that, I cover several novel and unusual features, including the following: Kinect gaming based around a version of the classic Breakout video game. Controls for the Kinect motor, LED, and accelerometer, which are not part of the standard OpenNI API. In fact, their absence is often held up as a serious drawback of the API. It’s actually quite easy to add these capabilities using a custom-built USB driver. 3D graphics programming in the point cloud and skeletal tracking examples, using Java 3D. A computer vision example that demonstrates how to link the Kinect to the popular (and powerful) OpenCV library. The creation of new body gestures (inspired by the FAAST system), which are not part of the limited NITE repertoire. A new type of GUI component controlled by hand gesturing, illustrated with three examples: a button, dial, and slider. These components are controlled without the help of mouse or keyboard. Depth with Brevity This book describes a lot of complicated code but, unlike some rather hefty programming tomes, you won’t find all the code tediously printed on these pages. Instead, you can download it from the book’s website. In addition, I’ve been adding supplementary chapters to the website, including ones discussing speech recognition and the Kinect microphone array.