Jump to content

Introduction to OpenKinect and as3Kinect

0
  imekinox's Photo
Posted Feb 01 2011 03:46 PM

This article functions as an introduction to building OpenKinect and as3kinect projects. This is the first in a series of articles on this topic. This opening article attempts to answer the following seven questions:

  • What is Kinect, and what can it do?
  • When and how did OpenKinect get started?
  • What is libfreenect?
  • How do you connect the Kinect to ActionScript?
  • How do you read from libfreenect (the driver) in your applications?
  • How do you send libfreenect data over the network?
  • How do you read and send data in AS3?


What is Kinect, and What Can It Do?

A Kinect is a hardware device that has two cameras. The first camera functions just like any other video camera you may have used (it captures RGB video). It’s the second camera that makes Kinect so interesting to many developers. It can capture infrared (IR) light (similar to the “night mode” in many camcorders). If you’re wondering how this camera makes Kinect so special, the answer is that while it captures IR light; the device also projects dots of IR light onto everything that is in front of it. It then analyzes the distortion and size of each dot to calculate the position and distance between the light source and that dot.

In addition, the Kinect has an array of built-in microphones that can be used to capture voice input, and a motor, that tilts the device up and down, so it can capture motion across a wider field of view.

When and How Did OpenKinect Get Started?

It was November 10, 2010 when Microsoft released the Kinect in Europe. And just three hours later, in Spain, Hector Martin got the Kinect working on Linux. He released his code in open source form along with a video that he uploaded to YouTube:



A few hours after that, the libfreenect project was born, headed by Joshua Blake, and maintained by Hector Martin and Kyle Machulis. By the next day, libfreenect was working on OSX, thanks to Theo Watson&—the idea of controlling Kinect from ActionScript was born, and by the end of the week, the OpenKinect community already had a few members.

As an ActionScript developer for a few years now, and with little experience in C, I initiated the quest to make this happen.

What Is libfreenect?

Libfreenect is the open source driver that enables users from almost any platform to gain access to the Kinect’s RGB video stream and to the processed dot array (dot position and distance) known as Depth stream. It also lets you to control the motor (over a range from -31 to 31 degrees), and change the LED to any of 5 available status values (there are 6 possible states, but values 5 and 6 are the same state).

At the beginning, this programming effort was claimed to be a hack—but it isn’t, because the device itself wasn’t hacked. Just to be clear, to control a Kinect in this manner, you do not need to open the device, nor cut any wires , nor connect it to custom hardware, and you do not need to overwrite the Kinect firmware for it to work on any computer. So basically nothing needs to be hacked.

To make this work, you just need to plug the USB adaptor that comes with the Kinect into your computer, and the libfreenect driver will initialize the device and start providing both streams (video and positional information) in memory byte arrays.

Here is a high-level overview diagram that shows how libfreenect works:

Attached Image

As you can see, libfreenect acts as a bridge between the USB device and an application; such a bridge is called a driver.

How Do You Connect the Kinect to ActionScript?

The most common way to make a device interact with Flash is through a socket connection. In fact, after considerable research, it turns out this still is the only way devices can interact with Flash. Because of the Flash Player’s security policies, an SWF application cannot interact directly with hardware.

The following diagram illustrates the basic concept to connect hardware to ActionScript:

Attached Image

First, you need an application that connects to the driver for the hardware you want to use. Next, you make the application connect to that driver. The server reads the data from that application and sends it through the network to a client. Although the server could send data to the client whenever that data becomes available, in this model the server sends data only when the client makes a request for it. For the Kinect/ libfreenect/ActionScript combination, I think this is the best choice, because the computer processes C faster than it can process ActionScript. So the slower script requests data from the faster script only when it can process it, which avoids packet loss.

So to connect a Kinect to ActionScript this more slightly more detailed model describes what must happen:

Attached Image

At the beginning of this diagram, moving right to left, you already have a Kinect and libfreenect, what you need is an application that acts like a server and sends the data to ActionScript over the network. That’s what the as3kinect project does. It’s an open source server and client that acts as a bridge between libfreenect and an ActionScript application.

Note: Both the libfreenect and as3kinect projects are completely open source, and some of the code shown in this document comes from their libraries.

How Do You Read from libfreenect (the Driver) in Your Applications?

Libfreenect provides an API for doing this. One interesting way to explore how it works is to open the glview.c demo provided by libfreenect and take a look into the code. Here’s what you’ll find:

The application needs to include the driver header file:
#include "libfreenect.h"

Next you need freenect_context and freenect_device variable types, declared globally:
freenect_context *f_ctx;
freenect_device *f_dev;

You need a global definition for the RGB data buffer:
uint8_t *rgb_back;

You initialize the RGB buffer in the main function:
rgb_back = (uint8_t*)malloc(640*480*3);

Now you can initialize the freenect driver the main function:
freenect_init(&f_ctx, NULL)

We set a log level for this context: (Debug level in this example)
freenect_set_log_level(f_ctx, FREENECT_LOG_DEBUG);

Query for the the number of Kinects detected: (usually 1)
Int nr_devices = freenect_num_devices (f_ctx);

Choose the device you want to open: (0 is the first device)
int user_device_number = 0;

Finally, tell the driver to open that device:
freenect_open_device(f_ctx, &f_dev, user_device_number)

After opening the device, you can immediately begin communicating with it, which you want to do in a new thread. For example, you can set the motor angle using:
freenect_set_tilt_degs(f_dev, freenect_angle);

Or set the led state:
freenect_set_led(f_dev, LED_RED);

You identify a function to be called (a callback function) when depth data is received:
freenect_set_depth_callback(f_dev, depth_cb);

And a callback function called when video data is received:
freenect_set_video_callback(f_dev, rgb_cb);

You need to set the format in which you want to receive the video data…
freenect_set_video_format(f_dev, FREENECT_VIDEO_RGB);

…and the depth data:
freenect_set_depth_format(f_dev, FREENECT_DEPTH_11BIT);

Identify the buffer for the video data (that you initialized earlier):
freenect_set_video_buffer(f_dev, rgb_back);

Finally, you tell the driver to begin producing depth and video data:
freenect_start_depth(f_dev);
freenect_start_video(f_dev);

Here is a very basic example of an RGB video callback:
void rgb_cb(freenect_device *dev, void *rgb, uint32_t timestamp)
{
 //…
 rgb_mid = (uint8_t*)rgb;
 got_rgb++;
 // …
}

The preceding function stores a byte array in rgb_mid that is 640 x 480 x 3 bytes in size. The video resolution received here is 640px x 480px, and there are 3 bytes per pixel. The first byte is the red color intensity, the second is the green color, and the third is the blue color. So this is basically a bitmap—a single frame of the RGB video.

Things are a little different for depth; here’s a very basic example of the depth callback function:
void depth_cb(freenect_device *dev, void *v_depth, uint32_t timestamp)
{
 //…
  uint16_t *depth = (uint16_t*)v_depth;
 got_depth++;
 // …
}

This function stores a byte array in depth_cb that is 640 x 480 x 2 bytes in size. Note that the depth resolution is the same (640px x 480px) but this time you have only 2 bytes per pixel (or dot). The two bytes represent a 16-bit integer, which is the distance from the device to that specific point in the space. These depth values can range from 0 to 2048 (11 bits).

Note: Remember the constant FREENECT_DEPTH_11BIT when setting depth format.

How Do You Send libfreenect Data Over the Network?

To send the data from libfreenect over the network, you need to create a socket server. This is relatively simple code to create a socket that listens on port 6001 on UNIX systems (error handling has been removed from the following code for demonstration purposes):

//Socket variables declaration:
struct sockaddr_in si_type;
char *conf_ip = "127.0.0.1";
int the_socket   = -1,
conf_port = 6001;
int optval = 1;

//Create the socket
the_socket = socket(AF_INET, SOCK_STREAM, 0);

//Set options to socket
setsockopt(the_socket, SOL_SOCKET, SO_REUSEADDR, 
   (const void *)&optval, sizeof(optval));

//Alloc memory for sockaddr 
memset((char *) &si_type, 0, sizeof(si_type));

//Configure the socket
si_type.sin_family = AF_INET;
si_type.sin_port = htons(conf_port);
si_type.sin_addr.s_addr = inet_addr(conf_ip);

//Bind the socket
bind(the_socket, (struct sockaddr *)&si_type, sizeof(si_type));

//Start listening on the socket
listen(*the_socket, 1);
Finally you wait for a client to connect:

//Init socket variables
int childlen;
struct sockaddr_in childaddr;

//Set the len of childaddr
childlen = sizeof(childaddr);

//Start accepting connections
data_child = accept(s_type, (struct sockaddr *)&childaddr, 
   (unsigned int *)&childlen);
//Send data to the client using the write function
//n = write(data_child, msg, msg_len);


How Do You Read Sent Data in AS3?
With the socket set up and a connected client, here’s the ActionScript code that reads the data sent by libfreenect.
//Import Socket and ByteArray libraries (inside the package)
import flash.net.Socket;
import flash.utils.ByteArray;

//Declare variables (inside the class)
private var _socket:Socket;
private var _buffer:ByteArray;

//Init socket and byteArray (inside the constructor or init method)
_socket = new Socket();
_buffer = new ByteArray();

//Listen to data
_socket.addEventListener(ProgressEvent.SOCKET_DATA, onSocketData);

//Process data 
private function onSocketData(event:ProgressEvent):void {
   if(_socket.bytesAvailable > 0) {
      _socket.readBytes(_buffer, 0, _packet_size);
   }
}


At this point, you have a complete overview of the code required to get OpenKinect and as3kinect working together. Future articles will discuss how to install, compile, and use the required libraries in more detail, but to start exploring for yourself, take a look at the openkinect.org wiki. To gain more insight into the ActionScript interaction, see the as3kinect.org blog.

About the Author

Attached ImageJuan Carlos del Valle, who you may know online as ekinox or imekinox, has been working with ActionScript, PHP and MySQL for five years, developing web applications for publishing digital media. He's also familiar with Objective-C, C, C++, C#, Python, node-js, Javascript, jQuery, HTML5 and CSS3. He is currently part of a small team that does research and development. He enjoys experimenting with new technologies and programming languages.


Tags:
1 Subscribe


2 Replies

0
  Omar K's Photo
Posted Mar 16 2011 09:59 AM

excellent!
thank you so much juan.
0
  informtomani's Photo
Posted Jun 03 2011 09:37 PM

Hi,

I am using AS3openni example from google group. how to start and server and client code from flex.. the code will be of which language?