Help
You can select the resolution, usage model, and camera angle as well as view status information. You can create this simple application in less than an hour after you have completed everything in the Getting Started section.
Getting Started
To get started you will need to download some software and perform the following steps:
- Verify your system meets the requirements.
- Download Microsoft Visual Studio 2010 Express if you do not have a retail version of Visual Studio 2010 already installed.
- Download the Kinect SDK from Microsoft Research.
- In this project, you’ll be the subject of the images you capture. Set up the Kinect so that it’s pointing at you, and verify that you are about four feet from the sensor.
After you have all obtained the required software, follow the installation instructions. That document walks you through the Kinect SDK installation process, so I won’t include that information here.
Creating the project
After you have successfully installed the Kinect SDK, you can create the skeleton project. This section shows how to create a C# WPF application that uses the Kinect, but tou can use any managed language or native C++ to communicate with the Kinect.
- Start Visual Studio 2010
- Open the “New Project” window by clicking File | New | Project
- Select the WPF Application template under Visual C | Windows and type in the name of the project you wish to create. To follow along with this article, name the project Video Kinect as shown in the following picture.
- Click the OK button to create the new project.
- Select Project | Add References from the menu, and then select the Microsoft.Research.Kinect assembly as shown in the following picture.
- Click the OK button.
- Open the Mainwindow.xaml.cs file and add using Microsoft.Research.Kinect.Nui to the top of the code. Adding this using reference will reduce the amount of typing you will need to do when referring to the Kinects SDK.
You should now have a WPF Application that can be used to connect to the Kinect Image SDK.
Understanding the SDK
Before you begin, you need to know about the Runtime class, and how to use it to retrieve data from the sensor.
Creating the Runtime Class
The Runtime class represents a Kinect sensor. You use it to communicate with the device. Begin by creating an instance of the class and initializing it. The following code demonstrates how to create the class.
m_nui = new Runtime();
Note: If you plan to communicate with more than one Kinect at a time, you will need to use the
Runtime(int index) constructor instead. Initializing the NUI Subsystem
After creating the Runtime instance, you need to initialize NUI subsystem that you plan on using by calling the Initialize method.
void Initialize(RuntimeOptions options);
The options argument tells the sensor which subsystem you plan on using. Valid choices are UseColor, UseDepth, UseDepthAndPlayerIndex, and UseSkeletalTracking. For this article you are interested in the camera images, so specify the UseColor option as follows.
m_nui.Initialize(RuntimeOptions.UseColor);
The method throws an InvalidOperationException if the Kinect isn’t actually connected to your computer, so you should wrap this code with a try/catch statement and handle the potential error appropriately.
Setting up the images
Next, you need to tell the sensor what type of color image data you want to receive. You do this by calling the Open method on the VideoStream property of the Runtime instance. The Open method is defined as
void Open(ImageStreamType streamType, int poolSize, ImageResolution resolution, ImageType image);
The streamType argument for the camera used in this article is ImageStreamType.Video. Future articles will discuss other types.
The poolSize argument sets the number of frames that the sensor will buffer before the runtime starts dropping un-retrieved frames. According to the help, a value of 2 is sufficient for most applications.
The resolution argument specifies the resolution of the images that you want to retrieve. For color RGB images the valid options are ImageResolution.Resolution640x480 and ImageResolution.Resolution1280x1024. Higher resolution images are limited to a lower FPS than the lower resolution images. To get images at 30 FPS, you need to specify the 640x480 image resolution option.
The image argument specifies the type of image you wish to receive. You may retrieve either RGB or YUV images. This article discusses how to display RGB images, so use ImageType.Color rather than of ImageType.ColorYuv or ImageType.ColorYuvRaw.
Retrieving the Images
With the image setup in place, you need to decide which Usage model you plan on using to retrieve images from the sensor. The valid options for the usage model are Event Model and Polling Model. The Event Model notifies you when a frame is ready by raising the VideoFrameReady event. You retrieve the available frame through that event’s ImageFrameReadyEventArgs parameter.
Using the Polling Model, you request an image from the sensor using the GetNextFrame method.
ImagFrame GetNextFrame(int millisecondsWait);
The millisecondsWait argument specifies how long the runtime should wait for a frame to be ready. If no frame is ready in the allotted time, the method returns null. The polling method is useful when you need to synchronize different subsystems. While there’s no guarantee that the frames from different subsystem can be synchronized using polling, polling gives you much better control than using events.
Adding Camera Support to the Project
Now that you have learned about the SDK it is time to apply it to the project you created earlier.
First, you should create the user interface (UI) for the WPF application. The application should allow the user to change the usage model, resolution, and elevation angle of the Kinect Sensor as well as display the images from the Kinect Sensor and connect and disconnect from the Kinect. The following image shows the UI that you will use to accomplish this.
Create the UI by changing the XAML in the Mainwindow.xaml file. When you first open the file, you’ll see this default XAML:
<Window x:Class="Video_Kinect.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Title="Video Kinect" Height="480" Width="840">
<Grid>
</Grid>
</Window>Replace the preceding XAML with this code:
<Window x:Class="Video_Kinect.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Title="Video Kinect" Height="480" Width="840">
<Grid>
<Grid.ColumnDefinitions>
<ColumnDefinition />
<ColumnDefinition Width="220"/>
</Grid.ColumnDefinitions>
<Image Name="ColorImage" Grid.Row="0" Opacity="1" />
<StackPanel Grid.Column="1" Margin="5">
<GroupBox Header="Device">
<StackPanel Orientation="Vertical">
<StackPanel>
<TextBlock FontWeight="Bold" Margin="3">Model:</TextBlock>
<ComboBox x:Name="UsageModelComboBox" Margin="3">
<ComboBoxItem IsSelected="True">Event</ComboBoxItem>
<ComboBoxItem>Polling</ComboBoxItem>
</ComboBox>
</StackPanel>
<StackPanel>
<TextBlock FontWeight="Bold" Margin="3">Resolution:</TextBlock>
<ComboBox x:Name="ResolutionComboBox" Margin="3">
<ComboBoxItem IsSelected="True">640x480</ComboBoxItem>
<ComboBoxItem>1280x1024</ComboBoxItem>
</ComboBox>
</StackPanel>
<Button x:Name="ConnectButton" Click="ConnectButton_Click"
Margin="3">Connect</Button>
<Button x:Name="DisconnectButton" IsEnabled="False"
Click="DisconnectButton_Click"
Margin="3">Disconnect</Button>
</StackPanel>
</GroupBox>
<GroupBox Header="Camera" Margin="5">
<StackPanel>
<TextBlock FontWeight="Bold" Margin="3">Name:</TextBlock>
<TextBlock x:Name="CameraName" Margin="3"></TextBlock>
<TextBlock FontWeight="Bold" Margin="3">Elevation Angle:</TextBlock>
<StackPanel Orientation="Horizontal">
<TextBox x:Name="CameraElevationAngle" Margin="3" Width="60"></TextBox>
<Button x:Name="CameraElevationAngleButton" IsEnabled="False"
Click="CameraElevationAngleButton_Click">Set</Button>
</StackPanel>
<TextBlock FontWeight="Bold" Margin="3">Frame Number:</TextBlock>
<TextBlock x:Name="FrameNumber" Margin="3"></TextBlock>
<TextBlock FontWeight="Bold" Margin="3">FPS:</TextBlock>
<TextBlock x:Name="FPS" Margin="3"></TextBlock>
<Button x:Name="PollButton" IsEnabled="False" Click="PollButton_Click"
Margin="3">Poll</Button>
</StackPanel>
</GroupBox>
</StackPanel>
</Grid>
</Window>Initializing the Kinects
Users connect to the Kinect by clicking the Connect button. The event handler of the ConnectButton calls the InitializeKinect method, as you can see in the following code.
private void ConnectButton_Click(object sender, RoutedEventArgs e)
{
InitializeKinect();
UpdateButtons();
}
void InitializeKinect()
{
try
{
m_nui = new Runtime();
m_nui.Initialize(RuntimeOptions.UseColor);
}
catch (InvalidOperationException)
{
MessageBox.Show("No Kinects detected!");
m_nui = null;
return;
}
try
{
var resolution = ImageResolution.Resolution1280x1024;
if (ResolutionComboBox.Text == "640x480")
{
resolution = ImageResolution.Resolution640x480;
}
m_nui.VideoStream.Open(ImageStreamType.Video, 2, resolution, ImageType.Color);
FPS.Text = "0";
if (UsageModelComboBox.Text == "Event")
{
m_nui.VideoFrameReady +=
(sender, e) =>
{
ShowImage(e.ImageFrame);
};
m_eventDriven = true;
FPS.IsEnabled = true;
m_samples = 0;
m_sw.Reset();
m_sw.Start();
}
else
{
FPS.IsEnabled = false;
m_eventDriven = false;
}
CameraName.Text = m_nui.NuiCamera.UniqueDeviceName;
CameraElevationAngle.Text = m_nui.NuiCamera.ElevationAngle.ToString();
CameraElevationAngle.Text = m_nui.NuiCamera.ElevationAngle.ToString();
}
catch (InvalidOperationException e)
{
MessageBox.Show( e.Message, "Error opening video stream!");
UninitializeKinect();
}
}The InitializeKinect method initializes the camera of the Kinect using the following code.
try
{
m_nui = new Runtime();
m_nui.Initialize(RuntimeOptions.UseColor);
}
catch (InvalidOperationException)
{
MessageBox.Show("No Kinects detected!");
m_nui = null;
return;
}The RuntimeOptions.UseColor enumeration parameter means that you want to initialize the RGB camera of the Kinect to get color
images.
The method then sets up the usage method (based on what the user specifies through the ResolutionComboBox combo box) with the following code.
var resolution = ImageResolution.Resolution1280x1024;
if (ResolutionComboBox.Text == "640x480")
{
resolution = ImageResolution.Resolution640x480;
}
m_nui.VideoStream.Open(ImageStreamType.Video, 2, resolution, ImageType.Color);When this code executes, the application creates and opens a video stream with the specified resolution and image type. Next, the code determines the requested usage model selected in the UsageModelComboBox combo box using the following code.
if (UsageModelComboBox.Text == "Event")
{
m_nui.VideoFrameReady +=
(sender, e) =>
{
ShowImage(e.ImageFrame);
};
m_eventDriven = true;
FPS.IsEnabled = true;
m_samples = 0;
m_sw.Reset();
m_sw.Start();
}
else
{
FPS.IsEnabled = false;
m_eventDriven = false;
}The code attaches to the Runtime.VideoFrameReady event when users select the Event model. Otherwise users must poll for an image by clicking the PollButton button. The following event handler shows the code that executes when a user clicks the Poll button.
private void PollButton_Click(object sender, RoutedEventArgs e)
{
ShowImage(m_nui.VideoStream.GetNextFrame(0));
}Unintializing the Kinect
Users disconnect from the Kinect device by clicking the Disconnect button, which fires the following code.
private void DisconnectButton_Click(object sender, RoutedEventArgs e)
{
UninitializeKinect();
UpdateButtons();
}
void UninitializeKinect()
{
if (m_nui != null)
{
m_nui.Uninitialize();
m_nui = null;
}
}The DisconnectButton event handler calls the UninitializeKinect method to uninitialize the kinect. The code that actually uninitializes the Kinect is the call to the Runtime.Uninitialize method.
Displaying the Image
The left panel of the UI displays the image from the Kinect sensor in an Image object named ColorImage, which shows the image that was set in the Source property. The following method updates the Source property of ColorImage.
void ShowImage(ImageFrame frame)
{
if (m_eventDriven)
{
CalculateFPS();
}
// 32-bit per pixel, RGBA image
PlanarImage planarImage = frame.Image;
FrameNumber.Text = frame.FrameNumber.ToString();
ColorImage.Source = BitmapSource.Create(planarImage.Width, planarImage.Height, 96, 96, PixelFormats.Bgr32, null, planarImage.Bits, planarImage.Width * planarImage.BytesPerPixel);
}For the event model, you obtain the frame parameter through the ImageFrame property of the VideoFrameReady ImageFrameReadyEventArgs. In contrast, the polling model requires you to call the Runtime.VideoStream.GetNextFrame method.
In addition to displaying the image to the user, the showImage method also keeps track of the framerate through the following CalculateFPS method.
void CalculateFPS()
{
++m_samples;
if (m_sw.Elapsed.TotalSeconds > 1.0)
{
m_sw.Stop();
FPS.Text = ((int)(((double)m_samples / m_sw.Elapsed.TotalSeconds))).ToString();
m_samples = 0;
m_sw.Reset();
m_sw.Start();
}
}The method increments the number of samples taken until the stopwatch reaches one second. Once the stopwatch hits one second, the method divides the number of samples taken by the elapsed time to calculate the FPS. The counter and stopwatch are then reset to start the process again.
Changing the Elevation Angle of the Camera
You can change the angle of the Kinect’s camera with the Framework.NuiCamera.ElevationAngle property. The specified value is in degrees and must be between the Camera.ElevationMininum and Camera.ElevationMaximum properties. The documentation also gives the following warning about changing the camera angle.
You should tilt the Kinect sensor as few times as possible, to minimize wear on the camera and to minimize tilting time. The camera motor is not designed for constant or repetitive movement, and attempts to use it that way may cause degradation of motor function. This beta SDK limits the rate at which applications can tilt the sensor, to protect the Kinect hardware. If the application tries to tilt the sensor too frequently, the runtime imposes a short lockout period during which any further calls return an error code.
Note: Please be sure the read the documentation before changing the camera angle.
The user sets the camera angle through the CameraElevationAngle textbox and changes the angle in the Kinect’s sensor by clicking the CameraElevationAngleButton button. Clicking the button fires the following event handler.
private void CameraElevationAngleButton_Click(object sender, RoutedEventArgs e)
{
int iValue;
if (!int.TryParse(CameraElevationAngle.Text, out iValue))
{
MessageBox.Show("The camera angle is in degrees!");
return;
}
if (iValue > Camera.ElevationMaximum)
{
MessageBox.Show(string.Format("The camera angle cannot be higher than {0}!",
Camera.ElevationMaximum));
return;
}
if (iValue < Camera.ElevationMinimum)
{
MessageBox.Show(string.Format("The camera angle cannot be lower than {0}!",
Camera.ElevationMinimum));
return;
}
if (m_nui.NuiCamera.ElevationAngle == iValue)
{
return;
}
if (m_warnUser)
{
var result = MessageBox.Show("You should tilt the Kinect sensor as few times as possible, to minimize wear on the camera and to minimize tilting time. The camera motor is not designed for constant or repetitive movement, and attempts to use it that way may cause degradation of motor function. This beta SDK limits the rate at which applications can tilt the sensor, to protect the Kinect hardware. If the application tries to tilt the sensor too frequently, the runtime imposes a short lockout period during which any further calls return an error code.\nDo you wish to continue?", "Warning", MessageBoxButton.YesNo, MessageBoxImage.Warning);
if (result == MessageBoxResult.No)
{
return;
}
m_warnUser = false;
}
try
{
m_nui.NuiCamera.ElevationAngle = iValue;
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}The event handler converts the text in the CameraElevationAngle textbox to an the integer stored in iValue. The variable iValue is validated and the passed to the Framework.NuiCamera.ElevationAngle after displaying a warning message to the user about changing the camera angle too often.
Summary
In this article you saw how to create a Kinect project and communicate with the Kinect camera. In the process, you’ve had a brief introduction to the Kinects API. The next article in this series shows how to obtain the depth stream from the Kinect sensor.
