Skeletal Tracking Fundamentals (Beta 2 SDK)

This content is no longer current. Our recommendation for up to date content: http://channel9.msdn.com/Series/KinectQuickstart/Camera-Fundamentals
Please use the newly updated Kinect for Windows SDK Quickstart series. The content below will only work with the Beta 2 version of the Kinect for Windows SDK.
This video covers the basics of reading camera data from the Kinect sensor. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides that have been updated for Beta 2 (Nov, 2011).
The video has not been updated for Beta 2, but the following changes have been made:
The steps below assume you have setup your development environment as explained in the Setting Up Your Development Environment video.
We’ll add two 320x240 Image controls to the MainWindow XAML file as shown in the following XAML:
XAML
<Window x:Class="WpfApplication1.MainWindow" xmlns="https://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="https://schemas.microsoft.com/winfx/2006/xaml" Title="MainWindow" Height="350" Width="800"> <Grid> <Image Height="240" HorizontalAlignment="Left" Margin="12,20,0,0" Name="image1" Stretch="Fill" VerticalAlignment="Top" Width="320" /> <Image Height="240" HorizontalAlignment="Left" Margin="355,20,0,0" Name="image2" Stretch="Fill" VerticalAlignment="Top" Width="320" /> </Grid> </Window>
In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseColor to use the RGB and depth camera:
C#
//Kinect Runtime Runtime nui; private void Window_Loaded(object sender, RoutedEventArgs e) { SetupKinect(); } private void SetupKinect() { if (Runtime.Kinects.Count == 0) { this.Title = "No Kinect connected"; } else { //use first Kinect nui = Runtime.Kinects[0]; //Initialize to return both Color & Depth images nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth); } }
Visual Basic
'Kinect Runtime Private nui As Runtime Private Sub Window_Loaded(ByVal sender As Object, ByVal e As RoutedEventArgs) SetupKinect() End Sub Private Sub SetupKinect() If Runtime.Kinects.Count = 0 Then Me.Title = "No Kinect connected" Else 'use first Kinect nui = Runtime.Kinects(0) 'Initialize to return both Color & Depth images nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepth) End If
To read values from the RGB camera, add an event handler for the VideoFrameReady event on the Kinect Runtime in the Window_Loaded event as shown below:
Tip: You can have Visual Studio automatically build the event handler for you by typing "nui.VideoFrameReady +=" and hitting the tab key twice.
C#
nui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady); void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e) { // event code goes here }
Visual Basic
AddHandler nui.VideoFrameReady, AddressOf nui_VideoFrameReady Private Sub nui_VideoFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs) ' event code goes here End Sub
Understanding the video frame ready returned values
The video frame returns an ImageFrameReadyEventArgs that contains an ImageFrame class. As shown below, the ImageFrame class contains two things:
To convert the byte[] array that represents the camera image to display it in an Image control (ex: image1 below), call the BitmapSource.Create method as shown below. The last parameter is stride. The stride is the number of bytes from one row of pixels in memory to the next row of pixels in memory. Stride is also called pitch. For more information, go to https://msdn.microsoft.com/en-us/library/aa473780(VS.85).aspx:
C#
PlanarImage imageData = e.ImageFrame.Image; image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, PixelFormats.Bgr32, null, imageData.Bits, data.Width * imageData.BytesPerPixel);
Visual Basic
Dim imageData As PlanarImage = e.ImageFrame.Image image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, _ PixelFormats.Bgr32, Nothing, imageData.Bits, imageData.Width * imageData.BytesPerPixel)
The Coding4Fun Kinect Toolkit has an extension method built into the ImageFrame class that simplifies creating the bitmap:
C#
image1.Source = e.ImageFrame.ToBitmapSource();
Visual Basic
image1.Source = e.ImageFrame.ToBitmapSource()
The Kinect cameras can support different resolutions depending on what ImageType is passed in. To determine these resolutions, call the GetValidResolutions on the ImageStream object:
C#
var x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex);
Visual Basic
Dim x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex)
Open the VideoStream
Before the event will fire, you must call the Open method on VideoStream as shown below. This opens a video stream, sets the PoolSize to 2, sets the resolution to 640x480, and sets the return image as a color image:
C#
nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
Visual Basic
nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color)
Understanding PoolSize
Pool Size is the number of buffers you want queued up. Usually, you want one displaying, and one being filled. The minimum is two but specifying more than two will give you slightly more latency in the frames you display, and so you’ll be more likely to have one waiting for you. The trade off, then, is smoother playback but higher latency.
To add depth data, we use the same steps as above.
First, let’s review and see that the Initialize method includes RuntimeOptions.UseDepth to receive depth data from Kinect:
C#
nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);
Visual Basic
nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepth)
Next, add the event handler to receive the Depth information. For simplicity, we’ll use the Coding4Fun Kinect toolkit to display the depth image in the image2 Image control:
C#
nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady); void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e) { image2.Source = e.ImageFrame.ToBitmapSource(); }
Visual Basic
AddHandler nui.DepthFrameReady, AddressOf nui_DepthFrameReady Private Sub nui_DepthFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs) image2.Source = e.ImageFrame.ToBitmapSource() End Sub
Finally, we’ll call DepthStream.Open to open a 320x240 depth stream:
C#
nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);
Visual Basic
nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth)
The result is shown below:
Warning: The tilt mechanism in the sensor array is not rated for frequent use. Your code should not make calls to tilt the device more than 15 times in any two-minute window. Changing the tilt too often results in an error message from the function.
Like before, create an instance of the Kinect Runtime class and call the Initialize method:
C#
Runtime nui = new Runtime(); nui.Initialize(RuntimeOptions.UseColor);
Visual Basic
nui.Initialize(RuntimeOptions.UseColor)
To adjust the tilt or pitch, set the ElevationAngle property to a value between –27 (Camera.ElevationMinimum) and +27 (Camera.ElevationMaximum). The code below sets the camera to the maximum elevation:
C#
nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum;
Visual Basic
nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum
Thanks a lot for the tutorial.
At first, my app was running but i wasn't getting any images. Then i realized it was because i connected Kinect thru USB hub. I read it is mentioned in a read me file but i couldn't find where it was so i decided to mention it here just in case someone else may do the same mistake. When i connected it to the chase directly it worked perfectly.
I have an issue. Im using WinForms (VB.net) and a ElementHost with an image control (i called the wpf control img and the actual control img). When I insert the code
image2.Source = e.ImageFrame.ToBitmapSource()it gives an error: 'ToBitmapSource' is not a member of 'Microsoft.Research.Kinect.Nui.ImageFrame'.
I tried importing the coding4fun wpf dll, but that didn't do anything. I do have SP1 if your wondering.
@ajpri:I did get it working, I referenced it instead of also importing.
In the VB example, converting Bytes to Image there is a mistake.
data.bits should equal imagedata.bits
It is correct in the download.
@bmsjr: fixed the typo
Hello, first i would like to thank you because of all tutorials which you provide us.... they are greatly useful.
I would like to know how to show the number of the Slider position in the Task: Adjust the Kinect Camera Tilt, i have to usea a text box and set it with the slider or what?
Im just a beginner in this languaje C# and i trying to learn how to use the knect whith this tool... So please, help me!!
Thanks!!!
@Ricardo: you can do it one of two ways. You can either do data binding in the XAML ala
<Label Content="{Binding ElementName=myAwesomeSliderControl, Path=Value}" />
or use the ValueChanged event on the slider to update the element.
Hi,
First of all let me thanks you for wonderful tutorial. It helps a lot.
Im trying to display the rgb camera image, but im running into XamlParseException.
It says "The invocation of the constructor on type "WpfApplication1.MainWindow" that matches the specified binding constraints threw an exception.' Line number 3 and the line position 9"
Did anyone run into this problem? Changing config file didnt help.
My xaml file looks like this
<Window x:Class="WpfApplication1.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Title="MainWindow" Height="350" Width="800" Loaded="Window_Loaded" Closed="Window_Closed">
<Grid>
<Image Height="240" HorizontalAlignment="Left" Name="image1" Stretch="Fill" VerticalAlignment="Top" Width="320" Grid.ColumnSpan="2" Margin="12,35,0,0" />
<Image Height="240" HorizontalAlignment="Left" Margin="172,35,0,0" Name="image2" Stretch="Fill" VerticalAlignment="Top" Width="320" Grid.Column="1" />
</Grid>
</Window>
nevermind, restarting the machine solved the problem
So I implemented my version of this program and I noticed that the corner of the depth image always has this area where depth is the same no matter what. I looked into your video closely and I noticed the same thing(in the upper right corner see that rectangle). If you look at the distance it produces it would be 65535 which is 2^16-1. Makes sense since it is 16 bit per pixel, only the documentation says they are using only 11 bits and 4 bits are unused which means we should not have values more than 4095. Do you know what causes that effect?
I was wondering has anyone tried using 2 kinect cameras with the sdk yet?
how did u do the camera tilt thing ? can u make a guide movie for that too ? i dont have any experince but im trying to learn i gotten everything sofar but it looked like u had another project for tilting and it wasent build in the video and depth project... would be greatful too get help since my camera is alitle high now :)
when i add
image2.Source = e.ImageFrame.ToBitmapSource();
it says an error
ToBitmapSource' is not a member of 'Microsoft.Research.Kinect.Nui.ImageFrame'.
I referenced the coding 4fun dll's but how do you import them
Anyone know of a good way to write the depth stream to a file? I have tried using StreamWriter to write the stream to a file however that seems not to work.
hello!thanks for sharing the guides for kinect
here is my problem
if i want to take a picture using kinect on c#
it is possible?
In kinect SDK documentation, it mentioned that kinect manage to provide regular quality RGB image which is compressed RGB image. So, by using the same SDK, how can i access the high quality uncompressed RGB image?
I followed the instructions, but when I press F5 (start debugging) the main windows appear without any image or depth image , just embty main window? why is that, any help?
Even I am having the same problem.. Any help will be greatly appreciable.. Thanks..
how 2 run the image???in Task: Display the RGB Camera Image
how 2 run the image??? I couldn't see the action which turn the word green.
(nui.DepthStream.Open(ImageStreamType.Depth, 2,
ImageResolution.Resolution320x240,ImageType.Depth);
)
thanks for the tutorial videos! i need to do a project based on Kinect and i have no basic fundamentals at all! this really helps :D
Thanks for the tutorials,
I'm having same issue where screen is blank or empty.
Any help?
Hi, Thanks for the tutorials!
I just have one question. Is there a direct way to get the image brightness of the camera from kinect? I know that the camera automatically adjusts brightness, however what I need is to know the ammount of light present in the environment using the kinect (as part of other stuff in my app...) I really hope kinect supports something like this.
Thanks a lot for any help!
Hi thank you for the tuto i have no diffecult in C# and visuel basic, now i learn the action script can someone help me how to creat a simple game flash like in this site of games for girls .
If you are having trouble getting video to display (blank window) after following the tutorial and your C# code looks the same:
I added ' Loaded="Window_Loaded" ' to the .xaml file and everything worked.
I don't know if I missed a step; I really don't understand Visual Studio or whether it should've added that for me or not, but it works now.
I still get build errors when I add ' Closed="Window_Closed" ' so I'm not sure what that's aboout.
From the post above,
if you having trouble with the blank screen edit your XAML code as follows:
Title="MainWindow" Height="350" Width="800" Loaded="Window_Loaded" Closed="Window_Closed>
Thank you for the info Radik :)
So this is a simple question. I am new to coding so bare with me. How do I display the videostream? As I was following the video he clicks a button and the kinect video pops up. How is this simple task done?
I also would like to know how to do this tilt thing. He already has it built and does not show us.
Thanks
I am trying to write a simple Metro App with Kinect SDK to show the RGB camera in a UI image (just like WPF), but I get a runtime error just at the very beginning. More details of the problem are available at http://social.msdn.microsoft.com/Forums/en-US/winappswithcsharp/thread/20ca8a68-e97f-4178-8443-9cbafd3205f8
Do you know if this is a known bug or am I doing something wrong?
Thanks
I wrote the code like tutorial, but when I run the program nothing being displayed. When I debugged the code, It read " nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);"
then skip rest of code;
------------------------------------------------------------
Code
------------------------------------------------------------
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Microsoft.Research.Kinect.Audio;
using Microsoft.Research.Kinect.Nui;
using Coding4Fun.Kinect.Wpf;
namespace PlayKinect1
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
}
Runtime nui = new Runtime();
private void Window_Loaded(object sender, RoutedEventArgs e)
{
if (Runtime.Kinects.Count != 0)
{
nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);
nui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
// nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);
nui.VideoStream.Open(ImageStreamType.Video, 2,
ImageResolution.Resolution320x240,
ImageType.Color);
}
}
void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
image1.Source = e.ImageFrame.ToBitmapSource();
}
void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
}
private void Window_Closed(object sender, EventArgs e)
{
nui.Uninitialize();
}
}
}
I'm a complete novice (in programming and software development) and I'd greatly appreciate some help.
I've got the video image and the grey scale depth image showing on my monitor, per the tutorial here, but, the images shown are mirror images (i.e. the images are flipped about a vertical axis). I can get them to show what the Kinect is actually seeing by changing the image properties "FlowDirection" from "LeftToRight" to "RightToLeft". Is there another way to do this in the C# window rather than in the Designer window?
The image from doing the "Working With Depth Data" tutorial doesn't flip the image like I've explained above...
Hi
I followed the tutorial step by step but when it runs, after a few seconds the image freezes (even using only the RBG camera)
Checking my task manager, I observed that my memory went to the roof, once I start the program.
The computer is relatively new late 2011, (Dell Optiplex 990 with 4Gbytes in RAM and i5 processor)
Any suggestions on how to dispose the images? (I set the buufer to 2 as shown in the Video, but it doesn't work.
Thank you for your help in advance
@Meminsky - I am having exactly the same problem. Please post a solution if you have found one...
Hey there,
I keep getting 2 errors, tested individually to avoid confusion:
The First is as follows below: (after I installed the Microsoft Kinect SDK as of recent).
XamlParseException.
It says "The invocation of the constructor on type "WpfApplication1.MainWindow" that matches the specified binding constraints threw an exception.' Line number 3 and the line position 9"
The Second is when I add image1.Source = e.ImageFrame.ToBitmapSource();
It is see as an ambiguous call I think conflicting with Microsoft.Kinect instead of Microsoft.Research.Kinect .
I can't seem to resolve these two issues. Help is greatly appreciated and thanks in advance.