Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Content Obsolete

This content is no longer current. We recommend you try the following url for more up to date content: http://channel9.msdn.com/Series/KinectQuickstart/Camera-Fundamentals

Camera Fundamentals (Beta 2 SDK)

Download

Right click “Save as…”

Update: Kinect for Window SDK v1 Quickstart Series now Available (Feb 1st)

Please use the newly updated Kinect for Windows SDK Quickstart series. The content below will only work with the Beta 2 version of the Kinect for Windows SDK.

 

This video covers the basics of reading camera data from the Kinect sensor.  You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides that have been updated for Beta 2 (Nov, 2011).

  • [00:25] Camera data information
  • [03:30] Creating the UI
  • [04:48] Initializing the Kinect runtime
  • [07:18] Reading values from the RGB camera
  • [11:26] Reading values from the Depth camera
  • [13:06] Adjusting camera tilt 

 

Updates for Kinect for Windows SDK Beta 2 (Nov, 2011)

The video has not been updated for Beta 2, but the following changes have been made:

  • Creating the Kinect runtime uses the SetupKinect method used in the Setting Up Your Development Environment example.
  • Beta 2 includes a sample (Microsoft.Samples.Kinect.WpfViewers) set of user controls for visualizing data from the Color/RGB and Depth cameras that you can reuse in your application. The samples can be found at: "\Program Files\Microsoft SDKs\Kinect\v1.0 Beta2\Samples\KinectSDKSamples.zip" in the “Managed\KinectWpfViewers” solution.

Setup

The steps below assume you have setup your development environment as explained in the Setting Up Your Development Environment video.

To get a depth or color image you must do the following

  1. Create an instance of a runtime object
  2. Initialize the instance with the correct options
  3. Subscribe to an event to receive camera data
  4. Open the video stream
  5. Convert the raw camera data into an image

Task: Display the RGB Camera Image

Designing your UI

We’ll add two 320x240 Image controls to the MainWindow XAML file as shown in the following XAML:

XAML

<Window x:Class="WpfApplication1.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" Height="350" Width="800">
    <Grid>
        <Image Height="240" HorizontalAlignment="Left" Margin="12,20,0,0" Name="image1" Stretch="Fill" VerticalAlignment="Top" Width="320" />
        <Image Height="240" HorizontalAlignment="Left" Margin="355,20,0,0" Name="image2" Stretch="Fill" VerticalAlignment="Top" Width="320" />
    </Grid>
</Window>

image

Initializing the runtime

In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseColor to use the RGB and depth camera:

C#

//Kinect Runtime
Runtime nui;
private void Window_Loaded(object sender, RoutedEventArgs e)
{
    SetupKinect();
}
private void SetupKinect()
{
    if (Runtime.Kinects.Count == 0)
    {
        this.Title = "No Kinect connected";
    }
    else
    {
        //use first Kinect
        nui = Runtime.Kinects[0];
 
        //Initialize to return both Color & Depth images
        nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);
    }
}

Visual Basic

'Kinect Runtime
Private nui As Runtime
Private Sub Window_Loaded(ByVal sender As Object, ByVal e As RoutedEventArgs)
    SetupKinect()
End Sub
Private Sub SetupKinect()
    If Runtime.Kinects.Count = 0 Then
        Me.Title = "No Kinect connected"
    Else
        'use first Kinect
        nui = Runtime.Kinects(0)
        'Initialize to return both Color & Depth images
        nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepth)
    End If

Subscribe to the VideoFrameReady event

To read values from the RGB camera, add an event handler for the VideoFrameReady event on the Kinect Runtime in the Window_Loaded event as shown below:

Tip: You can have Visual Studio automatically build the event handler for you by typing "nui.VideoFrameReady +=" and hitting the tab key twice.

C#

nui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);

void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
   // event code goes here
}

Visual Basic

AddHandler nui.VideoFrameReady, AddressOf nui_VideoFrameReady

Private Sub nui_VideoFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs)
    ' event code goes here
End Sub

Understanding the video frame ready returned values

The video frame returns an ImageFrameReadyEventArgs that contains an ImageFrame class. As shown below, the ImageFrame class contains two things:

  • Metadata about the image, such as ImageType to know if it’s a depth or color image, and resolution to know the size of the image
  • Image – the actual image data, which is stored in the Bits byte[] array

image

Converting the byte[] array to an image

To convert the byte[] array that represents the camera image to display it in an Image control (ex: image1 below), call the BitmapSource.Create method as shown below. The last parameter is stride. The stride is the number of bytes from one row of pixels in memory to the next row of pixels in memory. Stride is also called pitch.  For more information, go to http://msdn.microsoft.com/en-us/library/aa473780(VS.85).aspx:

C#

PlanarImage imageData = e.ImageFrame.Image;
image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, 
                PixelFormats.Bgr32, null, imageData.Bits, data.Width * imageData.BytesPerPixel);

Visual Basic

Dim imageData As PlanarImage = e.ImageFrame.Image
image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, _
    PixelFormats.Bgr32, Nothing, imageData.Bits, imageData.Width * imageData.BytesPerPixel)

The Coding4Fun Kinect Toolkit has an extension method built into the ImageFrame class that simplifies creating the bitmap:

C#

image1.Source = e.ImageFrame.ToBitmapSource();

Visual Basic

image1.Source = e.ImageFrame.ToBitmapSource()

Getting Supported Resolutions

The Kinect cameras can support different resolutions depending on what ImageType is passed in. To determine these resolutions, call the GetValidResolutions on the ImageStream object:

C#

var x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex);

Visual Basic

Dim x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex)

Open the VideoStream

Before the event will fire, you must call the Open method on VideoStream as shown below. This opens a video stream, sets the PoolSize to 2, sets the resolution to 640x480, and sets the return image as a color image:

C#

nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);

Visual Basic

nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color)

Understanding PoolSize

Pool Size is the number of buffers you want queued up. Usually, you want one displaying, and one being filled. The minimum is two but specifying more than two will give you slightly more latency in the frames you display, and so you’ll be more likely to have one waiting for you. The trade off, then, is smoother playback but higher latency.

Task: Displaying Depth Data

To add depth data, we use the same steps as above.

Update the Runtime Initialization

First, let’s review and see that the Initialize method includes RuntimeOptions.UseDepth to receive depth data from Kinect:

C#

nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);

Visual Basic

nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepth)

Subscribe to the DepthFrameReady event

Next, add the event handler to receive the Depth information. For simplicity, we’ll use the Coding4Fun Kinect toolkit to display the depth image in the image2 Image control:

C#

nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);

void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
    image2.Source = e.ImageFrame.ToBitmapSource(); 
}

Visual Basic

AddHandler nui.DepthFrameReady, AddressOf nui_DepthFrameReady

Private Sub nui_DepthFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs)
    image2.Source = e.ImageFrame.ToBitmapSource()
End Sub

Open the Depth Stream

Finally, we’ll call DepthStream.Open to open a 320x240 depth stream:

C#

nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);

Visual Basic

nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth)

The result is shown below:

Task: Adjust the Kinect Camera Tilt

Warning: The tilt mechanism in the sensor array is not rated for frequent use. Your code should not make calls to tilt the device more than 15 times in any two-minute window. Changing the tilt too often results in an error message from the function.

Setup

Like before, create an instance of the Kinect Runtime class and call the Initialize method:

C#

Runtime nui = new Runtime();
nui.Initialize(RuntimeOptions.UseColor);

Visual Basic

nui.Initialize(RuntimeOptions.UseColor)

Adjusting the tilt

To adjust the tilt or pitch, set the ElevationAngle property to a value between –27 (Camera.ElevationMinimum) and +27 (Camera.ElevationMaximum). The code below sets the camera to the maximum elevation:

C#

nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum;

Visual Basic

nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum

Follow the Discussion

  • Thanks a lot for the tutorial.

    At first, my app was running but i wasn't getting any images. Then i realized it was because i connected Kinect thru USB hub. I read it is mentioned in a read me file but i couldn't find where it was so i decided to mention it here just in case someone else may do the same mistake. When i connected it to the chase directly it worked perfectly.

  • I have an issue. Im using WinForms (VB.net) and a ElementHost with an image control (i called the wpf control img and the actual control img). When I insert the code

    image2.Source = e.ImageFrame.ToBitmapSource()
    it gives an error: 'ToBitmapSource' is not a member of 'Microsoft.Research.Kinect.Nui.ImageFrame'. 

    I tried importing the coding4fun wpf dll, but that didn't do anything. I do have SP1 if your wondering.

  • @ajpri:I did get it working, I referenced it instead of also importing.

  • bmsjrbmsjr

    In the VB example, converting Bytes to Image there is a mistake.

    data.bits should equal imagedata.bits

    It is correct in the download.

  • Clint RutkasClint I'm a "developer"

    @bmsjr: fixed the typo

  • RicardoRicardo

    Hello, first i would like to thank you because of all tutorials which you provide us.... they are greatly useful.

    I would like to know how to show the number of the Slider position in the Task: Adjust the Kinect Camera Tilt, i have to usea a text box and set it with the slider or what?

    Im just a beginner in this languaje C# and i trying to learn how to use the knect whith this tool... So please, help me!!

    Thanks!!!

  • Clint RutkasClint I'm a "developer"

    @Ricardo: you can do it one of two ways.  You can either do data binding in the XAML ala 

    <Label Content="{Binding ElementName=myAwesomeSliderControl, Path=Value}" />

    or use the ValueChanged event on the slider to update the element.

  • RufatRufat

    Hi,
    First of all let me thanks you for wonderful tutorial. It helps a lot.
    Im trying to display the rgb camera image, but im running into XamlParseException.
    It says "The invocation of the constructor on type "WpfApplication1.MainWindow" that matches the specified binding constraints threw an exception.' Line number 3 and the line position 9"
    Did anyone run into this problem? Changing config file didnt help.
    My xaml file looks like this
    <Window x:Class="WpfApplication1.MainWindow"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="MainWindow" Height="350" Width="800" Loaded="Window_Loaded" Closed="Window_Closed">
    <Grid>

    <Image Height="240" HorizontalAlignment="Left" Name="image1" Stretch="Fill" VerticalAlignment="Top" Width="320" Grid.ColumnSpan="2" Margin="12,35,0,0" />
    <Image Height="240" HorizontalAlignment="Left" Margin="172,35,0,0" Name="image2" Stretch="Fill" VerticalAlignment="Top" Width="320" Grid.Column="1" />
    </Grid>
    </Window>

  • RufatRufat

    nevermind, restarting the machine solved the problem

  • RufatRufat

    So I implemented my version of this program and I noticed that the corner of the depth image always has this area where depth is the same no matter what. I looked into your video closely and I noticed the same thing(in the upper right corner see that rectangle). If you look at the distance it produces it would be 65535 which is 2^16-1. Makes sense since it is 16 bit per pixel, only the documentation says they are using only 11 bits and 4 bits are unused which means we should not have values more than 4095. Do you know what causes that effect?

  • JermainJermain

    I was wondering has anyone tried using 2 kinect cameras with the sdk yet?

  • DanielDaniel

    how did u do the camera tilt thing ? can u make a guide movie for that too ? i dont have any experince but im trying to learn i gotten everything sofar but it looked like u had another project for tilting and it wasent build in the video and depth project... would be greatful too get help since my camera is alitle high now :)

  • RaajitRaajit

    when i add
    image2.Source = e.ImageFrame.ToBitmapSource();

    it says an error
    ToBitmapSource' is not a member of 'Microsoft.Research.Kinect.Nui.ImageFrame'.

    I referenced the coding 4fun dll's but how do you import them

  • JermainJermain

    Anyone know of a good way to write the depth stream to a file? I have tried using StreamWriter to write the stream to a file however that seems not to work.

  • jamesjames

    hello!thanks for sharing the guides for kinect
    here is my problem
    if i want to take a picture using kinect on c#
    it is possible?

  • amiramir

    In kinect SDK documentation, it mentioned that kinect manage to provide regular quality RGB image which is compressed RGB image. So, by using the same SDK, how can i access the high quality uncompressed RGB image?

  • RamiRami

    I followed the instructions, but when I press F5 (start debugging) the main windows appear without any image or depth image , just embty main window? why is that, any help?

  • sanchayitasanchayita

    Even I am having the same problem.. Any help will be greatly appreciable.. Thanks..

  • NyaoNyao

    how 2 run the image???in Task: Display the RGB Camera Image

  • NyaoNyao

    how 2 run the image??? I couldn't see the action which turn the word green.
    (nui.DepthStream.Open(ImageStreamType.Depth, 2,
    ImageResolution.Resolution320x240,ImageType.Depth);
    )

  • xl12344xl12344

    thanks for the tutorial videos! i need to do a project based on Kinect and i have no basic fundamentals at all! this really helps :D

  • KieferKiefer

    Thanks for the tutorials,

    I'm having same issue where screen is blank or empty.

    Any help?

  • SammySammy

    Hi, Thanks for the tutorials!
    I just have one question. Is there a direct way to get the image brightness of the camera from kinect? I know that the camera automatically adjusts brightness, however what I need is to know the ammount of light present in the environment using the kinect (as part of other stuff in my app...) I really hope kinect supports something like this.
    Thanks a lot for any help!

  • Hi thank you for the tuto Big Smile i have no diffecult in C# and visuel basic, now i learn the action script can someone help me how to creat a simple game flash like in this site of  games for girls . Smiley

  • DylanDylan

    If you are having trouble getting video to display (blank window) after following the tutorial and your C# code looks the same:

    I added ' Loaded="Window_Loaded" ' to the .xaml file and everything worked.

    I don't know if I missed a step; I really don't understand Visual Studio or whether it should've added that for me or not, but it works now.
    I still get build errors when I add ' Closed="Window_Closed" ' so I'm not sure what that's aboout.

  • RadikRadik

    From the post above,
    if you having trouble with the blank screen edit your XAML code as follows:
    Title="MainWindow" Height="350" Width="800" Loaded="Window_Loaded" Closed="Window_Closed>

  • TariTari

    Thank you for the info Radik :)

  • MazzinMazzin

    So this is a simple question. I am new to coding so bare with me. How do I display the videostream? As I was following the video he clicks a button and the kinect video pops up. How is this simple task done?

  • MazzinMazzin

    I also would like to know how to do this tilt thing. He already has it built and does not show us.

    Thanks

  • AdhyasAdhyas

    I am trying to write a simple Metro App with Kinect SDK to show the RGB camera in a UI image (just like WPF), but I get a runtime error just at the very beginning. More details of the problem are available at http://social.msdn.microsoft.com/Forums/en-US/winappswithcsharp/thread/20ca8a68-e97f-4178-8443-9cbafd3205f8

    Do you know if this is a known bug or am I doing something wrong?

    Thanks

  • NamerNamer

    I wrote the code like tutorial, but when I run the program nothing being displayed. When I debugged the code, It read " nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);"
    then skip rest of code;
    ------------------------------------------------------------
    Code
    ------------------------------------------------------------

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Windows;
    using System.Windows.Controls;
    using System.Windows.Data;
    using System.Windows.Documents;
    using System.Windows.Input;
    using System.Windows.Media;
    using System.Windows.Media.Imaging;
    using System.Windows.Navigation;
    using System.Windows.Shapes;
    using Microsoft.Research.Kinect.Audio;
    using Microsoft.Research.Kinect.Nui;
    using Coding4Fun.Kinect.Wpf;

    namespace PlayKinect1
    {
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
    public MainWindow()
    {
    InitializeComponent();
    }
    Runtime nui = new Runtime();
    private void Window_Loaded(object sender, RoutedEventArgs e)
    {
    if (Runtime.Kinects.Count != 0)
    {


    nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);
    nui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
    // nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);
    nui.VideoStream.Open(ImageStreamType.Video, 2,
    ImageResolution.Resolution320x240,
    ImageType.Color);

    }

    }

    void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
    {
    image1.Source = e.ImageFrame.ToBitmapSource();
    }
    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
    {

    }
    private void Window_Closed(object sender, EventArgs e)
    {
    nui.Uninitialize();
    }
    }
    }

  • thelumpofcoalthelumpofco​al

    I'm a complete novice (in programming and software development) and I'd greatly appreciate some help.

    I've got the video image and the grey scale depth image showing on my monitor, per the tutorial here, but, the images shown are mirror images (i.e. the images are flipped about a vertical axis). I can get them to show what the Kinect is actually seeing by changing the image properties "FlowDirection" from "LeftToRight" to "RightToLeft". Is there another way to do this in the C# window rather than in the Designer window?

    The image from doing the "Working With Depth Data" tutorial doesn't flip the image like I've explained above...

  • Hi

    I followed the tutorial step by step but when it runs, after a few seconds the image freezes (even using only the RBG camera)

    Checking my task manager, I observed that my memory went to the roof, once I start the program. 

    The computer is relatively new late 2011, (Dell Optiplex 990 with 4Gbytes in RAM and i5 processor) 

    Any suggestions on how to dispose the images? (I set the buufer to 2 as shown in the Video, but it doesn't work.

    Thank you for your help in advance

  • Bobble14988Bobble14988

    @Meminsky - I am having exactly the same problem. Please post a solution if you have found one...

  • Hey there,

    I keep getting 2 errors, tested individually to avoid confusion: 

    The First is as follows below:     (after I installed the Microsoft Kinect SDK as of recent).

    XamlParseException. 
    It says "The invocation of the constructor on type "WpfApplication1.MainWindow" that matches the specified binding constraints threw an exception.' Line number 3 and the line position 9" 

    The Second is when I add image1.Source = e.ImageFrame.ToBitmapSource();

    It is see as an ambiguous call I think conflicting with Microsoft.Kinect instead of Microsoft.Research.Kinect . 

    I can't seem to resolve these two issues. Help is greatly appreciated and thanks in advance.

     

Remove this comment

Remove this thread

close

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.