Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Content Obsolete

This content is no longer current. We recommend you try the following url for more up to date content: http://channel9.msdn.com/Series/KinectQuickstart/Working-with-Depth-Data

Working with Depth Data (Beta 2 SDK)

Download

Right click “Save as…”

Update: Kinect for Window SDK v1 Quickstart Series now Available (Feb 1st)

Please use the newly updated Kinect for Windows SDK Quickstart series. The content below will only work with the Beta 2 version of the Kinect for Windows SDK.

 

 

This video covers the basics of using depth data from Kinect. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides that have been updated for Beta 2 (Nov, 2011).

  • [00:43] Depth data overview
  • [04:56] Initializing the Kinect Runtime
  • [05:46] Using the depth data to create an image
  • [12:13] Using the PlayerIndex

Setup

The steps below assume you have setup your development environment as explained in the "Setting Up Your Development Environment" video.

 

Updates for Kinect for Windows SDK Beta 2 (Nov, 2011)

    • Creating the Kinect runtime uses the SetupKinect method used in the Setting Up Your Development Environment example.
    • Beta 2 includes a sample (Microsoft.Samples.Kinect.WpfViewers) set of user controls for visualizing data from the Color/RGB and Depth cameras that you can reuse in your application. The samples can be found at: "\Program Files\Microsoft SDKs\Kinect\v1.0 Beta2\Samples\KinectSDKSamples.zip" in the “Managed\KinectWpfViewers” solution.

 

Task Setup the Depth Camera event

Designing the UI

Add an image with Width=320 and Height=240 as shown below:

XAML

    <Image Height="240" HorizontalAlignment="Left" Margin="62,41,0,0" Name="image1" Stretch="Fill" VerticalAlignment="Top" Width="320" />

Setup the Depth Camera Event

Create an instance of the Kinect Runtime outside of the Window_Loaded event. Then, initialize the Runtime to use DepthAndPlayerIndex and UseSkeletalTracking. Finally register for DepthFrameReady event and open the Depth stream as shown below.

C#

//Kinect Runtime
Runtime nui = new Runtime(); 

private void Window_Loaded(object sender, RoutedEventArgs e)
{
    //UseDepthAndPlayerIndex and UseSkeletalTracking
    nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking);

    //register for event
    nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);

    //DepthAndPlayerIndex ImageType
    nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, 
        ImageType.DepthAndPlayerIndex); 
}

void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
}

Visual Basic

Private nui As New Runtime

Private Sub Window_Loaded(ByVal sender As Object, ByVal e As RoutedEventArgs)

    'UseDepthAndPlayerIndex and UseSkeletalTracking
    nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex Or RuntimeOptions.UseSkeletalTracking)

    'register for event
    AddHandler nui.DepthFrameReady, AddressOf nui_DepthFrameReady

    'DepthAndPlayerIndex ImageType
    nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex)

End Sub

Private Sub nui_DepthFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs)

End Sub

Understanding the PlanarImage byte[] Array

The DepthFrameReady event returns an ImageFrame class that contains a PlanarImage.  That PlanarImage contains a byte[] array which contains the distance of each pixel. The array:

  • Starts at top left of image
  • Moves left to right, then top to bottom
  • Represents distance for pixel

Calculating Distance

Because there are 2 bytes per pixel (16 bits) representing the distance, you will need to use bitshift operators to get the distance for a particular pixel. The exact bitshifting depends on the ImageType you use

Depth Image Type
Bitshift second byte left by 8
Distance (0,0) = (int)(Bits[0] | Bits[1] << 8 );

DepthAndPlayerIndex Image Type
Bitshift first byte right by 3 (to remove player index), and second byte left by 5
Distance (0,0) =(int)(Bits[0] >> 3 | Bits[1] << 5);

The method below shows returning the distance in millimeters using the 2 bytes for a particular pixel.

C#

private int GetDistanceWithPlayerIndex(byte firstFrame, byte secondFrame)
{
  //offset by 3 in first byte to get value after player index 
  int distance = (int)(firstFrame >> 3 | secondFrame << 5);
  return distance;
}

Visual Basic

Private Function GetDistanceWithPlayerIndex(ByVal firstFrame As Byte, ByVal secondFrame As Byte) As Integer

    'offset by 3 in first byte to get value after player index 
    Dim distance As Integer = (CInt(firstFrame) >> 3 Or CInt(secondFrame) << 5)
    Return distance

End Function

The DepthAndPlayerIndex image type is offset by 3 since the first three bits represent the Player Index. The Player Index has up to six possible values.

  • 0 – No player
  • 1 – Skeleton 0
  • 2 – Skeleton 1
  • ...

Note: While depth data is available for multiple players, you can only have skeletal/joint data for a maximum of two players.

To get a player index, we'll use the following formula. Since the player index is only in the first byte, we do not need the pixels second byte.

C#

private static int GetPlayerIndex(byte firstFrame)
{
    //returns 0 = no player, 1 = 1st player, 2 = 2nd player...
    return (int)firstFrame & 7; 
}

Visual Basic

Private Shared Function GetPlayerIndex(ByVal firstFrame As Byte) As Integer

    'returns 0 = no player, 1 = 1st player, 2 = 2nd player...
    'bitwise & on firstFrame
    Return CInt(firstFrame) And 7

End Function

Using distance to color values

Now that we know the distance for a particular pixel using the formula above, we can loop through all of the bytes in the byte[] array and dynamically assign colors to pixels based on their distance.

To begin with, we'll build a colorFrame byte[] array using the Bgr (Blue, Green, Red) pixel format. What this means is that each pixel (ex: the pixel at 0,0) is represented by four bytes, one each for Blue, Green, Red, and an empty one (or transparency byte if you choose the Bgra format).

For example, the snippet below would color the pixel blue if the current distance for that pixel was < 900 millimeters from the Kinect. To set the color to blue, we set the first byte (BlueIndex) to the maximum value, 255.

C#

if (distance <= 900)
{
    //we are very close
    colorFrame[index + BlueIndex] = 255;
    colorFrame[index + GreenIndex] = 0;
    colorFrame[index + RedIndex] = 0;
}

Visual Basic

If distance <= 900 Then

    'we are very close
    colorFrame(index + BlueIndex) = 255
    colorFrame(index + GreenIndex) = 0
    colorFrame(index + RedIndex) = 0

End If

Similarly, we can use the code snippet below to check if the current pixel represents a Player Index. If it does, we'll set the maximum value for both green and red which produces a yellow/gold-like color.

C#

if (GetPlayerIndex(depthData[depthIndex]) > 0)
{
    //we are the farthest
    colorFrame[index + BlueIndex] = 0;
    colorFrame[index + GreenIndex] = 255;
    colorFrame[index + RedIndex] = 255;
}

Visual Basic

If GetPlayerIndex(depthData(depthIndex)) > 0 Then

    'we are the farthest
    colorFrame(index + BlueIndex) = 0
    colorFrame(index + GreenIndex) = 255
    colorFrame(index + RedIndex) = 255

End If

Setting a monochrome intensity

While the previous examples we hard-coded colors, you can also build a monochrome histogram that scales the color range (0-255) proportionally based on the distance range (850-4000).

To get a monochrome appearance, the intensity is applied equally to all three colors (blue, green, red).

C#

//equal coloring for monochromatic histogram
var intensity = CalculateIntensityFromDepth(distance);

colorFrame[index + BlueIndex] = intensity;
colorFrame[index + GreenIndex] = intensity;
colorFrame[index + RedIndex] = intensity;

const float MaxDepthDistance = 4000; // max value returned
const float MinDepthDistance = 850; // min value returned
const float MaxDepthDistanceOffset = MaxDepthDistance - MinDepthDistance;

public static byte CalculateIntensityFromDepth(int distance)
{
   //formula for calculating monochrome intensity for histogram
   return (byte)(255 - (255 * Math.Max(distance - MinDepthDistance, 0)  / (MaxDepthDistanceOffset)));
}

Visual Basic

'equal coloring for monochromatic histogram
Dim intensity = CalculateIntensityFromDepth(distance)

colorFrame(index + BlueIndex) = intensity
colorFrame(index + GreenIndex) = intensity
colorFrame(index + RedIndex) = intensity

Private Const MaxDepthDistance As Single = 4000 ' max value returned
Private Const MinDepthDistance As Single = 850 ' min value returned
Private Const MaxDepthDistanceOffset As Single = MaxDepthDistance - MinDepthDistance

Public Shared Function CalculateIntensityFromDepth(ByVal distance As Integer) As Byte

    'formula for calculating monochrome intensity for histogram
    Return CByte(255 - (255 * Math.Max(distance - MinDepthDistance, 0) / (MaxDepthDistanceOffset)))

End Function

Putting it all together

Using the formulas above, we'll now return a byte[] array that has a color value for each pixel based on if the distance for that pixel:

  • > 900 = Blue
  • > 900 && < 2000 = Green
  • > 2000 = Red

If the pixel represents a player, the color will be yellow/gold.

C#

private byte[] GenerateColoredBytes(ImageFrame imageFrame)
{
    int height = imageFrame.Image.Height;
    int width = imageFrame.Image.Width;

    //Depth data for each pixel
    Byte[] depthData = imageFrame.Image.Bits; 

    //colorFrame contains color information for all pixels in image
    //Height x Width x 4 (Red, Green, Blue, empty byte)
    Byte[] colorFrame = new byte[imageFrame.Image.Height * imageFrame.Image.Width * 4];

    //Bgr32  - Blue, Green, Red, empty byte
    //Bgra32 - Blue, Green, Red, transparency 
    //You must set transparency for Bgra as .NET defaults a byte to 0 = fully transparent

    //hardcoded locations to Blue, Green, Red (BGR) index positions       
    const int BlueIndex = 0;
    const int GreenIndex = 1;
    const int RedIndex = 2;

    var depthIndex = 0;
    for (var y = 0; y < height; y++)
    {        
        var heightOffset = y * width;

        for (var x = 0; x < width; x++)
        {
            var index = ((width - x - 1) + heightOffset) * 4;

            //var distance = GetDistance(depthData[depthIndex], depthData[depthIndex + 1]);
            var distance = GetDistanceWithPlayerIndex(depthData[depthIndex], depthData[depthIndex + 1]);

            if (distance <= 900)
            {
                //we are very close
                colorFrame[index + BlueIndex] = 255;
                colorFrame[index + GreenIndex] = 0;
                colorFrame[index + RedIndex] = 0;       
            }
            else if (distance > 900 && distance < 2000)
            {
                //we are a bit further away
                colorFrame[index + BlueIndex] = 0;
                colorFrame[index + GreenIndex] = 255;
                colorFrame[index + RedIndex] = 0;
            }
            else if (distance > 2000)
            {
                //we are the farthest
                colorFrame[index + BlueIndex] = 0;
                colorFrame[index + GreenIndex] = 0;
                colorFrame[index + RedIndex] = 255;
            }

            //Color a player
            if (GetPlayerIndex(depthData[depthIndex]) > 0)
            {
                //we are the farthest
                colorFrame[index + BlueIndex] = 0;
                colorFrame[index + GreenIndex] = 255;
                colorFrame[index + RedIndex] = 255;
            }

            //jump two bytes at a time
            depthIndex += 2;
        }
    }

    return colorFrame;
}

Visual Basic

Private Function GenerateColoredBytes(ByVal imageFrame As ImageFrame) As Byte()

    Dim height As Integer = imageFrame.Image.Height
    Dim width As Integer = imageFrame.Image.Width

    'Depth data for each pixel
    Dim depthData() As Byte = imageFrame.Image.Bits

    'colorFrame contains color information for all pixels in image
    'Height x Width x 4 (Red, Green, Blue, empty byte)
    Dim colorFrame(imageFrame.Image.Height * imageFrame.Image.Width * 4 - 1) As Byte

    'Bgr32  - Blue, Green, Red, empty byte
    'Bgra32 - Blue, Green, Red, transparency 
    'You must set transparency for Bgra as .NET defaults a byte to 0 = fully transparent

    'hardcoded locations to Blue, Green, Red (BGR) index positions       
    Const BlueIndex As Integer = 0
    Const GreenIndex As Integer = 1
    Const RedIndex As Integer = 2

    Dim depthIndex = 0
    For y = 0 To height - 1

        Dim heightOffset = y * width

        For x = 0 To width - 1

            Dim index = ((width - x - 1) + heightOffset) * 4

            Dim distance = GetDistanceWithPlayerIndex(depthData(depthIndex), depthData(depthIndex + 1))

            If distance <= 900 Then

                'we are very close
                colorFrame(index + BlueIndex) = 255
                colorFrame(index + GreenIndex) = 0
                colorFrame(index + RedIndex) = 0

            ElseIf distance > 900 AndAlso distance < 2000 Then

                'we are a bit further away
                colorFrame(index + BlueIndex) = 0
                colorFrame(index + GreenIndex) = 255
                colorFrame(index + RedIndex) = 0

            ElseIf distance > 2000 Then

                'we are the farthest
                colorFrame(index + BlueIndex) = 0
                colorFrame(index + GreenIndex) = 0
                colorFrame(index + RedIndex) = 255

            End If

            '//Color a player
            If GetPlayerIndex(depthData(depthIndex)) > 0 Then

                'we are the farthest
                colorFrame(index + BlueIndex) = 0
                colorFrame(index + GreenIndex) = 255
                colorFrame(index + RedIndex) = 255

            End If

            'jump two bytes at a time
            depthIndex += 2

        Next x

    Next y

    Return colorFrame

End Function

Follow the Discussion

  • AlecAlec

    These videos and the accompanying material have been very helpful, Dan. However, concerning your comments at [1:45], I was under the assumption that the IR sensor does not detect depth via time-of-flight. Would you mind clarifying this?

    Thanks

  • Dan FernandezDan

    @Alec:Thanks for pointing this out, it's my goof Sad, I just corrected the first video in the series to clarify this too

  • benben

    Is it just me or is the depth data and video data mirror images of each other? (That is, one goes left to right and the goes right to left)

    Make is kinda hard to mix the two...

  • Pieter NijsPieter Nijs

    Hi Ben,

    I got the same issue. I changed the loop inside the GenerateColoredBytes-method from
    for (var x = 0; x < width; x++)
    to
    for (var x = width; x > 0; x--)

    Now the images are displayed correctly!

  • BasBas It finds lightbulbs.

    I was wondering the same thing. Why is the image mirrored?

    Also, I don't get why the index in GenerateColoredBytes is calculated as ((width - x - 1) + heightOffset) * 4. Why does it start off at 1276 instead of 0?

     

    Edit: instead of Peter Nijs's solution of making the x iteration go backwards, I changed the line

    var index = ((width - x - 1) + heightOffset) * 4;

    to

    var index = (x + heightOffset) * 4;

    Now the image is no longer mirrorred, and your x iteration can still go from 0 to width-1. The index calculation makes more sense to me this way anyway, because basically all you need is the heightOffset plus the current column, taking into account that it's four bytes per pixel. Hence, (x + heightOffset) * 4.

  • Pieter NijsPieter Nijs

    Hi Bas,

    your solution does make more sense than mine!
    Thanks!

  • Clint RutkasClint I'm a "developer"

    The issue is if you do depth versus depth with player index.  Coding4Fun toolkit's ToBitmapSource functions take this flipping effect into account for the image types for depth.

  • The above video guide included the method for nui_DepthFrameReady, but the text guide did not. It should read as follows:

    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)

    {     
         // Convert depth information for a pixel into color information
         byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);
         // create an image based on the colored bytes
         PlanarImage image = e.ImageFrame.Image;
         image1.Source = BitmapSource.Create(
              image.Width, image.Height, 96, 96, PixelFormats.Bgr32,     
              null, ColoredBytes, image.Width * PixelFormats.Bgr32.BitsPerPixel / 8);
    }

    Sorry for the strange formatting. Channel9 does not support multiline code postings in the comments.

     

  • LZHLZH

    Does anyone encounter problem with GenerateColoredBytes(ImageFrame imageFrame) ?

    The moment i put in this method, my video frame seems to stop working.

  • wizardliuwizardliu





    Im trying the depth data sample by setting a blue.red and green color depth screen.

    But my current project need me to show the distance value for player that been tracked.

    when i use this code below to execute, the programme will stuck without error...can somebody explain to me why?

    thxxxxxx a lot

    if (GetPlayerIndex(depthData[depthIndex]) >0)
    {

    colorFrame[index + BlueIndex] = 0;
    colorFrame[index + GreenIndex] = 255;
    colorFrame[index + RedIndex] = 255;
    txtdistance.Text = distance.ToString() + " mm";


    }

  • Clint RutkasClint I'm a "developer"

    @wizardliu: betting it doesn't get stuck, it is more you're overloading the user interface thread.  Looking at what your code bits look like, you're inside the double for loop statement. 

    commenting out: txtdistance.Text = distance.ToString() + " mm";
    I think will resolve your issue.

    You're updating that text element 76800 times in theory at 30 frames per second.  Set a varible outside the double for loop at the top and then set the text box's text property outside the loops on the bottom.  I'll bet that will work a lot better.  One thing to not then that will give you the right most point in the lowest row with a player index then since the value will always be overwritten.

  • Clint RutkasClint I'm a "developer"

    @LZH: can you share a bit more?  the samples download runs as expected

  • I am trying to uses the depth information to implement collision avoidance. My idea is to calculate the lowest depth in each column of pixels and save it into an array closeestDistance[].with that i can turn to the direction with the largest distance value when the center is below a certain threshold. I use the coding4fun.kinect.wpf... I am getting some weird findings on testing. see code:

    short resX = 320; short resY = 120;

    short[] closestDistances = newshort[320]; // setting the size of the array

    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgse)

    {

    shorti,j, least = 4000; //just using the 4000 for starters

    image2.Source = e.ImageFrame.ToBitmapSource();

    image2.Source = e.ImageFrame.ToBitmapSource();

    short[][] distances = e.ImageFrame.ToDepthArray2D();

    for(i = 0; i < resX; i++)

    {

    for(j = 0; j < resY; j++)

    {

    if((distances [i][j] < least) && (distances[i][j] != 0))

    {

    least = distances[i][j];

    }

    }

    closestDistances[i] = least;

    }

    textBox1.Text = closestDistances[8].ToString();

    textBox2.Text = closestDistances[80].ToString();

    textBox3.Text = closestDistances[160].ToString();

    textBox4.Text = closestDistances[240].ToString();

    textBox5.Text = closestDistances[319].ToString();

    }

    Code ends: Now if you run this code with all the initilisation and event handler depthandplayerindex and openstream etc.... When its runs and "an" obstacle is played close enough relatively to the corresponding directions of the pixels 8,80,160,240,319... when placed on the left of the image ie. right of kinect facing forward direction (8 or 80)... Its seems to change the other values in the  other texboxes  to the same value (ie textboxes for closestDistances -> 160, 240, 319). Meanwhile this other textboxes should be showing higher values. Please help, run the code and place an obstacle in different sectors to understand my question. Or maybe there is a problem with my algorithm

  • amiramir

    Ben i'm trying your code without any error during building process. But, the image doesn't not appear. I'm only get an empty window. After debug the code, it seem that the colorFrame array contain valid value. Maybe there are some problem in function BitmapSource Create() because i create my own byte array to pass to this function. Still, nothing appear except an empty window. Do you know why?

  • amiramir

    i already find my mistake. Actually for BitmapSource.Create() function, i shall put PixelFormat Bgr32 not Bgra32. That's why the image doesn't not appear. Thanks a lot

  • Clint RutkasClint I'm a "developer"

    @amir: if you are using Bgra32, you can set the 4th byte (aka index + 3) to 255 and your image will appear at well.

  • MarceMarce

    Hello, sorry if the question is silly, I'm new at this. Let me know if this example can be used for a web application. I need to know if I can put the video online of what the camera captures.

  • JoostJoost

    Great informative presentation.
    The only thing I would like to suggest is to think about what the spectator sees when you indicate "left to right" with your hands (1:30).
    What the spectator actually sees is "right to left" :)

  • ManishManish

    I am doing a project with a friend using kinect. We need to stand in front of the Kinect, calibrate, make movements etc to debug the application. We bought big screen monitor + wireless keyboard etc.. but still tough to debug. Is there any good way to debug kinect application?

  • RamiRami

    I need to extract this image data to any other form like: matlab, to work on it. is that possible? also is there possible way to display image depth in certain region only not every where in camera field of view?

  • ChrisChris

    I'm having trouble displaying anything. I've copied the code (mostly, I'm just using the depth stream as opposed to depth and player index) but when I run my program, just a blank window. I haven't installed the coding4fun dll, I'm wondering if some of the methods in there for dealing with bitmaps is what I'm missing. I notice that nowhere (as far as I can tell) in this code is the GenerateColoredBytes function actually called and I suspect this may be the root of my problem, but I am noob with WPF and using images to show "video" streams. Any pointers?

    If my explanation sucks, I'll be happy to post code.

    Thanks guys!

  • ChrisChris

    @Rami:

    in terms of showing only a portion (or region) of the frame, you could make your two loops (one for columns and one for rows) only loop through the pixels contained in the specific region you're looking for. If you use BGRa then any pixels outside of this region could be left as transparent, although you'd have to make sure that the pixels WITHIN the region were being set to be opaque.
    What you'd end up with would be an image the had transparent borders. This may be an inelegant solution but it may, depending on how you were going to use the data, be sufficient

  • guys,

    I need to send the distance information to txt file, I can't figure that out. I am using kinect with c# wpf application. I used the same sample above, but I do not know how to make loop and send all the depth information (320X240) to .txt file , then I want to define one object and get distance for just one objects? 

     

    Any help, please?

     

  • AndreAndre

    Guys,
    Could you, please, explain why there is a "golden" chair when the expectation is just a "golden" player?

    PS. Could you, please, post a starting point (sample/lab/docs) to build a 3D model.

    Thank you!

  • ChrisChris

    I'm trying to replicate this project using winform and a picturebox instead of an image. I'm running into trouble with

    <code>
    1

    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)

    1

    {
    1

    // Convert depth information for a pixel into color information
    1

    byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);
    1

    // create an image based on the colored bytes
    1

    PlanarImage image = e.ImageFrame.Image;
    1

    image1.Source = BitmapSource.Create(
    1

    image.Width, image.Height, 96, 96, PixelFormats.Bgr32,
    1

    null, ColoredBytes, image.Width * PixelFormats.Bgr32.BitsPerPixel / 8);

    </code>

    specifically the BitmapSource and PixelFormats. Neither of these exist in the current context. I've manually added in the using statements that these need, but then I get errors saying that those using statements aren't valid dlls. What I've gathered is that these are used specifically in WPF. If that is true, what would be the winform equivalents?

    Thanks a bunch for any feedback

  • omar olguinomar olguin

    checks mi error
    PixelFormats.Bgr32
    is not the same like PixelFormat.

    checks the 's'

  • Abe KarnikAbe Karnik

    This is really useful for starting off. From what I see, each pixel in the depth map has details about its z-depth value. Do you mind including a transform algorithm that also translates the pixel x-y (or row-column) into real world x-y (in mm)? What I am interested in is that given a any pixel I should be able to tell where on the view frustum it lies.

  • Is anyone getting ghost / double images with above source for "Working with Depth Data" example? Just cross checked that even with default gray scale depth Image, I see about 10 fingers for my left hand on Depth Image. Color Video Stream is fine, no doubles there.

  • victorvictor

    i'm beginer
    i want to use depth map to calculate distance and control my mobile robot.but my routine have some problem.plz help me
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Windows;
    using System.Windows.Controls;
    using System.Windows.Data;
    using System.Windows.Documents;
    using System.Windows.Input;
    using System.Windows.Markup;
    using System.Windows.Media;
    using System.Windows.Media.Imaging;
    using System.Windows.Navigation;
    using System.Windows.Shapes;
    using Microsoft.Research.Kinect.Nui;
    using System.IO.Ports;
    using System.Threading;


    using Coding4Fun.Kinect.Wpf;



    namespace KinectGettingStarted
    {
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
    Runtime _kinectNui;

    const int RED_IDX = 2;
    const int GREEN_IDX = 1;
    const int BLUE_IDX = 0;
    byte[] depthFrame32 = new byte[320 * 240 * 4];

    int totalFrames = 0;
    int lastFrames = 0;
    DateTime lastTime = DateTime.MaxValue;
    SerialPort comport = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);



    public MainWindow()
    {
    InitializeComponent();


    }








    private void InitializeNui()
    {
    try
    {
    //• Declares _kinectNui as a Runtime object, which represents the Kinect sensor instance.
    _kinectNui = new Runtime();

    //Open the video and depth streams, and sets up the event handlers that the runtime calls when a video, depth, or skeleton frame is ready
    //An application must initialize the Kinect sensor by calling Runtime.Initialize before calling any other methods on the Runtime object.
    _kinectNui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex |RuntimeOptions.UseDepth|RuntimeOptions.UseSkeletalTracking|
    RuntimeOptions.UseColor);

    //To stream color images:
    // • The options must include UseColor.
    // • Valid image resolutions are Resolution1280x1024 and Resolution640x480.
    // • Valid image types are Color, ColorYUV, and ColorYUVRaw.
    _kinectNui.VideoStream.Open(ImageStreamType.Video, 2,
    ImageResolution.Resolution640x480, ImageType.ColorYuv);

    //To stream depth and player index data:
    // • The options must include UseDepthAndPlayerIndex.
    // • Valid resolutions for depth and player index data are Resolution320x240 and Resolution80x60.
    // • The only valid image type is DepthAndPlayerIndex.
    _kinectNui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex);

    lastTime = DateTime.Now;

    _kinectNui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(NuiVideoFrameReady);
    _kinectNui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);


    }
    catch (InvalidOperationException ex)
    {
    MessageBox.Show(ex.Message);
    }
    }



    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
    {

    var Image = e.ImageFrame.Image;
    var convertedDepthFrame = convertDepthFrame(Image.Bits);

    depth.Source = BitmapSource.Create(
    Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, convertedDepthFrame, Image.Width * 4);

    CalculateFps();
    //che them vo
    int height = e.ImageFrame.Image.Height;
    int width = e.ImageFrame.Image.Width;

    //Depth data for each pixel
    Byte[] depthData = Image.Bits;


    var depthIndex = 0;
    for (var y = 0; y < height; y++)
    {
    SerialPort comport = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);
    var heightOffset = y * width;

    for (var x = 0; x < width; x++)
    {

    var index = ((width - x - 1) + heightOffset) * 4;

    //var distance = GetDistance(depthData[depthIndex], depthData[depthIndex + 1]);
    var distance = GetDistanceWithPlayerIndex(depthData[depthIndex], depthData[depthIndex + 1]);
    //jump two bytes at a time
    depthIndex += 2;
    if (distance > 0 && distance < 4000)
    {
    // Thread.Sleep(50);
    if (comport.IsOpen)
    comport.Close();
    comport.Open();
    comport.WriteLine("F");
    if (distance <= 850)
    {
    comport.WriteLine("S");

    }
    if (distance > 850 && distance < 1200)
    {
    comport.WriteLine("R");
    if (distance <= 850)
    {
    comport.WriteLine("B");
    comport.WriteLine("L");
    }


    }

    if (1100 < distance)
    {
    comport.WriteLine("F");
    }
    }

    }
    }


    }

    private int GetDistanceWithPlayerIndex(byte firstFrame, byte secondFrame)
    {
    //offset by 3 in first byte to get value after player index

    int distance = (int)(firstFrame >> 3 | secondFrame << 5);
    return distance;



    }


    // Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame
    // that displays different players in different colors
    byte[] convertDepthFrame(byte[] depthFrame16)
    {
    for (int i16 = 0, i32 = 0; i16 < depthFrame16.Length && i32 < depthFrame32.Length; i16 += 2, i32 += 4)
    {
    int player = depthFrame16[i16] & 0x07;
    int realDepth = (depthFrame16[i16 + 1] << 5) | (depthFrame16[i16] >> 3);
    // transform 13-bit depth information into an 8-bit intensity appropriate
    // for display (we disregard information in most significant bit)
    byte intensity = (byte)(255 - (255 * realDepth / 0x0fff));

    depthFrame32[i32 + RED_IDX] = intensity;
    depthFrame32[i32 + BLUE_IDX] = intensity;
    depthFrame32[i32 + GREEN_IDX] = intensity;
    }
    return depthFrame32;

    }

    private void WindowLoaded(object sender, RoutedEventArgs e)
    {
    InitializeNui();
    }

    void CalculateFps()
    {
    ++totalFrames;

    var cur = DateTime.Now;
    if (cur.Subtract(lastTime) > TimeSpan.FromSeconds(1))
    {
    int frameDiff = totalFrames - lastFrames;
    lastFrames = totalFrames;
    lastTime = cur;
    frameRate.Text = frameDiff.ToString() + " fps";
    }
    }

    void NuiVideoFrameReady(object sender, ImageFrameReadyEventArgs e)
    {
    PlanarImage Image = e.ImageFrame.Image;

    image.Source = BitmapSource.Create(
    Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null,
    Image.Bits, Image.Width * Image.BytesPerPixel);

    // imageCmyk32.Source = BitmapSource.Create(
    // Image.Width, Image.Height, 96, 96, PixelFormats.Cmyk32, null,
    //Image.Bits, Image.Width * Image.BytesPerPixel);
    }

    private void WindowClosed(object sender, EventArgs e)
    {
    _kinectNui.Uninitialize();
    }

    private void BtnStopClick(object sender, RoutedEventArgs e)
    {
    _kinectNui.Uninitialize();
    btnStop.IsEnabled = false;
    btnStart.IsEnabled = true;
    }

    private void BtnStartClick(object sender, RoutedEventArgs e)
    {
    InitializeNui();
    btnStop.IsEnabled = true;
    btnStart.IsEnabled = false;
    }

    private void START_Click(object sender, RoutedEventArgs e)
    {
    if (comport.IsOpen)
    comport.Close();
    comport.Open();

    label1.Content = " READY";
    }

    private void FORWAD_Click(object sender, RoutedEventArgs e)
    {
    comport.WriteLine("F");
    label1.Content = "FORWARD";
    }

    private void LEFT_Click(object sender, RoutedEventArgs e)
    {
    comport.WriteLine("L");
    label1.Content = "LEFT";
    }

    private void RIGHT_Click(object sender, RoutedEventArgs e)
    {
    comport.WriteLine("R");
    label1.Content = "RIGHT";
    }

    private void BACK_Click(object sender, RoutedEventArgs e)
    {
    comport.WriteLine("B");
    label1.Content = "BACK";
    }
    private void STOP_Click(object sender, RoutedEventArgs e)
    {
    comport.WriteLine("S");
    label1.Content = " STOP";
    }

    }
    }

  • Eunsung SongEunsung Song

    Actually I couldn't understand this code one hundred percent...

    I just copied this into my code with Skeleton code but "image1.Source" made one error.

  • amiramir

    My question is out from this topic. Based on my browsing through the internet regarding the kinect. There are some website showing the need of the calibration mechanism between color and depth sensor before using kinect. They said purpose is to map the depth pixel with the color pixel obtain from kinect accurately. The question is, do i have to do that calibration mechanism? It is because, i already try this kinect SDK, i assume that depth and color image obtained from this SDK is accurately map. Thanks

  • amiramir

    I change the depth resolution to 640x480 by changing the "nui.DepthStream.Open(ImageStreamType.Depth, 2,ImageResolution.Resolution640x480,ImageType.DepthAndPlayerIndex)" , but no image appear in window. Even the DepthFrameReady is not triggered. Why is this happen?

  • JimJim

    I'm trying the SDK and sample projects and am seeing ghost images too like the 'IlikeDotNet' post of Oct. 29. Any ideas? I didn't see that on Dan's video.

  • hey guys! newbie here! need help.

    coding:

    image1.Source = BitmapSource.Create(image.Width, image.Height, 96, 96, PixelFormats.Bgr32, null,

    ColoredBytes, image.Width = PixelFormats.Bgr32.BitsPerPixel /Eye Rolling;

     

    error: ArgumentException was unhandled

    Value does not fall within the expected range.

     

    what is it???

    need help Smiley

  • Don DeFordDon DeFord

    I am interested in getting the distance information but only for a particular item in the image (e.g. a target of a particular shape). Could you explain how to do this.

    Thank you.

  • Jean ThammJean Thamm

    Using processing, skelleton detection and drawing happened at about 30fps. Using this, skelleton detection is at about 5 to 10fps. On the same machine.

    Why is that so?

  • @LZH: hello.. i read your problem about freezing the video and i think i am facing the same problem now... have you succeeded to find out the solution to this problem ??

    thanks a lot,

  • kavekave

    Hi
    I tried the above code to get the depth data. But when I debug the code, after the debugger executed the following lines of code:
    //UseDepthAndPlayerIndex and UseSkeletalTracking
    nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseDepth);

    //register for event
    nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);

    //DepthAndPlayerIndex ImageType
    nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution80x60,
    ImageType.DepthAndPlayerIndex);

    The code terminates and the debugger never executed the following function
    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)

    And I got this error:
    “AccessViolationException was unhandled”
    “Attempted to read or write protected memory. This is often an indication that other memory is corrupt.”
    Do you guys know why it happens? What is the solution to this problem?
    thanks

Remove this comment

Remove this thread

close

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.