Content Archived

This content is no longer current. Our recommendation for up to date content: http://channel9.msdn.com/Series/KinectQuickstart/Working-with-Depth-Data

Working with Depth Data (Beta 2 SDK)

Sign in to queue

The Discussion

  • User profile image
    Alec

    These videos and the accompanying material have been very helpful, Dan. However, concerning your comments at [1:45], I was under the assumption that the IR sensor does not detect depth via time-of-flight. Would you mind clarifying this?

    Thanks

  • User profile image
    Dan

    @Alec:Thanks for pointing this out, it's my goof Sad, I just corrected the first video in the series to clarify this too

  • User profile image
    ben

    Is it just me or is the depth data and video data mirror images of each other? (That is, one goes left to right and the goes right to left)

    Make is kinda hard to mix the two...

  • User profile image
    Pieter Nijs

    Hi Ben,

    I got the same issue. I changed the loop inside the GenerateColoredBytes-method from
    for (var x = 0; x < width; x++)
    to
    for (var x = width; x > 0; x--)

    Now the images are displayed correctly!

  • User profile image
    Bas

    I was wondering the same thing. Why is the image mirrored?

    Also, I don't get why the index in GenerateColoredBytes is calculated as ((width - x - 1) + heightOffset) * 4. Why does it start off at 1276 instead of 0?

     

    Edit: instead of Peter Nijs's solution of making the x iteration go backwards, I changed the line

    var index = ((width - x - 1) + heightOffset) * 4;

    to

    var index = (x + heightOffset) * 4;

    Now the image is no longer mirrorred, and your x iteration can still go from 0 to width-1. The index calculation makes more sense to me this way anyway, because basically all you need is the heightOffset plus the current column, taking into account that it's four bytes per pixel. Hence, (x + heightOffset) * 4.

  • User profile image
    Pieter Nijs

    Hi Bas,

    your solution does make more sense than mine!
    Thanks!

  • User profile image
    Clint

    The issue is if you do depth versus depth with player index.  Coding4Fun toolkit's ToBitmapSource functions take this flipping effect into account for the image types for depth.

  • User profile image
    Tom_Anderson

    The above video guide included the method for nui_DepthFrameReady, but the text guide did not. It should read as follows:

    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)

    {     
         // Convert depth information for a pixel into color information
         byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);
         // create an image based on the colored bytes
         PlanarImage image = e.ImageFrame.Image;
         image1.Source = BitmapSource.Create(
              image.Width, image.Height, 96, 96, PixelFormats.Bgr32,     
              null, ColoredBytes, image.Width * PixelFormats.Bgr32.BitsPerPixel / 8);
    }

    Sorry for the strange formatting. Channel9 does not support multiline code postings in the comments.

     

  • User profile image
    LZH

    Does anyone encounter problem with GenerateColoredBytes(ImageFrame imageFrame) ?

    The moment i put in this method, my video frame seems to stop working.

  • User profile image
    wizardliu





    Im trying the depth data sample by setting a blue.red and green color depth screen.

    But my current project need me to show the distance value for player that been tracked.

    when i use this code below to execute, the programme will stuck without error...can somebody explain to me why?

    thxxxxxx a lot

    if (GetPlayerIndex(depthData[depthIndex]) >0)
    {

    colorFrame[index + BlueIndex] = 0;
    colorFrame[index + GreenIndex] = 255;
    colorFrame[index + RedIndex] = 255;
    txtdistance.Text = distance.ToString() + " mm";


    }

  • User profile image
    Clint

    @wizardliu: betting it doesn't get stuck, it is more you're overloading the user interface thread.  Looking at what your code bits look like, you're inside the double for loop statement. 

    commenting out: txtdistance.Text = distance.ToString() + " mm";
    I think will resolve your issue.

    You're updating that text element 76800 times in theory at 30 frames per second.  Set a varible outside the double for loop at the top and then set the text box's text property outside the loops on the bottom.  I'll bet that will work a lot better.  One thing to not then that will give you the right most point in the lowest row with a player index then since the value will always be overwritten.

  • User profile image
    Clint

    @LZH: can you share a bit more?  the samples download runs as expected

  • User profile image
    smoothik

    I am trying to uses the depth information to implement collision avoidance. My idea is to calculate the lowest depth in each column of pixels and save it into an array closeestDistance[].with that i can turn to the direction with the largest distance value when the center is below a certain threshold. I use the coding4fun.kinect.wpf... I am getting some weird findings on testing. see code:

    short resX = 320; short resY = 120;

    short[] closestDistances = newshort[320]; // setting the size of the array

    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgse)

    {

    shorti,j, least = 4000; //just using the 4000 for starters

    image2.Source = e.ImageFrame.ToBitmapSource();

    image2.Source = e.ImageFrame.ToBitmapSource();

    short[][] distances = e.ImageFrame.ToDepthArray2D();

    for(i = 0; i < resX; i++)

    {

    for(j = 0; j < resY; j++)

    {

    if((distances [i][j] < least) && (distances[i][j] != 0))

    {

    least = distances[i][j];

    }

    }

    closestDistances[i] = least;

    }

    textBox1.Text = closestDistances[8].ToString();

    textBox2.Text = closestDistances[80].ToString();

    textBox3.Text = closestDistances[160].ToString();

    textBox4.Text = closestDistances[240].ToString();

    textBox5.Text = closestDistances[319].ToString();

    }

    Code ends: Now if you run this code with all the initilisation and event handler depthandplayerindex and openstream etc.... When its runs and "an" obstacle is played close enough relatively to the corresponding directions of the pixels 8,80,160,240,319... when placed on the left of the image ie. right of kinect facing forward direction (8 or 80)... Its seems to change the other values in the  other texboxes  to the same value (ie textboxes for closestDistances -> 160, 240, 319). Meanwhile this other textboxes should be showing higher values. Please help, run the code and place an obstacle in different sectors to understand my question. Or maybe there is a problem with my algorithm

  • User profile image
    amir

    Ben i'm trying your code without any error during building process. But, the image doesn't not appear. I'm only get an empty window. After debug the code, it seem that the colorFrame array contain valid value. Maybe there are some problem in function BitmapSource Create() because i create my own byte array to pass to this function. Still, nothing appear except an empty window. Do you know why?

  • User profile image
    amir

    i already find my mistake. Actually for BitmapSource.Create() function, i shall put PixelFormat Bgr32 not Bgra32. That's why the image doesn't not appear. Thanks a lot

  • User profile image
    Clint

    @amir: if you are using Bgra32, you can set the 4th byte (aka index + 3) to 255 and your image will appear at well.

  • User profile image
    Marce

    Hello, sorry if the question is silly, I'm new at this. Let me know if this example can be used for a web application. I need to know if I can put the video online of what the camera captures.

  • User profile image
    Joost

    Great informative presentation.
    The only thing I would like to suggest is to think about what the spectator sees when you indicate "left to right" with your hands (1:30).
    What the spectator actually sees is "right to left" :)

  • User profile image
    Manish

    I am doing a project with a friend using kinect. We need to stand in front of the Kinect, calibrate, make movements etc to debug the application. We bought big screen monitor + wireless keyboard etc.. but still tough to debug. Is there any good way to debug kinect application?

  • User profile image
    Rami

    I need to extract this image data to any other form like: matlab, to work on it. is that possible? also is there possible way to display image depth in certain region only not every where in camera field of view?

  • User profile image
    Chris

    I'm having trouble displaying anything. I've copied the code (mostly, I'm just using the depth stream as opposed to depth and player index) but when I run my program, just a blank window. I haven't installed the coding4fun dll, I'm wondering if some of the methods in there for dealing with bitmaps is what I'm missing. I notice that nowhere (as far as I can tell) in this code is the GenerateColoredBytes function actually called and I suspect this may be the root of my problem, but I am noob with WPF and using images to show "video" streams. Any pointers?

    If my explanation sucks, I'll be happy to post code.

    Thanks guys!

  • User profile image
    Chris

    @Rami:

    in terms of showing only a portion (or region) of the frame, you could make your two loops (one for columns and one for rows) only loop through the pixels contained in the specific region you're looking for. If you use BGRa then any pixels outside of this region could be left as transparent, although you'd have to make sure that the pixels WITHIN the region were being set to be opaque.
    What you'd end up with would be an image the had transparent borders. This may be an inelegant solution but it may, depending on how you were going to use the data, be sufficient

  • User profile image
    rami82

    guys,

    I need to send the distance information to txt file, I can't figure that out. I am using kinect with c# wpf application. I used the same sample above, but I do not know how to make loop and send all the depth information (320X240) to .txt file , then I want to define one object and get distance for just one objects? 

     

    Any help, please?

     

  • User profile image
    Andre

    Guys,
    Could you, please, explain why there is a "golden" chair when the expectation is just a "golden" player?

    PS. Could you, please, post a starting point (sample/lab/docs) to build a 3D model.

    Thank you!

  • User profile image
    Chris

    I'm trying to replicate this project using winform and a picturebox instead of an image. I'm running into trouble with

    <code>
    1

    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)

    1

    {
    1

    // Convert depth information for a pixel into color information
    1

    byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);
    1

    // create an image based on the colored bytes
    1

    PlanarImage image = e.ImageFrame.Image;
    1

    image1.Source = BitmapSource.Create(
    1

    image.Width, image.Height, 96, 96, PixelFormats.Bgr32,
    1

    null, ColoredBytes, image.Width * PixelFormats.Bgr32.BitsPerPixel / 8);

    </code>

    specifically the BitmapSource and PixelFormats. Neither of these exist in the current context. I've manually added in the using statements that these need, but then I get errors saying that those using statements aren't valid dlls. What I've gathered is that these are used specifically in WPF. If that is true, what would be the winform equivalents?

    Thanks a bunch for any feedback

  • User profile image
    omar olguin

    checks mi error
    PixelFormats.Bgr32
    is not the same like PixelFormat.

    checks the 's'

  • User profile image
    Abe Karnik

    This is really useful for starting off. From what I see, each pixel in the depth map has details about its z-depth value. Do you mind including a transform algorithm that also translates the pixel x-y (or row-column) into real world x-y (in mm)? What I am interested in is that given a any pixel I should be able to tell where on the view frustum it lies.

  • User profile image
    iLikeDotNet

    Is anyone getting ghost / double images with above source for "Working with Depth Data" example? Just cross checked that even with default gray scale depth Image, I see about 10 fingers for my left hand on Depth Image. Color Video Stream is fine, no doubles there.

  • User profile image
    victor

    i'm beginer
    i want to use depth map to calculate distance and control my mobile robot.but my routine have some problem.plz help me
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Windows;
    using System.Windows.Controls;
    using System.Windows.Data;
    using System.Windows.Documents;
    using System.Windows.Input;
    using System.Windows.Markup;
    using System.Windows.Media;
    using System.Windows.Media.Imaging;
    using System.Windows.Navigation;
    using System.Windows.Shapes;
    using Microsoft.Research.Kinect.Nui;
    using System.IO.Ports;
    using System.Threading;


    using Coding4Fun.Kinect.Wpf;



    namespace KinectGettingStarted
    {
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
    Runtime _kinectNui;

    const int RED_IDX = 2;
    const int GREEN_IDX = 1;
    const int BLUE_IDX = 0;
    byte[] depthFrame32 = new byte[320 * 240 * 4];

    int totalFrames = 0;
    int lastFrames = 0;
    DateTime lastTime = DateTime.MaxValue;
    SerialPort comport = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);



    public MainWindow()
    {
    InitializeComponent();


    }








    private void InitializeNui()
    {
    try
    {
    //• Declares _kinectNui as a Runtime object, which represents the Kinect sensor instance.
    _kinectNui = new Runtime();

    //Open the video and depth streams, and sets up the event handlers that the runtime calls when a video, depth, or skeleton frame is ready
    //An application must initialize the Kinect sensor by calling Runtime.Initialize before calling any other methods on the Runtime object.
    _kinectNui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex |RuntimeOptions.UseDepth|RuntimeOptions.UseSkeletalTracking|
    RuntimeOptions.UseColor);

    //To stream color images:
    // • The options must include UseColor.
    // • Valid image resolutions are Resolution1280x1024 and Resolution640x480.
    // • Valid image types are Color, ColorYUV, and ColorYUVRaw.
    _kinectNui.VideoStream.Open(ImageStreamType.Video, 2,
    ImageResolution.Resolution640x480, ImageType.ColorYuv);

    //To stream depth and player index data:
    // • The options must include UseDepthAndPlayerIndex.
    // • Valid resolutions for depth and player index data are Resolution320x240 and Resolution80x60.
    // • The only valid image type is DepthAndPlayerIndex.
    _kinectNui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex);

    lastTime = DateTime.Now;

    _kinectNui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(NuiVideoFrameReady);
    _kinectNui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);


    }
    catch (InvalidOperationException ex)
    {
    MessageBox.Show(ex.Message);
    }
    }



    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
    {

    var Image = e.ImageFrame.Image;
    var convertedDepthFrame = convertDepthFrame(Image.Bits);

    depth.Source = BitmapSource.Create(
    Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, convertedDepthFrame, Image.Width * 4);

    CalculateFps();
    //che them vo
    int height = e.ImageFrame.Image.Height;
    int width = e.ImageFrame.Image.Width;

    //Depth data for each pixel
    Byte[] depthData = Image.Bits;


    var depthIndex = 0;
    for (var y = 0; y < height; y++)
    {
    SerialPort comport = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);
    var heightOffset = y * width;

    for (var x = 0; x < width; x++)
    {

    var index = ((width - x - 1) + heightOffset) * 4;

    //var distance = GetDistance(depthData[depthIndex], depthData[depthIndex + 1]);
    var distance = GetDistanceWithPlayerIndex(depthData[depthIndex], depthData[depthIndex + 1]);
    //jump two bytes at a time
    depthIndex += 2;
    if (distance > 0 && distance < 4000)
    {
    // Thread.Sleep(50);
    if (comport.IsOpen)
    comport.Close();
    comport.Open();
    comport.WriteLine("F");
    if (distance <= 850)
    {
    comport.WriteLine("S");

    }
    if (distance > 850 && distance < 1200)
    {
    comport.WriteLine("R");
    if (distance <= 850)
    {
    comport.WriteLine("B");
    comport.WriteLine("L");
    }


    }

    if (1100 < distance)
    {
    comport.WriteLine("F");
    }
    }

    }
    }


    }

    private int GetDistanceWithPlayerIndex(byte firstFrame, byte secondFrame)
    {
    //offset by 3 in first byte to get value after player index

    int distance = (int)(firstFrame >> 3 | secondFrame << 5);
    return distance;



    }


    // Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame
    // that displays different players in different colors
    byte[] convertDepthFrame(byte[] depthFrame16)
    {
    for (int i16 = 0, i32 = 0; i16 < depthFrame16.Length && i32 < depthFrame32.Length; i16 += 2, i32 += 4)
    {
    int player = depthFrame16[i16] & 0x07;
    int realDepth = (depthFrame16[i16 + 1] << 5) | (depthFrame16[i16] >> 3);
    // transform 13-bit depth information into an 8-bit intensity appropriate
    // for display (we disregard information in most significant bit)
    byte intensity = (byte)(255 - (255 * realDepth / 0x0fff));

    depthFrame32[i32 + RED_IDX] = intensity;
    depthFrame32[i32 + BLUE_IDX] = intensity;
    depthFrame32[i32 + GREEN_IDX] = intensity;
    }
    return depthFrame32;

    }

    private void WindowLoaded(object sender, RoutedEventArgs e)
    {
    InitializeNui();
    }

    void CalculateFps()
    {
    ++totalFrames;

    var cur = DateTime.Now;
    if (cur.Subtract(lastTime) > TimeSpan.FromSeconds(1))
    {
    int frameDiff = totalFrames - lastFrames;
    lastFrames = totalFrames;
    lastTime = cur;
    frameRate.Text = frameDiff.ToString() + " fps";
    }
    }

    void NuiVideoFrameReady(object sender, ImageFrameReadyEventArgs e)
    {
    PlanarImage Image = e.ImageFrame.Image;

    image.Source = BitmapSource.Create(
    Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null,
    Image.Bits, Image.Width * Image.BytesPerPixel);

    // imageCmyk32.Source = BitmapSource.Create(
    // Image.Width, Image.Height, 96, 96, PixelFormats.Cmyk32, null,
    //Image.Bits, Image.Width * Image.BytesPerPixel);
    }

    private void WindowClosed(object sender, EventArgs e)
    {
    _kinectNui.Uninitialize();
    }

    private void BtnStopClick(object sender, RoutedEventArgs e)
    {
    _kinectNui.Uninitialize();
    btnStop.IsEnabled = false;
    btnStart.IsEnabled = true;
    }

    private void BtnStartClick(object sender, RoutedEventArgs e)
    {
    InitializeNui();
    btnStop.IsEnabled = true;
    btnStart.IsEnabled = false;
    }

    private void START_Click(object sender, RoutedEventArgs e)
    {
    if (comport.IsOpen)
    comport.Close();
    comport.Open();

    label1.Content = " READY";
    }

    private void FORWAD_Click(object sender, RoutedEventArgs e)
    {
    comport.WriteLine("F");
    label1.Content = "FORWARD";
    }

    private void LEFT_Click(object sender, RoutedEventArgs e)
    {
    comport.WriteLine("L");
    label1.Content = "LEFT";
    }

    private void RIGHT_Click(object sender, RoutedEventArgs e)
    {
    comport.WriteLine("R");
    label1.Content = "RIGHT";
    }

    private void BACK_Click(object sender, RoutedEventArgs e)
    {
    comport.WriteLine("B");
    label1.Content = "BACK";
    }
    private void STOP_Click(object sender, RoutedEventArgs e)
    {
    comport.WriteLine("S");
    label1.Content = " STOP";
    }

    }
    }

  • User profile image
    Eunsung Song

    Actually I couldn't understand this code one hundred percent...

    I just copied this into my code with Skeleton code but "image1.Source" made one error.

  • User profile image
    amir

    My question is out from this topic. Based on my browsing through the internet regarding the kinect. There are some website showing the need of the calibration mechanism between color and depth sensor before using kinect. They said purpose is to map the depth pixel with the color pixel obtain from kinect accurately. The question is, do i have to do that calibration mechanism? It is because, i already try this kinect SDK, i assume that depth and color image obtained from this SDK is accurately map. Thanks

  • User profile image
    amir

    I change the depth resolution to 640x480 by changing the "nui.DepthStream.Open(ImageStreamType.Depth, 2,ImageResolution.Resolution640x480,ImageType.DepthAndPlayerIndex)" , but no image appear in window. Even the DepthFrameReady is not triggered. Why is this happen?

  • User profile image
    Jim

    I'm trying the SDK and sample projects and am seeing ghost images too like the 'IlikeDotNet' post of Oct. 29. Any ideas? I didn't see that on Dan's video.

  • User profile image
    kendrick0772

    hey guys! newbie here! need help.

    coding:

    image1.Source = BitmapSource.Create(image.Width, image.Height, 96, 96, PixelFormats.Bgr32, null,

    ColoredBytes, image.Width = PixelFormats.Bgr32.BitsPerPixel /Eye Rolling;

     

    error: ArgumentException was unhandled

    Value does not fall within the expected range.

     

    what is it???

    need help Smiley

  • User profile image
    Don DeFord

    I am interested in getting the distance information but only for a particular item in the image (e.g. a target of a particular shape). Could you explain how to do this.

    Thank you.

  • User profile image
    Jean Thamm

    Using processing, skelleton detection and drawing happened at about 30fps. Using this, skelleton detection is at about 5 to 10fps. On the same machine.

    Why is that so?

  • User profile image
    Colla

    @LZH: hello.. i read your problem about freezing the video and i think i am facing the same problem now... have you succeeded to find out the solution to this problem ??

    thanks a lot,

  • User profile image
    kave

    Hi
    I tried the above code to get the depth data. But when I debug the code, after the debugger executed the following lines of code:
    //UseDepthAndPlayerIndex and UseSkeletalTracking
    nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseDepth);

    //register for event
    nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);

    //DepthAndPlayerIndex ImageType
    nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution80x60,
    ImageType.DepthAndPlayerIndex);

    The code terminates and the debugger never executed the following function
    void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)

    And I got this error:
    “AccessViolationException was unhandled”
    “Attempted to read or write protected memory. This is often an indication that other memory is corrupt.”
    Do you guys know why it happens? What is the solution to this problem?
    thanks

Add Your 2 Cents