The Discussion

• 

  These videos and the accompanying material have been very helpful, Dan. However, concerning your comments at [1:45], I was under the assumption that the IR sensor does not detect depth via time-of-flight. Would you mind clarifying this?
  
  Thanks

• 

  @Alec:Thanks for pointing this out, it's my goof , I just corrected the first video in the series to clarify this too

• 

  Is it just me or is the depth data and video data mirror images of each other? (That is, one goes left to right and the goes right to left)
  
  Make is kinda hard to mix the two...

• 

  Hi Ben,
  
  I got the same issue. I changed the loop inside the GenerateColoredBytes-method from
  for (var x = 0; x < width; x++)
  to
  for (var x = width; x > 0; x--)
  
  Now the images are displayed correctly!

• 

  I was wondering the same thing. Why is the image mirrored?

  Also, I don't get why the index in GenerateColoredBytes is calculated as ((width - x - 1) + heightOffset) * 4. Why does it start off at 1276 instead of 0?

   

  Edit: instead of Peter Nijs's solution of making the x iteration go backwards, I changed the line

  var index = ((width - x - 1) + heightOffset) * 4;

  to

  var index = (x + heightOffset) * 4;

  Now the image is no longer mirrorred, and your x iteration can still go from 0 to width-1. The index calculation makes more sense to me this way anyway, because basically all you need is the heightOffset plus the current column, taking into account that it's four bytes per pixel. Hence, (x + heightOffset) * 4.

• 

  Hi Bas,
  
  your solution does make more sense than mine!
  Thanks!

• 

  The issue is if you do depth versus depth with player index.  Coding4Fun toolkit's ToBitmapSource functions take this flipping effect into account for the image types for depth.

• 

  The above video guide included the method for nui_DepthFrameReady, but the text guide did not. It should read as follows:

  void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)

  {     

       // Convert depth information for a pixel into color information

       byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);

       // create an image based on the colored bytes

       PlanarImage image = e.ImageFrame.Image;

       image1.Source = BitmapSource.Create(

            image.Width, image.Height, 96, 96, PixelFormats.Bgr32,     

            null, ColoredBytes, image.Width * PixelFormats.Bgr32.BitsPerPixel / 8);

  }

  Sorry for the strange formatting. Channel9 does not support multiline code postings in the comments.

   

• 

  Does anyone encounter problem with GenerateColoredBytes(ImageFrame imageFrame) ?

  The moment i put in this method, my video frame seems to stop working.

• 

  
  
  
  
  Im trying the depth data sample by setting a blue.red and green color depth screen.
  
  But my current project need me to show the distance value for player that been tracked.
  
  when i use this code below to execute, the programme will stuck without error...can somebody explain to me why?
  
  thxxxxxx a lot
  
  if (GetPlayerIndex(depthData[depthIndex]) >0)
  {
  
  colorFrame[index + BlueIndex] = 0;
  colorFrame[index + GreenIndex] = 255;
  colorFrame[index + RedIndex] = 255;
  txtdistance.Text = distance.ToString() + " mm";
  
  
  }
  

• 

  @wizardliu: betting it doesn't get stuck, it is more you're overloading the user interface thread.  Looking at what your code bits look like, you're inside the double for loop statement. 

  commenting out: txtdistance.Text = distance.ToString() + " mm";
  I think will resolve your issue.

  You're updating that text element 76800 times in theory at 30 frames per second.  Set a varible outside the double for loop at the top and then set the text box's text property outside the loops on the bottom.  I'll bet that will work a lot better.  One thing to not then that will give you the right most point in the lowest row with a player index then since the value will always be overwritten.

• 

  @LZH: can you share a bit more?  the samples download runs as expected

• 

  I am trying to uses the depth information to implement collision avoidance. My idea is to calculate the lowest depth in each column of pixels and save it into an array closeestDistance[].with that i can turn to the direction with the largest distance value when the center is below a certain threshold. I use the coding4fun.kinect.wpf... I am getting some weird findings on testing. see code:

  short resX = 320; short resY = 120;

  short[] closestDistances = newshort[320]; // setting the size of the array

  void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgse)

  {

  shorti,j, least = 4000; //just using the 4000 for starters

  image2.Source = e.ImageFrame.ToBitmapSource();

  image2.Source = e.ImageFrame.ToBitmapSource();

  short[][] distances = e.ImageFrame.ToDepthArray2D();

  for(i = 0; i < resX; i++)

  {

  for(j = 0; j < resY; j++)

  {

  if((distances [i][j] < least) && (distances[i][j] != 0))

  {

  least = distances[i][j];

  }

  }

  closestDistances[i] = least;

  }

  textBox1.Text = closestDistances[8].ToString();

  textBox2.Text = closestDistances[80].ToString();

  textBox3.Text = closestDistances[160].ToString();

  textBox4.Text = closestDistances[240].ToString();

  textBox5.Text = closestDistances[319].ToString();

  }

  Code ends: Now if you run this code with all the initilisation and event handler depthandplayerindex and openstream etc.... When its runs and "an" obstacle is played close enough relatively to the corresponding directions of the pixels 8,80,160,240,319... when placed on the left of the image ie. right of kinect facing forward direction (8 or 80)... Its seems to change the other values in the  other texboxes  to the same value (ie textboxes for closestDistances -> 160, 240, 319). Meanwhile this other textboxes should be showing higher values. Please help, run the code and place an obstacle in different sectors to understand my question. Or maybe there is a problem with my algorithm

• 

  Ben i'm trying your code without any error during building process. But, the image doesn't not appear. I'm only get an empty window. After debug the code, it seem that the colorFrame array contain valid value. Maybe there are some problem in function BitmapSource Create() because i create my own byte array to pass to this function. Still, nothing appear except an empty window. Do you know why?

• 

  i already find my mistake. Actually for BitmapSource.Create() function, i shall put PixelFormat Bgr32 not Bgra32. That's why the image doesn't not appear. Thanks a lot

• 

  @amir: if you are using Bgra32, you can set the 4th byte (aka index + 3) to 255 and your image will appear at well.

• 

  Hello, sorry if the question is silly, I'm new at this. Let me know if this example can be used for a web application. I need to know if I can put the video online of what the camera captures.

• 

  Great informative presentation.
  The only thing I would like to suggest is to think about what the spectator sees when you indicate "left to right" with your hands (1:30).
  What the spectator actually sees is "right to left" :)

• 

  I am doing a project with a friend using kinect. We need to stand in front of the Kinect, calibrate, make movements etc to debug the application. We bought big screen monitor + wireless keyboard etc.. but still tough to debug. Is there any good way to debug kinect application?

• 

  I need to extract this image data to any other form like: matlab, to work on it. is that possible? also is there possible way to display image depth in certain region only not every where in camera field of view?
  

• 

  I'm having trouble displaying anything. I've copied the code (mostly, I'm just using the depth stream as opposed to depth and player index) but when I run my program, just a blank window. I haven't installed the coding4fun dll, I'm wondering if some of the methods in there for dealing with bitmaps is what I'm missing. I notice that nowhere (as far as I can tell) in this code is the GenerateColoredBytes function actually called and I suspect this may be the root of my problem, but I am noob with WPF and using images to show "video" streams. Any pointers?
  
  If my explanation sucks, I'll be happy to post code.
  
  Thanks guys!

• 

  @Rami:
  
  in terms of showing only a portion (or region) of the frame, you could make your two loops (one for columns and one for rows) only loop through the pixels contained in the specific region you're looking for. If you use BGRa then any pixels outside of this region could be left as transparent, although you'd have to make sure that the pixels WITHIN the region were being set to be opaque.
  What you'd end up with would be an image the had transparent borders. This may be an inelegant solution but it may, depending on how you were going to use the data, be sufficient

• 

  guys,

  I need to send the distance information to txt file, I can't figure that out. I am using kinect with c# wpf application. I used the same sample above, but I do not know how to make loop and send all the depth information (320X240) to .txt file , then I want to define one object and get distance for just one objects? 

   

  Any help, please?

   

• 

  Guys,
  Could you, please, explain why there is a "golden" chair when the expectation is just a "golden" player?
  
  PS. Could you, please, post a starting point (sample/lab/docs) to build a 3D model.
  
  Thank you!

• 

  I'm trying to replicate this project using winform and a picturebox instead of an image. I'm running into trouble with
  
  <code>
  1
  
  void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
  
  1
  
  {
  1
  
  // Convert depth information for a pixel into color information
  1
  
  byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);
  1
  
  // create an image based on the colored bytes
  1
  
  PlanarImage image = e.ImageFrame.Image;
  1
  
  image1.Source = BitmapSource.Create(
  1
  
  image.Width, image.Height, 96, 96, PixelFormats.Bgr32,
  1
  
  null, ColoredBytes, image.Width * PixelFormats.Bgr32.BitsPerPixel / 8);
  
  </code>
  
  specifically the BitmapSource and PixelFormats. Neither of these exist in the current context. I've manually added in the using statements that these need, but then I get errors saying that those using statements aren't valid dlls. What I've gathered is that these are used specifically in WPF. If that is true, what would be the winform equivalents?
  
  Thanks a bunch for any feedback
  

• 

  checks mi error
  PixelFormats.Bgr32
  is not the same like PixelFormat.
  
  checks the 's'

• 

  This is really useful for starting off. From what I see, each pixel in the depth map has details about its z-depth value. Do you mind including a transform algorithm that also translates the pixel x-y (or row-column) into real world x-y (in mm)? What I am interested in is that given a any pixel I should be able to tell where on the view frustum it lies.

• 

  Is anyone getting ghost / double images with above source for "Working with Depth Data" example? Just cross checked that even with default gray scale depth Image, I see about 10 fingers for my left hand on Depth Image. Color Video Stream is fine, no doubles there.

• 

  i'm beginer
  i want to use depth map to calculate distance and control my mobile robot.but my routine have some problem.plz help me
  using System;
  using System.Collections.Generic;
  using System.Linq;
  using System.Text;
  using System.Windows;
  using System.Windows.Controls;
  using System.Windows.Data;
  using System.Windows.Documents;
  using System.Windows.Input;
  using System.Windows.Markup;
  using System.Windows.Media;
  using System.Windows.Media.Imaging;
  using System.Windows.Navigation;
  using System.Windows.Shapes;
  using Microsoft.Research.Kinect.Nui;
  using System.IO.Ports;
  using System.Threading;
  
  
  using Coding4Fun.Kinect.Wpf;
  
  
  
  namespace KinectGettingStarted
  {
  /// <summary>
  /// Interaction logic for MainWindow.xaml
  /// </summary>
  public partial class MainWindow : Window
  {
  Runtime _kinectNui;
  
  const int RED_IDX = 2;
  const int GREEN_IDX = 1;
  const int BLUE_IDX = 0;
  byte[] depthFrame32 = new byte[320 * 240 * 4];
  
  int totalFrames = 0;
  int lastFrames = 0;
  DateTime lastTime = DateTime.MaxValue;
  SerialPort comport = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);
  
  
  
  public MainWindow()
  {
  InitializeComponent();
  
  
  }
  
  
  
  
  
  
  
  
  private void InitializeNui()
  {
  try
  {
  //• Declares _kinectNui as a Runtime object, which represents the Kinect sensor instance.
  _kinectNui = new Runtime();
  
  //Open the video and depth streams, and sets up the event handlers that the runtime calls when a video, depth, or skeleton frame is ready
  //An application must initialize the Kinect sensor by calling Runtime.Initialize before calling any other methods on the Runtime object.
  _kinectNui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex |RuntimeOptions.UseDepth|RuntimeOptions.UseSkeletalTracking|
  RuntimeOptions.UseColor);
  
  //To stream color images:
  // • The options must include UseColor.
  // • Valid image resolutions are Resolution1280x1024 and Resolution640x480.
  // • Valid image types are Color, ColorYUV, and ColorYUVRaw.
  _kinectNui.VideoStream.Open(ImageStreamType.Video, 2,
  ImageResolution.Resolution640x480, ImageType.ColorYuv);
  
  //To stream depth and player index data:
  // • The options must include UseDepthAndPlayerIndex.
  // • Valid resolutions for depth and player index data are Resolution320x240 and Resolution80x60.
  // • The only valid image type is DepthAndPlayerIndex.
  _kinectNui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex);
  
  lastTime = DateTime.Now;
  
  _kinectNui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(NuiVideoFrameReady);
  _kinectNui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);
  
  
  }
  catch (InvalidOperationException ex)
  {
  MessageBox.Show(ex.Message);
  }
  }
  
  
  
  void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
  {
  
  var Image = e.ImageFrame.Image;
  var convertedDepthFrame = convertDepthFrame(Image.Bits);
  
  depth.Source = BitmapSource.Create(
  Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, convertedDepthFrame, Image.Width * 4);
  
  CalculateFps();
  //che them vo
  int height = e.ImageFrame.Image.Height;
  int width = e.ImageFrame.Image.Width;
  
  //Depth data for each pixel
  Byte[] depthData = Image.Bits;
  
  
  var depthIndex = 0;
  for (var y = 0; y < height; y++)
  {
  SerialPort comport = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);
  var heightOffset = y * width;
  
  for (var x = 0; x < width; x++)
  {
  
  var index = ((width - x - 1) + heightOffset) * 4;
  
  //var distance = GetDistance(depthData[depthIndex], depthData[depthIndex + 1]);
  var distance = GetDistanceWithPlayerIndex(depthData[depthIndex], depthData[depthIndex + 1]);
  //jump two bytes at a time
  depthIndex += 2;
  if (distance > 0 && distance < 4000)
  {
  // Thread.Sleep(50);
  if (comport.IsOpen)
  comport.Close();
  comport.Open();
  comport.WriteLine("F");
  if (distance <= 850)
  {
  comport.WriteLine("S");
  
  }
  if (distance > 850 && distance < 1200)
  {
  comport.WriteLine("R");
  if (distance <= 850)
  {
  comport.WriteLine("B");
  comport.WriteLine("L");
  }
  
  
  }
  
  if (1100 < distance)
  {
  comport.WriteLine("F");
  }
  }
  
  }
  }
  
  
  }
  
  private int GetDistanceWithPlayerIndex(byte firstFrame, byte secondFrame)
  {
  //offset by 3 in first byte to get value after player index
  
  int distance = (int)(firstFrame >> 3 | secondFrame << 5);
  return distance;
  
  
  
  }
  
  
  // Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame
  // that displays different players in different colors
  byte[] convertDepthFrame(byte[] depthFrame16)
  {
  for (int i16 = 0, i32 = 0; i16 < depthFrame16.Length && i32 < depthFrame32.Length; i16 += 2, i32 += 4)
  {
  int player = depthFrame16[i16] & 0x07;
  int realDepth = (depthFrame16[i16 + 1] << 5) | (depthFrame16[i16] >> 3);
  // transform 13-bit depth information into an 8-bit intensity appropriate
  // for display (we disregard information in most significant bit)
  byte intensity = (byte)(255 - (255 * realDepth / 0x0fff));
  
  depthFrame32[i32 + RED_IDX] = intensity;
  depthFrame32[i32 + BLUE_IDX] = intensity;
  depthFrame32[i32 + GREEN_IDX] = intensity;
  }
  return depthFrame32;
  
  }
  
  private void WindowLoaded(object sender, RoutedEventArgs e)
  {
  InitializeNui();
  }
  
  void CalculateFps()
  {
  ++totalFrames;
  
  var cur = DateTime.Now;
  if (cur.Subtract(lastTime) > TimeSpan.FromSeconds(1))
  {
  int frameDiff = totalFrames - lastFrames;
  lastFrames = totalFrames;
  lastTime = cur;
  frameRate.Text = frameDiff.ToString() + " fps";
  }
  }
  
  void NuiVideoFrameReady(object sender, ImageFrameReadyEventArgs e)
  {
  PlanarImage Image = e.ImageFrame.Image;
  
  image.Source = BitmapSource.Create(
  Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null,
  Image.Bits, Image.Width * Image.BytesPerPixel);
  
  // imageCmyk32.Source = BitmapSource.Create(
  // Image.Width, Image.Height, 96, 96, PixelFormats.Cmyk32, null,
  //Image.Bits, Image.Width * Image.BytesPerPixel);
  }
  
  private void WindowClosed(object sender, EventArgs e)
  {
  _kinectNui.Uninitialize();
  }
  
  private void BtnStopClick(object sender, RoutedEventArgs e)
  {
  _kinectNui.Uninitialize();
  btnStop.IsEnabled = false;
  btnStart.IsEnabled = true;
  }
  
  private void BtnStartClick(object sender, RoutedEventArgs e)
  {
  InitializeNui();
  btnStop.IsEnabled = true;
  btnStart.IsEnabled = false;
  }
  
  private void START_Click(object sender, RoutedEventArgs e)
  {
  if (comport.IsOpen)
  comport.Close();
  comport.Open();
  
  label1.Content = " READY";
  }
  
  private void FORWAD_Click(object sender, RoutedEventArgs e)
  {
  comport.WriteLine("F");
  label1.Content = "FORWARD";
  }
  
  private void LEFT_Click(object sender, RoutedEventArgs e)
  {
  comport.WriteLine("L");
  label1.Content = "LEFT";
  }
  
  private void RIGHT_Click(object sender, RoutedEventArgs e)
  {
  comport.WriteLine("R");
  label1.Content = "RIGHT";
  }
  
  private void BACK_Click(object sender, RoutedEventArgs e)
  {
  comport.WriteLine("B");
  label1.Content = "BACK";
  }
  private void STOP_Click(object sender, RoutedEventArgs e)
  {
  comport.WriteLine("S");
  label1.Content = " STOP";
  }
  
  }
  }
  
  

• 

  Actually I couldn't understand this code one hundred percent...
  
  I just copied this into my code with Skeleton code but "image1.Source" made one error.

• 

  My question is out from this topic. Based on my browsing through the internet regarding the kinect. There are some website showing the need of the calibration mechanism between color and depth sensor before using kinect. They said purpose is to map the depth pixel with the color pixel obtain from kinect accurately. The question is, do i have to do that calibration mechanism? It is because, i already try this kinect SDK, i assume that depth and color image obtained from this SDK is accurately map. Thanks

• 

  I change the depth resolution to 640x480 by changing the "nui.DepthStream.Open(ImageStreamType.Depth, 2,ImageResolution.Resolution640x480,ImageType.DepthAndPlayerIndex)" , but no image appear in window. Even the DepthFrameReady is not triggered. Why is this happen?

• 

  I'm trying the SDK and sample projects and am seeing ghost images too like the 'IlikeDotNet' post of Oct. 29. Any ideas? I didn't see that on Dan's video.

• 

  hey guys! newbie here! need help.

  coding:

  image1.Source = BitmapSource.Create(image.Width, image.Height, 96, 96, PixelFormats.Bgr32, null,

  ColoredBytes, image.Width = PixelFormats.Bgr32.BitsPerPixel /;

   

  error: ArgumentException was unhandled

  Value does not fall within the expected range.

   

  what is it???

  need help

• 

  I am interested in getting the distance information but only for a particular item in the image (e.g. a target of a particular shape). Could you explain how to do this.
  
  Thank you.

• 

  Using processing, skelleton detection and drawing happened at about 30fps. Using this, skelleton detection is at about 5 to 10fps. On the same machine.
  
  Why is that so?

• 

  @LZH: hello.. i read your problem about freezing the video and i think i am facing the same problem now... have you succeeded to find out the solution to this problem ??

  thanks a lot,

• 

  Hi
  I tried the above code to get the depth data. But when I debug the code, after the debugger executed the following lines of code:
  //UseDepthAndPlayerIndex and UseSkeletalTracking
  nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseDepth);
  
  //register for event
  nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);
  
  //DepthAndPlayerIndex ImageType
  nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution80x60,
  ImageType.DepthAndPlayerIndex);
  
  The code terminates and the debugger never executed the following function
  void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
  
  And I got this error:
  “AccessViolationException was unhandled”
  “Attempted to read or write protected memory. This is often an indication that other memory is corrupt.”
  Do you guys know why it happens? What is the solution to this problem?
  thanks