Camera Fundamentals (Beta 2 SDK)

This content is no longer current. Our recommendation for up to date content: http://channel9.msdn.com/Series/KinectQuickstart/Working-with-Depth-Data
Please use the newly updated Kinect for Windows SDK Quickstart series. The content below will only work with the Beta 2 version of the Kinect for Windows SDK.
This video covers the basics of using depth data from Kinect. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides that have been updated for Beta 2 (Nov, 2011).
The steps below assume you have setup your development environment as explained in the "Setting Up Your Development Environment" video.
Add an image with Width=320 and Height=240 as shown below:
XAML
<Image Height="240" HorizontalAlignment="Left" Margin="62,41,0,0" Name="image1" Stretch="Fill" VerticalAlignment="Top" Width="320" />
Create an instance of the Kinect Runtime outside of the Window_Loaded event. Then, initialize the Runtime to use DepthAndPlayerIndex and UseSkeletalTracking. Finally register for DepthFrameReady event and open the Depth stream as shown below.
C#
//Kinect Runtime Runtime nui = new Runtime(); private void Window_Loaded(object sender, RoutedEventArgs e) { //UseDepthAndPlayerIndex and UseSkeletalTracking nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking); //register for event nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady); //DepthAndPlayerIndex ImageType nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex); } void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e) { }
Visual Basic
Private nui As New Runtime Private Sub Window_Loaded(ByVal sender As Object, ByVal e As RoutedEventArgs) 'UseDepthAndPlayerIndex and UseSkeletalTracking nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex Or RuntimeOptions.UseSkeletalTracking) 'register for event AddHandler nui.DepthFrameReady, AddressOf nui_DepthFrameReady 'DepthAndPlayerIndex ImageType nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex) End Sub Private Sub nui_DepthFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs) End Sub
The DepthFrameReady event returns an ImageFrame class that contains a PlanarImage. That PlanarImage contains a byte[] array which contains the distance of each pixel. The array:
Because there are 2 bytes per pixel (16 bits) representing the distance, you will need to use bitshift operators to get the distance for a particular pixel. The exact bitshifting depends on the ImageType you use
Depth Image Type
Bitshift second byte left by 8
Distance (0,0) = (int)(Bits[0] | Bits[1] << 8 );
DepthAndPlayerIndex Image Type
Bitshift first byte right by 3 (to remove player index), and second byte left by 5
Distance (0,0) =(int)(Bits[0] >> 3 | Bits[1] << 5);
The method below shows returning the distance in millimeters using the 2 bytes for a particular pixel.
C#
private int GetDistanceWithPlayerIndex(byte firstFrame, byte secondFrame) { //offset by 3 in first byte to get value after player index int distance = (int)(firstFrame >> 3 | secondFrame << 5); return distance; }
Visual Basic
Private Function GetDistanceWithPlayerIndex(ByVal firstFrame As Byte, ByVal secondFrame As Byte) As Integer 'offset by 3 in first byte to get value after player index Dim distance As Integer = (CInt(firstFrame) >> 3 Or CInt(secondFrame) << 5) Return distance End Function
The DepthAndPlayerIndex image type is offset by 3 since the first three bits represent the Player Index. The Player Index has up to six possible values.
Note: While depth data is available for multiple players, you can only have skeletal/joint data for a maximum of two players.
To get a player index, we'll use the following formula. Since the player index is only in the first byte, we do not need the pixels second byte.
C#
private static int GetPlayerIndex(byte firstFrame) { //returns 0 = no player, 1 = 1st player, 2 = 2nd player... return (int)firstFrame & 7; }
Visual Basic
Private Shared Function GetPlayerIndex(ByVal firstFrame As Byte) As Integer 'returns 0 = no player, 1 = 1st player, 2 = 2nd player... 'bitwise & on firstFrame Return CInt(firstFrame) And 7 End Function
Now that we know the distance for a particular pixel using the formula above, we can loop through all of the bytes in the byte[] array and dynamically assign colors to pixels based on their distance.
To begin with, we'll build a colorFrame byte[] array using the Bgr (Blue, Green, Red) pixel format. What this means is that each pixel (ex: the pixel at 0,0) is represented by four bytes, one each for Blue, Green, Red, and an empty one (or transparency byte if you choose the Bgra format).
For example, the snippet below would color the pixel blue if the current distance for that pixel was < 900 millimeters from the Kinect. To set the color to blue, we set the first byte (BlueIndex) to the maximum value, 255.
C#
if (distance <= 900) { //we are very close colorFrame[index + BlueIndex] = 255; colorFrame[index + GreenIndex] = 0; colorFrame[index + RedIndex] = 0; }
Visual Basic
If distance <= 900 Then 'we are very close colorFrame(index + BlueIndex) = 255 colorFrame(index + GreenIndex) = 0 colorFrame(index + RedIndex) = 0 End If
Similarly, we can use the code snippet below to check if the current pixel represents a Player Index. If it does, we'll set the maximum value for both green and red which produces a yellow/gold-like color.
C#
if (GetPlayerIndex(depthData[depthIndex]) > 0) { //we are the farthest colorFrame[index + BlueIndex] = 0; colorFrame[index + GreenIndex] = 255; colorFrame[index + RedIndex] = 255; }
Visual Basic
If GetPlayerIndex(depthData(depthIndex)) > 0 Then 'we are the farthest colorFrame(index + BlueIndex) = 0 colorFrame(index + GreenIndex) = 255 colorFrame(index + RedIndex) = 255 End If
While the previous examples we hard-coded colors, you can also build a monochrome histogram that scales the color range (0-255) proportionally based on the distance range (850-4000).
To get a monochrome appearance, the intensity is applied equally to all three colors (blue, green, red).
C#
//equal coloring for monochromatic histogram var intensity = CalculateIntensityFromDepth(distance); colorFrame[index + BlueIndex] = intensity; colorFrame[index + GreenIndex] = intensity; colorFrame[index + RedIndex] = intensity; const float MaxDepthDistance = 4000; // max value returned const float MinDepthDistance = 850; // min value returned const float MaxDepthDistanceOffset = MaxDepthDistance - MinDepthDistance; public static byte CalculateIntensityFromDepth(int distance) { //formula for calculating monochrome intensity for histogram return (byte)(255 - (255 * Math.Max(distance - MinDepthDistance, 0) / (MaxDepthDistanceOffset))); }
Visual Basic
'equal coloring for monochromatic histogram Dim intensity = CalculateIntensityFromDepth(distance) colorFrame(index + BlueIndex) = intensity colorFrame(index + GreenIndex) = intensity colorFrame(index + RedIndex) = intensity Private Const MaxDepthDistance As Single = 4000 ' max value returned Private Const MinDepthDistance As Single = 850 ' min value returned Private Const MaxDepthDistanceOffset As Single = MaxDepthDistance - MinDepthDistance Public Shared Function CalculateIntensityFromDepth(ByVal distance As Integer) As Byte 'formula for calculating monochrome intensity for histogram Return CByte(255 - (255 * Math.Max(distance - MinDepthDistance, 0) / (MaxDepthDistanceOffset))) End Function
Using the formulas above, we'll now return a byte[] array that has a color value for each pixel based on if the distance for that pixel:
If the pixel represents a player, the color will be yellow/gold.
C#
private byte[] GenerateColoredBytes(ImageFrame imageFrame) { int height = imageFrame.Image.Height; int width = imageFrame.Image.Width; //Depth data for each pixel Byte[] depthData = imageFrame.Image.Bits; //colorFrame contains color information for all pixels in image //Height x Width x 4 (Red, Green, Blue, empty byte) Byte[] colorFrame = new byte[imageFrame.Image.Height * imageFrame.Image.Width * 4]; //Bgr32 - Blue, Green, Red, empty byte //Bgra32 - Blue, Green, Red, transparency //You must set transparency for Bgra as .NET defaults a byte to 0 = fully transparent //hardcoded locations to Blue, Green, Red (BGR) index positions const int BlueIndex = 0; const int GreenIndex = 1; const int RedIndex = 2; var depthIndex = 0; for (var y = 0; y < height; y++) { var heightOffset = y * width; for (var x = 0; x < width; x++) { var index = ((width - x - 1) + heightOffset) * 4; //var distance = GetDistance(depthData[depthIndex], depthData[depthIndex + 1]); var distance = GetDistanceWithPlayerIndex(depthData[depthIndex], depthData[depthIndex + 1]); if (distance <= 900) { //we are very close colorFrame[index + BlueIndex] = 255; colorFrame[index + GreenIndex] = 0; colorFrame[index + RedIndex] = 0; } else if (distance > 900 && distance < 2000) { //we are a bit further away colorFrame[index + BlueIndex] = 0; colorFrame[index + GreenIndex] = 255; colorFrame[index + RedIndex] = 0; } else if (distance > 2000) { //we are the farthest colorFrame[index + BlueIndex] = 0; colorFrame[index + GreenIndex] = 0; colorFrame[index + RedIndex] = 255; } //Color a player if (GetPlayerIndex(depthData[depthIndex]) > 0) { //we are the farthest colorFrame[index + BlueIndex] = 0; colorFrame[index + GreenIndex] = 255; colorFrame[index + RedIndex] = 255; } //jump two bytes at a time depthIndex += 2; } } return colorFrame; }
Visual Basic
Private Function GenerateColoredBytes(ByVal imageFrame As ImageFrame) As Byte() Dim height As Integer = imageFrame.Image.Height Dim width As Integer = imageFrame.Image.Width 'Depth data for each pixel Dim depthData() As Byte = imageFrame.Image.Bits 'colorFrame contains color information for all pixels in image 'Height x Width x 4 (Red, Green, Blue, empty byte) Dim colorFrame(imageFrame.Image.Height * imageFrame.Image.Width * 4 - 1) As Byte 'Bgr32 - Blue, Green, Red, empty byte 'Bgra32 - Blue, Green, Red, transparency 'You must set transparency for Bgra as .NET defaults a byte to 0 = fully transparent 'hardcoded locations to Blue, Green, Red (BGR) index positions Const BlueIndex As Integer = 0 Const GreenIndex As Integer = 1 Const RedIndex As Integer = 2 Dim depthIndex = 0 For y = 0 To height - 1 Dim heightOffset = y * width For x = 0 To width - 1 Dim index = ((width - x - 1) + heightOffset) * 4 Dim distance = GetDistanceWithPlayerIndex(depthData(depthIndex), depthData(depthIndex + 1)) If distance <= 900 Then 'we are very close colorFrame(index + BlueIndex) = 255 colorFrame(index + GreenIndex) = 0 colorFrame(index + RedIndex) = 0 ElseIf distance > 900 AndAlso distance < 2000 Then 'we are a bit further away colorFrame(index + BlueIndex) = 0 colorFrame(index + GreenIndex) = 255 colorFrame(index + RedIndex) = 0 ElseIf distance > 2000 Then 'we are the farthest colorFrame(index + BlueIndex) = 0 colorFrame(index + GreenIndex) = 0 colorFrame(index + RedIndex) = 255 End If '//Color a player If GetPlayerIndex(depthData(depthIndex)) > 0 Then 'we are the farthest colorFrame(index + BlueIndex) = 0 colorFrame(index + GreenIndex) = 255 colorFrame(index + RedIndex) = 255 End If 'jump two bytes at a time depthIndex += 2 Next x Next y Return colorFrame End Function
These videos and the accompanying material have been very helpful, Dan. However, concerning your comments at [1:45], I was under the assumption that the IR sensor does not detect depth via time-of-flight. Would you mind clarifying this?
Thanks
@Alec:Thanks for pointing this out, it's my goof , I just corrected the first video in the series to clarify this too
Is it just me or is the depth data and video data mirror images of each other? (That is, one goes left to right and the goes right to left)
Make is kinda hard to mix the two...
Hi Ben,
I got the same issue. I changed the loop inside the GenerateColoredBytes-method from
for (var x = 0; x < width; x++)
to
for (var x = width; x > 0; x--)
Now the images are displayed correctly!
I was wondering the same thing. Why is the image mirrored?
Also, I don't get why the index in GenerateColoredBytes is calculated as ((width - x - 1) + heightOffset) * 4. Why does it start off at 1276 instead of 0?
Edit: instead of Peter Nijs's solution of making the x iteration go backwards, I changed the line
var index = ((width - x - 1) + heightOffset) * 4;
to
var index = (x + heightOffset) * 4;
Now the image is no longer mirrorred, and your x iteration can still go from 0 to width-1. The index calculation makes more sense to me this way anyway, because basically all you need is the heightOffset plus the current column, taking into account that it's four bytes per pixel. Hence, (x + heightOffset) * 4.
Hi Bas,
your solution does make more sense than mine!
Thanks!
The issue is if you do depth versus depth with player index. Coding4Fun toolkit's ToBitmapSource functions take this flipping effect into account for the image types for depth.
The above video guide included the method for nui_DepthFrameReady, but the text guide did not. It should read as follows:
void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
// Convert depth information for a pixel into color information
byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);
// create an image based on the colored bytes
PlanarImage image = e.ImageFrame.Image;
image1.Source = BitmapSource.Create(
image.Width, image.Height, 96, 96, PixelFormats.Bgr32,
null, ColoredBytes, image.Width * PixelFormats.Bgr32.BitsPerPixel / 8);
}
Sorry for the strange formatting. Channel9 does not support multiline code postings in the comments.
Does anyone encounter problem with GenerateColoredBytes(ImageFrame imageFrame) ?
The moment i put in this method, my video frame seems to stop working.
Im trying the depth data sample by setting a blue.red and green color depth screen.
But my current project need me to show the distance value for player that been tracked.
when i use this code below to execute, the programme will stuck without error...can somebody explain to me why?
thxxxxxx a lot
if (GetPlayerIndex(depthData[depthIndex]) >0)
{
colorFrame[index + BlueIndex] = 0;
colorFrame[index + GreenIndex] = 255;
colorFrame[index + RedIndex] = 255;
txtdistance.Text = distance.ToString() + " mm";
}
@wizardliu: betting it doesn't get stuck, it is more you're overloading the user interface thread. Looking at what your code bits look like, you're inside the double for loop statement.
commenting out: txtdistance.Text = distance.ToString() + " mm";
I think will resolve your issue.
You're updating that text element 76800 times in theory at 30 frames per second. Set a varible outside the double for loop at the top and then set the text box's text property outside the loops on the bottom. I'll bet that will work a lot better. One thing to not then that will give you the right most point in the lowest row with a player index then since the value will always be overwritten.
@LZH: can you share a bit more? the samples download runs as expected
I am trying to uses the depth information to implement collision avoidance. My idea is to calculate the lowest depth in each column of pixels and save it into an array closeestDistance[].with that i can turn to the direction with the largest distance value when the center is below a certain threshold. I use the coding4fun.kinect.wpf... I am getting some weird findings on testing. see code:
short resX = 320; short resY = 120;
short[] closestDistances = newshort[320]; // setting the size of the array
void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgse)
{
shorti,j, least = 4000; //just using the 4000 for starters
image2.Source = e.ImageFrame.ToBitmapSource();
image2.Source = e.ImageFrame.ToBitmapSource();
short[][] distances = e.ImageFrame.ToDepthArray2D();
for(i = 0; i < resX; i++)
{
for(j = 0; j < resY; j++)
{
if((distances [i][j] < least) && (distances[i][j] != 0))
{
least = distances[i][j];
}
}
closestDistances[i] = least;
}
textBox1.Text = closestDistances[8].ToString();
textBox2.Text = closestDistances[80].ToString();
textBox3.Text = closestDistances[160].ToString();
textBox4.Text = closestDistances[240].ToString();
textBox5.Text = closestDistances[319].ToString();
}
Code ends: Now if you run this code with all the initilisation and event handler depthandplayerindex and openstream etc.... When its runs and "an" obstacle is played close enough relatively to the corresponding directions of the pixels 8,80,160,240,319... when placed on the left of the image ie. right of kinect facing forward direction (8 or 80)... Its seems to change the other values in the other texboxes to the same value (ie textboxes for closestDistances -> 160, 240, 319). Meanwhile this other textboxes should be showing higher values. Please help, run the code and place an obstacle in different sectors to understand my question. Or maybe there is a problem with my algorithm
Ben i'm trying your code without any error during building process. But, the image doesn't not appear. I'm only get an empty window. After debug the code, it seem that the colorFrame array contain valid value. Maybe there are some problem in function BitmapSource Create() because i create my own byte array to pass to this function. Still, nothing appear except an empty window. Do you know why?
i already find my mistake. Actually for BitmapSource.Create() function, i shall put PixelFormat Bgr32 not Bgra32. That's why the image doesn't not appear. Thanks a lot
@amir: if you are using Bgra32, you can set the 4th byte (aka index + 3) to 255 and your image will appear at well.
Hello, sorry if the question is silly, I'm new at this. Let me know if this example can be used for a web application. I need to know if I can put the video online of what the camera captures.
Great informative presentation.
The only thing I would like to suggest is to think about what the spectator sees when you indicate "left to right" with your hands (1:30).
What the spectator actually sees is "right to left" :)
I am doing a project with a friend using kinect. We need to stand in front of the Kinect, calibrate, make movements etc to debug the application. We bought big screen monitor + wireless keyboard etc.. but still tough to debug. Is there any good way to debug kinect application?
I need to extract this image data to any other form like: matlab, to work on it. is that possible? also is there possible way to display image depth in certain region only not every where in camera field of view?
I'm having trouble displaying anything. I've copied the code (mostly, I'm just using the depth stream as opposed to depth and player index) but when I run my program, just a blank window. I haven't installed the coding4fun dll, I'm wondering if some of the methods in there for dealing with bitmaps is what I'm missing. I notice that nowhere (as far as I can tell) in this code is the GenerateColoredBytes function actually called and I suspect this may be the root of my problem, but I am noob with WPF and using images to show "video" streams. Any pointers?
If my explanation sucks, I'll be happy to post code.
Thanks guys!
@Rami:
in terms of showing only a portion (or region) of the frame, you could make your two loops (one for columns and one for rows) only loop through the pixels contained in the specific region you're looking for. If you use BGRa then any pixels outside of this region could be left as transparent, although you'd have to make sure that the pixels WITHIN the region were being set to be opaque.
What you'd end up with would be an image the had transparent borders. This may be an inelegant solution but it may, depending on how you were going to use the data, be sufficient
guys,
I need to send the distance information to txt file, I can't figure that out. I am using kinect with c# wpf application. I used the same sample above, but I do not know how to make loop and send all the depth information (320X240) to .txt file , then I want to define one object and get distance for just one objects?
Any help, please?
Guys,
Could you, please, explain why there is a "golden" chair when the expectation is just a "golden" player?
PS. Could you, please, post a starting point (sample/lab/docs) to build a 3D model.
Thank you!
I'm trying to replicate this project using winform and a picturebox instead of an image. I'm running into trouble with
<code>
1
void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
1
{
1
// Convert depth information for a pixel into color information
1
byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);
1
// create an image based on the colored bytes
1
PlanarImage image = e.ImageFrame.Image;
1
image1.Source = BitmapSource.Create(
1
image.Width, image.Height, 96, 96, PixelFormats.Bgr32,
1
null, ColoredBytes, image.Width * PixelFormats.Bgr32.BitsPerPixel / 8);
</code>
specifically the BitmapSource and PixelFormats. Neither of these exist in the current context. I've manually added in the using statements that these need, but then I get errors saying that those using statements aren't valid dlls. What I've gathered is that these are used specifically in WPF. If that is true, what would be the winform equivalents?
Thanks a bunch for any feedback
checks mi error
PixelFormats.Bgr32
is not the same like PixelFormat.
checks the 's'
This is really useful for starting off. From what I see, each pixel in the depth map has details about its z-depth value. Do you mind including a transform algorithm that also translates the pixel x-y (or row-column) into real world x-y (in mm)? What I am interested in is that given a any pixel I should be able to tell where on the view frustum it lies.
Is anyone getting ghost / double images with above source for "Working with Depth Data" example? Just cross checked that even with default gray scale depth Image, I see about 10 fingers for my left hand on Depth Image. Color Video Stream is fine, no doubles there.
i'm beginer
i want to use depth map to calculate distance and control my mobile robot.but my routine have some problem.plz help me
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Markup;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Microsoft.Research.Kinect.Nui;
using System.IO.Ports;
using System.Threading;
using Coding4Fun.Kinect.Wpf;
namespace KinectGettingStarted
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
Runtime _kinectNui;
const int RED_IDX = 2;
const int GREEN_IDX = 1;
const int BLUE_IDX = 0;
byte[] depthFrame32 = new byte[320 * 240 * 4];
int totalFrames = 0;
int lastFrames = 0;
DateTime lastTime = DateTime.MaxValue;
SerialPort comport = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);
public MainWindow()
{
InitializeComponent();
}
private void InitializeNui()
{
try
{
//• Declares _kinectNui as a Runtime object, which represents the Kinect sensor instance.
_kinectNui = new Runtime();
//Open the video and depth streams, and sets up the event handlers that the runtime calls when a video, depth, or skeleton frame is ready
//An application must initialize the Kinect sensor by calling Runtime.Initialize before calling any other methods on the Runtime object.
_kinectNui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex |RuntimeOptions.UseDepth|RuntimeOptions.UseSkeletalTracking|
RuntimeOptions.UseColor);
//To stream color images:
// • The options must include UseColor.
// • Valid image resolutions are Resolution1280x1024 and Resolution640x480.
// • Valid image types are Color, ColorYUV, and ColorYUVRaw.
_kinectNui.VideoStream.Open(ImageStreamType.Video, 2,
ImageResolution.Resolution640x480, ImageType.ColorYuv);
//To stream depth and player index data:
// • The options must include UseDepthAndPlayerIndex.
// • Valid resolutions for depth and player index data are Resolution320x240 and Resolution80x60.
// • The only valid image type is DepthAndPlayerIndex.
_kinectNui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex);
lastTime = DateTime.Now;
_kinectNui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(NuiVideoFrameReady);
_kinectNui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);
}
catch (InvalidOperationException ex)
{
MessageBox.Show(ex.Message);
}
}
void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
var Image = e.ImageFrame.Image;
var convertedDepthFrame = convertDepthFrame(Image.Bits);
depth.Source = BitmapSource.Create(
Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, convertedDepthFrame, Image.Width * 4);
CalculateFps();
//che them vo
int height = e.ImageFrame.Image.Height;
int width = e.ImageFrame.Image.Width;
//Depth data for each pixel
Byte[] depthData = Image.Bits;
var depthIndex = 0;
for (var y = 0; y < height; y++)
{
SerialPort comport = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);
var heightOffset = y * width;
for (var x = 0; x < width; x++)
{
var index = ((width - x - 1) + heightOffset) * 4;
//var distance = GetDistance(depthData[depthIndex], depthData[depthIndex + 1]);
var distance = GetDistanceWithPlayerIndex(depthData[depthIndex], depthData[depthIndex + 1]);
//jump two bytes at a time
depthIndex += 2;
if (distance > 0 && distance < 4000)
{
// Thread.Sleep(50);
if (comport.IsOpen)
comport.Close();
comport.Open();
comport.WriteLine("F");
if (distance <= 850)
{
comport.WriteLine("S");
}
if (distance > 850 && distance < 1200)
{
comport.WriteLine("R");
if (distance <= 850)
{
comport.WriteLine("B");
comport.WriteLine("L");
}
}
if (1100 < distance)
{
comport.WriteLine("F");
}
}
}
}
}
private int GetDistanceWithPlayerIndex(byte firstFrame, byte secondFrame)
{
//offset by 3 in first byte to get value after player index
int distance = (int)(firstFrame >> 3 | secondFrame << 5);
return distance;
}
// Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame
// that displays different players in different colors
byte[] convertDepthFrame(byte[] depthFrame16)
{
for (int i16 = 0, i32 = 0; i16 < depthFrame16.Length && i32 < depthFrame32.Length; i16 += 2, i32 += 4)
{
int player = depthFrame16[i16] & 0x07;
int realDepth = (depthFrame16[i16 + 1] << 5) | (depthFrame16[i16] >> 3);
// transform 13-bit depth information into an 8-bit intensity appropriate
// for display (we disregard information in most significant bit)
byte intensity = (byte)(255 - (255 * realDepth / 0x0fff));
depthFrame32[i32 + RED_IDX] = intensity;
depthFrame32[i32 + BLUE_IDX] = intensity;
depthFrame32[i32 + GREEN_IDX] = intensity;
}
return depthFrame32;
}
private void WindowLoaded(object sender, RoutedEventArgs e)
{
InitializeNui();
}
void CalculateFps()
{
++totalFrames;
var cur = DateTime.Now;
if (cur.Subtract(lastTime) > TimeSpan.FromSeconds(1))
{
int frameDiff = totalFrames - lastFrames;
lastFrames = totalFrames;
lastTime = cur;
frameRate.Text = frameDiff.ToString() + " fps";
}
}
void NuiVideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
PlanarImage Image = e.ImageFrame.Image;
image.Source = BitmapSource.Create(
Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null,
Image.Bits, Image.Width * Image.BytesPerPixel);
// imageCmyk32.Source = BitmapSource.Create(
// Image.Width, Image.Height, 96, 96, PixelFormats.Cmyk32, null,
//Image.Bits, Image.Width * Image.BytesPerPixel);
}
private void WindowClosed(object sender, EventArgs e)
{
_kinectNui.Uninitialize();
}
private void BtnStopClick(object sender, RoutedEventArgs e)
{
_kinectNui.Uninitialize();
btnStop.IsEnabled = false;
btnStart.IsEnabled = true;
}
private void BtnStartClick(object sender, RoutedEventArgs e)
{
InitializeNui();
btnStop.IsEnabled = true;
btnStart.IsEnabled = false;
}
private void START_Click(object sender, RoutedEventArgs e)
{
if (comport.IsOpen)
comport.Close();
comport.Open();
label1.Content = " READY";
}
private void FORWAD_Click(object sender, RoutedEventArgs e)
{
comport.WriteLine("F");
label1.Content = "FORWARD";
}
private void LEFT_Click(object sender, RoutedEventArgs e)
{
comport.WriteLine("L");
label1.Content = "LEFT";
}
private void RIGHT_Click(object sender, RoutedEventArgs e)
{
comport.WriteLine("R");
label1.Content = "RIGHT";
}
private void BACK_Click(object sender, RoutedEventArgs e)
{
comport.WriteLine("B");
label1.Content = "BACK";
}
private void STOP_Click(object sender, RoutedEventArgs e)
{
comport.WriteLine("S");
label1.Content = " STOP";
}
}
}
Actually I couldn't understand this code one hundred percent...
I just copied this into my code with Skeleton code but "image1.Source" made one error.
My question is out from this topic. Based on my browsing through the internet regarding the kinect. There are some website showing the need of the calibration mechanism between color and depth sensor before using kinect. They said purpose is to map the depth pixel with the color pixel obtain from kinect accurately. The question is, do i have to do that calibration mechanism? It is because, i already try this kinect SDK, i assume that depth and color image obtained from this SDK is accurately map. Thanks
I change the depth resolution to 640x480 by changing the "nui.DepthStream.Open(ImageStreamType.Depth, 2,ImageResolution.Resolution640x480,ImageType.DepthAndPlayerIndex)" , but no image appear in window. Even the DepthFrameReady is not triggered. Why is this happen?
I'm trying the SDK and sample projects and am seeing ghost images too like the 'IlikeDotNet' post of Oct. 29. Any ideas? I didn't see that on Dan's video.
hey guys! newbie here! need help.
coding:
image1.Source = BitmapSource.Create(image.Width, image.Height, 96, 96, PixelFormats.Bgr32, null,
ColoredBytes, image.Width = PixelFormats.Bgr32.BitsPerPixel /;
error: ArgumentException was unhandled
Value does not fall within the expected range.
what is it???
need help
I am interested in getting the distance information but only for a particular item in the image (e.g. a target of a particular shape). Could you explain how to do this.
Thank you.
Using processing, skelleton detection and drawing happened at about 30fps. Using this, skelleton detection is at about 5 to 10fps. On the same machine.
Why is that so?
@LZH: hello.. i read your problem about freezing the video and i think i am facing the same problem now... have you succeeded to find out the solution to this problem ??
thanks a lot,
Hi
I tried the above code to get the depth data. But when I debug the code, after the debugger executed the following lines of code:
//UseDepthAndPlayerIndex and UseSkeletalTracking
nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseDepth);
//register for event
nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);
//DepthAndPlayerIndex ImageType
nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution80x60,
ImageType.DepthAndPlayerIndex);
The code terminates and the debugger never executed the following function
void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
And I got this error:
“AccessViolationException was unhandled”
“Attempted to read or write protected memory. This is often an indication that other memory is corrupt.”
Do you guys know why it happens? What is the solution to this problem?
thanks