Aligning the Kinect skeleton with video or depth images

The final post in out "Skeleton week" provides a final piece of the skeleton puzzle...

Aligning a Kinect Skeleton with the Video or Depth images

The most asked for snippet of code at our recent workshop that I didn’t have was how to properly align a Kinect skeleton to either the Video image or the depth image. It turns out this is non trivial and most samples that just scale all three items to fit the screen are wrong. I took the time this weekend to make a good sample for others to follow. If you just want the two snippets scroll to the very bottom of the article otherwise continue reading for a full walkthrough, it starts with Kinect Beta2 clean WPF code.

To make this super easy for you to drop into your project I’m going to put this scaling into my skeleton control and let you choose by setting a mode. In order to scale to the depth image or the video image you need to do up to 3 things:

1. Set the ScaleMode property to ToFill, ToDepth, ToVideo
2. Set reference to the Kinect Runtime being used, set property Runtime.
3. (Video only) We need the Depth PlanerImage, pass into method DepthImage.

...

Comments

As you can see it is not as simple or as clean as you like, a simple extension method to the Joint would be ideal, instead we need access to the Runtime itself to get the depth pixel and a depth image frame to get the colour pixel. Lastly during testing I found that 640x480 video with 320x240 depth worked fine, but up the depth to 640x480 and the whole thing was off, hence the fix, weird. You may have noticed this all works with/without the playerindex being captured in the depth stream but even with that off it fails for colour image about 640x480. I don’t see why this shouldn’t work with 1280x1024 image but it doesn’t. Let’s hope this all gets addressed for the next beta but until then feel free to use the code from this article in any form.

Project Information URL: http://www.soulsolutions.com.au/Blog/tabid/73/EntryId/770/Aligning-a-Kinect-Skeleton-with-the-Video-or-Depth-images.aspx

Project Source URL: http://www.soulsolutions.com.au/examples/videoskeleton.zip

Contact Information:

• Blog: http://www.soulsolutions.com.au/Blog
• Twitter: @soulsolutions

Page image, Lego Skeleton,courtesy of onathanb1989