Kinect for Windows v2 Face Tracking Managed and Native
Peter Daukintis, Microsoft Technical Evangelist, recently posted about Face Tracking with the Kinect for Windows v2. What was a little different is that he posted both managed and native looks...
Since working on the virtual rail project I haven’t had much chance to carry out much in the way of Kinect programming. That all changes now since I ordered a Kinect 4 Windows v2 as I have been hacking away over the last few weeks and wanted to share my experiences here. I don’t intend to cover anything introductory so for that please see the Programming for Kinect for Windows v2 Jumpstart videos here https://channel9.msdn.com/Series/Programming-Kinect-for-Windows-v2 and also the blog series here http://mtaulty.com/CommunityServer/blogs/mike_taultys_blog/archive/tags/Kinect/default.aspx as Mike takes you right from opening the box to getting dancing skeletons and onwards to whichever direction it takes him.
I’m going to ease into this gently with a look into face tracking and I’m going to start out with Managed c# code within a Windows Store application but I suspect that this might change as I learn how best to translate the concepts over to native c++ code. I have been re-learning c++ for a while now and have taken the opportunity also to look into DirectX 11 so I expect a lot of my future samples to go in this direction.
For me, Reactive Extensions is a natural fit for Kinect programming in an event driven application as it removes the need to think about the mechanics of how to compose and orchestrate the associated event streams. A similar example to that above but using Rx shows how to subscribe to an Observable and shows how to specify which threading context the subscription and event delivery should occur on.
It is necessary to use the tracking ID of the body to start the face tracking and also I will need to get colour frames (so we can see the face). As a result I will use a MultiSourceFrameReader which I will ask to deliver body and colour frames in the same event (at the same time). The body data will contain the skeletons in the camera’s view but we just require to get a tracking ID from he body data which is required for face tracking. The colour frames will be copied into a WriteableBitmap for display.
For the Face frame handler I will hook up the data for the bounding box to a display rectangle and whether the face is happy and/or engaged with the camera will be represented by some simple ui controls.
Here’s a screenshot of the resulting sample running:
Project Information URL: http://peted.azurewebsites.net/face-tracking-kinect-4-windows-v2/
Project Download URL: http://1drv.ms/1uJ9bFV
Well, with the managed version working in c# (see Face Tracking Kinect 4 Windows V2 – Managed) it should be a breeze to convert the code to a native c++ windows store xaml app :).
Converting an event-based c#/xaml app seemed like a gentle introduction to a more native environment which I hope to get into later. Also, I was asked recently whether face tracking worked in a native app.
There are a few things I’ve become accustomed to in the managed development environment; nuget for one thing and also I’ve been using Reactive extensions in my Kinect projects. A quick Bing-search later and I found this:
Another difference I introduced to the native version was that I couldn’t find a way to schedule rx to deliver the Kinect events via the Thread Pool so I subscribed to the multi frame event from a Thread Pool thread and used the CurrentThreadScheduler. Aside from these minor differences the rest of the code looked pretty much the same.
Project Information URL: http://peted.azurewebsites.net/face-tracking-native-kinect-4-windows-v2/
Project Download URL: http://1drv.ms/1th57Lb.