Today's post from Carmine Sirignano, a Developer Support Escalation Engineer on the Kinect for Windows talks in detail on how the Kinect for Windows Face Tracking actually works, providing some cool code samples too...
Since the release of Kinect for Windows version 1.5, developers have been able to use the Face Tracking software development kit (SDK) to create applications that can track human faces in real time. Figure 1, an illustration from the Face Tracking documentation, displays 87 of the points used to track the face. Thirteen points are not illustrated here—more on those points later.
You have questions...
Based on feedback we received via comments and forum posts, it is clear there is some confusion regarding the face tracking points and the data values found when using the SDK sample code. The managed sample, FaceTrackingBasics-WPF, demonstrates how to visualize mesh data by displaying a 3D model representation on top of the color camera image.
By exploring this sample source code, you will find a set of helper functions under the Microsoft.Kinect.Toolkit.FaceTracking project, in particular GetProjected3DShape(). What many have found was the function returned a collection where the length of the array was 121 values. Additionally, some have also found an enum list, called “FeaturePoint”, that includes 70 items.
We have answers...
As you can see, we have two main sets of numbers that don't seem to add up. This is because these are two sets of values that are provided by the SDK:
- 3D Shape Points (mesh representation of the face): 121
- Tracked Points: 87 + 13
The 3D Shape Points (121 of them) are the mesh vertices that make a 3D face model based on the Candide-3 wireframe.
To get the 100 tracked points mentioned above, we need to dive more deeply into the APIs. The managed APIs, provide an FtInterop.cs file that defines an interface, IFTResult, which contains a Get2DShapePoints function. FtInterop is a wrapper for the native library that exposes its functionality to managed languages. Users of the unmanaged C++ API may have already seen this and figured it out. Get2DShapePoints is the function that will provide the 100 tracked points.
If we have a look at the function, it doesn’t seem to be useful to a managed code developer:
Pulling it all together...
As we have seen, there are two types of data points that are available from the Face Tracking SDK:
- Shape Points: data used to track the face
- Mesh Data: vertices of the 3D model from the GetProjected3DShape() function
- FeaturePoints: named vertices on the 3D model that play a significant role in face tracking
To get the shape point data, we have to extend the current managed wrapper with a new function that will handle the interop with the native API.