Kinect, UWP and Windows 10 Anniversary Update
Wondering about using your Kinect for Windows v2 device with your new UWP app?
Here's some great news, and a demo of it in action, for you!
UWP apps today cannot access most of the data from a Kinect sensor. This post provides an overview of what we plan to deliver in the near future to begin bringing Kinect for Windows into the Universal Windows Platform (UWP) world.
With the Windows 10 Anniversary Update and a Kinect driver update, Kinect developers will be able to read RGB, infrared (IR), and depth data from the Kinect sensor. The Kinect sensor’s microphone array is already visible to the UWP audio and speech recognition APIs today (although not all features from the Kinect for Windows SDK are accessible to UWP APIs)—this remains unchanged.
Skeleton data is important to many Kinect apps. Through a supplemental SDK that will be available later this year, UWP developers will have access to skeletal poses. With the new capabilities in the Windows 10 Anniversary Update, developers can also use the existing Windows.Media.FaceAnalysis APIs with Kinect to detect and track faces.
Existing Kinect Win32 apps and SDK capabilities will not be affected by this new UWP support.
Details: how it works
Today, there are several sensors that can provide rich correlated data, such as RGB, IR, or depth information. To make such data available to app developers in a device-independent manner, we introduced the Windows.Media.Capture.Frames APIs, a set of extensions to Media Capture that adds frame-by-frame access to RGB, IR, and depth data, as well as sensor correlation to the traditional image and video capture features, all with a consistent, familiar programming model. In addition, this extended model can give UWP apps access to custom streams from a sensor.
In the case of Kinect for Windows, we plan to make use of the custom streams capability to deliver skeleton data from Kinect to UWP apps. This will require a small supplemental Kinect-specific SDK to decode the custom streams on the app side, which we’ll make available later this year.
In summary, these are the elements that enable UWP apps to access Kinect:
- The new Windows.Media.Capture.Frames extensions to Media Capture are available in the Windows 10 SDK preview, build 14332 or newer. These API extensions are device independent. To access IR and depth data, a compatible device and a matching device driver are also required.
- A matching driver update for Kinect will be available later this spring.
- To access skeleton data from Kinect, which is delivered through Windows.Media.Capture as a custom stream, a Kinect-specific supplemental SDK is needed, which we plan to release in the second half of 2016.
Join the Windows Insider Program so that you can access Windows 10 build 14332 and its preview SDK. Watch the Channel 9 video for a brief preview of the full capabilities that will be available when the Windows 10 Anniversary SDK and the supplemental Kinect SDK are released to the public, including full skeleton support. The video also tours the code for a simple demo app that demonstrates how to get frames from Kinect or any other RGB camera on your system.
In this preview, Senior Program Manager Alex Turner demonstrates the new Media Capture Frames API that will be available as part of the Windows 10 Anniversary Update this summer. This API will provide a unified way for desktop apps and UWP apps to access color, infrared, depth, and other frame-based data from the Kinect sensor and other compatible sensor devices.