Kinect for Windows v2 will make you green (with envy at its background removal features)
That last project in our v2 week comes from Vangos Pterneas where he shows off the new background removal, replacement capabilities...
Other recent Gallery Posts for Vangos Pterneas;
- Looking at the Kinect for Windows v2
- Color, depth and infrared streams in the Kinect for Windows v2 world (here's how)
- Kinect gestures, an implementation walk through
- Put down the measuring tape. Using the Kinect to find your height
- Body Tracking with the Kinect for Windows v2
Throughout the past few days, I got many requests about Kinect color to depth pixel mapping. As you probably already know, Kinect streams are not properly aligned. The RGB and depth cameras have a different resolution and their point of view is slightly shifted. As a result, more and more people have been asking me (either in the blog comments or by email) about properly aligning the color and depth streams. The most common application they want to build is a cool green-screen effect, just like the following video:
As you can see, the pretty girl is tracked by the Kinect sensor and the background is totally removed. I can replace the background with a solid color, a gradient fill, or even a random image!
Nice, huh? So, I created a simple project that maps a player’s depth values to the corresponding color pixels. This way, I could remove the background and replace it with something else. The source code is hosted on GitHub as a separate project. It is also part of Vitruvius.
How background removal works
When we refer to “background removal”, we need to keep the pixels which form the user and remove anything else that does not belong to the user. The depth camera of the Kinect sensor comes in handy for determining a user’s body. However, we need to find the RGB color values, not the depth distances. We need to specify which RGB values correspond to the user’s depth values. Confused? Please don’t.
Using Kinect, each point in space has the following information:
- Color value: Red + Green + Blue
- Depth value: The distance from the sensor
The depth camera gives us the depth value and the RGB camera provides us with the color value. We map those values using CoordinateMapper. CoordinateMapper is a useful Kinect property that determines which color values correspond to each depth distances (and vice-versa).
Please note that the RGB frames (1920×1080) are wider than the depth frames (512×424). As a result, not every color pixel has a corresponding depth mapping. However, body tracking is performed primarily using the depth sensor, so no need to worry about missing values.
In the GitHub project I shared, you can use the following code to remove the background and get the green-screen effect:
The BackgroundRemovalTool is part of Vitruvius, an open-source library that will speed-up the development of your Kinect projects. Vitruvius supports both version 1 and version 2 sensors, so you can use it for any kind of Kinect project. Download it and give it a try.
PS 2: New Kinect book - 20% off [Scroll down to the bottom of the post]
Project Information URL: http://pterneas.com/2014/04/11/kinect-background-removal/
Project Download URL: https://github.com/Vangos/kinect-2-background-removal
Project Source URL: https://github.com/Vangos/kinect-2-background-removal