Display Kinect color image containing only players (aka background removal)

Today's post is short and sweet and I'm hoping does two things.

One, I hope the snip helps those trying to solve this project or implement something like this

Two, I wanted to highlight the goodness you can find in the Kinect for Windows SDK Beta Forums. They are an outstanding resource to help you solve problems and for sharing your successes.

HOWTO: Display color image containing only players (aka background removal)

I wanted to render an image containing *just* recognized players while the rest of the image is transparent so interesting things could be drawn behind it. This turned out to be way trickier than I anticipated and a number of threads seem to suggest that other people are struggling with the same.

The code below takes in an Image control and a Runtime object. It assumes the runtime object has been initialized for color, depth, and player tracking. It hooks into the depth & color frame ready events, generates the image I described above, and then updates the given Image control with it.

There's plenty of room for improvement here (ensuring the depth and video timestamps are in sync, blending in a few surrounding pixels so the result doesn't look so pixelated, etc) but hopefully this will help others get a jump start.

Project Information URL: http://social.msdn.microsoft.com/Forums/en-US/kinectsdknuiapi/thread/24276df9-0e4c-49cb-92ca-b233599f94ba

Project Source URL: http://social.msdn.microsoft.com/Forums/en-US/kinectsdknuiapi/thread/24276df9-0e4c-49cb-92ca-b233599f94ba

[Code Snip Updated 7/20/2011]

Usage example:

            kinect = new Runtime();
            kinect.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseDepthAndPlayerIndex);
            kinect.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex);
            kinect.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
            this.Person.Source = kinect.CreateLivePlayerRenderer(); 

Complete code block:

    public static class RuntimeExtensions
    {
        public static WriteableBitmap CreateLivePlayerRenderer(this Runtime runtime)
        {
            if (runtime.DepthStream.Width == 0)
                throw new InvalidOperationException("Either open the depth stream before calling this method or use the overload which takes in the resolution that the depth stream will later be opened with.");
            return runtime.CreateLivePlayerRenderer(runtime.DepthStream.Width, runtime.DepthStream.Height);
        }
        public static WriteableBitmap CreateLivePlayerRenderer(this Runtime runtime, int depthWidth, int depthHeight)
        {
            PlanarImage depthImage = new PlanarImage();
            WriteableBitmap target = new WriteableBitmap(depthWidth, depthHeight, 96, 96, PixelFormats.Bgra32, null);
            var depthRect = new System.Windows.Int32Rect(0, 0, depthWidth, depthHeight);

            runtime.DepthFrameReady += (s, e) =>
            {
                depthImage = e.ImageFrame.Image;
                Debug.Assert(depthImage.Height == depthHeight && depthImage.Width == depthWidth);
            };

            runtime.VideoFrameReady += (s, e) =>
            {
                // don't do anything if we don't yet have a depth image
                if (depthImage.Bits == null) return;

                byte[] color = e.ImageFrame.Image.Bits;

                byte[] output = new byte[depthWidth * depthHeight * 4];

                // loop over each pixel in the depth image
                int outputIndex = 0;
                for (int depthY = 0, depthIndex = 0; depthY < depthHeight; depthY++)
                {
                    for (int depthX = 0; depthX < depthWidth; depthX++, depthIndex += 2)
                    {
                        // combine the 2 bytes of depth data representing this pixel
                        short depthValue = (short)(depthImage.Bits[depthIndex] | (depthImage.Bits[depthIndex + 1] << 8));

                        // extract the id of a tracked player from the first bit of depth data for this pixel
                        int player = depthImage.Bits[depthIndex] & 7;

                        // find a pixel in the color image which matches this coordinate from the depth image
                        int colorX, colorY;
                        runtime.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(
                            e.ImageFrame.Resolution,
                            e.ImageFrame.ViewArea,
                            depthX, depthY, // depth coordinate
                            depthValue,  // depth value
                            out colorX, out colorY);  // color coordinate

                        // ensure that the calculated color location is within the bounds of the image
                        colorX = Math.Max(0, Math.Min(colorX, e.ImageFrame.Image.Width - 1));
                        colorY = Math.Max(0, Math.Min(colorY, e.ImageFrame.Image.Height - 1));

                        output[outputIndex++] = color[(4 * (colorX + (colorY * e.ImageFrame.Image.Width))) + 0];
                        output[outputIndex++] = color[(4 * (colorX + (colorY * e.ImageFrame.Image.Width))) + 1];
                        output[outputIndex++] = color[(4 * (colorX + (colorY * e.ImageFrame.Image.Width))) + 2];
                        output[outputIndex++] = player > 0 ? (byte)255 : (byte)0;
                    }
                }
                target.WritePixels(depthRect, output, depthWidth * PixelFormats.Bgra32.BitsPerPixel / 8, 0);
            };
            return target;
        }
    }