Today's project is from Friend of the Gallery Vangos Pterneas responding to a question from the Net on how to use the Kinect to draw in the air...
A few days ago, I received an interesting comment from one of the fans of this blog. The fellow-developer was asking me about a good way to draw on top of a virtual canvas using his hands. So, today, I’ll show you how you can easily use a Kinect sensor to draw in the air! This video shows what we are about to build:
- Kinect for XBOX v2 sensor with an adapter (or Kinect for Windows v2 sensor)
- Kinect for Windows v2 SDK
- Windows 8.1 or higher
- Visual Studio 2013 or higher
- A dedicated USB 3 port
Video & Source Code
As usual, I’m providing you with the complete source code, as well as a demo video.
The XAML User Interface
First things first. Launch Visual Studio and create a new WPF or Windows Store project. Select .NET Framework 4.5 or higher.
If using WPF, add a reference to Microsoft.Kinect.dll.
If using Windows Store (WinRT), add a reference to WindowsPreview.Kinect.dll.
Navigate to the XAML code of your main page. Nothing fancy – we just want to display the feed of the Color camera and the drawing brush. We’ll use an Image component for the RGB stream and a Canvas component for the drawing brush. For proper scaling reasons, I’ve placed both the Image and a Canvas into a Viewbox element. The Viewbox element will automatically scale its contents according to the size of your screen.
The C# code
Our user interface is ready to accept the drawing functionality. Let’s get started. Navigate to your .xaml.cs file and add the C# code below.
Step 1 – Declare the required Kinect members
First things first! We need to specify the objects we’ll work with. If you are following this blog, you already know what you need:
Step 2 – Initialize Kinect
After you have declared the required members, you need to initialize the Kinect sensor and create the event handlers. You can do it in the constructor of your page/window.
Step 3 – Display the RGB Color stream
The event named “ColorReader_FrameArrived” will be called whenever Kinect has a new RGB frame. All we need to do is grab the frame and transform it into a WriteableBitmap. The following source code acquires the raw RGB values, copies the values into our byte array, and, finally, creates the displayable bitmap:
Step 4 – Detect the hand
Now, let’s move to our BodyReader_FrameArrived event handler. This where we’ll detect the active body and its joints. We only need one joint: the right (or left) hand. We detect the desired hand by using the JointType enumeration.
Hint: for educational purposes, I am only showing you how to get the hand joint. In a real-world project, you could check a number of additional parameters, such as the distance between the hand and the body, or the distance between the body and the sensor. You could even have custom gestures to start and stop the painting process.
Step 5 – Coordinate Mapping
As you know, Kinect provides us with the position of the hand in the 3D space (X, Y, and Z values). To detect where the hand is pointing in the 2D space, we’ll need to convert the 3D coordinates to 2D coordinates. The 3D coordinates are measured in meters. The 2D coordinates are measured in, well, pixels. So, how could we convert meters to pixels? The SDK includes a powerful utility, called CoordinateMapper. Using CoordinateMapper, we can find the position of the hand in the 2D space!
Step 6 – Draw!
Finally, we’ll update the point collection of our Polyline component by adding the new color point.
So, this is it, folks! Hope you enjoyed this tutorial. Feel free to extend the source code, add your own effects, constraints, and functionality, and use it in your own projects.
Project Information URL: http://pterneas.com/2016/03/15/kinect-drawing/
Project Source URL: https://github.com/LightBuzz/Kinect-Drawing