Kinect gestures, an implementation walk through
Vangos Pterneas brings us today's project where he not only shows off how we can easily add gesture support to our app's but also a little about is other project too, Vitruvius. Vitruvius a free & open-source library he built, which provides many utilities for your Kinect applications.
Gesture recognition is a fundamental element when developing Kinect-based applications (or any other Natural User Interfaces). Gestures are used for navigation, interaction or data input. The most common gesture examples include waving, sweeping, zooming, joining hands, and much more. Unfortunately, the current Kinect for Windows SDK does not include a gesture-detection mechanism out of the box. So, you thought that recognizing gestures using Kinect is a pain in the ass? Not any more. Today I’ll show you how you can implement your own gestures using some really easy techniques. There is no need to be a Math guru or an Artificial Intelligence Yoda to build a simple gesture detection mechanism.
What is a gesture?
Before implementing something, it is always good to define it. Kinect provides you with the position (X, Y and Z) of the users’ joints 30 times (or frames) per second. If some specific points move to specific relative positions for a given amount of time, then you have a gesture. So, in terms of Kinect, a gesture is the relative position of some joints for a given number of frames. Let’s take the wave gesture as an example. People wave by raising their left or right hand and moving it from side to side. Throughout the gesture, the hand usually remains above the elbow and moves periodically from left to right. Here is a graphical representation of the movement:
In the wave gesture, the hand remains above the elbow and moves periodically from left to right. Each position (left / right) is a discrete part of the gesture. Formally, these parts are called segments.
So, the first segment would contain the conditions “hand above elbow” and “hand right of elbow”:
- Hand.Position.Y > Elbow.Position.Y AND
- Hand.Position.X > Elbow.Position.X
Similarly, the second segment would contain the conditions “hand above elbow” and “hand left of elbow”:
- Hand.Position.Y > Elbow.Position.Y AND
- Hand.Position.X < Elbow.Position.X
The gesture class
Having decided on the window size parameter, we can now build the WaveGesture class. Notice the process:
- In the constructor, we create the gesture parts and we specify their order in the the _segments array. You can use as many occurrences of each segment as you like!
- In the Update method, we keep track of the frame index and check every segment for success or failure.
- If we succeed, we throw the GestureRecognized event and reset the gesture
- If we fail or the window size has been reached, we reset the gesture and start over
Here is the final class for our wave gesture:
Using the code
Using the code we created is straightforward. Create an instance of the WaveGesture class inside your program and subscribe to the GestureRecognized event. Remember to call the Update method whenever you have a new Skeleton frame. Here is a complete Console app example:
Something to note
Obviously, you cannot expect your users to do everything right. One might wave but not perform the entire movement. Another might just perform the movement too quickly or too slowly. When developing a business app targeting the Kinect platform, you have to be aware of all these issues and add conditions to your code. In a common situation, you’ll need to specify whether the user is “almost” performing a gesture. That is, you’ll need to bypass a number of frames before determining the final gesture result. This is why I mentioned the Undetermined statement before.
So, if you want more production-ready gestures right now, consider downloading Vitruvius. Vitruvius is a free & open-source library I built, which provides many utilities for your Kinect applications. It currently supports 9 gestures, waiting for more to come. The code is more generic and you can easily build your own extensions on top of it. Give it a try, enjoy and even contribute yourself!
Project Information URL: http://pterneas.com/2014/01/27/implementing-kinect-gestures/
Project Source URL: http://pterneas.com/wp-content/uploads/2014/01/KinectSimpleGesture.zip