May the Kinect be with you...
- Posted: Feb 11, 2016 at 6:00AM
How could I not highlight this project? Kinect, Sphere, Star Wars and source? Oh yeah...
Unless you have been living under a rock for the last six months, you probably know that the new installment in the Star Wars film series is coming up. And it’s going to be huge! “The Force Awakens” in the movie and so did it in our geeky minds. We at ELEKS spent a lot of time meditating with Yoda and practicing with Obi-Wan to find the Force. But what we really had to do was just use Kinect for Windows 2.0 and Sphero 2.0. Ladies and gentlemen, we give you the power to move objects (almost) telepathically.
You should know Kinect is very cool – I mean come on, it can make us a little Darth Vader-ish! But it was quite demanding to work with. Kinect uses only USB 3.0 because it sends the amount of data USB 2.0 can`t handle. It also requires DirectX 11 and drivers supporting Windows 8 and higher. But when all the requirements are satisfied, you get 120 Mb/sec Color data, 13 Mb/sec Depth data, 13 Mb/sec Infrared data, 32 Kb/sec audio and other cool features. Cool, huh? Plus, Kinect SDK allows skeletal tracking for up to 6 people at the same time. All this makes Kinect the right sensor for creating human-technology interaction.
Certainly, BB-8 was an inspiration for our team. And since BB-8 is based on Sphero, a device we already had, we chose it for our experiment. We used the unofficial .NET SDK, which allowed us to make a custom control system for Sphero using Kinect and Kinect SDK for Windows.
.NET SDK has actually limited functionality. If you want to move it, you must set angle and velocity. But we already had the necessary coordinates thanks to JediGeustureRecognizer, the class we created to detect Sphero’s points of movement. So, how can we move Sphero from its current point to the next one? Let’s see:
According to Microsoft’s guidelines on working with Kinect, the first thing you have to do is determine the PHIZ – physical human interaction zone. Usually, Kinect for Windows is used to interact with the screen, so the PHIZ is in front of the person’s body.
How do we Track the Force?
To recognize hand gestures, we used Body data. Since the algorithms that determine the human skeleton are far from perfect, we had to filter out accurate data from the noise according to the formula FrameRecived – HandTrackedFrames <= Epsilon (where FrameRecived – the number of frames we covered, HandTrackedFrames – the number of frames in which the fist was open, Epsilon – a constant that determines noise sensibility). This formula helps to determine whether a person holds an open hand in front of oneself. For the research we set Epsilon to 3 and verification every 8 frames. This means that if we tracked an open hand at least 5 times out of 8, the Force would be applied.
The tracking logic can be found here.
Behold the Force!
First, set up the scene in the application:
1. Put Sphero on the floor.
2. Set the correct angle (the backlight has to be opposite the Kinect).
3. Set up the speed.
Take your place on the scene like a young padawan. Feel the Force in you. Stretch your hand out and focus your palm (and mind) on Sphero. Now telekinetically move Sphero with your hand. To stop Sphero just clench your fist. When you feel the force again, change your position and continue moving Sphero.
Cool! Where can I get the Source Code?
Project Information URL: http://elekslabs.com/2015/12/awaken-the-force-with-kinect-and-sphero.html
Project Source URL: https://github.com/eleks/rnd-kinect-sphero-jedi