Kinect's power of gesture and speech as an education tool

Sign in to queue


[Extra Edition]

While this week's posts are already done and published, and the post from earlier today closed out our Education Week theme, the desire and drive of this project's author warranted a "Extra Edition" post. She was driven to get her project into our Education Week, going above and beyond to get her project posted, online and source uploaded. Plus being "Daughter Tested" kind of tugged at my heart strings... Smiley

HCI Project - Using gesture and speech with Kinect for education

This fall term, I took a course in Human-Computer Interaction (HCI) CSC2524 with Prof. Daniel Wigdor. The topics in interactive computing focused on user experiences with next-generation input & output technologies. For my course project I decided to work with novel input modalities of gesture and speech with Microsoft Kinect as new hardware and developed a proof of concept for use in education.

I was inspired by our class topic on case studies in Multi-Modal Input: Gesture + Speech where we discussed two papers "Put that There" by Richard Bolt and Multimodal multiplayer tabletop gaming by Tse et al. The papers discuss the use of deictic references which are speech terms such as "this" and "that" which can be used with spatial gestures like pointing to an object or location. I got the idea that deixis would be an efficient mode of interaction with the computer for younger children in a learning environment. Gestures and speech used as multimodal input could simplify complex learning and make classroom education interactive and fun. So I decided to do my project on this topic.

As proof of concept, I developed few prototypes using Microsoft Visual C# and Kinect SDK beta 2, which demonstrate pointing to objects on screen and using voice commands to select them. The Coding4Fun Kinect Toolkit provides a hover button control. I referred to the tutorial by Michael Crump. Another project posted on MSDN channel9 made use of the hover button implementing it as a "hand" cursor that has a timer based ring used to press the buttons. In my implementation I have used voice command recognition to fire the hovering event inside the hover button control, when it is found to be on top of other buttons on the canvas.


My daughters, aged 4 and 7 tried out my system and as mother involved in their early years learning development, I definitely see the potential use of this technology in education. I had the opportunity to study the prototypes in my daughter's Junior Kindergarten classroom with her classmates. It was very encouraging and I find myself motivated to continue exploring this space even beyond the course

Project Information URL:

Project Source URL:






Contact Information:

The Discussion

  • User profile image
    vincent john vincent

    I'm really glad to see this. Francis Macdougall and I invented video gesture control technology from 1986 onward, and we invented the 3d gesture control that is in the Kinect in 2000. With our company GestureTek we have invented, pioneered, patented, evangelized and licensed this technology to the tech world in a myrid of ways over the years. But it was always intended to get people active and more embodied in the learning process. We created hundreds of immersive gestural educational applications but it was mostly for science centers. We did produce the technology behind Hasbro's ION immersive educational gaming system toy that came out in 2008, with learning lessons with characters from Nicholodean cartoons that did quite well. I am glad to see it being used for general education....Vincent john Vincent

  • User profile image

    This is great! I am sorry but I am very new to this. How would I use this program with my Xbox in the classroom?

    kind regards

  • User profile image

    @toph: This actually uses a Kinect (a "retail Kinect," one that is bought standalone and not bundled with a 360... those done have the extra power cord that's needed when using it with a PC) with a Windows PC... For more details, please check out,  

  • User profile image

    @toph:if you have a look on my blog i've made some how too videos to show you how to make a simple app and i've also put a lot of source for examples of applications. Theres a blog post or two in there which show you what you need as well. 

Add Your 2 Cents