Kinect's power of gesture and speech as an education tool


[Extra Edition]

While this week's posts are already done and published, and the post from earlier today closed out our Education Week theme, the desire and drive of this project's author warranted a "Extra Edition" post. She was driven to get her project into our Education Week, going above and beyond to get her project posted, online and source uploaded. Plus being "Daughter Tested" kind of tugged at my heart strings... Smiley

HCI Project - Using gesture and speech with Kinect for education

This fall term, I took a course in Human-Computer Interaction (HCI) CSC2524 with Prof. Daniel Wigdor. The topics in interactive computing focused on user experiences with next-generation input & output technologies. For my course project I decided to work with novel input modalities of gesture and speech with Microsoft Kinect as new hardware and developed a proof of concept for use in education.

I was inspired by our class topic on case studies in Multi-Modal Input: Gesture + Speech where we discussed two papers "Put that There" by Richard Bolt and Multimodal multiplayer tabletop gaming by Tse et al. The papers discuss the use of deictic references which are speech terms such as "this" and "that" which can be used with spatial gestures like pointing to an object or location. I got the idea that deixis would be an efficient mode of interaction with the computer for younger children in a learning environment. Gestures and speech used as multimodal input could simplify complex learning and make classroom education interactive and fun. So I decided to do my project on this topic.

As proof of concept, I developed few prototypes using Microsoft Visual C# and Kinect SDK beta 2, which demonstrate pointing to objects on screen and using voice commands to select them. The Coding4Fun Kinect Toolkit provides a hover button control. I referred to the tutorial by Michael Crump. Another project posted on MSDN channel9 made use of the hover button implementing it as a "hand" cursor that has a timer based ring used to press the buttons. In my implementation I have used voice command recognition to fire the hovering event inside the hover button control, when it is found to be on top of other buttons on the canvas.


My daughters, aged 4 and 7 tried out my system and as mother involved in their early years learning development, I definitely see the potential use of this technology in education. I had the opportunity to study the prototypes in my daughter's Junior Kindergarten classroom with her classmates. It was very encouraging and I find myself motivated to continue exploring this space even beyond the course

Project Information URL:

Project Source URL:






Contact Information: