Home Automation with a little help from Netduino, Microsoft Kinect Point Cloud and Speech Recognition


Today's Hardware Friday comes to us from Friend of the Blog, Dan Thyer, who continues to push the boundaries of the Kinect, Netduino and Home Automation that geeks will love...

Some of Dan's past highlights;

Home Automation with Microsoft Kinect Point Cloud and Speech Recognition

I love using the Microsoft Kinect for my home automation projects. The Microsoft Kinect for Windows API is really amazing and can be used to track our movement in the physical world in unique and creative ways outside of the traditional game controller. Traditional computer vision systems are too slow to track normal human motion, but the Kinect is able to give you coordinates of 20 joints 30 times a second. The Kinect is able to simplify the computer vision problem by creating what is called a Point Cloud out of infrared light. This infrared light is similar to visible light but has a longer wavelength than what we can see. The Point Cloud is able to be seen with a special camera or night vision goggles as shown in the image below.

The Kinect has a special lens that sends out a known pattern of spaced lines of infrared light. The light makes dots on the objects it touches creating a Point Cloud. The Kinect has a special camera for seeing the infrared dots. The vision system on the Kinect measures the distance between the dots and analyses the displacement in the pattern to know how far away an object is. See the image below to see how close up objects have dots closer together and further objects have dots spaced further apart. The Kinect is able to analyze the spacing of the infrared dots to build a depth map and quickly see a human outline because the human is in front of other objects. 


Create a Natural UI with the Kinect

There are some great user interfaces with the Kinect but most require you to be looking at a computer screen. I built a system that does not require you to look at a computer in order to select a device and turn it on or off. You can simply point to a device with one hand and raise your other hand above your head and wave one direction to turn on and the other direction to turn off. In addition to using gestures, I use the Kinect speech recognition engine to turn devices on or off.

Vectors ...

Finding the Coordinates of an Object ...


Figuring Out Which Object You Are Pointing At ...


Kinect Gestures ...

Speech Recognition ...

Z-Wave ...


Netduino is a wonderful open-source electronics prototyping platform based on the .NET Micro Framework. I use the netduino plus 2 and a custom circuit that I built to control many devices in my home including turning on my fireplace, aiming a squirt gun at the pool, watering the garden and opening the garage door. I use the Kinect.Living gestures and audio commands for turning on the fireplace. We have a new kitten in the house who is very interested in the fireplace. I quickly became concerned that the kitten would craw into the fireplace at the wrong time while someone was doing the gesture or audio command to turn it on. For safety I wired up a mesh screen curtain that she cannot get behind! Please read my previous articles on the netduino and jQuery Mobile:


It is really fun to use the Microsoft Kinect for Windows API for home automation projects. This project presents a much more natural UI for controlling your devices in your house. It is really nice to be able to control devices without needing a remote control. In our house, the remote control is always lost in the sofa somewhere, but no worries anymore. With the Microsoft Kinect and this project, you are the remote control to control the devices in the entire house.

The ideas in this article can reach far beyond home automation. There are many other useful applications for having a computer know what object you are pointing to. We are living in an exciting time where vision systems are packaged up into inexpensive devices and are readily available such as the Microsoft Kinect. The Kinect and the Kinect for Windows SDK enables us to build incredible applications with minimal effort that would seem like science fiction 10 years ago. 

[Read the entire post and get the source]

The Discussion

  • User profile image

    Thanks for the post Greg!  The ch9 highlight on "Connecting your Netduino to your Kinect" was actually done by a friend of mine, Brady Gaster, who now works for Microsoft.  

    This blog post does make 6 times that ch9 has highlighted my work though!  The blog post "Home Automation with some help from jQuery Mobile, MVC and Netduino" is another one of my channel 9 highlights.

    I'm really stoked to have my work highlighted again on ch9!!  Thank you!

  • User profile image

    @DanThyer: Doh! Thanks for catching that.. And thanks for the great work and how you give make to the community... :)

Conversation locked

This conversation has been locked by the site admins. No new comments can be made.