Kinect Interactions with(out) WPF – Part III: Demystifying the Interaction Stream

 In this part of my Kinect Interaction blog post series, we go deep into the rabbit hole, and examine the foundation of Kinect Interactions – the InteractionStream, upon which the entire library is built. This is a risky ride – with no official documentation, we can only count on our trusty reflector, the source code of the Kinect Interaction SDK and careful exploration.

You only need to access the treasures of InteractionStream, if you want to go beyond what the KinectRegion and other controls provide. For example, you want to create your own KinectRegion, you want to zoom a map by gripping it with two hands, or want to build your entirely new interaction model, using two hands along with the press and grip gestures.

Initializing the InteractionStream

Initializing the InteractionStream is much like initializing the DepthStream or the SkeletonStream. If you have a KinectSensor object, all you need are the next two lines of code:

...

It seems like we are on track. Just pass an initialized KinectSensor object and this DummyInteractionClient to the constructort of the InteractionStream, and we should be all set, right? Well, not quite. The InteractionFrameReady event does not fire.

Interaction Needs Skeleton and Depth

It turns out, that for the InteractionStream to work, it needs to process the data from both the depth and the skeleton streams. So, we need to initiate all three of the streams. This is what the entire OnLoaded method (which you have to wire up either in XAML or in the constructor of the page) looks like:

...

It doesn’t seem so simple now, so let's walk through each line of code and see what they do. The first five lines define local variables that hold a reference to the Kinect sensor the application will use, the InteractionStream itself, the skeletons identified by the SkeletonStream, and the hand position information about the users as determined by the InteractionStream.

...

Dissecting InteractionFrameReadyEventArgs

So, now that we have finally made the InteractionStream call the InteractionFrameReady event, we can start analyzing what kind of data we can get to. The first step is to get our hand on an InteractionFrame, and the interaction data it contains:

Summary

I’ve shown you how you can get started with the InteractionStream and acquire information about the hands of the users, detect active hands, pressing (IsPressed and PressExtent), and open / closed hands (HandEventType and our custom LastHandEventType). What I’ve shown here does not require WPF – you can use this from any .NET application, be it XNA, Windows Forms or even Command Line.

Project Information URL: http://dotneteers.net/blogs/vbandi/archive/2013/05/03/kinect-interactions-with-wpf-part-iii-demystifying-the-interaction-stream.aspx

Project Source URL: Source Download

private Dictionary<int, InteractionHandEventType> _lastLeftHandEvents = new Dictionary<int, InteractionHandEventType>(); private Dictionary<int, InteractionHandEventType> _lastRightHandEvents = new Dictionary<int, InteractionHandEventType>();
Dictionary<int, InteractionHandEventType>();
  
private void InteractionStreamOnInteractionFrameReady(object sender, 
ctionFrameReadyEventArgs args)
{
     using (var iaf = args.OpenInteractionFrame()) //dispose as soon as 
le
     {
         if (iaf == null)
             return;
  
         iaf.CopyInteractionDataTo(_userInfos);
     }
  
     StringBuilder dump = new StringBuilder();
  
     var hasUser = false;
     foreach (var userInfo in _userInfos)
     {
         var userID = userInfo.SkeletonTrackingId;
         if (userID == 0)
             continue;
  
         hasUser = true;
         dump.AppendLine("User ID = " + userID);
         dump.AppendLine("  Hands: ");
         var hands = userInfo.HandPointers;
         if (hands.Count == 0)
             dump.AppendLine("    No hands");
         else
         {
             foreach (var hand in hands)
             {
                 var lastHandEvents = hand.HandType == 
ctionHandType.Left
                                             ? _lastLeftHandEvents
                                             : _lastRightHandEvents;
  
                 if (hand.HandEventType != 
ctionHandEventType.None)
                     lastHandEvents[userID] = hand.HandEventType;
  
                 var lastHandEvent = lastHandEvents.ContainsKey(userID)
                                         ? lastHandEvents[userID]
                                         : 
ctionHandEventType.None;
  
                 dump.AppendLine();
                 dump.AppendLine("    HandType: " + hand.HandType);
                 dump.AppendLine("    HandEventType: " + 
andEventType);
                 dump.AppendLine("    LastHandEventType: " + 
ndEvent);
                 dump.AppendLine("    IsActive: " + hand.IsActive);
                 dump.AppendLine("    IsPrimaryForUser: " + 
sPrimaryForUser);
                 dump.AppendLine("    IsInteractive: " + 
sInteractive);
                 dump.AppendLine("    PressExtent: " + 
ressExtent.ToString("N3"));
                 dump.AppendLine("    IsPressed: " + hand.IsPressed);
                 dump.AppendLine("    IsTracked: " + hand.IsTracked);
                 dump.AppendLine("    X: " + hand.X.ToString("N3"));
                 dump.AppendLine("    Y: " + hand.Y.ToString("N3"));
                 dump.AppendLine("    RawX: " + 
awX.ToString("N3"));
                 dump.AppendLine("    RawY: " + 
awY.ToString("N3"));
                 dump.AppendLine("    RawZ: " + 
awZ.ToString("N3"));
             }
         }
  
         tb.Text = dump.ToString();
     }
  
     if (!hasUser)
         tb.Text = "No user detected.";
}

Contact Information:

• Blog: http://dotneteers.net/blogs/vbandi/
• Twitter: @vbandi

Marcus has provided a great example of streamlining Kinect for Windows SDK development. It's simple, yet, when you see it, you'll know you want to grab this... (plus you've got to love that he's released the source too)

Fluent Kinect

Since I have been playing around with the Kinect for Windows SDK I’ve created a lot of little new projects and samples to try things out. Starting point was always something like this:

Because I most often use the 640×480 option anyway, the format is an optional parameter when enabling the streams and it defaults to *640x480Fps30.

I’ve extracted the two little lines that gets the first connected Kinect Sensor to a class called KinectConnector. At the moment an exception is thrown when no Kinect unit is connected. This is not a very good way of handling this scenario and will be changed in the future.

Now the code is even cleaner:

var sensor = KinectConnector.GetKinect()
.EnableColorStream()
.EnableSkeletonStream()
.EnableDepthStream()
.Seated()
.NearMode()
.Start();

For an even shorter and quicker Setup I’ve implemented the method ‘KickStart’ which enables the three streams and calls Start() on the KinectSensor object.

For future ‘try out samples’ I’ll just have to code this now:

var sensor = KinectConnector.GetKinect()
.KickStart();

Project Information URL: http://passiondev.wordpress.com/2013/05/03/fluent-kinect/

Project Download URL: FluentKinect

Project Source URL: FluentKinect

Contact Information:

• Blog: http://passiondev.wordpress.com/
• Twitter: @MarcusKohnert

New Kinect capabilities

Viral Search for Twitter

Valve console

Office 13 transfers

Microsoft .NET Framework 4.5 and Kinect for Windows v1.6 Templates Are Now on ECE

The following feature updates are now available on ECE:

• Microsoft .NET Framework 4.5 template and redistributable installer for Windows Embedded 7 SP1.
• Microsoft .NET Framework 4.5 redistributable installer for Windows Embedded POSReady 7. 
• Microsoft Kinect for Windows v1.6 template for Windows Embedded 7 SP1.

...

Microsoft Kinect for Windows v1.6 template allows for the installation of Kinect for Windows Runtime v1.6 on Windows Embedded Standard 7 SP1.  Highlights of new features supported in v1.6 runtime are:

• .NET Framework 4.5 support
• Accelerometer Data APIs - sensor's accelerometer is now exposed in the API
• Extended Depth Data Is Now Available - provides details beyond 4 meters
• Color Camera Setting APIs - Camera Settings can now be optimized to your environment
• More Control over Decoding- Bayer to RGB conversions on CPU or GPU
• German Language Pack for Speech Recognition
• Infrared Emitter Control API - new API for turning the infrared emitter off.
• Infrared Stream Is Now Exposed in the API

Project Information URL: http://blogs.msdn.com/b/windows-embedded/archive/2013/01/03/microsoft-net-framework-4-5-and-kinect-for-windows-v1-6-templates-are-now-on-ece.aspx

Contact Information:

• Blog: http://blogs.msdn.com/b/windows-embedded
• Twitter: @MSFTWEB

Toyota Prius Xbox 360 Kinect for Windows Los Angeles Auto Show 2012 December

I was walking around the auto booths and saw some interactive demos, using the Xbox 360 Kinect for Windows system. This was Toyota making an interactive video game with the Toyota Prius. You can interact and look around the car and the cockpit. You had to run to the car. And you can drive the Prius in a little video game. Walked the show on December 4, 2012.

Project Information URL: http://www.youtube.com/watch?v=G_xEltK_dvk

CastleOS

How it Works

CastleOS is an integrated software suite for controlling the automation equipment in your home - an operating system for your castle, if you will. The first piece of the suite is what we call the "Core Service" - it acts as the central controller for the whole system. This runs on any relatively recent Windows computer (or more specifically, the computer that has an Insteon PLM or USB stick plugged in to it), and creates a network connection to both your home automation devices, and the second piece of the integrated suite - the remote access apps like the HTML5 app, Kinect voice control app, and future Android/iOS apps.

Working in concert with the Core Service, these apps allow you to interact with the system from any device, and, with a Kinect, in any room of your house by speaking out loud. The HTML5 app can be accessed from your homes internal WiFi/network, and if you choose to configure your router to allow it, from outside your home as well (for instance, on your smartphone).

The Kinect voice control app similarly runs on a Windows computer (could be the same one as the Core Service, or not) with an Xbox360 or Windows Kinect plugged in. Once enabled, it persistantly listens (does not record) to any speech in the room, and when a command is spoken aloud, it acts accordingly. We are very proud to say that despite running 24/7 in a living room blasting loud music, movies, TV, and other noise at it, we have reduced false positives to all but zero.

...

Project Information URL: http://www.castleos.com/

Project Download URL: http://www.castleos.com/Download.aspx

Contact Information:

• Blog: http://www.castleos.com/home.aspx#blog
• Twitter: [Twitter]

Listening with Kinect

In previous articles, I walked you through the capabilities you can tap into when developing Kinect for Windows applications. You have seen how to use the skeleton tracker, how to create test scenarios and how to enable 3D sight with the depth sensor. In this article, I explain how to use the Kinect hardware for speech recognition. Those of you who are already familiar with the Speech.Recognition library will notice many similarities between that and Microsoft.Speech, which Kinect uses, and should feel right at home. Let’s begin by looking at those two libraries.

A Tale of Two Libraries

Speech recognition isn’t new. For some time, Windows developers have been connecting microphones and other hardware to their Windows machines to listen and respond to human speech. Using the System.Speech library, applications can build grammars, direct where to “listen” from, respond with semantic correctness and even take dictation with increasing accuracy and ease. Now enters the Kinect for Windows SDK with the Microsoft.Speech library—but why do we need a new Speech library?

On a closer look, the System.Speech and Microsoft.Speech libraries appear comparable. In fact, they are a lot alike: they have the same classes, the same methods and so on. The main difference between these two libraries is that Microsoft.Speech is specifically optimized for the Kinect hardware. Although using System.Speech with the Kinect hardware or, conversely, using Microsoft.Speech with a regular microphone is possible, in both cases the results are less than optimum. (We’ll actually try this later in the article and see how it works.)

The other major distinction between the two libraries is that as of Kinect for Windows SDK 1.6, Microsoft.Speech doesn’t support the dictation model. The recognition engine doesn’t support DictationGrammar. That means that to use the speech recognition aspect of the SDK, you must know all the spoken values and incorporate them into a fixed grammar. Although this constraint is serious, Microsoft.Speech still gives you an enormous amount of freedom to interact with users via speech. Later in this article, I’ll return to the concept of dictation with a little experiment, but first, let’s go over the basics of speech recognition for the Kinect.

...

What Do You Have to Say?

In this article, I demonstrated some of the techniques and tools available when using the Kinect for Windows sensor to facilitate speech recognition. I showed how you can define a grammar within code as well as in an XML document. I also attempted to use Kinect to receive dictation. By using both Microsoft.Speech and System.Speech in conjunction with the Kinect for Windows SDK (1.6), I was able to get some decent results with limited effort.

Using these techniques opens the door for some really creative user experiences. Couple the speech-recognition capabilities with the depth and skeleton-tracking capabilities, and you can provide Kinect users with even more sophisticated experiences. For example, you could have the speech recognizer respond only if the user is looking directly at the Kinect, more closely simulating humanistic eye contact.

...

Project Information URL: http://msdn.microsoft.com/en-us/magazine/jj884371.aspx

Project Source URL: http://archive.msdn.microsoft.com/201212Kinect

Contact Information:

• Blog: http://lelandholmquest.wordpress.com/

Speech Recognition in Kinect

There are few things that I have seen while developing Kinect examples, I wouldn't want to say apps, because I have not yet developed a full-fledged application. Kinect user interface, is different from the traditional user interface, where we will see buttons that needs to be clicked. Using voice you can control or do things that you used to do with a mouse, using hand gesture you can control your application like you used to do with a mouse.

In this article I will demonstrate to you, on how you can control your application using voice commands.

Objective

The objective of this article is to demonstrate to you on how to control your application using voice commands instead of a mouse.     

When I started creating the example of this article, I first created buttons and later i removed those buttons. The reason I finally removed the buttons, is because I did not need them anymore, I could control my example app using voice commands and the nice part is that I controlled the application using my mother tongue. ...

Project Information URL: http://www.dotnetfunda.com/articles/article2076-speech-recognition-in-kinect.aspx

Project Source: http://www.dotnetfunda.com/UserFiles/ArticlesFiles/Vuyiswamb_Articles_634917069066188907_VoiceCommandsInKinect.zip

Contact Information:

• Blog: http://www.dotnetfunda.com/

Build-A-Bear Selects Kinect for Windows for "Store of the Future"

Build-A-Bear Workshop stores have been delivering custom-tailored experiences to children for 15 years in the form of make-your-own stuffed animals, but the company recently recognized that its target audience was gravitating toward digital devices. So it has begun advancing its in-store experiences to match the preferences of its core customers by incorporating digital screens throughout the stores—from the entrance to the stations where the magic of creating new fluffy friends happens.

A key part of Build-A-Bear's digital shift is their interactive storefront that's powered by Kinect for Windows. It enables shoppers to play digital games on either a screen adjacent to the store entrance or directly through the storefront window simply by using their bodies and natural gestures to control the game.

...

"We wanted something that we could build on, that's a platform for ever-improving experiences," Elliott said. "With Kinect for Windows, there’s no learning curve. People can interact naturally with technology by simply speaking and gesturing the way they do when communicating with other people. The Kinect for Windows sensor sees and hears them."

"Right now, we're just using the skeletal tracking, but we could use voice recognition components or transform the kids into on-screen avatars," he added. "The possibilities are endless."
Part of the Build-A-Bear's vision is to create Kinect for Windows apps that tie into the seasonal marketing themes that permeate the stores. Elliott said that Build-A-Bear selected the combination of the Microsoft .NET Framework, Kinect for Windows SDK, and Kinect for Windows sensor specifically so that they can take advantage of existing developer platforms to build these new apps quickly.

“We appreciate that the Kinect for Windows SDK is developing so rapidly. We appreciate the investment Microsoft is making to continue to open up features within the Kinect for Windows sensor to us,” Elliott said. "The combination of Kinect for Windows hardware and software unlocks a world of new UI possibilities for us."

...

Project Information URL: http://blogs.msdn.com/b/kinectforwindows/archive/2012/12/13/build-a-bear-selects-kinect-for-windows-for-quot-store-of-the-future-quot.aspx

Contact Information:

• Blog: http://blogs.msdn.com/b/kinectforwindows/
• Twitter: @KinectWindows

Khristmas is here

I just published an augmented reality game on Github. Here’s how you play it:

• Download the source code and extract it somewhere safe.
• Make sure you have the latest Kinect SDK installed. If not, get it from here.
• Connect a Kinect sensor to your PC and power it on.
• Open the solution in Visual Studio and run it.

When the game starts, hover your hand over the Go button. You need to collect as many falling gifts as possible with your right hand in a duration of 60 seconds. Pretty simple! You can retry the game as many times as you want.

...

Khristmas v1.0 This is a simple game that uses the Kinect sensor to add some augmented reality. All you need to do is collect as many falling gifts as you can with your right hand in 60 seconds.

The K stands for Kinect....

Project Information URL: http://sandreddy.wordpress.com/2012/12/25/khristmas-is-here/

Project Source URL: https://github.com/sandeep-reddy/Khristmas

Contact Information:

• Blog: http://sandreddy.wordpress.com/
• Twitter: @rdsandeep

Spying On Our Walking Habits With Kinects, To Create Smarter Spaces

A project at MIT has been using the video game system to help model how pedestrians move through space and what they do when things get in their way. With that knowledge, the scientists are figuring out better ways to manage crowds in tight urban places.

Microsoft unveiled the Kinect two years ago as an add-on for the Xbox. The infrared motion-detecting sensor was designed to enhance video games, to turn you, as Xbox puts it, into your own controller. But immediately, of course, researchers at MIT started hacking the things.

A tool initially intended for your virtual dance moves on Michael Jackson: The Experience had obvious potential for scientists as well to study how people move through all kinds of scenarios and spaces. Researchers at MIT’s Senseable City Lab figured the simple gadget--available for about $80 at the scientific warehouse of Best Buy--could change how scientists study all kinds of crowd behaviors in our increasingly congested cities.

Tracking Pedestrians

They bought some Kinects and mounted them on campus. Now their project–-Kinect Kinetics, they call it–-has demonstrated a new technique for tracking human behavior that could ultimately help us build smarter airports, train stations, public plazas, shopping malls and supermarkets. With 60% of the world’s population projected to cram into urban areas by 2030, this is no trivial invention.

...

Project Information URL:  http://www.fastcoexist.com/1681002/spying-on-our-walking-habits-with-kinnects-to-create-smarter-spaces

KinectSEN - Using Kinect in Special Schools for Pupils with Severe Learning Difficulties

This is a guide for starting to use Kinect programs as a tool for sensory interaction and engagement with pupils with special educational needs (SEN), especially severe learning difficulties (SLD). It is hopefully practical and easy to use. You do need some basic knowledge of computers to get going and a little perserverance but it will be worth it though, below are about 5 programs you can try in school with minimal fuss!

This information is teacher created and all of the programs can be accessed for free. All of the programs have also been tried out and used in lessons in special needs schools with pupils with SEN and SLD, typically P4 level to NC1.

Updated 26th September 2012 with the excellent Kinect Flow and Physics from Amnon Owed and another kinect-ed Processing sketch.

...

This is the beginnings of a Professional Learning Community (PLC) involving special needs schools. There is talk of a 'Kinectogether' event with the Somantics team, a whole host of teachers, programmers and other agencies and businesses to give everything a good kick start.

I got a bit fed up of waiting for SEN companies to develop Kinect programs so I tried out a few things, got involved in the Somantics project and tried out programs that digital artists had created and put online for free with pupils in a special needs school setting. The results with the pupils have been very positive

...

Conclusion
I hope more special schools start to use the kinect very soon. Like most things these kinect sessions take time to prepare for, and set up but then they evolve and develop and the benefits start to appear. With a little bit of space and a small outlay in money an interactive area can be set up to try to engage our pupils and help their interactions, creativity and movement. Different sessions and different programs produce different responses and these responses also evolve over time. Some pupils love it and take to it straight away, others need some encouragement and patience, others aren't bothered with it at all, but in my experience that is true of everything! What is important is that these kinect programs have no real parallel in our schools, they use movement to create visuals and sound without needing the pupils to use equipment or to learn any specific skills first. They are natural and intuative.

The technical issues are often difficult to overcome, although with new technology it's to be expected!

It is hard for people who do not understand the complex needs of our pupils to understand the significance of enabling and motivating some of our pupils to interact meaningfully with anything in their environment. The more enabling technologies we use the more chance we have of finding something that will increase motivation, creativity, interaction and movement when other traditional methods have failed. The Kinect is not replacing these other methods but adding a whole new level of opportunity to the tools at our disposal. I am certain that given the resourcefulness and inventiveness of the special needs teaching community worldwide and the support of the business and technical community that these opportunities will multiply and flourish.

Project Information URL: http://kinectsen.wikispaces.com/

Kinect GUI Framework for XNA Games

Introduction

This example will show you how to build a simple GUI framework for the Kinect to use with XNA applications.  XNA doesn't have a native set of GUI widgets like WPF does, so if you need a button or a scrollbox, you have to build your own.  I have done this several times and I finally created a reusable framework to save myself time on projects.

This example assumes that you have a Kinect for Windows and the SDK version 1.5 for the Kinect.  It also assumes that you have installed the XNA game studio 4.0 or higher.

Building the Sample

If you have the prerequisites, you should be able to build and run the example with no problem provided that your Kinect device is hooked up and running on your PC.

Description

The solution is divided into two major parts, the KSGui framework project, and an XNA game example project that use the KSGui framework.  There is a utility project also in the solution, but it is just a hodgepodge of freeware libraries that I lifted from various websites.

First let us look at the KSGui framework.  Every good framework starts with a contract (or interface as we say in .NET land), and so the KSGui has IKSComponent.  This interface contains the essential properties and methods that are needed for a component to be a KSWidget.  Below is the source code for this interface.

...

It's a pretty straightforward interface for anyone familiar with XNA games.  Notice that Update() is not required by the interface.  You will see that some of our components don't need Update(), so I left it out of the interface.  You can put it back if you like symmetry more than pragmatism.

The base class for the KSGui framework is KSWidget, and of course it is based on the IKSComponent interface.  I will not post all of the source code for KSWidget here, but there are a couple of interesting parts to this class.  For instance, since we are dealing with a Kinect as the input device, we don't have any method for a person to click anything in the GUI.  So we implement a sort of hover time for widgets.  If the user hovers over a widget for some set time, we raise events.  That code looks like this:

...

Project Information URL: http://code.msdn.microsoft.com/Kinect-GUI-Framework-for-496b9806

Project Download URL: http://code.msdn.microsoft.com/Kinect-GUI-Framework-for-496b9806

Project Source URL: http://code.msdn.microsoft.com/Kinect-GUI-Framework-for-496b9806

Kinect for Windows releases SDK update and launches in China

I’m very pleased to announce that the latest Kinect for Windows runtime and software development kit (SDK) have been released today. I am also thrilled to announce that the Kinect for Windows sensor is now available in China.

...

The updated SDK gives developers more powerful sensor data tools and better ease of use, while offering businesses the ability to deploy in more places. The updated SDK includes:

Extended sensor data access

• Data from the sensor's 3-axis accelerometer is now exposed in the API. This enables detection of the sensor's orientation.
• Extended-range depth data now provides details beyond 4 meters. Extended-range depth data is data beyond the tested and certified ranges and is therefore lower accuracy. For those developers who want access to this data, it’s now available.
• Color camera settings, such as brightness and exposure, can now be set explicitly by the application, allowing developers to tune a Kinect for Windows sensor’s environment.
• The infrared stream is now exposed in the API. This means developers can use the infrared stream in many scenarios, such as calibrating other color cameras to the depth sensor or capturing grayscale images in low-light situations.
• The updated SDK used with the Kinect for Windows sensors allows for faster toggling of IR to support multiple overlapping sensors.

Access to all this data means new experiences are possible: Whole new scenarios open up, such as monitoring manufacturing processes with extended-range depth data...

Improved developer tools

• Kinect Studio has been updated to support all new sensor data features.
• The SDK ships with a German speech recognition language pack that has been optimized for the sensor's microphone array.
• Skeletal tracking is now supported on multiple sensors within a single application.
• New samples show how to use all the new SDK features. Additionally, a fantastic new sample has been released that demonstrates a best-in-class UI based on the Kinect for Windows Human Interface Guidelines called the Basic User Interactions – WPF sample.

...

Greater support for operating systems

• Windows 8 compatibility. By using the updated Kinect for Windows SDK, you can develop a Kinect for Windows solution for Windows 8 desktop applications.
• The latest SDK supports development with Visual Studio 2012 and the new Microsoft .NET Framework 4.5.
• The Kinect for Windows sensor now works on Windows running in a virtual machine (VM) and has been tested with the following VM environments: Microsoft Hyper-V, VMWare, and Parallels.

Windows 8 compatibility and VM support now mean Kinect for Windows can be in more places, on more devices. We want our business customers to be able to build and deploy their solutions where they want, using the latest tools, operating systems, and programming languages available today.

This updated version of the SDK is fully compatible with previous commercial versions, so we recommend that all developers upgrade their applications to get access to the latest improvements and to ensure that Windows 8 deployments have a fully tested and supported experience...."

Kinect for Windows: SDK and Runtime version 1.6 Released

"Version 1.6 is an in-place update to version 1.0 or 1.5 that is fully compatible. (You should be able to move easily to use v1.6 if you are already using or targeting v1.5 or v1.0.)

• The launch announcementis on our blog
  • Our website has been updated, including new and updated developer resources:
  • Updated docs which continue to be hosted in MSDN or downloadable for offline use
  • Human Interface Guidelines which helps developers get started with NUI interaction design. We've added a sample in v1.6 called "Basic Interactions-WPF" which shows a concrete example of applying many of the guidelines.
• Sensors are now available for purchase in China, with more countries coming later this fall.
• Download the SDK (and Developer Toolkit) here

Thank you all for your interest, enthusiasm, questions, and feedback – your contributions are a key part of making the product great. ..."

Easy Kinect Mouse Controller for Windows

This sample will show you how to build a simple controller for the Kinect that will let you control the mouse on your Windows desktop using a Kinect device.  This sample assumes that you have a Kinect for Windows device installed on your desktop and that you have the Kinect for Windows SDK Version 1.5 or greater.  This sample also uses the WPF Kinect framework samples from Microsoft and I have included the necessary parts in the solution as a separate project.

Description

This sample is very simple to understand if you already have a basic understanding of how the Kinect works.  If you do not, you might want to look at my other sample on Beginning Kinect Programming.  The main tasks we will be doing in this sample are:

1) Intercept the skeleton information from the Kinect and pick out the right hand.

2) Figure out the position of the right hand relative to the screen of the PC.

3) Pass that position to a Microsoft API for mouse control.

That's it really.  As I said, it's a simple thing to do once you know how.  We are also going to simulate a mouse click() function by setting a timer when the user is hovering over any spot on the screen for a certain length of time.  We do that by creating a stopwatch and timing the hovers.  So in the MainWindow constructor for the WPF XAML you will see a stopwatch being created, like this:

...

Project Information URL: http://code.msdn.microsoft.com/Easy-Kinect-Mouse-09233c52

Project Download URL: http://code.msdn.microsoft.com/Easy-Kinect-Mouse-09233c52

Project Source URL: http://code.msdn.microsoft.com/Easy-Kinect-Mouse-09233c52

• [0:30] Anders Hejlsberg: Introducing TypeScript (Ander Hejlsberg), TypeScript: JavaScript Development at Application Scale (Soma), Announcing TypeScript Preview for Visual Studio (Chris Koenig)
• [2:16] WOWZAPP 2012 Hackathon for Windows (Larry Larsen, Andrew Parsons)
• [3:23] SharpDX 2.3 new version released
• [4:47] C++ Runtime for Windows 8 Store apps (Raman Sharma)
• [6:02] Windows Store app development: Include that privacy policy (Pete Brown)
• [7:36] Microsoft Visual Studio 2012 Documentation
• [8:24] Episode 50 of Visual Studio Toolbox: Collecting IntelliTrace Data in Production (Larry Guger, Robert Green)
• [9:40] Introduction to Fakes and migration from Moles (Jan Fajfr)

Picks of the Week!

• Mark's Pick of the Week:[11:07] Grid Computing with 256 Windows Azure Worker Roles (Bjorn Eriksen, Alan Smith), 256 Windows Azure Worker Roles, Windows Kinect and a 90's Text-Based Ray-Tracer (Alan Smith)
• Dan's Pick of the Week:[12:51] NASA’s Curiosity Rover Checks In On Foursquare From Mars, One Check-In Closer From Becoming Mayor [IMAGE]

Kinect Maths–Early Stages / Source Code

You will remember a month or two back, that I started working on a Kinect Game. The idea of the game was to use more active gestures to answer questions in Mathematics lessons. To use the game, students will calibrate their custom movements for addition, subtractions, multiplication and division. Once they have answer the questions, It will give them a sound if correct. If they have got it incorrect then it will give the students an error noise.

I have decided to hide the operator so that the student must guess what symbol they should be using.

To use it, you must click on the drop down menu and then press the calibrate button. It will count down to 1 and then the student can start recording their movement. Each movement should be unique in it’s own way. For example, I wave my right hand for addition, left hand for subtraction, clap for multiplication and then move my arms up in the air for division. I have noticed that it doesn’t always give you the count down. If it stalls on 3 then start moving your arms and you will notice it start recording.

It has taken me a while to get most of it together but there are still a few bugs in it. I’m looking to add more features into the game such as scoring and timers in the corner. This is the early stages and it might help others who are trying to use custom (dynamic) gestures.

I have programmed it to store up to 4 gestures, however you can adjust it to include more.

Project Information URL: http://raychambers.wordpress.com/2012/09/23/kinect-mathsearly-stages-source-code/

Project Download URL: https://skydrive.live.com/redir?resid=FD734F2CFC605887!127&authkey=!AJ-tvzqbrGW4EhA

Project Source URL: https://skydrive.live.com/redir?resid=FD734F2CFC605887!127&authkey=!AJ-tvzqbrGW4EhA

Contact Information:

• Blog: http://raychambers.wordpress.com
• Twitter: @Lanky_Boi_Ray

devinh / KinectRCCar

This is the code for my Master Project. It allows you to control a remote control car with the Microsoft Kinect.

The program was written in C# using VS2010 Ultimate. You should be able to use any other version of VS2010, only the Modeling project probably won't load. I haven't tried loading this in Visual C# Express, but I don't see why it won't work.

In order to use the software, you need to have a remote control car that the program is able to interface to. To do this you must use a hardware solution that interfaces between the remote control car's transmitter and the computer. You must then create a plugin DLL the Kinect/RC Car program then uses to communicate with your hardware solution. You can look at my solution in the KinectRC.CarControllerService project which communicates with an mbed microcontroller prototype board which in turn controls an MCP4261.

To make your own plugin DLL:

1. Create a class library project

2. Reference the KinectRCCar.Infrastructure project

3. Reference the Unity IOC Library and the PRISM library from NuGet or download from their respective websites.

4. Create a class that implements the ICarControllerService interface.

5. Write the code in this class that communicates with your hardware solution.

6. Create a Module class that inherits from the PRISM IModule interface.

7. Register your CarControllerService in the Initilization method.

8. Add the following attribute above the class declaration in your Module class: [Module(ModuleName = "CarControllerModule")]

9. Build the project then place the DLL in the Modules folder in the KinectRCCar build folder.

10. Remove the current CarControllerService DLL from the Modules folder.

11. Run the program.

Additional information on how the program was written is in the Project Report...

"Additional information?" I'd say so! We're talking a 66 page PDF!

The objective of this project is to create a program (herein after called Kinect/RC) that allows a user to easily control a remote control car using a natural user interface (NUI), specifically, the Microsoft Kinect. To accomplish this project, various goals need to be met. Video needs to be provided from a camera mounted on the remote control car to allow the user to view the direction in which the car is headed. This is a more natural and easier way to control the car than watching the car directly. Additionally, the program needs to provide telemetry data such as direction and velocity of the car to confirm to the user that their intended hand motions are being recognized the way they wish (Hands in a position that expresses turning left should show that a left turn is being sent to the remote control car). The program should also display the skeletal data that is determined by the Kinect so the user is able to verify they are being recognized correctly. Thus, for example, the user can correct their positioning should part of their body be cut off from the view of the Kinect. The Kinect has an adjustable elevation of ±27° [1] so the program needs to provide the user the ability to adjust this elevation. This adjustment will allow the placement of the Kinect at any height within its tolerance of 2 ft. to 6 ft. [1] above the floor. The project also must be written using design patterns and other object-oriented principles to develop a modular and easily maintainable program.

Oh and all the source is available too!

Now check out the video to see it in action.

See, awesome isn't it?

RTM’d today: Programming with the Kinect for Windows Software Development Kit

We’re happy to announce that Programming with the Kinect for Windows Software Development Kit (ISBN 9780735666818) has shipped to the printer! If you are a developer who wants to learn how to add gesture and posture recognition to your applications, check out David Catuhe’s book. This guide to the Kinect for Windows Software Development Kit is full of insights and code samples that teach you how to integrate Kinect in your apps and help you begin writing UIs and controls that can handle Kinect interaction.

The book will be shipped to retailers in just a couple weeks. In the meantime, preorder your copy here, here, or here.

Catuhe is a developer evangelist at Microsoft who says that “fell in love” with Kinect for Windows the first time he used it. Read on to learn more about his passion for this technology in this excerpt from the book’s Introduction.

But first, a quick look at the book’s contents:

Contents at a Glance

PART I  KINECT AT A GLANCE
CHAPTER 1  A bit of background
CHAPTER 2  Who’s there?

PART II INTEGRATE KINECT IN YOUR APPLICATION
CHAPTER 3  Displaying Kinect data
CHAPTER 4  Recording and playing a Kinect session

PART III  POSTURES AND GESTURES
CHAPTER 5  Capturing the context
CHAPTER 6  Algorithmic gestures and postures
CHAPTER 7  Templated gestures and postures
CHAPTER 8  Using gestures and postures in an application

PART IV  CREATING A USER INTERFACE FOR KINECT
CHAPTER 9  You are the mouse!
CHAPTER 10  Controls for Kinect
CHAPTER 11  Creating augmented reality with Kinect

...

Project Information URL: http://blogs.msdn.com/b/microsoft_press/archive/2012/09/11/rtm-d-today-programming-with-the-kinect-for-windows-software-development-kit.aspx

Contact Information:

• Blog: Eternal Coding, <http://blogs.msdn.com/b/eternalcoding>
• Twitter: @deltakosh

Check out the video below and you'll see why I think it's so cool. Also make sure you click through and check out the ProjectReport.PDF. That's a great 66 page document all about this project (I know, imagine that?)

Kinect/RC Car

Kinect/RC Car This is the code for my Master Project. It allows you to control a remote control car with the Microsoft Kinect. The program was written in C# using VS2010 Ultimate. You should be able to use any other version of VS2010, only the Modeling project probably won't load. I haven't tried loading this in Visual C# Express, but I don't see why it won't work. In order to use the software, you need to have a remote control car that the program is able to interface to. To do this you must use a hardware solution that interfaces between the remote control car's transmitter and the computer. You must then create a plugin DLL the Kinect/RC Car program then uses to communicate with your hardware solution. You can look at my solution in the KinectRC.CarControllerService project which communicates with an mbed microcontroller prototype board which in turn controls an MCP4261.

...

Project Information URL: https://github.com/devinh/KinectRCCar

Project Download URL: https://github.com/devinh/KinectRCCar

Project Source URL: https://github.com/devinh/KinectRCCar

NatuLearn

What problem we are going to solve?

We realize that education in schools is boring for students. Teachers have quite a problem when they want to get pupils interested in their lessons. As far as we all remember students were always taught at schools in a sitting position, often prone to yawning.

We know that pupils, especially at primary schools, are at the age when they simply like to move. Unfortunately, according to what has been widely accepted by teachers around the world, they cannot do this in class, they just have to sit and listen.

What is our solution?

Our mission is to make lessons in schools more fun for teachers and students. We want to prove that learning and having fun at the same time is possible.

We believe that movement is the pupils` best friend so we are trying to connect these things together by making educational applications operated by body gestures.

Kinect Quiz

Quiz allows teachers to build many databases for questions and answers. After that they are able to use them in classroom, where pupils choose their answers simply by moving their hands.

Kinect LearnLang

LearnLang is based on voice recognition, teachers have to build some kind of database too where each question consists of a picture and some text. Pupils have to say their answer in the foreign language and our software, through the Kinect device, will check if their pronunciation is correct

Project Information URL: http://natulearn.com/

Project Download URL: http://natulearn.com/

Contact Information:

• Blog: http://natulearn.com/
• Twitter: @tomkowalczyk

Kinect Games v2.42

Following on from my last post and while I was creating an all in one installer for xGames, I thought I should do the same for my Kinect Games. The installer below will install Kinect Angles v2.4, Kinect Time v1.1 and Kinect Magic Cursor v1.2. It will also install .net4, xna4 runtime and Kinect Runtime v1.5, basically it will install everything you need to run my Kinect Games. All you have to do is run it and then plug in a Kinect for XBOX360 or Kinect for Windows and you are set.

...

[Later, in a follow-up post, URL below]

...I have also changed all 3 applications so that an error message will appear and ask you to plug in Kinect if you haven’t done so; previously if you ran them without Kinect plugged in nothing would happen, they wouldn’t even launch.

So go ahead and download the new and improved Kinect Games all in one installer by clicking here. This time it will work (new filesize is a healthy 291MB).

Project Information URL: http://drenton72.wordpress.com/2012/09/16/kinect-games-v2-42/

Project Download URL: Kinect Games v2.42

Contact Information:

• Blog: http://drenton72.wordpress.com/
• Twitter: @drenton72

Kinect Quiz–Source Code

So I have had a few requests over the summer from a number of people asking me for the source code the quiz. I have been trying to modify it so that it runs smoother and I have slightly adjusted the head, hand movements. If you have any questions about the running of the code then you can email me at my new email address: [Click through to his blog for his email address...]

Now that school has started back I’m looking to get back into the coding on the Kinect and I’m looking at starting a number of projects within my new school with the Kinect. I thought that a good way to start the new school year would be to improve the code slightly. I have a few things up my sleeve but this should keep people new to the world of the Kinect on their toes.

I’m currently looking at re-developing this quiz to change it into a format you will all be familiar with (Who wants to be a millionaire)… watch this space.

Project Information URL: http://raychambers.wordpress.com/2012/09/09/kinect-quizsource-code/

Project Download URL: https://skydrive.live.com/?cid=fd734f2cfc605887&id=FD734F2CFC605887%21126

Project Source URL: https://skydrive.live.com/?cid=fd734f2cfc605887&id=FD734F2CFC605887%21126

Contact Information:

• Blog: http://raychambers.wordpress.com
• Twitter: @Lanky_Boi_Ray

CyberNanny: Remote Access via Distributed Components

This article is about an application called CyberNanny, which I recently wrote to allow me to remotely see my baby daughter Miranda at home from anywhere at any time. It’s written in Visual C++ (MFC) and it comprises different technologies such as Kinect and its SDK, Windows Azure, Web services and Office automation via Outlook. The project is hosted on CodePlex (cybernanny.codeplex.com), where you can check out the code or contribute to it.

Before I get into the nuts and bolts of the application, I’ll briefly explain the technologies used to build it.

C++ has been—and still is—the workhorse in many software shops. Saying that, the new standard C++ 11 takes the language to a new level. Three terms to describe it would be modern, elegant and extremely fast. Also, MFC is still around and Microsoft has been upgrading it with every new release of its Visual C++ compiler.

The Kinect technology is amazing, to say the least; it changes the way we interact with games and computers. And with Microsoft providing developers with an SDK, a new world of opportunities is unveiled for creating software that requires human interaction. Interestingly, though, the Kinect SDK is based on COM (as well as the new programming model in Windows 8, called Windows Runtime, often abbreviated as WinRT). The SDK is also available to Microsoft .NET Framework languages.

Windows Azure is the Microsoft Platform as a Service (PaaS) offering that has been around for a couple of years. It provides a series of services that allow building solutions on top of them (such as Compute and Storage). One of the requirements I had with CyberNanny was the reliable delivery of messages via a highly available queue, and Windows Azure provides that.

The native use and consumption of Web services is possible using the Windows Web Services API (WWSAPI), which was introduced with Windows 7. I have a blog post (bit.ly/LiygQY) that describes a Windows Presentation Foundation (WPF) application implementing a native component using WWSAPI. It’s important to mention that WWSAPI is built in to the OS, so there’s no need to download or install anything but the Windows SDK (for header and lib files).

Why reinvent the wheel? One of the requirements for CyberNanny was the ability to send e-mails with attached pictures, so instead of writing my own e-mailing class, I preferred to reuse the functionality provided by Outlook for this task. This allowed me to focus on the main objective: building a distributed application for looking after my baby.

This article is organized in four main sections:

1. Overview of the general architectural solution
2. Kinect architecture
3. Locally deployed components (native)
4. Cloud-hosted components (managed)

...

Locally Deployed Components (Native)

The CyberNanny project comprises the following:

• Application
  • CCyberNannyApp (inherited from CWinApp). The application has a single member of type Nui_Core for interacting with the sensor.
• UI Elements
  • CCyberNannyDlg (Main Window, inherited from CDialogEx)
  • CAboutDlg (About Dialog, inherited from CDialogEx)
• Web Service Client
  • Files auto-generated after executing WSUTIL against a service, Web Services Description Language (WSDL). These files contain the messages, structures and methods exposed by the WCF Web service.
• Outlook Object Classes
  • In order to manipulate some of the Outlook objects, you have to import them into your project by selecting “Add MFC Class” from ActiveX Control Wizard. The objects used in this solution are Application, Attachment, Mail-Item and Namespace.
• Proxy
  • This is a custom class that encapsulates the creation of the required objects to interact with WWSAPI.
• Helper Classes
  • These classes are used to support the functionality of the application, such as converting a bitmap into a JPEG to reduce the file size, providing a wrapper to send e-mails and interact with Outlook, and so on.

When the application starts, the following events occur:

...

Project Information URL: http://msdn.microsoft.com/en-us/magazine/jj553517.aspx, http://cybernanny.codeplex.com/

Project Download URL: http://msdn.microsoft.com/en-us/magazine/jj553517.aspx,

Project Source URL: http://cybernanny.codeplex.com/SourceControl/list/changesets

void Nui_Core::TakePicture(std::shared_ptr<BYTE>& imageBytes, int& bytesCount) {
  byte *bytes;
  NUI_IMAGE_FRAME imageFrame;
  NUI_LOCKED_RECT LockedRect;
  if (SUCCEEDED(m_pSensor->NuiImageStreamGetNextFrame(m_hVideoStream,
    m_millisecondsToWait, &imageFrame))) {
    auto pTexture = imageFrame.pFrameTexture;
    pTexture->LockRect(0, &LockedRect, NULL, 0);
    if (LockedRect.Pitch != 0) {
      bytes = static_cast<BYTE *>(LockedRect.pBits);
      m_pDrawColor->Draw(bytes, LockedRect.size);
    }
    pTexture->UnlockRect(0);
    imageBytes.reset(new BYTE[LockedRect.size]);
    memcpy(imageBytes.get(), bytes, LockedRect.size);
    bytesCount = LockedRect.size;
    m_pSensor->NuiImageStreamReleaseFrame(m_hVideoStream, &imageFrame);
  }
}

Contact Information:

• Blog: http://www.bonafideideas.com/
• Twitter: @angeljesus76

Simple Gesture Processing using the Kinect for Windows

This sample will show you the basic components of a gesture processing system for the Microsoft Kinect for Windows.  Gesture processing is one of the most powerful abilities of an augmented reality application and the Kinect makes processing gestures straightforward.  This sample is very rudimentary, but it demonstrates the basic gesture processing pipeline that you would have to implement in a real world application.  In this case, the application is a simple Kinect controller for PowerPoint applications using keyboard shortcuts to control the PowerPoint application.  Along the way, it also demonstrates some other useful concepts, like how to draw a skeleton from the Kinect depth stream.  I hope you find this sample useful.  Please leave any feedback or questions that you may have.

Description

This sample builds on the Beginning Kinect for Windows Programming sample.  If you have not looked at that sample AND you are not familiar with Kinect for Windows programming, I would recommend that you start there first before tackling this sample.

This sample assumes that you already have a Kinect test environment set up with all of the tools and technologies installed.  If you do not have that, the previous sample will also help you with that.  

When you are ready to proceed with this sample, open up the source code file that is attached and take a look at the gesture.cs file within the GestureFramework project.

This simple gesture framework is based upon the relationships of skeletal joints to each other over some period of time.  The enumeration JointRelationship describes the possible relationships of joints that are supported by the framework.  

The framework has two major class hierarchies.  One is a static description of the gestures, and the other is a really simple state engine that examines incoming skeletal information from the Kinect and tries to discern the relationships between joints that it is interested in using the static model.  The figure below shows the relationships between the major classes in the framework.

Here are the descriptions of the major classes in the hierarchy:

GestureComponent -Describes a single relationship between two joints.   The relationship contains only two joints and has two temporal parts; a beginning relationship and an ending relationship.

Gesture - Essentially this is a list of type GestureComponents with a unique identifier and a description.

GestureMap - Contains all of the utility functions to load a set of gestures from an XML file and manage their lifecycle.

GestureComponentState - Tracks the state of the relationships for a GestureComponent.

GestureState - Manages the state model for a set of GestureComponents and reports on the state.  This class in our application also maps the keycode for the PowerPoint application to the gesture model, although this is probably sub-optimal design.

GestureMapState - Rolls up the current state of all gestures and is the primary interface of the application into the gesture state model.

If you look through the above classes, you will see that there is absolutely no magic or rocket science inside of them.  The gesture recognition that they provide is very primitive, but they provide a platform onto which you could build a much more sophisticated model if you desire.

Project Information URL: http://code.msdn.microsoft.com/Simple-Gesture-Processing-097c5527

Project Download URL: http://code.msdn.microsoft.com/Simple-Gesture-Processing-097c5527

Project Source URL: http://code.msdn.microsoft.com/Simple-Gesture-Processing-097c5527

void SensorSkeletonFrameReady(AllFramesReadyEventArgs e) 
{ 
    using (SkeletonFrame skeletonFrameData = e.OpenSkeletonFrame()) 
    { 
        if (skeletonFrameData == null) 
        { 
            return; 
        } 

        var allSkeletons = new Skeleton[skeletonFrameData.SkeletonArrayLength]; 
        skeletonFrameData.CopySkeletonDataTo(allSkeletons); 

        foreach (Skeleton sd in allSkeletons) 
        { 
            // If this skeleton is no longer being tracked, skip it 
            if (sd.TrackingState != SkeletonTrackingState.Tracked) 
            { 
                continue; 
            } 

            // If there is not already a gesture state map for this skeleton, then create one 
            if (!_gestureMaps.ContainsKey(sd.TrackingId)) 
            { 
                var mapstate = new GestureMapState(_gestureMap); 
                _gestureMaps.Add(sd.TrackingId, mapstate); 
            } 


                var keycode = _gestureMaps[sd.TrackingId].Evaluate(sd, false, _bitmap.Width, _bitmap.Height); 
                GetWaitingMessages(_gestureMaps); 

                if (keycode != VirtualKeyCode.NONAME) 
                { 
                    rtbMessages.AppendText("Gesture accepted from player " + sd.TrackingId + "\r"); 
                    rtbMessages.ScrollToCaret(); 
                    rtbMessages.AppendText("Command passed to System: " + keycode + "\r"); 
                    rtbMessages.ScrollToCaret(); 
                    InputSimulator.SimulateKeyPress(keycode); 
                    _gestureMaps[sd.TrackingId].ResetAll(sd); 
                } 

            PlayerId = sd.TrackingId; 

            if (_bitmap != null) 
                _bitmap = AddSkeletonToDepthBitmap(sd, _bitmap, false); 
             
        } 
    } 
}