SPEECH RECOGNITION FOR THE KINECT: Kinect speech media controller

The Kinect speech media controller turns your windows media player into a voice controlled media player. Forget remote controls. Forget waving your arms and hands around to control your media player. Control it purely by voice.

Not only that, but to make it a bit more interesting, you get to choose the words for the commands. While this is fun, I found this to be necessary. Kinect speech works great, but it has a few problems with some strong accents.

When working on the voice controlled maths game for Kin-educate the Kinect refused to recognise the word 'four' in a British accent. After trying to spell it different ways (fore, for) and having no luck I gave up. I just made sure the answers could never be four. Problem solved. But not really.

The real solution? We all speak differently and use different phrases, so why not choose what words and phrases you use to control the Kinect? Give your Kinect a unique name, and when you call it, the Kinect will activate and start listening for commands.

Try different phrases and words to see which work well for you, and have a bit of fun with it too. This way the Kinect will be more responsive to your particular way of speaking. You can change the volume, play / pause, rewind / fast-forward  open and close the media player and browse for files all through voice control.

This is a really simple demo that could be expanded in to much more. With a quick change to the code this can be used to control any windows based program. A whole range of other voice commands could be added.

...

After a few seconds of silence the Kinect will go into standby mode. It will stay like this until you call it by its name. When you are not using it call the 'hide' command to minimise the app to the task bar.

Check out this video of the app in action. As you can see it works pretty good. You could have a lot of fun changing the commands around.

I'll make the source code available to download next week. In the meantime remember to have a look at the speech recognition tutorial. ...

Project Information URL: http://kinectmediacontroller.blogspot.co.uk/2013/05/speech-recognition-for-kinect-kinect.html

Project Source URL: Download here

Contact Information:

• Blog: http://www.michaelpalmerwebdesign.com/index.php

Kinect For Windows Interactions Gallery

As the Kinect For Windows SDK has started to evolve, the team has been adding some nice little controls which are quite useful and also controls everyone was writing in one way or another to solve the same issues. I think it’s a really good step so we’re not all spending a bunch of time writing similar controls plus it means there should be some consistency going forward if people use the supplied controls. This will help users with the learning curve with many of the applications.

When you first look through the interactions gallery it’s a bit overwhelming as there’s a bunch of controls and the interaction stream to deal with all at once. For this reason I wanted to do a set of posts so we can concentrate on them one at a time.

• KinectSensorUI
• KinectRegion
• KinectUserViewer
• KinectTileButton
• KinectCirceButton
• KinectScrollViewer
• InteractionStream

Project Information URL: http://www.soulsolutions.com.au/Blog/tabid/73/EntryId/851/Kinect-For-Windows-Interactions-Gallery.aspx

Contact Information:

• Blog: http://www.soulsolutions.com.au/Blog/

New course: Building a Real Application with Kinect

John Sonmez has published a new course: Building a Real Application with Kinect

Have you played with Microsoft Kinect? Do you think it’s legit? Wouldn’t it be awesome if you could write your own Kinect enabled applications to take advantage of this awesome piece of technology? Well, I have great news… John has written a course so you can 

In this course he will walk you through the creation of a real application using the Kinect. You’ll be building a Fruit Ninja clone called Shape Ninja which will be capable of detecting chopping gestures and responding to audio commands. You will start off by learning a little bit about the Kinect itself and the Kinect SDK. Then, John shows you just how easy it is to get color image and depth data from the Kinect. You’ll make your application be able to detect and respond to a chopping gesture. He wraps things up by teaching you how to use the Microsoft Speech Platform SDK in combination with the Kinect’s audio sensors to implement real voice commands for our application.

If you’ve been waiting to check out the Kinect, but you didn’t know where to start, or perhaps you thought it would be difficult to learn; this course will get you up and running with the Kinect in no time. Click here to get started now. We hope you enjoy the course!

Project Information URL: http://blog.pluralsight.com/2013/05/01/new-course-building-a-real-application-with-kinect/, http://www.pluralsight.com/training/Courses/TableOfContents/building-real-kinect-application

Contact Information:

• Blog: http://blog.pluralsight.com
• Twitter: @pluralsight

Kinect No Need 4 Green – The Green Screen without a Green Screen

This little piece of software allows you to produce easily, quickly and cheaply the type of picture that you would normally need a proper green screen setup to create. The software uses a Kinect for Windows or Kinect for XBOX360 device connected to a Windows 7 or 8 pc.

All you need to do is stand in front of the device and it will cut you out. You can choose between different backdrops, which you can add to by copying your own pictures (JPEG or PNG) into the PICTURES sub-folder. You can zoom in & out and move the cut out image using the keyboard or by on-screen controls.  You can take snapshot pictures of what is displayed in the window and these pictures are saved into the SNAPSHOTS sub-folder.

The keyboard controls are as below:-
SPACE bar takes a snapshot photo
W,A,S,D keys move the cut-out image left, right, up and down
+ and – keys zoom the cut-out image in and out
UP and DOWN keys allow you to adjust the viewing angle of the Kinect Device
LEFT and RIGHT keys allow you to choose the backdrop picture
M toggles mirroring mode on and off
F11 toggles full screen mode on and off
C toggles depth cut off mode on and off. This mode changes the way Kinect cuts the image out, by cutting out based on the distance from Kinect, rather than trying to cut out individual people. When in this mode the < key and the > key allow you to adjust the cut off distance.

Project Information URL: http://drenton72.wordpress.com/2013/05/10/kinect-no-need-4-green-the-green-screen-without-a-green-screen/

Project Download URL: Click here to download No Need 4 Green version 1

Project Source URL: Click here to download No Need 4 Green version 1 with C# source code

Contact Information:

• Blog: http://drenton72.wordpress.com
• Twitter: @drenton72

Update: Since the writing of this post David has already released a v2!

Kinect NoNeed4Green v2

I've already made some changes to my NoNeed4Green, so here is version 2. The main addition being the facility to add foreground pictures. The foreground pictures have their own sub-folder called foreground and they use PNG files only, as you need images with transparent backgrounds for it to work. This allows you to put objects in front of the live cut-outs of people as well as having a background behind them. This lets you do things like putting someone behind the desk of the oval office or behind the desk of the BBC newsroom or on the bow of the Titanic. You can also now resize and move all 3 layers. Layer 1 is the background, layer 2 is the live cut-outs of people, while layer 3 are the foreground objects.

...

There is a few new keyboard controls as well :-

H toggles between hiding all on-screen buttons and revealing them. Keys 1,2,3 select layers 1,2 and 3 to allow you to move and resize them.

...

Project Information URL: http://drenton72.wordpress.com/2013/05/12/kinect-noneed4green-v2/

Project Download URL: Click here to download No Need 4 Green version 2

Project Source URL: Click here to download No Need 4 Green version 2

Lose your Occlusion

Depth-sensing cameras like the Kinect give us the opportunity to mix physical environments and virtual environments, creating new immersive experiences. In this Second Story Labs experiment, we demonstrate how the use of multiple cameras helps solve problems with occlusion or “holes” the use of a single camera creates.

If you’ve ever worked with a Microsoft Kinect, you know that occlusion is no laughing matter. It’s responsible for gaping holes in people’s chests, disappearing necks, the noseless faces of zombies. Humans are full of convexities, and to a Kinect that means that we are also full of holes. We’ve dealt with this in lots of creative ways—filling in gaps with best estimates, “blurring” data, hiding holes with smoke and mirrors. But the best and perhaps most obvious solution to the dilemma of occlusion is simply to add more Kinects. A second gunman, if you will, shooting from an angle that will cover the first Kinect’s blind spot.

In this case, we are calibrating two Kinects in space about a meter apart and angled inward toward their subject. This way we can “see” our way around noses, arms, and other pesky occlusions. Then all we have to do is combine their data into a single mesh, and the rest is up to our imaginations.

At full resolution, we can actually get a pretty accurate model of a person’s face. And all of this is being rendered in real time, so that a user’s reality can be “augmented” while they interact. Here we have added three-dimensional models and rain particles to the virtual space to put the user into an imaginary landscape.

Project Information URL: http://blog.secondstory.com/2013/03/18/lose-your-occlusion-2/

from Vimeo.

Contact Information:

• Blog: http://blog.secondstory.com/
• Twitter: @2storypdx

Today...

Kinect Magic Cursor version 1.7 with Gesture support

I am releasing a new version of Kinect Magic Cursor which works much like the last version except it now uses gestures to simulate the left mouse button, instead of raising your left hand. So now your right hand controls the mouse pointer and you can PRESS with your left hand to simulate a single left mouse button click. You can also GRIP (make a fist) with your right hand to simulate holding down the left mouse button for dragging, selecting etc. To stop holding it down you simply RELEASE (stop making a fist). I went with the LEFT hand for PRESS rather than the right hand as I found pressing with the RIGHT hand tending to move the cursor and made it hard to click on small buttons.

Project Information URL: http://drenton72.wordpress.com/2013/05/09/kinect-magic-cursor-version-1-7-with-gesture-support/

Project Download URL: Click here to download Kinect Magic Cursor V1.7

Project Source URL: Click here to download Kinect Magic Cursor v1.7 with C# source

Contact Information:

• Blog: http://drenton72.wordpress.com
• Twitter: @drenton72

Kinect Interactions with(out) WPF – Part III: Demystifying the Interaction Stream

 In this part of my Kinect Interaction blog post series, we go deep into the rabbit hole, and examine the foundation of Kinect Interactions – the InteractionStream, upon which the entire library is built. This is a risky ride – with no official documentation, we can only count on our trusty reflector, the source code of the Kinect Interaction SDK and careful exploration.

You only need to access the treasures of InteractionStream, if you want to go beyond what the KinectRegion and other controls provide. For example, you want to create your own KinectRegion, you want to zoom a map by gripping it with two hands, or want to build your entirely new interaction model, using two hands along with the press and grip gestures.

Initializing the InteractionStream

Initializing the InteractionStream is much like initializing the DepthStream or the SkeletonStream. If you have a KinectSensor object, all you need are the next two lines of code:

...

It seems like we are on track. Just pass an initialized KinectSensor object and this DummyInteractionClient to the constructort of the InteractionStream, and we should be all set, right? Well, not quite. The InteractionFrameReady event does not fire.

Interaction Needs Skeleton and Depth

It turns out, that for the InteractionStream to work, it needs to process the data from both the depth and the skeleton streams. So, we need to initiate all three of the streams. This is what the entire OnLoaded method (which you have to wire up either in XAML or in the constructor of the page) looks like:

...

It doesn’t seem so simple now, so let's walk through each line of code and see what they do. The first five lines define local variables that hold a reference to the Kinect sensor the application will use, the InteractionStream itself, the skeletons identified by the SkeletonStream, and the hand position information about the users as determined by the InteractionStream.

...

Dissecting InteractionFrameReadyEventArgs

So, now that we have finally made the InteractionStream call the InteractionFrameReady event, we can start analyzing what kind of data we can get to. The first step is to get our hand on an InteractionFrame, and the interaction data it contains:

Summary

I’ve shown you how you can get started with the InteractionStream and acquire information about the hands of the users, detect active hands, pressing (IsPressed and PressExtent), and open / closed hands (HandEventType and our custom LastHandEventType). What I’ve shown here does not require WPF – you can use this from any .NET application, be it XNA, Windows Forms or even Command Line.

Project Information URL: http://dotneteers.net/blogs/vbandi/archive/2013/05/03/kinect-interactions-with-wpf-part-iii-demystifying-the-interaction-stream.aspx

Project Source URL: Source Download

private Dictionary<int, InteractionHandEventType> _lastLeftHandEvents = new Dictionary<int, InteractionHandEventType>(); private Dictionary<int, InteractionHandEventType> _lastRightHandEvents = new Dictionary<int, InteractionHandEventType>();
Dictionary<int, InteractionHandEventType>();
  
private void InteractionStreamOnInteractionFrameReady(object sender, 
ctionFrameReadyEventArgs args)
{
     using (var iaf = args.OpenInteractionFrame()) //dispose as soon as 
le
     {
         if (iaf == null)
             return;
  
         iaf.CopyInteractionDataTo(_userInfos);
     }
  
     StringBuilder dump = new StringBuilder();
  
     var hasUser = false;
     foreach (var userInfo in _userInfos)
     {
         var userID = userInfo.SkeletonTrackingId;
         if (userID == 0)
             continue;
  
         hasUser = true;
         dump.AppendLine("User ID = " + userID);
         dump.AppendLine("  Hands: ");
         var hands = userInfo.HandPointers;
         if (hands.Count == 0)
             dump.AppendLine("    No hands");
         else
         {
             foreach (var hand in hands)
             {
                 var lastHandEvents = hand.HandType == 
ctionHandType.Left
                                             ? _lastLeftHandEvents
                                             : _lastRightHandEvents;
  
                 if (hand.HandEventType != 
ctionHandEventType.None)
                     lastHandEvents[userID] = hand.HandEventType;
  
                 var lastHandEvent = lastHandEvents.ContainsKey(userID)
                                         ? lastHandEvents[userID]
                                         : 
ctionHandEventType.None;
  
                 dump.AppendLine();
                 dump.AppendLine("    HandType: " + hand.HandType);
                 dump.AppendLine("    HandEventType: " + 
andEventType);
                 dump.AppendLine("    LastHandEventType: " + 
ndEvent);
                 dump.AppendLine("    IsActive: " + hand.IsActive);
                 dump.AppendLine("    IsPrimaryForUser: " + 
sPrimaryForUser);
                 dump.AppendLine("    IsInteractive: " + 
sInteractive);
                 dump.AppendLine("    PressExtent: " + 
ressExtent.ToString("N3"));
                 dump.AppendLine("    IsPressed: " + hand.IsPressed);
                 dump.AppendLine("    IsTracked: " + hand.IsTracked);
                 dump.AppendLine("    X: " + hand.X.ToString("N3"));
                 dump.AppendLine("    Y: " + hand.Y.ToString("N3"));
                 dump.AppendLine("    RawX: " + 
awX.ToString("N3"));
                 dump.AppendLine("    RawY: " + 
awY.ToString("N3"));
                 dump.AppendLine("    RawZ: " + 
awZ.ToString("N3"));
             }
         }
  
         tb.Text = dump.ToString();
     }
  
     if (!hasUser)
         tb.Text = "No user detected.";
}

Contact Information:

• Blog: http://dotneteers.net/blogs/vbandi/
• Twitter: @vbandi

Today's comes from Ben Lower...

Using Kinect InteractionStream Outside of WPF

Last month with the release of version 1.7 of our SDK and toolkit we introduced something called the InteractionStream.  Included in this release were two new samples called Controls Basics and Interaction Gallery which, among other things, show how to use the new InteractionStream along with new interactions like Press and Grip.  Both of these new samples are written using managed code (C#) and WPF.

One question I’ve been hearing from developers is, “I don’t want to use WPF but I still want to use InteractionStream with managed code.  How do I do this?”  In this post I’m going to show how to do exactly that.  I’m going to take it to the extreme by removing the UI layer completely:  we’ll use a console app using C#.

The way our application will work is summarized in the diagram below:

There are a few things to note here:

1. Upon starting the program, we initialize our sensor, interactions, and create FrameReady event handlers.
2. Our sensor is generating data for every frame.  We use our FrameReady event handlers to respond and handle depth, skeleton, and interaction frames.
3. The program implements the IInteractionClient interface which requires us to implement a method called GetInteractionInfoAtLocation which gives us back information about interactions happening with a particular user at a specified location:
   ...

4. The other noteworthy part of our program is in the InteractionFrameReady method.  This is where we process information about our users, route our UI events, handle things like Grip and GripRelease, etc.

Project Information URL: http://blogs.msdn.com/b/k4wdev/archive/2013/05/01/using-kinect-interactionstream-outside-of-wpf.aspx

Project Source URL: sample code

Contact Information:

• Blog: http://blogs.msdn.com/b/k4wdev

Marcus has provided a great example of streamlining Kinect for Windows SDK development. It's simple, yet, when you see it, you'll know you want to grab this... (plus you've got to love that he's released the source too)

Fluent Kinect

Since I have been playing around with the Kinect for Windows SDK I’ve created a lot of little new projects and samples to try things out. Starting point was always something like this:

Because I most often use the 640×480 option anyway, the format is an optional parameter when enabling the streams and it defaults to *640x480Fps30.

I’ve extracted the two little lines that gets the first connected Kinect Sensor to a class called KinectConnector. At the moment an exception is thrown when no Kinect unit is connected. This is not a very good way of handling this scenario and will be changed in the future.

Now the code is even cleaner:

var sensor = KinectConnector.GetKinect()
.EnableColorStream()
.EnableSkeletonStream()
.EnableDepthStream()
.Seated()
.NearMode()
.Start();

For an even shorter and quicker Setup I’ve implemented the method ‘KickStart’ which enables the three streams and calls Start() on the KinectSensor object.

For future ‘try out samples’ I’ll just have to code this now:

var sensor = KinectConnector.GetKinect()
.KickStart();

Project Information URL: http://passiondev.wordpress.com/2013/05/03/fluent-kinect/

Project Download URL: FluentKinect

Project Source URL: FluentKinect

Contact Information:

• Blog: http://passiondev.wordpress.com/
• Twitter: @MarcusKohnert

Use your face as mouse control with Kinect

Replace your mouse with your face. Control the cursor just by moving your head. Click by winking your eyes, scroll by raising and lowering your eyebrows. All of that is possible now with FaceMouse Kinect.

During the usage of our KinectMouse, we figured out that it is very exhausting for your arm, if you have to control the mouse cursor all the time with your hand. So we were looking for a more easier method to control the mouse cursor with Kinect. All you have to do is moving your head and using some facial expressions for certain mouse actions.

...

How does our application work?

Cursor Moving

It is really easy: Just move your head to control the cursor. Make sure that Kinect can see your face as well as your chest. Sometimes the inital recognition is better when you are waving. It is normal that Kinect needs a few seconds to identify your face correctly. In contrast to KinectMouse, FaceMouse Kinect is more precise when you are more close to the sensor (but not too close) as the sensor has a more detailed view on your face this way. One meter should be a sufficient distance.

Left Click

Just wink with your right eye about a second...

Right Click...

Double Click...

Scrolling

Raise your eyebrows for scrolling up and lower it for scrolling down

Drag & Drop

Open your mouth for starting drag & drop. Move your head to move the cursor and keep your mouth open. For dropping, just close your mouth

Find the correct settings for yourself

Every face is different. It could be that the preselected settings in FaceMouse Kinect are not optimal for you. Just play a bit with the thresholds until you are satisfied.

• ClickDelay: .
• Headrotation Smoothing Filter Values: ...
• Percentage of horizontal edge Pixels: ....
• Used frames for closed eye detection: ....
• Eye closed filter threshold: ...
• Double click second eye threshold: ....
• Brow raiser start threshold:  ...
• Brow lowerer start threshold: ....
• Mouth open start threshold: ....
• Mouth open confirmation: ...
• Mouth open end threshold: ...
• Scroll multiplier up: ...
• Scroll multiplier down: ...
• Head to Screen relation X – Width: ...
• Head to Screen relation Y – Height: ...

I’m always happy to get some feedback. So please comment and let me know, if you are satisfied with this app. If you have any questions, feel free to ask. [Please click through and comment on the original post]

Project Information URL: http://futuretechblog.com/?p=71

Project Download URL: Download FaceMouse Kinect

Contact Information:

• Blog: http://futuretechblog.com/

Use your hand instead a mouse (with Kinect for Windows)

Within an university programming project we tried out several methods to control the mouse cursor with Kinect for Windows. One of the results is a small application, which lets you control the Windows mouse cursor with your hands. Generally the cursor moves like the Kinect cursor on Xbox, but in our opinion there are several disadvantages on the Xbox Kinect mouse control:

1. On Xbox you need to hold the cursor for about a second over a tile if you want to click on it. This is an unnecessary delay. So we were looking for a clicking method, which is as fast as a click on a PC mouse.
2. With the Xbox Kinect control you only have the opportunity to make a standard click. There is neither a possibility for a right click nor a possibility for a double click and also no possibilty for Drag & Drop. But you need all of these special mouse actions, when you are working with your PC.

How does our application work?

Cursor Moving

Just move your hand for controlling the cursor. The recognition of your hand is a bit insensitive when you are too close to the sensor. If the cursor is too much jumping, just go one or two steps back. When you start the application it takes a moment for the sensor to recognize your hand. Make sure that Kinect can see your face as well as your chest and both of your hands. Sometimes the inital recognition is better when you are waving.

Left Click ...

Double Click ...

Right Click ...

Drag & Drop ...

Project Information URL: http://futuretechblog.com/?p=26

Project Download URL: Download KinectMouse

Contact Information:

• Blog: http://futuretechblog.com

Frog Makes Star Trek’s Voice-Controlled Computers A Reality

Those are iconic words to any Star Trek fan--it’s the preferred drink of Captain Picard, as ordered from the Enterprise’s always-listening computer system. They also represent a vision of voice-activated, ubiquitous computing interfaces that took hold in sci-fi books and film nearly 70 years ago.

It’s taken a long time for our world to sync up to Picard’s, but with the advent of Kinect and voice recognition systems, it’s finally happening. "Writers from Heinlein to Doctorow envisioned a far more heads-up, cooperative, and simple way of engaging with technology," explain a team of Frog technologists behind RoomE, a heads-up computing project. “We think we’ve also reached a point in our technological evolution that will allow many of these visions to become real.“ Devised and built by Frog Fellow Jared Ficklin, RoomE is one of the first working examples of a type of ubiquitous computing interface only imagined for decades.

...

Installed at Frog’s Austin offices, RoomE’s hardware is all off-the-shelf: two Kinects provide an array of voice and motion sensors, while a series of projectors are positioned to turn any surface into a screen. The software is custom-built, using Microsoft Speech Recognition Engine, Computer Vision, and the Kinect SDK. “A lot of people seem to be working on various pieces, but no one has yet to combine them,” says Ficklin. “That’s one reason we had to build one for ourselves.” On YouTube, Ficklin uploads videos of himself pointing to certain lamps in an office and, in a friendly Texas lilt, telling RoomE to “turn on those lights.” He also turns them back on with a hand motion and then orders takeout from Yelp, tweets, controls the thermostat, and checks out the CCTV feed from the backyard--all using voice commands. "RoomE leverages the elegance of context," the team at Frog explains. "The system knows who is in the room and what is in the room via computer vision. Therefore when the command is issued, RoomE calculates where the command came from, and, by putting the context together, the results can be placed where they best serve the user."

...

Read more about RoomE here. Or, for those interested in building their own system, the team has published a great cookbook-style guide (PDF).

Project Information URL: http://www.fastcodesign.com/1672284/frog-makes-star-treks-voice-controlled-computers-a-reality#1

PowerShell and the Kinect

A few months back I wrote a “module” for scripting the Kinect with PowerShell. It’s located at: https://github.com/adminian/PowerKinect. Recently a new Kinect SDK was released. I’m going to be testing out the new SDK to make sure everything works as expected.

More info about the new SDK: http://blogs.msdn.com/b/kinectforwindows/archive/2013/03/18/the-latest-kinect-for-windows-sdk-is-here.aspx.

Currently there are only two gestures: right hand swipe and left hand swipe. There is also a function that starts PowerPoint and allows you to control your presentation.

Gary Siepser, a fellow PFE and PowerShell Guru, jumped in and started helping out. He’s written the Audio control functions for the module. We’ll be integrating them into the repo soon! I’ll be posting more about Scripting the Kinect, but for now go clone the repo and give it a try!

Project Information URL: http://adminian.com/2013/03/24/powershell-and-the-kinect/

PowerKinect

A module to control the Kinect from PowerShell.

Currently we only support Kinect SDK version 1.6. Download located here: http://go.microsoft.com/fwlink/?LinkID=262831

Sample-KinectScripts.ps1 has a dependency on ShowUI: http://showui.codeplex.com/

The following is PowerKinects Hello world...

Start-Kinect

Enable-SkeletonStream

Add-RightHandGesture -action { notepad }

Add-LeftHandGesture -action { ps notepad | kill }

That's it!

Project Source URL: https://github.com/adminian/PowerKinect

$global:ie = New-Object -ComObject InternetExplorer.Application
$global:ie.GoHome()
$global:ie.Visible = $true
sleep 1
$global:ie.Navigate("http://www.microsoft.com")
sleep 1
$global:ie.Navigate("http://www.adminian.com")
sleep 1
$global:ie.Navigate("http://powershellsaturday.com/003/")
sleep 1

Add-RightHandGesture -action { $global:ie.GoForward() }
Add-LeftHandGesture -action { $global:ie.GoBack() }

Contact Information:

• Blog: http://adminian.com/
• Twitter: @Admnian

Kinect Interaction with WPF Part II: Getting Started Easier [KinectChooserEx]

In the previous post of this series, I have introduced the Kinect for Windows SDK 1.7, and the key controls and concepts within its Kinect Interactions toolkit. If you have followed through that post, you have seen all the work and code you need to get the basics running. Most of the initialization code was biolerplate, and code that you can easily copy and paste into your own project.

I have packaged up the initialization code into a UserControl called KinectChooserEx, and wired up a couple of dependency properties.

Note: I could have inherited from KinectChooser instead of packaging it into a user control. But I ran into some serious strong name issues, and wanted more consistent naming for the properties, so it was better to hide some internal workings.

The NearMode property is a boolean value, and can be used to turn Near Mode on and off. Changing Near Mode also sets the EnableTrackingInNearRange of the SkeletonStream. You will probably want to keep NearMode as false if you are using a Kinect for XBox sensor instead of a Kinect for Windows one. Otherwise the interaction engine works better at close range, so I have set NearMode to be true by default.

The other property is KinectSensor, which contains null if no sensor is initialized, or a KinectSensor object if one is.

Both of these properties are bindable. With KinectChooserEx, you can finally get started with your own Kinect project purely in Blend, without writing a single line of code (at least for the interaction part). Here is a step by step tutorial on how to do this:

• Create a New Project and Add KinectChooserEx.xaml and KinectChooserEx.xaml.cs

Create a new WPF project, and add the necessary references as described in my previous post. Drag and drop the KinectChooserEx.xaml and KinectChooserEx.xaml.cs files from Explorer into the project. You will also want to add the KinectInteraction170_32.dll and KinectInteraction170_64.dll files at this point (again, see my previous post).

• Add KinectChooserEx to the MainPage
• Adding a KinectRegion
• Adding a KinectUserViewer control
• Let’s See it Work!
• Using the Near Mode

Project Information URL: http://dotneteers.net/blogs/vbandi/archive/2013/04/19/kinect-interaction-with-wpf-part-ii-getting-started-easier.aspx

Project Download URL: http://sdrv.ms/117qGxo

Project Source URL: http://sdrv.ms/117qGxo

Contact Information:

• Blog: http://dotneteers.net/blogs/vbandi/
• Twitter: @vbandi

The Kinectinator

The Kinectinator is a Kinect-controlled turret. If a human walks in range of the Kinect, the Kinectinator starts tracking the human with a pan/tilt foam dart shooter and after a few seconds of tracking, it fires the dart. The original version was built as a Kinect demo for the Tech Olympics in Cincinnati, OH on February 2013.

Here's an early video of it in action: http://www.facebook.com/photo.php?v=10151321555418932&l=1891434071247027160 (it was later changed to target mid-chest because getting shot in the face with a foam dart over and over again gets old)

The software is in two portions:

• A PC program that works with the Kinect to track humans, identify target points and send control data to the sensor. It also displays some targeting info an graphics on the screen. (ooh, pretty).
• An Arduino program that controls the pan/tilt servos and the pneumatic valve that makes the foam fly. It talks to the PC via a USB serial connection.

The Kinectinator hardware is a bracket that slides onto a Kinect. It holds the base servo of a small pan/tilt system. A bracket/manifold on the head of the pan/tilt takes tubing in from a 1/8" high-pressure line and connects it to an aluminum tube that holds the foam dart around it.

The Arduino source, 3D printer models and more detailed pictures of the build are forthcoming.

Project Information URL: http://kinectinator.codeplex.com/

Project Source URL: http://kinectinator.codeplex.com/SourceControl/BrowseLatest

Tutorial – Kinect Television

Welcome to the very first Kinecting for Windows tutorial.

Introduction

The goal of this application is to create a television where you are in the picture!
It starts with a static tv untill you plug your Kinect in & it is connected, that’s where you jump in.

You’ll be able to see yourself and change the angle or output type in the settings panel.

Last but least, when you disconnect your sensor it will go back to static and vice versa so that our application is more stable as well.

I wrote a post that introduces you to the SDK, I recommend you read it before you move on.

What you will learn

• Connect a Kinect sensor
• Enable Color-data
• Show camera feed
• Tilt the camera
• Selecting camera output at runtime
• Supporting different Kinect states

Project Information URL: http://www.kinectingforwindows.com/2013/04/09/tutorial-kinect-television/

Project Source URL: https://github.com/KinectingForWindows/KinectTelevision

Contact Information:

• Blog: http://www.kinectingforwindows.com/
• Twitter: @KinectingforWin

Just For Fun–KINECTing Orchestrator Runbooks

Hello Readers/Viewers!

As you can see, I am really trying hard to get both of my quotas met: blog posts + fun…

That said, this post, while dealing a very small amount with Automation and System Center Orchestrator, is really just me getting my geek on for a few technologies I was exploring at the time – yes, at the time – this video coming up on 2 years old now.

Which is something that should be noted… it is old and has some older references (including my job title). Again, this is just for fun!

So why bring it back up?

Why not?!?

We have new audience, the content is still valid, and hey, the blog post did have “Just For Fun” in the title…

How did I do this?

Well, with a bit of help from existing example projects created using the KINECT for Windows SDK Beta, I was able to create a simplistic “app” which not only displays the KINECT “Skeleton”, but also tracks movement in the right and left hands to determine mouse events (movement and click). Leveraging this NUI, I was able to [slowly] create a simple Runbook with ONLY hand and arm gestures.

REMINDER - This approach to Runbook creation is completely impractical for day-to-day Runbook design. Mouse and keyboard usage are still the only supported method for creating Orchestrator Runbooks.

If you are curious, I used a mix of the following KINECT examples to throw this “app” together:

• http://www.microsoft.com/en-us/kinectforwindows/
• http://kinectmouse.codeplex.com/

Project Information URL: http://blogs.technet.com/b/privatecloud/archive/2013/03/01/just-for-fun-kinecting-orchestrator-runbooks.aspx

Contact Information:

• Blog: http://blogs.technet.com/b/privatecloud/

Creating a PowerPoint add-in that uses Kinect to move through slides

If you are interested in developing Kinect applications, you can start with the latest version of Kinect SDK, available here:

http://www.microsoft.com/en-us/kinectforwindows/develop/developer-downloads.aspx

After installing the SDK, you might want to install the Developer Toolkit also, in order to study the source code for different functionalities. The Toolkit is available here:

http://www.microsoft.com/en-us/download/details.aspx?id=34807

You have plenty of sample projects for the various things that you can program for Kinect, from gesture recognition to face or voice recognition.

Once you installed those two things, you can start developing something interesting. For example, in the sample projects you have an application that demonstrates how to use swipe gestures to browse through a photo archive. In this tutorial, we will use the same functionality to create a more useful application: an add-in that allows a presenter to swipe through the slides of his PowerPoint presentation.

In order to do that, we will use a few elements:

1. The Kinect drivers and SDK. Be aware that the Kinect drivers are not available separately, but only inside the SDK, and if you will want to install your final application on another machine, you will need
   to install the SDK on that machine also
2. The Kinect sensor
3. An add-in containing code for gesture recognition and for PowerPoint slide change
4. Visual Studio 2010

First, create a PowerPoint 2010 Add-in in VS 2010, by choosing the standard project template.

In the Kinect Developer Toolkit, you will find a sample application named “Slideshow Gestures – WPF”. Install the project and open it with VS 2010.
Look at the references and you will see that it uses a .dll named Microsoft.Samples.Kinect.SwipeGestureRecognizer.dll.
Copy this dll to your Add-in project and add it to references, as it will be in charge of the swipe gesture recognition.

Add these two using directives to the main page of your project:

...

Start the PowerPoint slideshow.

Stand in front of the Kinect sensor at a distance between 1.5 meters and 4 meters

To move to the next slide, do a swiping movement with your right hand, from right to left. The hand should be at chest-level and the movement should be continuous

To move to the previous slide, do a swiping movement with your left hand, from left to right

That would be all. You can use this sample in your team presentations, to leverage the Kinect technology and demonstrate that it can have a multitude of uses, outside the gaming world.

You can download the sample from here: http://sdrv.ms/13PMDX0

Project Information URL: http://blogs.msdn.com/b/alexpetr/archive/2013/03/10/creating-a-powerpoint-add-in-that-uses-kinect-to-move-through-slides.aspx

Project Source URL: http://sdrv.ms/13PMDX0

Contact Information:

• Blog: http://blogs.msdn.com/b/alexpetr/

How to do Push to Press with Kinect SDK 1.7

Microsoft officially released the 1.7 Kinect for Windows SDK. In this article I will demonstrate or explain what interest me most. The Push-to-Press functionality.

Introduction

Kinect 1.7 sdk has been released, when I first played around with the examples like ‘Interaction Gallery’ and the fact that I don’t to adjust the jittering anymore , I knew I had to stop whatever I was doing in my Kinect project to learn “Press to Push” and implement in my current project. Before with the help of Coding4Fun toolkit we would place a hand over a button and some nice animation cycling your hand will appear telling you that you are about to select something, I had to struggle teaching my 2 years old daughter to wait for you, I was teaching her un-natural user interface, with Kinect 1.7, we have a natural user interface gesture, because you push to press a button. I must say a struggled a bit to isolate this functionality. I hope it will help someone.

...

Objective

In this article I wanted to show you on how to create a basic navigation system in wpf that uses hand gestures to select menu items. This is an updated version from the one I created from Coding4Fun toolkit. This one uses the press to push. So this means to select a menu, you need to push the button to press it. 

Code Explanation

Few things have changed from the way we used to do things in sdk 1.6. Well you can still choose to Coding4Fun toolkit to do hand gestures that interact with your controls, but in this article I will explain how you can use the nice things found in the new sdk. The first thing that is new is the Kinect Region.

• KinectRegion
• KinectTileButton
• KinectScrollViewer
• UserViewer

Generic Class to store the KinectSensorChooser

...

Project Information URL: http://www.dotnetfunda.com/articles/article2214-how-to-do-push-to-press-with-kinect-sdk-17.aspx

Project Download URL: How to do Push to Press with Kinect SDK 1.7

Project Source URL: How to do Push to Press with Kinect SDK 1.7

Kinect Interactions with WPF - Part I: Getting Started

... Kinect Interactions. Kinect Interactions allow developers to create interactive applications with a new, well-designed and ergonomically friendlier interaction language than before. The Kinect SDK can now detect a “push towards the screen” gesture to activate buttons, and this is the recommended “activate” gesture now instead of the old “hover, wait until the circle fills and its pressed” gesture.

Browsing lists larger than a dozen or so items has been even more difficult using Kinect. The recommended solution was to implement a “swipe from side” gesture, which paged through items on the screen. Here is such a list, straight from my XBox:

You can see that the list has 358 games, and 5 of them are visible at the same time. To get through even half of this list, you would have to perform the swipe gesture 36 times! This is extremely slow, frustrating and physically exhausting.

The latest K4W SDK solves this issue by differentiating between an open and a closed hand. When you close your hand, you “grip” whatever’s underneath it (such as a scrollviewer), and then you can move your hand to scroll. If you release the scroller while your hand is still moving, the scrolling keeps happening – similarly to the inertia scroll you are already used to on touch devices. You can close your hand again at any time, and stop the inertia – again, as you expected.

In this post, I will help you to get started with Kinect Interactions by creating a Hello World WPF app that uses the features mentioned above. In later posts, I will dive deeper into the Kinect Interactions SDK, show you how to customize the existing controls, create new ones and go deep into the lower levels of the InteractionStream to invent new ways of interacting with your app. But first, let’s see how you can get started with Kinect Interactions today!

Note: I am assuming that you are familiar with C# and WPF for this article series.

...

• First Things First – Download and Install the bits
• Create the Project
• Initializing the Kinect Sensor with KinectSensorChooser
• Setting up the Kinect for Interactions
• The Kinect Region
• User Viewer
• Interactive Controls
  • The KinectTileButton
  • The KinectCircleButton
• They See Me Scrollin’

...

Download

You can download the demo project for this article here.

Summary

In this post, I’ve introduced you to the basics to get started with a Kinect Interactions application – getting the Kinect sensor up and running, defining a KinectRegion, adding buttons and scrollable areas to it.

In the next posts of this series, I will dive deeper into the world of Kinect Interactions. We will dive deeper into the – unfortunately very sparsely documented – controls I discussed above, and see the customization options they offer. And finally, going deeper into the rabbit hole, we will examine the low level Interaction Stream.

Project Information URL: http://dotneteers.net/blogs/vbandi/archive/2013/03/25/kinect-interactions-with-wpf-part-i-getting-started.aspx

Project Source URL: http://sdrv.ms/10tWqi9

Contact Information:

• Blog: http://dotneteers.net/blogs/vbandi
• Twitter: @vbandi

Karl is continuing to improve his project and has recently moved it to CodePlex and updated with the Kinect for Windows v1.7 SDK.

Kinect Depth Smoothing

Removes noise from the Kinect Depth Frames in real-time using pixel filters and weighted moving average techniques.

More here: http://www.codeproject.com/Articles/317974/KinectDepthSmoothing

This uses version 1.7 of the Kinect for Windows SDK: http://www.microsoft.com/en-us/kinectforwindows/develop/developer-downloads.aspx

Project Information URL: http://kinectdepthsmoothing.codeplex.com

Project Download URL: http://kinectdepthsmoothing.codeplex.com/releases/

Project Source URL: http://kinectdepthsmoothing.codeplex.com/SourceControl/BrowseLatest

Contact Information:

• Twitter: @krsanford

Kinect + Brain Scan = Augmented Reality for Neurosurgeons

With a little duct tape, a touch screen tablet, and their new Kinect API, the Microsoft Research Cambridge team built an augmented reality system to help brain surgeons visualize 3D brain scans. Kinect Fusion supplies 3D modeling of anything, which could fuel some seriously neat medical innovations. (The Cambridge team also built KinEtre, which lets you posses anything.) At the 13th annual Microsoft TechFest, Ben Glocker demoed a prototype system that would allow neurosurgeons to prepare for surgery by looking inside a patient's brain before they cut it open. Doctors could see the skeleton, brain, blood vessels, and the targeted tumor on a tablet—which they can move around the patient's head—helping them to plot the best brain surgery path.

Project Information URL: http://spectrum.ieee.org/automaton/robotics/medical-robots/microsoft-kinect-fusion-augmented-reality-neurosurgeons

Found via: New Kinect Mod Lets You See Your Brain Inside Your Head, Redefines The Word ‘Creepy’ [VIDEO]

Please join me in welcoming back Kean Walmsley and his combining the Kinect and AutoCAD (and with Fusion, that seems like a match made in... well, maybe after reading this, not Heaven, but pretty close!)

Kinect Fusion inside AutoCAD

OK, here goes: my first (public) attempt at integrating the brand new Kinect Fusion functionality – made available this week in v1.7 of Microsoft’s Kinect for Windows SDK – into AutoCAD. There are still a few quirks, so I dare say I’ll be posting an update in due course.

As mentioned in the last post, I’ve been working on this for some time but can now show it publicly, as the required SDK capabilities have now been published. As part of this effort, I’ve gone ahead and made sure the other Kinect samples I’ve written for AutoCAD work with this version of the SDK: all can be found here.

Much of the work was clearly to integrate the appropriate Kinect API calls into an AutoCAD-resident jig, much in the way we’ve seen before when displaying/importing a single depth frame. Kinect Fusion introduces the idea of a reconstruction volume that gets gradually populated with data streamed in from a Kinect sensor, building up an underlying mesh that represents the 3D model.

AutoCAD is OK with meshes to a certain size, but I wanted to get at the raw point data, instead. The Kinect team has kindly provided the Reconstruction.ExportVolumeBlock() method for just this purpose – it’s intended to populate an array with voxel data which you can interpolate trilinearly to extract model/mesh information (erk) – but I haven’t yet been able to have it return anything but an array of zeroes. So the code is currently asking the Kinect Fusion runtime to calculate a mesh from the reconstruction volume and we then use the vertices from that mesh as points to display.

The typical Kinect Fusion sample makes use of a quite different technique: it generates a shaded view of the mesh from a particular viewpoint – the underlying API casts rays into the reconstruction volume – which is very quick. Calculating a mesh and extracting its vertices is slower – especially when we get into the millions of points – so we have to accept the responsiveness is going to be different.

And that’s mostly OK: we simply drop incoming frames when we’re already processing one, as otherwise we build up a queue of unprocessed frames leading to a significant lag between the movement of the sensor and the population of the reconstruction volume. But this also means that there’s a much bigger risk of the Kinect Fusion runtime not being able to track the movement – as the time between processed frames is larger and so are the differences – at which point we receive “tracking failures”.

...

Project Information URL: http://through-the-interface.typepad.com/through_the_interface/2013/03/kinect-fusion-inside-autocad.html

Project Source URL: complete samples

Contact Information:

• Blog: http://through-the-interface.typepad.com/
• Twitter: @keanw

Seems fitting to jump forward a couple years, and highlight another such project, yet one that's one step (or two) beyond. Light Saber duels anyone? Hulk Smash?

And there's source of course!

Slides and Demos for Programming With Kinect for Windows Talk, and Introducing JediSmash

I have been playing a lot with the Kinect for Windows this year, and gave presentations on it at LRTechFest and Cowtown CodeCamp recently. As promised, you can find the slides and demos here.

I also showed a fun WPF application that combines multiple Kinect capabilities like Color, Depth, Skeletal tracking and speech recognition. It’s called JediSmash.

I also got a chance to show the Kinect Replay application that I built that lets you record and replay Kinect sessions without having the actual Kinect hardware. I’ll blog about it soon. I am excited about the version 1.7 of the SDK that came out this week, and am planning to talk about the new features as I do more Kinect related talks in user groups/conferences in Texas and surrounding areas this year. If you lead a user group and would like me to present, feel free to get in touch

Project Information URL: http://dotnetsurfers.com/blog/2013/03/17/slides-and-demos-for-programming-with-kinect-for-windows-talk/

Project Source URL: JediSmash

You know what? We're going to make this a two for one day! Here's another great post by Latish...

Kinect.Replay: Record and Replay Your Kinect Sessions

One of the limitations of programming with Kinect is that you always need the Kinect hardware plugged in during development time. You can use Kinect Studio to record and replay sessions (it gets tiring after a while to get up repeatedly in front of the Kinect and debug at the same time, and your spouse/kids/pets start looking at you funny), but you still need the Kinect hardware to replay a Kinect Studio session. Being hardware independent can be useful in scenarios like programming while on the move (Most of my side projects are done in coffee shops), or running integration tests on a server. I created Kinect.Replay as an experiment to see if I can remove the dependency on the hardware during some parts of development. The core of the code is based off the work done by David Catuhe in Kinect Toolbox. My initial approach was to enhance Kinect Toolbox, but the direction I was taking soon started to look like I was butchering the code, and I ended up creating a separate project for my needs. Some of the enhancements I have done include storing ColorToDepthRelationalParameters to recreate CoordinateMapper during replay, and also support for audio recording/replay. You can see a small demo of what the using Kinect.Recorder looks like in the video below.

Project Information URL: http://dotnetsurfers.com/blog/2013/03/19/kinect-dot-replay-record-and-replay-your-kinect-sessions/

Project Source URL: Kinect.Replay

Contact Information:

• Blog: http://dotnetsurfers.com/blog
• Twitter: @Latish

Kinect Educational App: River Crossing by Kinems

River Crossing” is a new Kinect educational app available for download, freely provided by Kinems.  Kinems is a company that provides learning games utilizing Kinect for K-9 children and enriches the traditional therapeutic method with their motion-sensored educational games.

Here is some more info about River Crossing from the Kinems Game Launcher:

Age: 4+

Objective: Visual-kinetic coordination and development of critical thinking.

Description: In the ‘River Crossing’ game, the child undertakes the task to lead a boat in a river and transfers a duck, a fox and a sack of seeds from one shore to the other. The child-skipper should be very careful so as not to throw his boat on the rocks that exist in the river. Sometimes the passage becomes narrower or wider, depending on the difficulty of the game! For more experienced skippers, the task becomes more complex and as a result the duck should not be abandoned with the fox or the duck with the seeds at the same river shore, since there is always the risk that one disappears from the other. The correct option, in conjunction with careful driving, is to practice the delicate mobility, the visual-kinetic coordination and the grapho-kinetic skills of the children, contributing in this way in the development of the necessary skills for the conquest of writing.”

Project Information URL: http://www.kinecteducation.com/blog/2013/02/12/kinect-educational-app-river-crossing-by-kinems/

Project Download URL: http://apps.kinecteducation.com/catalog/river-crossing-by-kinems.html

from Vimeo.

Contact Information:

• Blog: http://www.kinecteducation.com/blog/
• Twitter: @johnnyeducation