Kinect for Windows SDK Beta 2 Quickstarts - Channel 9

Audio Fundamentals (Beta 2 SDK)

Clint Rutkas, Dan Fernandez — Thu, 16 Jun 2011 17:15:43 GMT

Update: Kinect for Window SDK v1 Quickstart Series now Available (Feb 1st)

Please use the newly updated Kinect for Windows SDK Quickstart series. The content below will only work with the Beta 2 version of the Kinect for Windows SDK.

This video covers the basics of reading audio data from the Kinect microphone array, a demo adapted from the built in audio recorder. The video also covers speech recognition using Kinect. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides that have been updated for Beta 2 (Nov, 2011).

[00:35] Kinect microphone information
[01:10] Audio data
[02:15] Speech recognition information
[05:08] Recording audio
[08:17] Speech recognition demo

Updates for Kinect for Windows SDK Beta 2 (Nov, 2011)

The video has not been updated for Beta 2, but the following changes have been made:

Beta 2 now enables you to record audio on a Single-Threaded Apartment (STA) thread, the default thread that is used for WPF applications. Previously, you had to create a new thread marked as a Multi-Threaded Apartment (MTA) for audio processing to work.
Beta 2 includes a new WPF audio example, KinectAudioDemo, that demonstrates speech recognition and calculating the angle of the current sound source.

Setup

The steps below assume you have setup your development environment as explained in the "Setting Up Your Development Environment" video.

Task: Designing Your UI

We’ll add in a Slider and two Button controls, and we'll also use some stack panels to be sure everything lines up nicely:

XAML

http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="Audio Recorder Sample" Height="159" Width="525">

Creating Click events

For each button, we'll want to create a click event. Go to the properties window (F4), select the RecordButton, select the Events tab, and double click on the Click event to create the RecordButton_Click event. Do the same for the Play Button so we have the PlayButton_Click event wired up as well

Task: Working with the KinectAudioSource

The first task is to add in the Kinect Audio library:

using Microsoft.Research.Kinect.Audio;

Visual Basic

Imports Microsoft.Research.Kinect.Audio

Synchronous and asynchronous recording

There are two ways we can record audio. You can record audio synchronously, meaning that the UI thread will in effect be “frozen” while we record audio using it. Alternatively, you can record audio on a separate thread so that the UI thread remains responsive to events while the recording happens in parallel. Our sample includes both methods so you can choose which one is required for your application.

We’ll build variables to hold the amount of time we’ll record, the file name of the recording, and to enable asynchronous recording, we’ll use the FinishedRecording event to notify the UI thread that we're done recording:

double _amountOfTimeToRecord;
string _lastRecordedFileName;
private event RoutedEventHandler FinishedRecording;

Visual Basic

Private _amountOfTimeToRecord As Double
Private _lastRecordedFileName As String
Private Event FinishedRecording As RoutedEventHandler

Next we’ll create the RecordAudio method that will do the actual audio recording.

private void RecordAudio()
{
}

Visual Basic

Private Sub RecordAudio()
End Sub

To create threads, we'll add in the System.Threading namespace:

using System.Threading;

Visual Basic

Imports System.Threading

Now we'll create the thread and do some simple end-user management in the RecordButton_Click event. First we'll disable the two buttons, record the audio, and create a unique file name.

Then we have the option of calling the RecordAudio method either synchronously or asynchronously as shown below:

‘C#

private void RecordButton_Click(object sender, RoutedEventArgs e)
{
    RecordButton.IsEnabled = false;
    PlayButton.IsEnabled = false;
    _amountOfTimeToRecord = RecordForTimeSpan.Value; 
    _lastRecordedFileName = DateTime.Now.ToString("yyyyMMddHHmmss") + "_wav.wav";
            
    var t = new Thread(new ThreadStart(RecordAudio));
    t.SetApartmentState(ApartmentState.MTA);
    t.Start();
}

Visual Basic

Private Sub RecordButton_Click(ByVal sender As Object, ByVal e As RoutedEventArgs)

    RecordButton.IsEnabled = False
    PlayButton.IsEnabled = False
    _amountOfTimeToRecord = RecordForTimeSpan.Value
    _lastRecordedFileName = Date.Now.ToString("yyyyMMddHHmmss") & "_wav.wav"

    Dim t = New Thread(New ThreadStart(AddressOf RecordAudio))
    t.SetApartmentState(ApartmentState.MTA)
    t.Start()

End Sub

Task: Capturing Audio Data

From here, this sample and the built-in sample are pretty much the same. We'll only add three differences: the FinishedRecording event, a dynamic playback time, and the dynamic file name. Note that the WriteWavHeader function is the exact same as the one in the built-in demo as well. Since we leverage different types of streams, we'll add the System.IO namespace:

using System.IO;

Visual Basic

Imports System.IO

The entire RecordAudio method:

private void RecordAudio()
{
    using (var source = new KinectAudioSource())
    {
        var recordingLength = (int) _amountOfTimeToRecord * 2 * 16000;
        var buffer = new byte[1024];
        source.SystemMode = SystemMode.OptibeamArrayOnly;
        using (var fileStream = new FileStream(_lastRecordedFileName, FileMode.Create))
        {
            WriteWavHeader(fileStream, recordingLength);

            //Start capturing audio                               
            using (var audioStream = source.Start())
            {
                //Simply copy the data from the stream down to the file
                int count, totalCount = 0;
                while ((count = audioStream.Read(buffer, 0, buffer.Length)) > 0 && totalCount < recordingLength)
                {
                    fileStream.Write(buffer, 0, count);
                    totalCount += count;
                }
            }
        }

        if (FinishedRecording != null)
            FinishedRecording(null, null);
    }
}

Visual Basic

Private Sub RecordAudio()
    Using source = New KinectAudioSource

        Dim recordingLength = CInt(Fix(_amountOfTimeToRecord)) * 2 * 16000
        Dim buffer = New Byte(1023) {}

        source.SystemMode = SystemMode.OptibeamArrayOnly

        Using fileStream = New FileStream(_lastRecordedFileName, FileMode.Create)

            WriteWavHeader(fileStream, recordingLength)

            'Start capturing audio                               
            Using audioStream = source.Start()

                'Simply copy the data from the stream down to the file
                Dim count As Integer, totalCount As Integer = 0
                count = audioStream.Read(buffer, 0, buffer.Length)
                Do While count > 0 AndAlso totalCount < recordingLength

                    fileStream.Write(buffer, 0, count)
                    totalCount += count

                    count = audioStream.Read(buffer, 0, buffer.Length)
                Loop

            End Using

        End Using

        RaiseEvent FinishedRecording(Nothing, Nothing)

    End Using

End Sub

Task: Playing Back the Audio We Just Captured

So we've recorded the audio, saved it, and fired off an event that said we're done—let's hook into it. We'll wire up that event in the MainWindow constructor:

public MainWindow()
{
    InitializeComponent();

    FinishedRecording += new RoutedEventHandler(MainWindow_FinishedRecording);
}

Visual Basic

Public Sub New()
    InitializeComponent()

    AddHandler FinishedRecording, AddressOf MainWindow_FinishedRecording
End Sub

Since that event will return on a non-UI thread, we'll need to use the Dispatcher to get us back on a UI thread so we can reenable those buttons:

void MainWindow_FinishedRecording(object sender, RoutedEventArgs e)
{
    Dispatcher.BeginInvoke(new ThreadStart(ReenableButtons));
}

private void ReenableButtons()
{
    RecordButton.IsEnabled = true;
    PlayButton.IsEnabled = true;
}

Visual Basic

Private Sub MainWindow_FinishedRecording(sender As Object, e As RoutedEventArgs)
    Dispatcher.BeginInvoke(New ThreadStart(ReenableButtons))
End Sub

Private Sub ReenableButtons()
    RecordButton.IsEnabled = True
    PlayButton.IsEnabled = True
End Sub

And finally, we'll make the Media element play back the audio we just saved! We'll also verify both that the file exists and that the user recorded some audio:

private void PlayButton_Click(object sender, RoutedEventArgs e)
{
    if (!string.IsNullOrEmpty(_lastRecordedFileName) && File.Exists(_lastRecordedFileName))
    {
        audioPlayer.Source = new Uri(_lastRecordedFileName, UriKind.RelativeOrAbsolute);
        audioPlayer.LoadedBehavior = MediaState.Play;
        audioPlayer.UnloadedBehavior = MediaState.Close;
    }
}

Visual Basic

Private Sub PlayButton_Click(sender As Object, e As RoutedEventArgs)

    If (Not String.IsNullOrEmpty(_lastRecordedFileName)) AndAlso File.Exists(_lastRecordedFileName) Then

        audioPlayer.Source = New Uri(_lastRecordedFileName, UriKind.RelativeOrAbsolute)
        audioPlayer.LoadedBehavior = MediaState.Play
        audioPlayer.UnloadedBehavior = MediaState.Close

    End If

End Sub

Task: Speech Recognition

To do speech recognition, we need to bring in the speech recognition namespaces from the speech SDK:

using Microsoft.Speech.AudioFormat;
using Microsoft.Speech.Recognition;

Visual Basic

Imports Microsoft.Speech.AudioFormat
Imports Microsoft.Speech.Recognition

In VB we'll also need to add in a MTA flag as well under the Sub Main. C# does not need this.

Visual Basic

 _
Shared Sub Main(ByVal args() As String)

Next, we need to setup the KinectAudioSource in a way that's compatbile for speech recognition:

using (var source = new KinectAudioSource())
{
    source.FeatureMode = true;
    source.AutomaticGainControl = false; //Important to turn this off for speech recognition
    source.SystemMode = SystemMode.OptibeamArrayOnly; //No AEC for this sample
}

Visual Basic

Using source = New KinectAudioSource

source.FeatureMode = True
source.AutomaticGainControl = False 'Important to turn this off for speech recognition
source.SystemMode = SystemMode.OptibeamArrayOnly 'No AEC for this sample

End Using

With that in place, we can initialize the SpeechRecognitionEngine to use the Kinect recognizer, which was downloaded earlier:

private const string RecognizerId = "SR_MS_en-US_Kinect_10.0";
RecognizerInfo ri = SpeechRecognitionEngine.InstalledRecognizers().Where(r => r.Id == RecognizerId).FirstOrDefault();

Visual Basic

Private Const RecognizerId As String = "SR_MS_en-US_Kinect_10.0"
Dim ri As RecognizerInfo = SpeechRecognitionEngine.InstalledRecognizers().Where(Function(r) r.Id = RecognizerId).FirstOrDefault()

Next, a "grammar" needs to be setup, which specifies which words the speech recognition engine should listen for. The following code creates a grammar for the words "red", "blue" and "green".

using (var sre = new SpeechRecognitionEngine(ri.Id))
{                
    var colors = new Choices();
    colors.Add("red");
    colors.Add("green");
    colors.Add("blue");
    var gb = new GrammarBuilder();
    //Specify the culture to match the recognizer in case we are running in a different culture.                                 
    gb.Culture = ri.Culture;
    gb.Append(colors);
  
    // Create the actual Grammar instance, and then load it into the speech recognizer.
    var g = new Grammar(gb);                  
    sre.LoadGrammar(g);
}

Visual Basic

Using sre = New SpeechRecognitionEngine(ri.Id)

Dim colors = New Choices
colors.Add("red")
colors.Add("green")
colors.Add("blue")

Dim gb = New GrammarBuilder
'Specify the culture to match the recognizer in case we are running in a different culture
gb.Culture = ri.Culture
gb.Append(colors)

' Create the actual Grammar instance, and then load it into the speech recognizer.
Dim g = New Grammar(gb)

sre.LoadGrammar(g)

End Using

Next, several events are hooked up so you can be notified when a word is recognized, hypothesized, or rejected:

sre.SpeechRecognized += SreSpeechRecognized;
sre.SpeechHypothesized += SreSpeechHypothesized;
sre.SpeechRecognitionRejected += SreSpeechRecognitionRejected;

Visual Basic

AddHandler sre.SpeechRecognized, AddressOf SreSpeechRecognized
AddHandler sre.SpeechHypothesized, AddressOf SreSpeechHypothesized
AddHandler sre.SpeechRecognitionRejected, AddressOf SreSpeechRecognitionRejected

Finally, the audio stream source from the Kinect is applied to the speech recognition engine:

using (Stream s = source.Start())
{
    sre.SetInputToAudioStream(s,
                              new SpeechAudioFormatInfo(
                                  EncodingFormat.Pcm, 16000, 16, 1,
                                  32000, 2, null));
    Console.WriteLine("Recognizing. Say: 'red', 'green' or 'blue'. Press ENTER to stop");
    sre.RecognizeAsync(RecognizeMode.Multiple);
    Console.ReadLine();
    Console.WriteLine("Stopping recognizer ...");
    sre.RecognizeAsyncStop();                       
}

Visual Basic

Using s As Stream = source.Start()

sre.SetInputToAudioStream(s, New SpeechAudioFormatInfo(EncodingFormat.Pcm, 16000, 16, 1, 32000, 2, Nothing))

Console.WriteLine("Recognizing. Say: 'red', 'green' or 'blue'. Press ENTER to stop")

sre.RecognizeAsync(RecognizeMode.Multiple)
Console.ReadLine()
Console.WriteLine("Stopping recognizer ...")
sre.RecognizeAsyncStop()

End Using

The event handlers specified earlier display information based on the result of the user's speech being recognized:

static void SreSpeechRecognitionRejected(object sender, SpeechRecognitionRejectedEventArgs e)
{
    Console.WriteLine("\nSpeech Rejected");
    if (e.Result != null)
        DumpRecordedAudio(e.Result.Audio);
}

static void SreSpeechHypothesized(object sender, SpeechHypothesizedEventArgs e)
{
    Console.Write("\rSpeech Hypothesized: \t{0}\tConf:\t{1}", e.Result.Text, e.Result.Confidence);
}

static void SreSpeechRecognized(object sender, SpeechRecognizedEventArgs e)
{
    Console.WriteLine("\nSpeech Recognized: \t{0}", e.Result.Text);
}

private static void DumpRecordedAudio(RecognizedAudio audio)
{
    if (audio == null)
        return;

    int fileId = 0;
    string filename;
    while (File.Exists((filename = "RetainedAudio_" + fileId + ".wav")))
        fileId++;

    Console.WriteLine("\nWriting file: {0}", filename);
    using (var file = new FileStream(filename, System.IO.FileMode.CreateNew))
        audio.WriteToWaveStream(file);
}

Visual Basic

Private Shared Sub SreSpeechRecognitionRejected(ByVal sender As Object, ByVal e As SpeechRecognitionRejectedEventArgs)

     Console.WriteLine(vbLf & "Speech Rejected")
     If e.Result IsNot Nothing Then
          DumpRecordedAudio(e.Result.Audio)
     End If

End Sub
Private Shared Sub SreSpeechHypothesized(ByVal sender As Object, ByVal e As SpeechHypothesizedEventArgs)

     Console.Write(vbCr & "Speech Hypothesized: " & vbTab & "{0}" & vbTab & "Conf:" & vbTab & "{1}", e.Result.Text, e.Result.Confidence)

End Sub
Private Shared Sub SreSpeechRecognized(ByVal sender As Object, ByVal e As SpeechRecognizedEventArgs)

     Console.WriteLine(vbLf & "Speech Recognized: " & vbTab & "{0}", e.Result.Text)

End Sub

Private Shared Sub DumpRecordedAudio(ByVal audio As RecognizedAudio)
     If audio Is Nothing Then
          Return
     End If

     Dim fileId As Integer = 0
     Dim filename As String
     filename = "RetainedAudio_" & fileId & ".wav"
     Do While File.Exists(filename)
          fileId += 1
          filename = "RetainedAudio_" & fileId & ".wav"
     Loop

     Console.WriteLine(vbLf & "Writing file: {0}", filename)
     Using file = New FileStream(filename, System.IO.FileMode.CreateNew)
          audio.WriteToWaveStream(file)
     End Using

End Sub

In the case of a word being rejected, the audio is written out to a WAV file so it can be listened to later.

Recap

We've created an application that can record audio for a variable amount of time with Kinect!

Working with Depth Data (Beta 2 SDK)

Dan Fernandez — Thu, 16 Jun 2011 17:15:36 GMT

Update: Kinect for Window SDK v1 Quickstart Series now Available (Feb 1st)

Please use the newly updated Kinect for Windows SDK Quickstart series. The content below will only work with the Beta 2 version of the Kinect for Windows SDK.

This video covers the basics of using depth data from Kinect. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides that have been updated for Beta 2 (Nov, 2011).

[00:43] Depth data overview
[04:56] Initializing the Kinect Runtime
[05:46] Using the depth data to create an image
[12:13] Using the PlayerIndex

Setup

The steps below assume you have setup your development environment as explained in the "Setting Up Your Development Environment" video.

Updates for Kinect for Windows SDK Beta 2 (Nov, 2011)

Creating the Kinect runtime uses the SetupKinect method used in the Setting Up Your Development Environment example.
Beta 2 includes a sample (Microsoft.Samples.Kinect.WpfViewers) set of user controls for visualizing data from the Color/RGB and Depth cameras that you can reuse in your application. The samples can be found at: "\Program Files\Microsoft SDKs\Kinect\v1.0 Beta2\Samples\KinectSDKSamples.zip" in the “Managed\KinectWpfViewers” solution.

Task Setup the Depth Camera event

Designing the UI

Add an image with Width=320 and Height=240 as shown below:

XAML

Setup the Depth Camera Event

Create an instance of the Kinect Runtime outside of the Window_Loaded event. Then, initialize the Runtime to use DepthAndPlayerIndex and UseSkeletalTracking. Finally register for DepthFrameReady event and open the Depth stream as shown below.

//Kinect Runtime
Runtime nui = new Runtime(); 

private void Window_Loaded(object sender, RoutedEventArgs e)
{
    //UseDepthAndPlayerIndex and UseSkeletalTracking
    nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking);

    //register for event
    nui.DepthFrameReady += new EventHandler(nui_DepthFrameReady);

    //DepthAndPlayerIndex ImageType
    nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, 
        ImageType.DepthAndPlayerIndex); 
}

void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
}

Visual Basic

Private nui As New Runtime

Private Sub Window_Loaded(ByVal sender As Object, ByVal e As RoutedEventArgs)

    'UseDepthAndPlayerIndex and UseSkeletalTracking
    nui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex Or RuntimeOptions.UseSkeletalTracking)

    'register for event
    AddHandler nui.DepthFrameReady, AddressOf nui_DepthFrameReady

    'DepthAndPlayerIndex ImageType
    nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex)

End Sub

Private Sub nui_DepthFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs)

End Sub

Understanding the PlanarImage byte[] Array

The DepthFrameReady event returns an ImageFrame class that contains a PlanarImage. That PlanarImage contains a byte[] array which contains the distance of each pixel. The array:

Starts at top left of image
Moves left to right, then top to bottom
Represents distance for pixel

Calculating Distance

Because there are 2 bytes per pixel (16 bits) representing the distance, you will need to use bitshift operators to get the distance for a particular pixel. The exact bitshifting depends on the ImageType you use

Depth Image Type
Bitshift second byte left by 8
Distance (0,0) = (int)(Bits[0] | Bits[1] << 8 );

DepthAndPlayerIndex Image Type
Bitshift first byte right by 3 (to remove player index), and second byte left by 5
Distance (0,0) =(int)(Bits[0] >> 3 | Bits[1] << 5);

The method below shows returning the distance in millimeters using the 2 bytes for a particular pixel.

private int GetDistanceWithPlayerIndex(byte firstFrame, byte secondFrame)
{
  //offset by 3 in first byte to get value after player index 
  int distance = (int)(firstFrame >> 3 | secondFrame << 5);
  return distance;
}

Visual Basic

Private Function GetDistanceWithPlayerIndex(ByVal firstFrame As Byte, ByVal secondFrame As Byte) As Integer

    'offset by 3 in first byte to get value after player index 
    Dim distance As Integer = (CInt(firstFrame) >> 3 Or CInt(secondFrame) << 5)
    Return distance

End Function

The DepthAndPlayerIndex image type is offset by 3 since the first three bits represent the Player Index. The Player Index has up to six possible values.

0 – No player
1 – Skeleton 0
2 – Skeleton 1
...

Note: While depth data is available for multiple players, you can only have skeletal/joint data for a maximum of two players.

To get a player index, we'll use the following formula. Since the player index is only in the first byte, we do not need the pixels second byte.

private static int GetPlayerIndex(byte firstFrame)
{
    //returns 0 = no player, 1 = 1st player, 2 = 2nd player...
    return (int)firstFrame & 7; 
}

Visual Basic

Private Shared Function GetPlayerIndex(ByVal firstFrame As Byte) As Integer

    'returns 0 = no player, 1 = 1st player, 2 = 2nd player...
    'bitwise & on firstFrame
    Return CInt(firstFrame) And 7

End Function

Using distance to color values

Now that we know the distance for a particular pixel using the formula above, we can loop through all of the bytes in the byte[] array and dynamically assign colors to pixels based on their distance.

To begin with, we'll build a colorFrame byte[] array using the Bgr (Blue, Green, Red) pixel format. What this means is that each pixel (ex: the pixel at 0,0) is represented by four bytes, one each for Blue, Green, Red, and an empty one (or transparency byte if you choose the Bgra format).

For example, the snippet below would color the pixel blue if the current distance for that pixel was < 900 millimeters from the Kinect. To set the color to blue, we set the first byte (BlueIndex) to the maximum value, 255.

if (distance <= 900)
{
    //we are very close
    colorFrame[index + BlueIndex] = 255;
    colorFrame[index + GreenIndex] = 0;
    colorFrame[index + RedIndex] = 0;
}

Visual Basic

If distance <= 900 Then

    'we are very close
    colorFrame(index + BlueIndex) = 255
    colorFrame(index + GreenIndex) = 0
    colorFrame(index + RedIndex) = 0

End If

Similarly, we can use the code snippet below to check if the current pixel represents a Player Index. If it does, we'll set the maximum value for both green and red which produces a yellow/gold-like color.

if (GetPlayerIndex(depthData[depthIndex]) > 0)
{
    //we are the farthest
    colorFrame[index + BlueIndex] = 0;
    colorFrame[index + GreenIndex] = 255;
    colorFrame[index + RedIndex] = 255;
}

Visual Basic

If GetPlayerIndex(depthData(depthIndex)) > 0 Then

    'we are the farthest
    colorFrame(index + BlueIndex) = 0
    colorFrame(index + GreenIndex) = 255
    colorFrame(index + RedIndex) = 255

End If

Setting a monochrome intensity

While the previous examples we hard-coded colors, you can also build a monochrome histogram that scales the color range (0-255) proportionally based on the distance range (850-4000).

To get a monochrome appearance, the intensity is applied equally to all three colors (blue, green, red).

//equal coloring for monochromatic histogram
var intensity = CalculateIntensityFromDepth(distance);

colorFrame[index + BlueIndex] = intensity;
colorFrame[index + GreenIndex] = intensity;
colorFrame[index + RedIndex] = intensity;

const float MaxDepthDistance = 4000; // max value returned
const float MinDepthDistance = 850; // min value returned
const float MaxDepthDistanceOffset = MaxDepthDistance - MinDepthDistance;

public static byte CalculateIntensityFromDepth(int distance)
{
   //formula for calculating monochrome intensity for histogram
   return (byte)(255 - (255 * Math.Max(distance - MinDepthDistance, 0)  / (MaxDepthDistanceOffset)));
}

Visual Basic

'equal coloring for monochromatic histogram
Dim intensity = CalculateIntensityFromDepth(distance)

colorFrame(index + BlueIndex) = intensity
colorFrame(index + GreenIndex) = intensity
colorFrame(index + RedIndex) = intensity

Private Const MaxDepthDistance As Single = 4000 ' max value returned
Private Const MinDepthDistance As Single = 850 ' min value returned
Private Const MaxDepthDistanceOffset As Single = MaxDepthDistance - MinDepthDistance

Public Shared Function CalculateIntensityFromDepth(ByVal distance As Integer) As Byte

    'formula for calculating monochrome intensity for histogram
    Return CByte(255 - (255 * Math.Max(distance - MinDepthDistance, 0) / (MaxDepthDistanceOffset)))

End Function

Putting it all together

Using the formulas above, we'll now return a byte[] array that has a color value for each pixel based on if the distance for that pixel:

> 900 = Blue
> 900 && < 2000 = Green
> 2000 = Red

If the pixel represents a player, the color will be yellow/gold.

private byte[] GenerateColoredBytes(ImageFrame imageFrame)
{
    int height = imageFrame.Image.Height;
    int width = imageFrame.Image.Width;

    //Depth data for each pixel
    Byte[] depthData = imageFrame.Image.Bits; 

    //colorFrame contains color information for all pixels in image
    //Height x Width x 4 (Red, Green, Blue, empty byte)
    Byte[] colorFrame = new byte[imageFrame.Image.Height * imageFrame.Image.Width * 4];

    //Bgr32  - Blue, Green, Red, empty byte
    //Bgra32 - Blue, Green, Red, transparency 
    //You must set transparency for Bgra as .NET defaults a byte to 0 = fully transparent

    //hardcoded locations to Blue, Green, Red (BGR) index positions       
    const int BlueIndex = 0;
    const int GreenIndex = 1;
    const int RedIndex = 2;

    var depthIndex = 0;
    for (var y = 0; y < height; y++)
    {        
        var heightOffset = y * width;

        for (var x = 0; x < width; x++)
        {
            var index = ((width - x - 1) + heightOffset) * 4;

            //var distance = GetDistance(depthData[depthIndex], depthData[depthIndex + 1]);
            var distance = GetDistanceWithPlayerIndex(depthData[depthIndex], depthData[depthIndex + 1]);

            if (distance <= 900)
            {
                //we are very close
                colorFrame[index + BlueIndex] = 255;
                colorFrame[index + GreenIndex] = 0;
                colorFrame[index + RedIndex] = 0;       
            }
            else if (distance > 900 && distance < 2000)
            {
                //we are a bit further away
                colorFrame[index + BlueIndex] = 0;
                colorFrame[index + GreenIndex] = 255;
                colorFrame[index + RedIndex] = 0;
            }
            else if (distance > 2000)
            {
                //we are the farthest
                colorFrame[index + BlueIndex] = 0;
                colorFrame[index + GreenIndex] = 0;
                colorFrame[index + RedIndex] = 255;
            }

            //Color a player
            if (GetPlayerIndex(depthData[depthIndex]) > 0)
            {
                //we are the farthest
                colorFrame[index + BlueIndex] = 0;
                colorFrame[index + GreenIndex] = 255;
                colorFrame[index + RedIndex] = 255;
            }

            //jump two bytes at a time
            depthIndex += 2;
        }
    }

    return colorFrame;
}

Visual Basic

Private Function GenerateColoredBytes(ByVal imageFrame As ImageFrame) As Byte()

    Dim height As Integer = imageFrame.Image.Height
    Dim width As Integer = imageFrame.Image.Width

    'Depth data for each pixel
    Dim depthData() As Byte = imageFrame.Image.Bits

    'colorFrame contains color information for all pixels in image
    'Height x Width x 4 (Red, Green, Blue, empty byte)
    Dim colorFrame(imageFrame.Image.Height * imageFrame.Image.Width * 4 - 1) As Byte

    'Bgr32  - Blue, Green, Red, empty byte
    'Bgra32 - Blue, Green, Red, transparency 
    'You must set transparency for Bgra as .NET defaults a byte to 0 = fully transparent

    'hardcoded locations to Blue, Green, Red (BGR) index positions       
    Const BlueIndex As Integer = 0
    Const GreenIndex As Integer = 1
    Const RedIndex As Integer = 2

    Dim depthIndex = 0
    For y = 0 To height - 1

        Dim heightOffset = y * width

        For x = 0 To width - 1

            Dim index = ((width - x - 1) + heightOffset) * 4

            Dim distance = GetDistanceWithPlayerIndex(depthData(depthIndex), depthData(depthIndex + 1))

            If distance <= 900 Then

                'we are very close
                colorFrame(index + BlueIndex) = 255
                colorFrame(index + GreenIndex) = 0
                colorFrame(index + RedIndex) = 0

            ElseIf distance > 900 AndAlso distance < 2000 Then

                'we are a bit further away
                colorFrame(index + BlueIndex) = 0
                colorFrame(index + GreenIndex) = 255
                colorFrame(index + RedIndex) = 0

            ElseIf distance > 2000 Then

                'we are the farthest
                colorFrame(index + BlueIndex) = 0
                colorFrame(index + GreenIndex) = 0
                colorFrame(index + RedIndex) = 255

            End If

            '//Color a player
            If GetPlayerIndex(depthData(depthIndex)) > 0 Then

                'we are the farthest
                colorFrame(index + BlueIndex) = 0
                colorFrame(index + GreenIndex) = 255
                colorFrame(index + RedIndex) = 255

            End If

            'jump two bytes at a time
            depthIndex += 2

        Next x

    Next y

    Return colorFrame

End Function

Camera Fundamentals (Beta 2 SDK)

Dan Fernandez — Thu, 16 Jun 2011 17:14:38 GMT

Update: Kinect for Window SDK v1 Quickstart Series now Available (Feb 1st)

Please use the newly updated Kinect for Windows SDK Quickstart series. The content below will only work with the Beta 2 version of the Kinect for Windows SDK.

This video covers the basics of reading camera data from the Kinect sensor. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides that have been updated for Beta 2 (Nov, 2011).

[00:25] Camera data information
[03:30] Creating the UI
[04:48] Initializing the Kinect runtime
[07:18] Reading values from the RGB camera
[11:26] Reading values from the Depth camera
[13:06] Adjusting camera tilt

Updates for Kinect for Windows SDK Beta 2 (Nov, 2011)

The video has not been updated for Beta 2, but the following changes have been made:

Creating the Kinect runtime uses the SetupKinect method used in the Setting Up Your Development Environment example.
Beta 2 includes a sample (Microsoft.Samples.Kinect.WpfViewers) set of user controls for visualizing data from the Color/RGB and Depth cameras that you can reuse in your application. The samples can be found at: "\Program Files\Microsoft SDKs\Kinect\v1.0 Beta2\Samples\KinectSDKSamples.zip" in the “Managed\KinectWpfViewers” solution.

Setup

The steps below assume you have setup your development environment as explained in the Setting Up Your Development Environment video.

To get a depth or color image you must do the following

Create an instance of a runtime object
Initialize the instance with the correct options
Subscribe to an event to receive camera data
Open the video stream
Convert the raw camera data into an image

Task: Display the RGB Camera Image

Designing your UI

We’ll add two 320x240 Image controls to the MainWindow XAML file as shown in the following XAML:

XAML

http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" Height="350" Width="800">

Initializing the runtime

In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseColor to use the RGB and depth camera:

//Kinect Runtime
Runtime nui;
private void Window_Loaded(object sender, RoutedEventArgs e)
{
    SetupKinect();
}
private void SetupKinect()
{
    if (Runtime.Kinects.Count == 0)
    {
        this.Title = "No Kinect connected";
    }
    else
    {
        //use first Kinect
        nui = Runtime.Kinects[0];
 
        //Initialize to return both Color & Depth images
        nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);
    }
}

Visual Basic

'Kinect Runtime
Private nui As Runtime
Private Sub Window_Loaded(ByVal sender As Object, ByVal e As RoutedEventArgs)
    SetupKinect()
End Sub
Private Sub SetupKinect()
    If Runtime.Kinects.Count = 0 Then
        Me.Title = "No Kinect connected"
    Else
        'use first Kinect
        nui = Runtime.Kinects(0)
        'Initialize to return both Color & Depth images
        nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepth)
    End If

Subscribe to the VideoFrameReady event

To read values from the RGB camera, add an event handler for the VideoFrameReady event on the Kinect Runtime in the Window_Loaded event as shown below:

Tip: You can have Visual Studio automatically build the event handler for you by typing "nui.VideoFrameReady +=" and hitting the tab key twice.

nui.VideoFrameReady += new EventHandler(nui_VideoFrameReady);

void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
   // event code goes here
}

Visual Basic

AddHandler nui.VideoFrameReady, AddressOf nui_VideoFrameReady

Private Sub nui_VideoFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs)
    ' event code goes here
End Sub

Understanding the video frame ready returned values

The video frame returns an ImageFrameReadyEventArgs that contains an ImageFrame class. As shown below, the ImageFrame class contains two things:

Metadata about the image, such as ImageType to know if it’s a depth or color image, and resolution to know the size of the image
Image – the actual image data, which is stored in the Bits byte[] array

Converting the byte[] array to an image

To convert the byte[] array that represents the camera image to display it in an Image control (ex: image1 below), call the BitmapSource.Create method as shown below. The last parameter is stride. The stride is the number of bytes from one row of pixels in memory to the next row of pixels in memory. Stride is also called pitch. For more information, go to http://msdn.microsoft.com/en-us/library/aa473780(VS.85).aspx:

PlanarImage imageData = e.ImageFrame.Image;
image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, 
                PixelFormats.Bgr32, null, imageData.Bits, data.Width * imageData.BytesPerPixel);

Visual Basic

Dim imageData As PlanarImage = e.ImageFrame.Image
image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, _
    PixelFormats.Bgr32, Nothing, imageData.Bits, imageData.Width * imageData.BytesPerPixel)

The Coding4Fun Kinect Toolkit has an extension method built into the ImageFrame class that simplifies creating the bitmap:

image1.Source = e.ImageFrame.ToBitmapSource();

Visual Basic

image1.Source = e.ImageFrame.ToBitmapSource()

Getting Supported Resolutions

The Kinect cameras can support different resolutions depending on what ImageType is passed in. To determine these resolutions, call the GetValidResolutions on the ImageStream object:

var x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex);

Visual Basic

Dim x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex)

Open the VideoStream

Before the event will fire, you must call the Open method on VideoStream as shown below. This opens a video stream, sets the PoolSize to 2, sets the resolution to 640x480, and sets the return image as a color image:

nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);

Visual Basic

nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color)

Understanding PoolSize

Pool Size is the number of buffers you want queued up. Usually, you want one displaying, and one being filled. The minimum is two but specifying more than two will give you slightly more latency in the frames you display, and so you’ll be more likely to have one waiting for you. The trade off, then, is smoother playback but higher latency.

Task: Displaying Depth Data

To add depth data, we use the same steps as above.

Update the Runtime Initialization

First, let’s review and see that the Initialize method includes RuntimeOptions.UseDepth to receive depth data from Kinect:

nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);

Visual Basic

nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepth)

Subscribe to the DepthFrameReady event

Next, add the event handler to receive the Depth information. For simplicity, we’ll use the Coding4Fun Kinect toolkit to display the depth image in the image2 Image control:

nui.DepthFrameReady += new EventHandler(nui_DepthFrameReady);

void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
    image2.Source = e.ImageFrame.ToBitmapSource(); 
}

Visual Basic

AddHandler nui.DepthFrameReady, AddressOf nui_DepthFrameReady

Private Sub nui_DepthFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs)
    image2.Source = e.ImageFrame.ToBitmapSource()
End Sub

Open the Depth Stream

Finally, we’ll call DepthStream.Open to open a 320x240 depth stream:

nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);

Visual Basic

nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth)

The result is shown below:

Task: Adjust the Kinect Camera Tilt

Warning: The tilt mechanism in the sensor array is not rated for frequent use. Your code should not make calls to tilt the device more than 15 times in any two-minute window. Changing the tilt too often results in an error message from the function.

Setup

Like before, create an instance of the Kinect Runtime class and call the Initialize method:

Runtime nui = new Runtime();
nui.Initialize(RuntimeOptions.UseColor);

Visual Basic

nui.Initialize(RuntimeOptions.UseColor)

Adjusting the tilt

To adjust the tilt or pitch, set the ElevationAngle property to a value between –27 (Camera.ElevationMinimum) and +27 (Camera.ElevationMaximum). The code below sets the camera to the maximum elevation:

nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum;

Visual Basic

nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum

Skeletal Tracking Fundamentals (Beta 2 SDK)

Dan Fernandez — Thu, 16 Jun 2011 17:13:16 GMT

Update: Kinect for Window SDK v1 Quickstart Series now Available (Feb 1st)

Please use the newly updated Kinect for Windows SDK Quickstart series. The content below will only work with the Beta 2 version of the Kinect for Windows SDK.

This video covers the basics of skeletal tracking using the Kinect sensor. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides that have been updated for Beta 2 (Nov, 2011).

[00:31] Skeleton Tracking API
[01:24] Understanding Skeleton Quality and Joint data
[03:27] Setup skeleton tracking
[03:44] Adding a basic hand tracked cursor
[09:12] Using TransformSmoothing to remove “skeletal jitter”

Updates for Kinect for Windows SDK Beta 2 (Nov, 2011)

The video has not been updated for Beta 2, but the following changes have been made:

Previously, the SkeletonFrameReady event would only fire when there was a tracked skeleton. The SkeletonFrameReady has been updated to fire continuously even when there are no skeletons being tracked. To get tracked skeletons, you can filter the list of skeletons to see if the TrackingState property is set to SkeletonTrackingState.Tracked.
Beta 2 includes a sample (Microsoft.Samples.Kinect.WpfViewers) set of user controls for visualizing data from the Color/RGB and Depth cameras that you can reuse in your application. The samples can be found at: "\Program Files\Microsoft SDKs\Kinect\v1.0 Beta2\Samples\KinectSDKSamples.zip"

Setup

The steps below assume you have setup your development environment as explained in the Setting Up Your Development Environment video.

Task: Setup skeleton tracking

Create the Window_Loaded event

Go to the properties window (F4), select the MainWindow, select the Events tab, and double click on the Loaded event to create the Window_Loaded event

Initializing the runtime

Create a new variable outside of the Window_Loaded event to reference the Kinect runtime.

Runtime nui = new Runtime();

Visual Basic

Dim nui As New Runtime()

In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseSkeletalTracking to receive skeletal data and register for the SkeletonFrameReady event.

nui.Initialize(RuntimeOptions.UseSkeletalTracking);

nui.SkeletonFrameReady += new EventHandler(nui_SkeletonFrameReady);

void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
}

Visual Basic

nui.Initialize(RuntimeOptions.UseSkeletalTracking)

AddHandler nui.SkeletonFrameReady, AddressOf nui_SkeletonFrameReady

Private Sub nui_SkeletonFrameReady(ByVal sender As Object, ByVal e As SkeletonFrameReadyEventArgs)

End Sub

Running the application

Add a breakpoint inside the SkeletonFrameReady event and run the application.

Note – You will need to stand far enough away that the Kinect can see all or most of your skeleton for the SkeletonFrameReady event to fire.

When the breakpoint fires, you can inspect the SkeletonFrameReadyEventArgs to see that the SkeletonFrame returns a collection of six skeletons as shown below.

Task: Add a basic hand tracked cursor

In this example, we're going to use the position of the tracked skeleton's, head, left hand, and right hand to move ellipse controls.

Designing your UI

Starting from the project above, switch to MainWindow.xaml and make sure you can see the XAML code. We will add three three ellipses of varying color onto a Canvas control named MainCanvas as shown in the XAML below.

XAML

Get the Tracked Skeleton

In the SkeletonFrameReady event, we'll use a LINQ query to get the first tracked skeleton.

void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    SkeletonFrame allSkeletons = e.SkeletonFrame;

    //get the first tracked skeleton
    SkeletonData skeleton = (from s in allSkeletons.Skeletons
                             where s.TrackingState == SkeletonTrackingState.Tracked
                             select s).FirstOrDefault();
}

Visual Basic

Private Sub nui_SkeletonFrameReady(ByVal sender As Object, ByVal e As SkeletonFrameReadyEventArgs)
    Dim allSkeletons As SkeletonFrame = e.SkeletonFrame

    'get the first tracked skeleton
    Dim skeleton As SkeletonData = ( _
        From s In allSkeletons.Skeletons _
        Where s.TrackingState = SkeletonTrackingState.Tracked _
        Select s).FirstOrDefault()
End Sub

Getting a Joint position

A Joint position returns X,Y,Z values as explained below

X = Horizontal position measured as the distance, in meters from the Kinect along the X Axis
Y = Vertical position measured as the distance, in meters from the Kinect along the Y Axis
Z = Distance from Kinect measured in meters

Scaling a Joint value

Given that the X and Y positions are distance measurements, we can use the Coding4Fun Kinect Toolkit ScaleTo method to scale the value to a maximum X and Y position as shown below.

Joint HandRight =  skeleton.Joints[JointID.HandRight].ScaleTo(640, 480);

Visual Basic

Dim HandRight = skeleton.Joints(JointID.HandRight).ScaleTo(640, 480)

You can also constrain the maximum values for the skeletal joints to a smaller range. For example, if you were designing an application where the user can reach around the screen to touch things, you may not want them to have to walk two feet left or right to reach the edges. By using the second version of the ScaleTo method, you can specify the range of the specified joint to something smaller as shown:

Joint HandRight =  skeleton.Joints[JointID.HandRight].ScaleTo(640, 480, .5f, .5f);

Visual Basic

Dim HandRight = skeleton.Joints(JointID.HandRight).ScaleTo(640, 480, .5f, .5f)

This code will make it so that your right hand only has to travel one meter (joint range of -0.5 to +0.5) to traverse the pixels at 0 to 640. If you set this value to –1 to +1, for example, then the right hand would have to travel two meters total, one meter to the left, one meter to the right to traverse the 640 pixels.

Also, as you can see above, you can get a particular Joint, like the HandRight Joint, by using the skeleton indexer for the Joints collection.

Setting an Ellipse Position

To move the ellipses in our MainWindow to the location of a Joint, we will use the method below that sets the Canvas.Left and Canvas.Top position to the X (Left) and Y (Top) value from the Joint parameter.

private void SetEllipsePosition(FrameworkElement ellipse, Joint joint)
{
    var scaledJoint = joint.ScaleTo(640, 480, .5f, .5f);

    Canvas.SetLeft(ellipse, scaledJoint.Position.X);
    Canvas.SetTop(ellipse, scaledJoint.Position.Y);
}

Visual Basic

Private Sub SetEllipsePosition(ByVal ellipse As FrameworkElement, ByVal joint As Joint)
    Dim scaledJoint = joint.ScaleTo(640, 480,.5f,.5f)

    Canvas.SetLeft(ellipse, scaledJoint.Position.X)
    Canvas.SetTop(ellipse, scaledJoint.Position.Y)
End Sub

Putting it all together

Finally, in the nui_SkeletonFrameReady event, call the SetEllipsePosition methods for the three joints as shown below:

SetEllipsePosition(headEllipse, skeleton.Joints[JointID.Head]);
SetEllipsePosition(leftEllipse, skeleton.Joints[JointID.HandLeft]);
SetEllipsePosition(rightEllipse, skeleton.Joints[JointID.HandRight]);

Visual Basic

SetEllipsePosition(headEllipse, skeleton.Joints(JointID.Head))
SetEllipsePosition(leftEllipse, skeleton.Joints(JointID.HandLeft))
SetEllipsePosition(rightEllipse, skeleton.Joints(JointID.HandRight))

Task: Using TransformSmoothing to for less skeletal jitter

Using the same application as above, you may notice jitteriness in the hand positions as small changes between updates to the SkeletalFrameReady event change the location of the ellipses.

Using TransformSmoothParameters

To see the difference between using and not using TransformSmoothing, toggle the true/false TransformSmooth property. There are two ways to use TransformSmoothing, you can set it to true and it will use a default set of parameters, or you can customize and experiment with each of the parameters to find the parameters that work best for your application. To do that, you’ll need create the TransformSmoothParameters struct yourself and define the parameters.

You must set the TransformSmoothParameters after calling the nui.Initialize method.

Note: Since every application is different, you will need to experiment with the parameters to understand what's right for your application.

private void SetupKinect()
{
     if (Runtime.Kinects.Count == 0)
     {
         this.Title = "No Kinect connected"; 
     }
     else
     {
         //use first Kinect
         nui = Runtime.Kinects[0];
         //Initialize to do skeletal tracking
         nui.Initialize(RuntimeOptions.UseSkeletalTracking);
         //add event to receive skeleton data
         nui.SkeletonFrameReady += new EventHandler(nui_SkeletonFrameReady);
         //to experiment, toggle TransformSmooth between true & false
         // parameters used to smooth the skeleton data
         nui.SkeletonEngine.TransformSmooth = true;
         TransformSmoothParameters parameters = new TransformSmoothParameters();
         parameters.Smoothing = 0.7f;
         parameters.Correction = 0.3f;
         parameters.Prediction = 0.4f;
         parameters.JitterRadius = 1.0f;
         parameters.MaxDeviationRadius = 0.5f;
         nui.SkeletonEngine.SmoothParameters = parameters;
     }
}

Visual Basic

        Private Sub SetupKinect()
            If Runtime.Kinects.Count = 0 Then
                Me.Title = "No Kinect connected"
            Else
                'use first Kinect
                nui = Runtime.Kinects(0)
                'Initialize to do skeletal tracking
                nui.Initialize(RuntimeOptions.UseSkeletalTracking)
                'add event to receive skeleton data
                AddHandler nui.SkeletonFrameReady, AddressOf nui_SkeletonFrameReady
                'to experiment, toggle TransformSmooth between true & false
                ' parameters used to smooth the skeleton data
                nui.SkeletonEngine.TransformSmooth = True
                Dim parameters As New TransformSmoothParameters()
                parameters.Smoothing = 0.7f
                parameters.Correction = 0.3f
                parameters.Prediction = 0.4f
                parameters.JitterRadius = 1.0f
                parameters.MaxDeviationRadius = 0.5f
                nui.SkeletonEngine.SmoothParameters = parameters
            End If
        End Sub

Setting Up Your Development Environment (Beta 2 SDK)

Dan Fernandez — Thu, 16 Jun 2011 17:11:58 GMT

Update: Kinect for Window SDK v1 Quickstart Series now Available (Feb 1st)

Please use the newly updated Kinect for Windows SDK Quickstart series. The content below will only work with the Beta 2 version of the Kinect for Windows SDK.

This video covers how to set up your development environment. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides that have been updated for Beta 2 (Nov, 2011).

[00:35] Sample Requirements
[01:50] Starting the Demo
[02:13] Adding References
[02:45] Coding4Fun Kinect Library
[03:23] Initializing the Kinect Runtime
[05:41] Uninitializing the Kinect Runtime

Updates for Kinect for Windows SDK Beta 2 (Nov, 2011)

The video in the Kinect for Windows SDK has not been updated, but the tutorial below has the following changes:

The Microsoft.Research.Kinect DLL version is now 1.0.0.45.
The requirements for DirectX samples have been clarified to say that they are only required for C++.
The speech sample requirements have been updated for Beta 2. If you are using an x64 PC, note that you should install both the x86 and x64 Speech Platform Runtime (see Speech samples below).
The source code for shipping examples is now in "\Program Files\Microsoft SDKs\Kinect\v1.0 Beta2\Samples\KinectSDKSamples.zip"
The Kinect runtime now provides a StatusChanged event that will fire if a Kinect is connected, disconnected, and so forth. See below for a code snippet for how to subscribe and handle the StatusChanged event.
For an example on setting up and handling Kinect connect/disconnect events, refer to the ShapeGame sample.
For an example on how to show two Kinects at the same time, refer to the SkeletalViewer sample.

Task: Download Sample Requirements

Some samples use DirectX and the Microsoft Speech APIs. For these demos to work, you must download the correct pre-requisite software.

Kinect for Windows SDK:

http://www.kinectforwindows.org

Visual Studio:

Visual Studio 2010 Express edition of the demo in the appropriate programming language or Visual Studio Professional or higher

DirectX Samples (for C++ Skeletal Viewer)

Microsoft DirectX® SDK - June 2010 or later version
Current runtime for Microsoft DirectX® 9

Speech Samples:

For Speech samples:

Microsoft Speech Platform Runtime, version 10.2 – select 32-bit if you are running 32-bit Windows. If you have 64-bit Windows, we suggest that you install both the 32-bit and 64-bit runtime.
Microsoft Speech Platform - Software Development Kit, version 10.2 – select 32-bit or 64-bit according to your Windows installation
Kinect for Windows Runtime Language Pack, version 0.9

Task: Setting up a new Visual Studio 2010 Project

In this task, you’ll create a new Visual Studio 2010 project and add references to the correct libraries
Start Visual Studio 2010 (Express Editions or higher)
Select File –> New Project –> Windows Presentation Foundation and set the name for your project

Add a reference

C# – From the Solution Explorer, right click on References and select Add Reference as shown below

Visual Basic – From the Solution Explorer, right click on the project name and select Add Reference as shown below

Visual Basic / C# – In the Add Reference dialog, switch to the .NET tab, click the Component Name header to sort references alphabetically, then scroll down to select Microsoft.Research.Kinect and click OK. Make sure to select 1.0.0.45.

Add a using statement

Within your project, you can now add a using statement to reference the Kinect namespaces:

using Microsoft.Research.Kinect.Nui;
using Microsoft.Research.Kinect.Audio;

Visual Basic

Imports Microsoft.Research.Kinect.Nui
Imports Microsoft.Research.Kinect.Audio

Optional: Using the Coding4Fun Kinect Toolkit

Go to http://c4fkinect.codeplex.com and download the latest version of the Coding4Fun Kinect Toolkit. This library is filled with useful items to help with a variety of tasks, such as converting the data from the Kinect to a data array or Bitmap.

Depending on if you are building a WPF or Windows Form application, we have two different DLLs:

WPF Applications: use the Coding4Fun.Kinect.Wpf.dll
Windows Form Applications: use the Coding4Fun.Kinect.WinForm.dll

// if you're using WPF
using Coding4Fun.Kinect.Wpf;

// if you're using WinForms
using Coding4Fun.Kinect.WinForm;

Visual Basic

' if you're using WPF
Imports Coding4Fun.Kinect.Wpf

' if you're using WinForms
Imports Coding4Fun.Kinect.WinForm

Initializing and Uninitializing the runtime

For NUI, we have to initialize and uninitialize the runtime. In our examples, we typically will initialize on the Loaded event on the Window and we'll uninitialize on the Closed event. Depending on what needs to be done, the Initialize method on the runtime will be tweaked though the concepts stay the same.

Create the Window_Loaded event

Go to the properties window (F4), select the MainWindow, select the Events tab, and double click on the Loaded event to create the Window_Loaded event

Initializing the runtime

Create a new Runtime variable named “nui” outside of the Window_Loaded event. When the Window_Loaded event fires, we will call the SetupKinect method that checks to see if there is at least one Kinect plugged in, and if there is, it uses the first Kinect (0 representing the first Kinect) and initializes the Runtime (more details on that later).

//Kinect Runtime
Runtime nui;
private void Window_Loaded(object sender, RoutedEventArgs e)
{
    SetupKinect(); 
    
}
private void SetupKinect()
{
    if (Runtime.Kinects.Count == 0)
    {
        this.Title = "No Kinect connected";
    }
    else
    {
        //use first Kinect
        nui = Runtime.Kinects[0];
        nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking);
    }
}

Visual Basic

       'Kinect Runtime
        Private nui As Runtime

        Private Sub Window_Loaded(ByVal sender As Object, ByVal e As RoutedEventArgs)
            SetupKinect()
            AddHandler Runtime.Kinects.StatusChanged, AddressOf Kinects_StatusChanged
        End Sub
        Private Sub SetupKinect()
            If Runtime.Kinects.Count = 0 Then
                Me.Title = "No Kinect connected"
            Else
                'use first Kinect
                nui = Runtime.Kinects(0)
                nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepthAndPlayerIndex Or RuntimeOptions.UseSkeletalTracking)
            End If
        End Sub

Handling multiple Kinects

You can see how many Kinects are connected by getting the count of Kinects as shown in the code below. Note that for multiple Kinects, you can only have one Kinect do skeletal tracking at a time (color and depth camera work for all) and each Kinect needs to be in its own USB hub, otherwise it won’t have enough USB bandwidth.

[code lang=”csharp”]

int KinectCount = Runtime.Kinects.Count;

[/code]

Visual Basic

[code lang=”vb”]

Dim KinectCount as Integer = Runtime.Kinects.Count

[/code]

Handling Kinect Status Change events

You can also handle events when the status changes for Kinects that are plugged in. To do this, first register for the StatusChanged event as shown below. When this event fires, it returns a KinectStatus enum with the following possible values: Connected, Error, Disconnected, NotReady, or NotPowered.

[code lang=”csharp”]

Runtime.Kinects.StatusChanged += new EventHandler(Kinects_StatusChanged);

…

void Kinects_StatusChanged(object sender, StatusChangedEventArgs e)
{
    string message = "Status Changed: ";
    switch (e.Status)
    {
        case KinectStatus.Connected:
            message += "Connected";
            break;
        case KinectStatus.Disconnected:
            message += "Disconnected";
            break;
        case KinectStatus.Error:
            message += "Error";
            break;
        case KinectStatus.NotPowered:
            message += "Not Powered";
            break;
        case KinectStatus.NotReady:
            message += "Not Ready";
            break;
        default:
            if (e.Status.HasFlag(KinectStatus.Error))
            {
                message += "Kinect error";
            }
            break;
    }
    this.Title = message;
}

[/code]

Visual Basic

[code lang=”vb”]

AddHandler Runtime.Kinects.StatusChanged, AddressOf Kinects_StatusChanged

…

Private Sub Kinects_StatusChanged(sender As Object, e As StatusChangedEventArgs)
    Dim message As String = "Status Changed: "
    Select Case e.Status
        Case KinectStatus.Connected
            message += "Connected"
            Exit Select
        Case KinectStatus.Disconnected
            message += "Disconnected"
            Exit Select
        Case KinectStatus.[Error]
            message += "Error"
            Exit Select
        Case KinectStatus.NotPowered
            message += "Not Powered"
            Exit Select
        Case KinectStatus.NotReady
            message += "Not Ready"
            Exit Select
        Case Else
            If e.Status.HasFlag(KinectStatus.[Error]) Then
                message += "Kinect error"
            End If
            Exit Select
    End Select
    Me.Title = message

End Sub

[/code]

Setting Runtime Options

In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseColor to use the RGB camera:

nui.Initialize(RuntimeOptions.UseColor);

Visual Basic

nui.Initialize(RuntimeOptions.UseColor)

RuntimeOptions is a Flag enumeration, this means you can set multiple options as parameters in the Initialize method by separating them with the pipe "|" character (the "or" operator) in c# or the "Or" operator in Visual Basic. The example below sets the runtime to use the color camera, a depth camera, and skeletal tracking:

nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking );

Visual Basic

nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepthAndPlayerIndex Or RuntimeOptions.UseSkeletalTracking)

Uninitializing the Runtime

Remember that when you are done using the Kinect Runtime, you should call the Uninitialize method. For a WPF application, you would typically do this in the Windows_Closed event:

nui.Uninitialize();

Visual Basic

nui.Uninitialize()

Installing and Using the Kinect Sensor (Beta 2 SDK)

Dan Fernandez — Thu, 16 Jun 2011 16:52:09 GMT

Update: Kinect for Window SDK v1 Quickstart Series now Available (Feb 1st)

Please use the newly updated Kinect for Windows SDK Quickstart series. The content below will only work with the Beta 2 version of the Kinect for Windows SDK.

This video covers the basics of installing and using the Kinect sensor. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides that have been updated for Beta 2 (Nov, 2011).

[02:34] Overview of Kinect Sensor
[05:06] Device Manager
[05:36] Using Kinect as an Audio Device
[07:11] Using Kinect for Speech Recognition

Updates for Kinect for Windows SDK Beta 2 (Nov, 2011)

The video has not been updated for Beta 2, but the following changes have been made:

The new Web site for Kinect is http://kinectforwindows.org

Installing the Kinect for Windows SDK

Download the 32-bit or 64-bit version of the Kinect for Windows SDK at http://kinectforwindows.org

Kinect Hardware Basics

The Kinect Sensor consists of the following:

RGB Camera – A regular video camera that has a resolution of 640x480 at 30 frames per second
3D Depth Sensors – This is a combination of a depth projector (left reticule) and sensor (right reticule) to calculate distance. For more information, see the Tech behind Kinect video
Motorized Tilt – The motorized tilt enables you to adjust the camera up or down 27 degrees to ensure that the camera has the best possible view of you
Multi-Array Microphone – A four-microphone array that is mounted as a single microphone in Windows

Device manager

After installing the Kinect for Windows SDK and plugging in the Kinect to a USB port on your computer. Click Control Panel –> Device Manager. You should see Microsoft Kinect as shown below

Audio Microphone

The Kinect microphone can be used for any application that uses a microphone. In the picture below, you’ll see that you can go to Control Panel –> Sounds and select the Recording tab to see the Kinect microphone array

Using the Kinect Microphone with Windows Sound Recorder

To use Kinect as a microphone for Windows Sound Recorder, simply ensure that the Kinect microphone is setup in the Sound control panel (above) as the default sound recorder. Click the Start Recording button to start recording using the Kinect as a microphone and click the Stop Recording button to be prompted where to save your file

How to use Windows Speech Recognition with the Kinect microphone

Click Start and type “Windows Speech Recognition” and hit enter or, in the Sound control panel, right click and select Configure Speech Recognition as shown below

If you’ve never used speech recognition before, consider using the Take Speech Tutorial option and reviewing the Speech Reference Card. If you are familiar with speech recognition, select Start Speech Recognition

Since the Kinect is a microphone array, make sure to select the Other option as shown below

Once the Windows Speech Recognition application has started, your screen will display the following widget, which you can use to interact with Speech Recognition application if required