Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Unity and the Kinect SDK

Unity is is officially huge in the gaming industry, so when I saw this I thought that there had to be some of your that might find it interesting or useful...

Bringing the Kinect SDK in to Unity

Install the Unity Package

The pre-built Unity package includes all of the scripts required to start running your world.

To install, simply import the package, and you all of the required assets will appear in your project.

The following items are included within the package:

Scenes
  • KinectExample - This scene shows you how a skeleton is generated / tracked by placing spheres at each of the bones tracked by the Kinect, and how to use kinect to control your model. Use this to get a feel for what the Kinect is capable of. It also shows you how to prepare your GameObjects
Prefabs
  • Kinect_Prefab- This prefab has an empty node which contains all of the necessary scripts for your world to start using the Kinect - this does NOT include the controller, which you will need to use to actually control your models. See setup
  • KinectPointMan - This prefab shows how KinectPointController works.
Kinect
  • KinectModelControllerV2 - This is the script that you will attach to your model that you want to manipulate - you will drag each of the bones to be controlled into the appropriate slot, and determine which player controls which model.
  • KinectPointController - This script will place GameObjects that you define onto points that are tracked by the Kinect, generating a skeleton. The starting scene comes with an example of how this looks / should be done.
  • DisplayDepth - This script will get the depth image. Attach it to a game object with renderer. NOTE: In unity, you need to restart unity everytime after running your world, otherwise this script will not work (because of SDK issue).
  • DisplayColor - This script will get the RGB image. Attach it to a game object with renderer. NOTE: In unity, you need to restart unity everytime after running your world, otherwise this script will not work (because of SDK issue).
  • KinectRecorder - This script will record your movement and output playback files for the emulator.
  • KinectEmulator - This script will act as a virtual Kinect. It works with playback files. For now it only simulate the skeleton data.
  • KinectSensor - This script gets data from the physic Kinect.
  • DeviceOrEmulator - This script sets whether to use physic Kinect or the emulator.
  • SkeletonWrapper - This script grabs skeleton data.
  • DepthWrapper - This script grabs depth image data.
  • KinectInterop - This script grabs data from Microsoft Kinect SDK.
  • Recordings/playbackDefault - This is the default playback file for emulator. Do NOT remove this file.
Models and Materials
  • rainbowMan
This is an example model that you can use for testing / learning how to use the wrapper. You can drag the KinectModelControllerV2 script onto this model and attach the bones and immediately begin testing.

Using the Scripts

After importing the asset package, you have the KinectExample to test and get familiar with the KinectPointController and KinectModelControllerV2. You can use this as your base scene, or you can drag the appropriate GameObjects into a new one.

Setting up the Kinect_Prefab

As previously mentioned, the Kinect_Prefab GameObject is required to have Unity talk with / use the Kinect. This empty object needs to exist somewhere in your scene. However, if you are switching scenes, it will persist because of the DontDestroyOnLoad call in the KinectWrapper script, so you don't need to instantiate a new Kinect_Prefab in each scene.

To get the best use out of the Kinect, you may need to tweak some settings within the Kinect_Prefab object itself - here is a breakdown of each option.

Device Or Emulator
  • Use Emulator - Check this to use emulator.
Kinect Sensor

NOTE: do NOT enable this script manually, it is controlled by DeviceOrEmulator.

  • Sensor Height - How high (in meters) off the ground is the sensor.
  • Kinect Center - This tells the Kinect where it should be looking for it's 0,0,0 point (relative to the ground directly under the sensor). The default works pretty well.
  • Look At - Tells the Kinect how to orient the camera using the motor control.

NOTE: The following values allow you to smooth the skeleton data. Usually the default values work fine. Do NOT change them unless you find magic numbers for you project.

  • Smoothing - Default value 0.5.
  • Correction - Default value 0.5.
  • Prediction - Default value 0.5.
  • Jitter Radius - Default value 0.05.
  • Max Deviation Radius - Default value 0.04.

Project Information URL: http://wiki.etc.cmu.edu/unity3d/index.php/Microsoft_Kinect_-_Microsoft_SDK

UPDATE:

This project looks to be using the beta version of the SDK and not v1 (as of the time of this post). With the release of the Kinect for Windows SDK v1, I've meant to only highlight projects that are v1 compatible, but missed a few phrase on the page, "Also, since we are not using the new version of Kinect SDK, do NOT download it from Microsoft website...."

I would NOT recommend installing the Kinect for Windows SDK from any other source than Microsoft and so suggest, if you've already upgraded to v1 for the SDK, that you hold off going to far with this project. If you are interested in the project, because it does look pretty darn cool, then you encourage the authors to upgrade to v1 as soon as they can.

Tags:

Follow the Discussion

  • YormanYorman

    Hello

    Is this wrapper for version 1.0 or the beta?

  • Greg Duncangduncan411 It's amazing what a professional photographer can do...

    @Yorman: Based on the dates of the updates, pages, the files in the zip's, etc, I believe it's for v1 (at least that's what I hope, I'm trying to only highlight v1 based projects).

  • Hello

    I think it's the beta version because it tries to access:
    C: \ Program Files (x86) \ Microsoft Research KinectSDK \ MSRKINECTNUI.DLL

  • Greg Duncangduncan411 It's amazing what a professional photographer can do...

    @yormanh:GRRR... I missed this line on the site, "NOTE: Before installing, ensure that the Kinect is NOT connected to your machine. Also, since we are not using the new version of Kinect SDK, do NOT download it from Microsoft website."

    So yeah, this looks like it's NOT v1. My bad causing the confusion...

    Still if you find this project interesting, I would recommend you contact the authors and encourage them to release a v1 compatible version...

  • Greg Duncangduncan411 It's amazing what a professional photographer can do...

    @yormanh:Besides my comment above, I've also added an Update note to the post...

    I'm on the verge of pulling the whole post entirely, but I think the idea behind it is cool and really hope to generate some interest and support for a v1 version, so am leaving it up for now.

    Thanks for pointing this out and helping me on this...

  • Hello

    if the idea is interesting but I don't know to get the name of the class

    for example

    [DllImportAttribute (@ "C: \ Program Files (x86) \ Microsoft Research KinectSDK \MSRKINECTNUI.DLL", EntryPoint = "NuiInitialize")]
    public static extern int NuiInitialize (NuiInitializeFlags dwFlags);

    This version does not get the EntryPoint or method names, you know? or the class KinectSensor?

    for example (this doesn't  work)
    [DllImportAttribute (@ "C: \ Program Files \ Microsoft SDKs \ Kinect \ v1.0 \ Assemblies \ Microsoft.Kinect.dll", EntryPoint = "Start")]
    public static extern int Start ();

  • lucabertinettolucabertine​tto

    Actually I'm using the v1.0 SDK...so this process won't work with me?

  • lucabertinettolucabertine​tto

    No, it doesn't work :(
    DllNotFoundException: C:\Program Files (x86)\Microsoft Research KinectSDK\MSRKINECTNUI.DLL

    I would really appreciate if somebody could link me a way to bridge the Kinect SDK 1.0 to Unity.

  • Well, this is definitely great and would be very useful to a lot of devs. That it doesn't work with v1 is obviously a turn off. Where would be the right place to ask and see if an update would be possible ?

  • Greg Duncangduncan411 It's amazing what a professional photographer can do...

    @haknick: Good question. Using the page history I believe I've found a point of contact for that project. I've just shot them an email asking if they are indeed a good PoC and if there's any work on a v1 version.

    If/when I hear more, I'll post a follow-up comment.

  • Greg Duncangduncan411 It's amazing what a professional photographer can do...

    There is indeed work being done on a v1 version... As I learn more I'll let you know. Smiley

  • NiranjanNiranjan

    Thw wrapper works with v1.0, all have you have to is change the .dll path in the script "KinectInterop" to "C:\Windows\System32\Kinect10.dll"

  • Greg Duncangduncan411 It's amazing what a professional photographer can do...

    Also just found this, thought you might find it interesting...

    https://github.com/adevine1618/KinectSDK-Unity3D_Interface_Plugin

    "This is a wrapper that allows the user to access Microsoft's Kinect SDK v.1.0 data from inside the free Unity3D game engine much faster than a TCP socket connection"

    This project was just updated for the v1 SDK a few days ago...

  • hello

    sorry I had not read this post.

    Thanks Niranjan, it is true that by changing the path works but I have a problem, thefirst run and it starts but the second time Unity3D dies and does not respond,does  someone this happen?

  • @Niranjan: Thanks, now it's working Smiley

    @yormanh: I modified the source KinectInterop.cs that you can find in the 'NewKinet' example project, change all the dll at the end of the file with "C:\Windows\System32\Kinect10.dll", and THEN I opened the project with unity.
    I tried to modify it one time inside unity, with mono, but it crashed afterwards.

    @all: if there is participation it would be great to open a Google group, I think Big Smile 

  • Hello

     

    I was reading in the file KinectSensor.cs 

    // The MSR Kinect DLL (native code) is going to load into the Unity process and stay resident even between debug runs of the game.
    // So our component must be resilient to starting up on a second run when the Kinect DLL is already loaded and
    // perhaps even left in a running state. Kinect does not appear to like having NuiInitialize called when it is already initialized as
    // it messes up the internal state and stops functioning. It is resilient to having Shutdown called right before initializing even if it
    // hasn't been initialized yet. So calling this first puts us in a good state on a first or second run.
    // However, calling NuiShutdown before starting prevents the image streams from being read, so if you want to use image data
    // (either depth or RGB), comment this line out.
    //NuiShutdown();

     

    I don't know if there is solve for this problem, I am now investigating more. The group is a good idea.

    Anyone know how to contact the developers of the wrapper?

    a greeting

     

  • Hello,

    Thanks Niranjan, I changed the DLL path, Unity didnt crashed (lie - now is crashing when I try to play the app) and when I have an GameObject to show the RGB or Depth Camera from Kinect i'm get the following error: 


    NullReferenceException: Object reference not set to an instance of an object
    System.String.memcpy1 (System.Byte* dest, System.Byte* src, Int32 size)
    System.String.memcpy (System.Byte* dest, System.Byte* src, Int32 size)
    (wrapper unknown) Kinect.ColorBuffer:PtrToStructure (intptr,object)
    KinectSensor.extractColorImage (Kinect.NuiImageBuffer buf) (at Assets/Kinect/KinectSensor.cs:259)
    KinectSensor.Kinect.KinectInterface.pollColor () (at Assets/Kinect/KinectSensor.cs:212)
    DisplayColor.Update () (at Assets/Kinect/DisplayColor.cs:22)

    The line is:

    ColorBuffer cb = (ColorBuffer)Marshal.PtrToStructure(buf.m_pBuffer,typeof(ColorBuffer));

    Can anyone help me out?

     

    Anyone created the the Google Group? I can join?

     

    Cheers,

    Felipe Fragoso

  • Thnx, Niranjan. I changed the paths and now it works, but only first time I running it in Unity. Trying run it second time Unity doesn't respond at all. I have no ideas to solve it.

    @fdfragoso: Hi, You said " Unity didnt crashed". What else you did besides changing paths.

    Thanks for any advice.

  • @AkhmadMax: I didnt do anything else. But after few days using, sometimes when I play the Unity crash, but sometimes works fine. Really dont know.

  • AAAAAA

    i have an error in unity that it can't find MSRKINECTNUI.dll

    dllnot found exception

    i have beta2 installed
    can anyone help me

  • Hi..

    The wrapper is awesome!! Just a quick question:

    I am completely new to unity. And I have to use it for integration of my 
    Kinect gesture recognition code with the 3d module of a car.  Please help 
    me as i am not able to find which script of C# to use for coding my module!! 
    Plus it would be great if you can also tell me how to integrate c# 
    and JavaScript because the car module is in JavaScript and kinect code is 
    in c#!! 

    Please reply as soon as possible!! Need it for my project in college!! 

    Thanks. 
    Mayank 

  • DennisDennis

    I'm using the Kinect SDK 1.0 (not beta) and Unity crashed every second time I ran my game. Here's how to fix it: http://www.rozengain.com/blog/2012/05/10/microsoft-kinect-sdk-wrapper-for-unity-crash-bug-fix/

  • Greg Duncangduncan411 It's amazing what a professional photographer can do...

    @Dennis:Nice, thanks for the write on this...

  • WahooneyWahooney

    Has anyone been able to get DisplayDepth and DisplayColor working in v1.x?

    I can get the bones working but I need the colour feedback too?

    Thanks.

  • mikmik

    Unfortunatelly depth and color don t work anymore:((((

    The plugin is not anymore updated?

  • THIBAULT DavidTHIBAULT David

    Hello,
    I wonder which device is compatible with KinectSDK-Unity3D_Interface_Plugin ?
    The XBOX360 kinect (http://emea.microsoftstore.com/UK/en-GB/Microsoft/Xbox-+-Games/Xbox-360-Kinect) or the Windows Kinect (http://emea.microsoftstore.com/UK/en-GB/Microsoft/Kinect-for-Windows)?

    Thanks.

  • Greg Duncangduncan411 It's amazing what a professional photographer can do...

    @THIBAULT David: The only fully supported device is the Kinect for Windows device.

  • Stevie GiovanniStevie Giovanni

    has anyone succeeded on using the color image? i tried to use the kinect sdk provided in the link but i keep getting cannot find dll MSRKINECTNUI.dll although i've given the correct path.

  • Steven HicksonSteven Hickson

    I just hacked it to work with the SDK 1.5 version.
    It was just a matter of rewrapping the c++ SDK with a DLL and changing a little C# in KinectInterop.cs

    If anyone wants the DLL and C# that work with V1.5. Feel free to e-mail me at me@stevehickson.com

  • @Steven Hickson: I tried e-mail you, but your mail server refuses it. 
    Maybe you could share your version of the wrapper by other means.

    I'm sure it can be very useful for a lot of people.

    Thank you.

     

  • milton ortizmilton ortiz

    i have a question maybe a little offset the main topic.
    how can i use the samples from kinect browser kit (face tracking) inside unity3d? thanks a lot

  • hi,

    Just started thinking of embarking my new project using kinect and unity. I have downloaded Kinect SDK 1.5 since this version is compatible with Kinect XBox sensor, I am using VC# Express 2010. Toolbox also downloaded. How about Unity? Does the free version workable for this? or I need to consider spending my money for Unity 4.

     

  • Greg Duncangduncan411 It's amazing what a professional photographer can do...

    Re-opening the comments...

Remove this comment

Remove this thread

close

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.