Comment Feed for Channel 9 - Unity and the Kinect SDK http://files.channel9.msdn.com/thumbnail/05bfcf24-5e3c-47d0-b101-53fbcbd0a56c.png Channel 9 - Unity and the Kinect SDK Unity is is officially huge in the gaming industry, so when I saw this I thought that there had to be some of your that might find it interesting or useful... Bringing the Kinect SDK in to UnityInstall the Unity PackageThe pre-built Unity package includes all of the scripts required to start running your world. To install, simply import the package, and you all of the required assets will appear in your project. The following items are included within the package: ScenesKinectExample - This scene shows you how a skeleton is generated / tracked by placing spheres at each of the bones tracked by the Kinect, and how to use kinect to control your model. Use this to get a feel for what the Kinect is capable of. It also shows you how to prepare your GameObjects PrefabsKinect_Prefab- This prefab has an empty node which contains all of the necessary scripts for your world to start using the Kinect - this does NOT include the controller, which you will need to use to actually control your models. See setup KinectPointMan - This prefab shows how KinectPointController works. KinectKinectModelControllerV2 - This is the script that you will attach to your model that you want to manipulate - you will drag each of the bones to be controlled into the appropriate slot, and determine which player controls which model. KinectPointController - This script will place GameObjects that you define onto points that are tracked by the Kinect, generating a skeleton. The starting scene comes with an example of how this looks / should be done. DisplayDepth - This script will get the depth image. Attach it to a game object with renderer. NOTE: In unity, you need to restart unity everytime after running your world, otherwise this script will not work (because of SDK issue). DisplayColor - This script will get the RGB image. Attach it to a game object with renderer. NOTE: In unity, you need to restart unity everytime after running your world, otherwise this script will not work (because of SDK issue). KinectRecorder - This script will record your movement and output playback files for the emulator. KinectEmulator - This script will act as a virtual Kinect. It works with playback files. For now it only simulate the skeleton data. KinectSensor - This script gets data from the physic Kinect. DeviceOrEmulator - This script sets whether to use physic Kinect or the emulator. SkeletonWrapper - This script grabs skeleton data. DepthWrapper - This script grabs depth image data. KinectInterop - This script grabs data from Microsoft Kinect SDK. Recordings/playbackDefault - This is the default playback file for emulator. Do NOT remove this file. Models and MaterialsrainbowMan This is an example model that you can use for testing / learning how to use the wrapper. You can drag the KinectModelControllerV2 script onto this model and attach the bones and immediately begin testing.Using the ScriptsAfter importing the asset package, you have the KinectExample to test and get familiar with the KinectPointController and KinectModelControllerV2. You can use this as your base scene, or you can drag the appropriate GameObjects into a new one. Setting up the Kinect_PrefabAs previously mentioned, the Kinect_Prefab GameObject is required to have Unity talk with / use the Kinect. This empty object needs to exist somewhere in your scene. However, if you are switching scenes, it will persist because of the DontDestroyOnLoad call in the KinectWrapper script, so you don't need to instantiate a new Kinect_Prefab in each scene. To get the best use out of the Kinect, you may need to tweak some settings within the Kinect_Prefab object itself - here is a breakdown of each option. Device Or EmulatorUse Emulator - Check this to use emulator. Kinect SensorNOTE: do NOT enable this script manually, it is controlled by DeviceOrEmulator. Sensor Height - How high (in meters) off the ground is the sensor. Kinect Center - This tells the Kinect where it should be looking for it's 0,0,0 point (relative to the ground directly under the sensor). The default works pretty well. Look At - Tells the Kinect how to orient the camera using the motor control. NOTE: The following values allow you to smooth the skeleton data. Usually the default values work fine. Do NOT change them unless you find magic numbers for you project. Smoothing - Default value 0.5. Correction - Default value 0.5. Prediction - Default value 0.5. Jitter Radius - Default value 0.05. Max Deviation Radius - Default value 0.04. Project Information URL: http://wiki.etc.cmu.edu/unity3d/index.php/Microsoft_Kinect_-_Microsoft_SDK UPDATE: This project looks to be using the beta version of the SDK and not v1 (as of the time of this post). With the release of the Kinect for Windows SDK v1, I've meant to only highlight projects that are v1 compatible, but missed a few phrase on the page, "Also, since we are not using the new version of Kinect SDK, do NOT download it from Microsoft website...." I would NOT recommend installing the Kinect for Windows SDK from any other source than Microsoft and so suggest, if you've already upgraded to v1 for the SDK, that you hold off going to far with this project. If you are interested in the project, because it does look pretty darn cool, then you encourage the authors to upgrade to v1 as soon as they can. en Tue, 21 May 2013 19:51:50 GMT Tue, 21 May 2013 19:51:50 GMT Rev9 Re: Unity and the Kinect SDK
Is this wrapper for version 1.0 or the beta?

posted by Yorman

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660485074041432 Tue, 28 Feb 2012 17:55:07 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660485074041432 Yorman Re: Unity and the Kinect SDK @Yorman: Based on the dates of the updates, pages, the files in the zip's, etc, I believe it's for v1 (at least that's what I hope, I'm trying to only highlight v1 based projects).

posted by gduncan411

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660498918509292 Tue, 28 Feb 2012 18:18:11 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660498918509292 gduncan411 Re: Unity and the Kinect SDK Hello

I think it's the beta version because it tries to access:
C: \ Program Files (x86) \ Microsoft Research KinectSDK \ MSRKINECTNUI.DLL

posted by yormanh

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660567419601107 Tue, 28 Feb 2012 20:12:21 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660567419601107 yormanh Re: Unity and the Kinect SDK @yormanh:GRRR... I missed this line on the site, "NOTE: Before installing, ensure that the Kinect is NOT connected to your machine. Also, since we are not using the new version of Kinect SDK, do NOT download it from Microsoft website."

So yeah, this looks like it's NOT v1. My bad causing the confusion...

Still if you find this project interesting, I would recommend you contact the authors and encourage them to release a v1 compatible version...

posted by gduncan411

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660591661978048 Tue, 28 Feb 2012 20:52:46 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660591661978048 gduncan411 Re: Unity and the Kinect SDK @yormanh:Besides my comment above, I've also added an Update note to the post...

I'm on the verge of pulling the whole post entirely, but I think the idea behind it is cool and really hope to generate some interest and support for a v1 version, so am leaving it up for now.

Thanks for pointing this out and helping me on this...

posted by gduncan411

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660599883811133 Tue, 28 Feb 2012 21:06:28 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660599883811133 gduncan411 Re: Unity and the Kinect SDK Hello

if the idea is interesting but I don't know to get the name of the class

for example

[DllImportAttribute (@ "C: \ Program Files (x86) \ Microsoft Research KinectSDK \MSRKINECTNUI.DLL", EntryPoint = "NuiInitialize")]
public static extern int NuiInitialize (NuiInitializeFlags dwFlags);

This version does not get the EntryPoint or method names, you know? or the class KinectSensor?

for example (this doesn't  work)
[DllImportAttribute (@ "C: \ Program Files \ Microsoft SDKs \ Kinect \ v1.0 \ Assemblies \ Microsoft.Kinect.dll", EntryPoint = "Start")]
public static extern int Start ();

posted by yormanh

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660606162979969 Tue, 28 Feb 2012 21:16:56 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634660606162979969 yormanh Re: Unity and the Kinect SDK posted by lucabertinetto

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634671625652488540 Mon, 12 Mar 2012 15:22:45 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634671625652488540 lucabertinetto Re: Unity and the Kinect SDK DllNotFoundException: C:\Program Files (x86)\Microsoft Research KinectSDK\MSRKINECTNUI.DLL

I would really appreciate if somebody could link me a way to bridge the Kinect SDK 1.0 to Unity.

posted by lucabertinetto

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634671660107180766 Mon, 12 Mar 2012 16:20:10 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634671660107180766 lucabertinetto Re: Unity and the Kinect SDK Well, this is definitely great and would be very useful to a lot of devs. That it doesn't work with v1 is obviously a turn off. Where would be the right place to ask and see if an update would be possible ?

posted by haknick

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634673368371398446 Wed, 14 Mar 2012 15:47:17 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634673368371398446 haknick Re: Unity and the Kinect SDK @haknick: Good question. Using the page history I believe I've found a point of contact for that project. I've just shot them an email asking if they are indeed a good PoC and if there's any work on a v1 version.

If/when I hear more, I'll post a follow-up comment.

posted by gduncan411

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634673411849979523 Wed, 14 Mar 2012 16:59:44 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634673411849979523 gduncan411 Re: Unity and the Kinect SDK There is indeed work being done on a v1 version... As I learn more I'll let you know. Smiley

posted by gduncan411

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634673483878353134 Wed, 14 Mar 2012 18:59:47 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634673483878353134 gduncan411 Re: Unity and the Kinect SDK posted by Niranjan

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634674021495736615 Thu, 15 Mar 2012 09:55:49 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634674021495736615 Niranjan Re: Unity and the Kinect SDK Also just found this, thought you might find it interesting...

https://github.com/adevine1618/KinectSDK-Unity3D_Interface_Plugin

"This is a wrapper that allows the user to access Microsoft's Kinect SDK v.1.0 data from inside the free Unity3D game engine much faster than a TCP socket connection"

This project was just updated for the v1 SDK a few days ago...

posted by gduncan411

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634674246652017014 Thu, 15 Mar 2012 16:11:05 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634674246652017014 gduncan411 Re: Unity and the Kinect SDK hello

sorry I had not read this post.

Thanks Niranjan, it is true that by changing the path works but I have a problem, thefirst run and it starts but the second time Unity3D dies and does not respond,does  someone this happen?

posted by yormanh

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634675204446292633 Fri, 16 Mar 2012 18:47:24 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634675204446292633 yormanh Re: Unity and the Kinect SDK @Niranjan: Thanks, now it's working Smiley

@yormanh: I modified the source KinectInterop.cs that you can find in the 'NewKinet' example project, change all the dll at the end of the file with "C:\Windows\System32\Kinect10.dll", and THEN I opened the project with unity.
I tried to modify it one time inside unity, with mono, but it crashed afterwards.

@all: if there is participation it would be great to open a Google group, I think Big Smile 

posted by lucabertinetto

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634675761653667111 Sat, 17 Mar 2012 10:16:05 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634675761653667111 lucabertinetto Re: Unity and the Kinect SDK Hello

 

I was reading in the file KinectSensor.cs 

// The MSR Kinect DLL (native code) is going to load into the Unity process and stay resident even between debug runs of the game.
// So our component must be resilient to starting up on a second run when the Kinect DLL is already loaded and
// perhaps even left in a running state. Kinect does not appear to like having NuiInitialize called when it is already initialized as
// it messes up the internal state and stops functioning. It is resilient to having Shutdown called right before initializing even if it
// hasn't been initialized yet. So calling this first puts us in a good state on a first or second run.
// However, calling NuiShutdown before starting prevents the image streams from being read, so if you want to use image data
// (either depth or RGB), comment this line out.
//NuiShutdown();

 

I don't know if there is solve for this problem, I am now investigating more. The group is a good idea.

Anyone know how to contact the developers of the wrapper?

a greeting

 

posted by yormanh

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634677608121061037 Mon, 19 Mar 2012 13:33:32 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634677608121061037 yormanh Re: Unity and the Kinect SDK Hello,

Thanks Niranjan, I changed the DLL path, Unity didnt crashed (lie - now is crashing when I try to play the app) and when I have an GameObject to show the RGB or Depth Camera from Kinect i'm get the following error: 


NullReferenceException: Object reference not set to an instance of an object
System.String.memcpy1 (System.Byte* dest, System.Byte* src, Int32 size)
System.String.memcpy (System.Byte* dest, System.Byte* src, Int32 size)
(wrapper unknown) Kinect.ColorBuffer:PtrToStructure (intptr,object)
KinectSensor.extractColorImage (Kinect.NuiImageBuffer buf) (at Assets/Kinect/KinectSensor.cs:259)
KinectSensor.Kinect.KinectInterface.pollColor () (at Assets/Kinect/KinectSensor.cs:212)
DisplayColor.Update () (at Assets/Kinect/DisplayColor.cs:22)

The line is:

ColorBuffer cb = (ColorBuffer)Marshal.PtrToStructure(buf.m_pBuffer,typeof(ColorBuffer));

Can anyone help me out?

 

Anyone created the the Google Group? I can join?

 

Cheers,

Felipe Fragoso

posted by fdfragoso

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634690765772435707 Tue, 03 Apr 2012 19:02:57 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634690765772435707 fdfragoso Re: Unity and the Kinect SDK Thnx, Niranjan. I changed the paths and now it works, but only first time I running it in Unity. Trying run it second time Unity doesn't respond at all. I have no ideas to solve it.

@fdfragoso: Hi, You said " Unity didnt crashed". What else you did besides changing paths.

Thanks for any advice.

posted by AkhmadMax

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634692909687105454 Fri, 06 Apr 2012 06:36:08 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634692909687105454 AkhmadMax Re: Unity and the Kinect SDK @AkhmadMax: I didnt do anything else. But after few days using, sometimes when I play the Unity crash, but sometimes works fine. Really dont know.

posted by fdfragoso

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634697635462859213 Wed, 11 Apr 2012 17:52:26 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634697635462859213 fdfragoso Re: Unity and the Kinect SDK
dllnot found exception

i have beta2 installed
can anyone help me

posted by AAA

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634714768326232082 Tue, 01 May 2012 13:47:12 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634714768326232082 AAA Re: Unity and the Kinect SDK Hi..

The wrapper is awesome!! Just a quick question:

I am completely new to unity. And I have to use it for integration of my 
Kinect gesture recognition code with the 3d module of a car.  Please help 
me as i am not able to find which script of C# to use for coding my module!! 
Plus it would be great if you can also tell me how to integrate c# 
and JavaScript because the car module is in JavaScript and kinect code is 
in c#!! 

Please reply as soon as possible!! Need it for my project in college!! 

Thanks. 
Mayank 

posted by mayanktaneja

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634718651315480416 Sun, 06 May 2012 01:38:51 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634718651315480416 mayanktaneja Re: Unity and the Kinect SDK posted by Dennis

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634722440948826474 Thu, 10 May 2012 10:54:54 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634722440948826474 Dennis Re: Unity and the Kinect SDK @Dennis:Nice, thanks for the write on this...

posted by gduncan411

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634722621930100631 Thu, 10 May 2012 15:56:33 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634722621930100631 gduncan411 Re: Unity and the Kinect SDK
I can get the bones working but I need the colour feedback too?

Thanks.

posted by Wahooney

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634756229626982287 Mon, 18 Jun 2012 13:29:22 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634756229626982287 Wahooney Re: Unity and the Kinect SDK
The plugin is not anymore updated?

posted by mik

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634760499537351774 Sat, 23 Jun 2012 12:05:53 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634760499537351774 mik Re: Unity and the Kinect SDK I wonder which device is compatible with KinectSDK-Unity3D_Interface_Plugin ?
The XBOX360 kinect (http://emea.microsoftstore.com/UK/en-GB/Microsoft/Xbox-+-Games/Xbox-360-Kinect) or the Windows Kinect (http://emea.microsoftstore.com/UK/en-GB/Microsoft/Kinect-for-Windows)?

Thanks.

posted by THIBAULT David

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634812415151321985 Wed, 22 Aug 2012 14:11:55 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634812415151321985 THIBAULT David Re: Unity and the Kinect SDK @THIBAULT David: The only fully supported device is the Kinect for Windows device.

posted by gduncan411

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634812459444049963 Wed, 22 Aug 2012 15:25:44 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634812459444049963 gduncan411 Re: Unity and the Kinect SDK posted by Stevie Giovanni

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634850078155574702 Fri, 05 Oct 2012 04:23:35 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634850078155574702 Stevie Giovanni Re: Unity and the Kinect SDK It was just a matter of rewrapping the c++ SDK with a DLL and changing a little C# in KinectInterop.cs

If anyone wants the DLL and C# that work with V1.5. Feel free to e-mail me at me@stevehickson.com

posted by Steven Hickson

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634892001818332659 Thu, 22 Nov 2012 16:56:21 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634892001818332659 Steven Hickson Re: Unity and the Kinect SDK @Steven Hickson: I tried e-mail you, but your mail server refuses it. 
Maybe you could share your version of the wrapper by other means.

I'm sure it can be very useful for a lot of people.

Thank you.

 

posted by DyegoMaas

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634903054804278006 Wed, 05 Dec 2012 11:58:00 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634903054804278006 DyegoMaas Re: Unity and the Kinect SDK how can i use the samples from kinect browser kit (face tracking) inside unity3d? thanks a lot

posted by milton ortiz

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634907654348279370 Mon, 10 Dec 2012 19:43:54 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634907654348279370 milton ortiz Re: Unity and the Kinect SDK hi,

Just started thinking of embarking my new project using kinect and unity. I have downloaded Kinect SDK 1.5 since this version is compatible with Kinect XBox sensor, I am using VC# Express 2010. Toolbox also downloaded. How about Unity? Does the free version workable for this? or I need to consider spending my money for Unity 4.

 

posted by arrezes

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634907965450925935 Tue, 11 Dec 2012 04:22:25 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634907965450925935 arrezes Re: Unity and the Kinect SDK Re-opening the comments...

posted by gduncan411

]]> http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634986902785790123 Tue, 12 Mar 2013 13:04:38 GMT http://channel9.msdn.com/coding4fun/kinect/Unity-and-the-Kinect-SDK#c634986902785790123 gduncan411