Kinect v2 + Unity3D + C# = Interactive Artist's Books

Dec 03, 2015 at 6:00AM

by Greg Duncan

Description

I came across this project in the Kinect 2 SDK forum and thought you all would like it...

Interactive Artist's Books - direct gestural interaction with 3d objects

[Post copied in full]
I recently finished an installation for the Swiss National Library (CH) using Kinect v2, Unity3D and the C# SDK. It's about presentation and direct interaction (e.g. rotation, zoom) with sensitive exhibit's in a museum and exhibition context using a large venue projection.
Showcase video: https://vimeo.com/146547262
I'd like to share some experiences back to the forum as I was very happy to see and use answers from other projects / questions. If you have questions on specific topics, feel free to ask.
Equipment: gaming notebook with Nvidia graphics card (Lenovo Y50-70), Kinect v2 for Windows, Windows 8.1 Pro, Visual Studio CE for C# code
In the video, you can see two 3d cursors for the hands. In contrast to the UI SDK, the hands cursor also transmit depth (Z) of the hands. Most important: use the joint smoothing code from here: https://social.msdn.microsoft.com/Forums/en-US/045b058a-ae3a-4d01-beb6-b756631b4b42/joint-smoothing-code?forum=kinectv2sdk
To get a nice cursor, you need to implement something like a 3D PHIZ (2D PHIZ described in https://channel9.msdn.com/Series/Programming-Kinect-for-Windows-v2/07). This normalizes with body/shoulder rotation, arm lengths, and makes the movement of each hand relative to the body.
Hand states: if a person is more than 2m away from the sensor, the hand states get very jittery. In my installation, I use a closed hand state for zooming and rotating. I smooth out hand state at 0.25s - which seems quite stable and yet is not too long to be really noticeable.
For the exposition context, only the foremost person in front of the sensor is able to control the object (artificial limit)
Body movement and position in the room is also used for walking "around" the object or coming closer. This really works very well and stable.
I used floor marks to show best standing position to the visitors. When walking out of reach of the sensor, the object gets into a screensaver state but just moves slowly away from the position. So when a person gets closer again, the position seems quite natural.
Implementing 3d cursor to drive the physx physics body (touch, pull) is not trivial in Unity3D. The problem is that the hands can pass through the object or a pull with closed hands collides with the object mesh. In the installation, a mesh collider is used with custom raycasting to prevent pass-throughs and with custom push/slide mechanism (similar to a character controller).
The 3d models themselves were digitized using photogrammetry and have a high quality.
The chocolate cake shown in the video is a real exposition object that must not be exhibited because the chocolate would melt. So virtual touch is the ideal means to present this object publicly.
There's more information on the objects, including a WebGL viewer at http://nationalbibliothek.ch/3d.
Regards, Kai
[Please click through for the original post]