Demystifying the Interaction Stream (and how to use it without WPF)

Description

The second in our two part InteractionStream without WPF series comes from András Velvárt, where he dives into a good deal of depth, not only showing how it works, but explains the details behind it too...

Kinect Interactions with(out) WPF – Part III: Demystifying the Interaction Stream

 In this part of my Kinect Interaction blog post series, we go deep into the rabbit hole, and examine the foundation of Kinect Interactions – the InteractionStream, upon which the entire library is built. This is a risky ride – with no official documentation, we can only count on our trusty reflector, the source code of the Kinect Interaction SDK and careful exploration.

You only need to access the treasures of InteractionStream, if you want to go beyond what the KinectRegion and other controls provide. For example, you want to create your own KinectRegion, you want to zoom a map by gripping it with two hands, or want to build your entirely new interaction model, using two hands along with the press and grip gestures.

Initializing the InteractionStream

Initializing the InteractionStream is much like initializing the DepthStream or the SkeletonStream. If you have a KinectSensor object, all you need are the next two lines of code:

...

It seems like we are on track. Just pass an initialized KinectSensor object and this DummyInteractionClient to the constructort of the InteractionStream, and we should be all set, right? Well, not quite. The InteractionFrameReady event does not fire.

Interaction Needs Skeleton and Depth

It turns out, that for the InteractionStream to work, it needs to process the data from both the depth and the skeleton streams. So, we need to initiate all three of the streams. This is what the entire OnLoaded method (which you have to wire up either in XAML or in the constructor of the page) looks like:

...

It doesn’t seem so simple now, so let's walk through each line of code and see what they do. The first five lines define local variables that hold a reference to the Kinect sensor the application will use, the InteractionStream itself, the skeletons identified by the SkeletonStream, and the hand position information about the users as determined by the InteractionStream.

...

Dissecting InteractionFrameReadyEventArgs

So, now that we have finally made the InteractionStream call the InteractionFrameReady event, we can start analyzing what kind of data we can get to. The first step is to get our hand on an InteractionFrame, and the interaction data it contains:

image

Summary

I’ve shown you how you can get started with the InteractionStream and acquire information about the hands of the users, detect active hands, pressing (IsPressed and PressExtent), and open / closed hands (HandEventType and our custom LastHandEventType). What I’ve shown here does not require WPF – you can use this from any .NET application, be it XNA, Windows Forms or even Command Line.

Project Information URL: http://dotneteers.net/blogs/vbandi/archive/2013/05/03/kinect-interactions-with-wpf-part-iii-demystifying-the-interaction-stream.aspx

Project Source URL: Source Download

private Dictionary<int, InteractionHandEventType> _lastLeftHandEvents = new Dictionary<int, InteractionHandEventType>(); private Dictionary<int, InteractionHandEventType> _lastRightHandEvents = new Dictionary<int, InteractionHandEventType>();
Dictionary<int, InteractionHandEventType>();
  
private void InteractionStreamOnInteractionFrameReady(object sender, 
ctionFrameReadyEventArgs args)
{
     using (var iaf = args.OpenInteractionFrame()) //dispose as soon as 
le
     {
         if (iaf == null)
             return;
  
         iaf.CopyInteractionDataTo(_userInfos);
     }
  
     StringBuilder dump = new StringBuilder();
  
     var hasUser = false;
     foreach (var userInfo in _userInfos)
     {
         var userID = userInfo.SkeletonTrackingId;
         if (userID == 0)
             continue;
  
         hasUser = true;
         dump.AppendLine("User ID = " + userID);
         dump.AppendLine("  Hands: ");
         var hands = userInfo.HandPointers;
         if (hands.Count == 0)
             dump.AppendLine("    No hands");
         else
         {
             foreach (var hand in hands)
             {
                 var lastHandEvents = hand.HandType == 
ctionHandType.Left
                                             ? _lastLeftHandEvents
                                             : _lastRightHandEvents;
  
                 if (hand.HandEventType != 
ctionHandEventType.None)
                     lastHandEvents[userID] = hand.HandEventType;
  
                 var lastHandEvent = lastHandEvents.ContainsKey(userID)
                                         ? lastHandEvents[userID]
                                         : 
ctionHandEventType.None;
  
                 dump.AppendLine();
                 dump.AppendLine("    HandType: " + hand.HandType);
                 dump.AppendLine("    HandEventType: " + 
andEventType);
                 dump.AppendLine("    LastHandEventType: " + 
ndEvent);
                 dump.AppendLine("    IsActive: " + hand.IsActive);
                 dump.AppendLine("    IsPrimaryForUser: " + 
sPrimaryForUser);
                 dump.AppendLine("    IsInteractive: " + 
sInteractive);
                 dump.AppendLine("    PressExtent: " + 
ressExtent.ToString("N3"));
                 dump.AppendLine("    IsPressed: " + hand.IsPressed);
                 dump.AppendLine("    IsTracked: " + hand.IsTracked);
                 dump.AppendLine("    X: " + hand.X.ToString("N3"));
                 dump.AppendLine("    Y: " + hand.Y.ToString("N3"));
                 dump.AppendLine("    RawX: " + 
awX.ToString("N3"));
                 dump.AppendLine("    RawY: " + 
awY.ToString("N3"));
                 dump.AppendLine("    RawZ: " + 
awZ.ToString("N3"));
             }
         }
  
         tb.Text = dump.ToString();
     }
  
     if (!hasUser)
         tb.Text = "No user detected.";
}

Contact Information:

The Discussion

  • User profile image
    Ben

    Thank you, i have been waiting forever to get the Interaction Stream Rockin'! This was informative.

Comments closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums, or Contact Us and let us know.