Today...

Kinect Magic Cursor version 1.7 with Gesture support

I am releasing a new version of Kinect Magic Cursor which works much like the last version except it now uses gestures to simulate the left mouse button, instead of raising your left hand. So now your right hand controls the mouse pointer and you can PRESS with your left hand to simulate a single left mouse button click. You can also GRIP (make a fist) with your right hand to simulate holding down the left mouse button for dragging, selecting etc. To stop holding it down you simply RELEASE (stop making a fist). I went with the LEFT hand for PRESS rather than the right hand as I found pressing with the RIGHT hand tending to move the cursor and made it hard to click on small buttons.

Project Information URL: http://drenton72.wordpress.com/2013/05/09/kinect-magic-cursor-version-1-7-with-gesture-support/

Project Download URL: Click here to download Kinect Magic Cursor V1.7

Project Source URL: Click here to download Kinect Magic Cursor v1.7 with C# source

Contact Information:

• Blog: http://drenton72.wordpress.com
• Twitter: @drenton72

And who doesn't like a new SDK to play with?

HeadTexter, A Head Movement to Text Conversion using Intel Perceptual Computing SDK

HeadTexter is a simple App targeted to convert head movement to English text. It uses Intel Perceptual Computing SDK  for head tracking and is a contest entry in recently concluded perceptual computing challenge.

Background

Before you read the rest of article you must know the reasons behind this madness of trying to convert the head movement to text.   It is to carry a research and give a direction for communication to Alzheimer's disease.

Alzheimer's Disease is a disease which is mainly caused by stroke. It paralyzes the patients and they looses capability to move any other parts of their body other than their head. The head movement also becomes limited with time and only way they can convey any message is through their eyes. So I basically wanted to build a system that converts the eye movement to text. But the limitations of Beta SDK in face tracking and more in eye tracking has made me to shift the design a bit towards face tracking. 

The interaction may not be quite realistic for Alzheimer's patients, but somehow it had to be started at a point. So I decided to build the basic framework with head movement ( which turned out more complicated than I thought it would be) and to transform the work to eye tracking once the sdk issues are fixed. That would mean some simple changes in the transform part and should come easy.

Now coming to my decision of writing a tutorial on head tracking based system rather than hand gesture work is pretty simple. It works with any normal web cam. Yes that is right. All of you can actually install the sdk and get on with the programming without a creative gesture camera. So my motive of  introducing Intel Perceptual Computing SDK before the community ( especially those crazy C#'rs was to give everybody an opportunity and a simple walk through with the sdk which they can carry forward).

So what basically we are learning here in this tutorial?

a) Working with Perceptual Computing SDK

b) Head Tracking system with Perceptual Computing SDK

c) Do something funny ( or meaningful) with the tracked data.

d) Learn how to use a 175 years old computer concept effectively.

Let's not waste any more digital bytes in clarifying my motives or article's objective and start with what we do best! Code.

Using the code

First download the sdk from  Perceptual Computing SDK Download Page

Start with SDK:

The SDK is mainly written in C++ as you might expect it to due to speed constraints and what you get for C# is a managed dll. So Start a project and add a reference to libpixclr.dll located in sdk/bin/X64 or  sdk/bin/X86 folder. If you are really that inclined to a 64 bit application then you must also change the  project properties to 64 bit. If you select x86, don't forget to change the project properties to x86. "Any CPU" will not work. Unlike some other solutions like Microsoft's Ink technology that  does work only in x86 architecture and the applications fails in x64, no such worries with this one. Select x86 and it runs well on both the architecture.

All of us who are well versed with OpenCV and C++ style of coding use an infinite loop  for acquiring data. However as it had to be  C#, I wanted the solution to be BackgroundWorker based. So what we will do is come out of those for(; stuff of OpenCv and capture the frames from DoWork, processing would be performed from progress changed. UI and SDK threads are entirely different stack, so we will use a delegate to update UI with the result from SDk thread.

The first thing you got to do with any PerC work is create a session with sdk. So you need an object of PXCMSession.  Let us call it session;

...

So install the Perceptual Computing SDK Download Page (I used Beta 3). Don't worry about not having the mentioned Camera, for this project, your built in camera should work (at least it did for me).

Download the project's source and fire it up in Visual Studio. In my case, I had to switch to the Debug configuration for the project to find the references, compile and run

And ran it did (you've got to love my "What am I doing! It's Saturday!" look

Make sure you read the rest of the CodeProject post to see how he uses the SDK and some of the hoops he had to jump through...

Is this going to set the world on fire? Doubt it. Is this something that you could do with OpenCV? Probably. But come on! This is a new cool SDK too play with and bend to your will!!!

 For more information, please see the additional topic here:

• Make your site touch-ready

See also related videos here:

• Web Standards: How to use XMLHttpRequest

Book about Kinect and Gestures, "Programming with the Kinect for Windows Software Development Kit: Add gesture and posture recognition to your applications"

You want to learn how to develop with Kinect?

You want to learn how to detect gestures or postures?

You want to learn how to create reality augmented application using Kinect?

If yes, this book is for you:

http://www.amazon.com/Programming-Kinect-Windows-Software-Development/dp/0735666814/ref=sr_1_11?ie=UTF8&qid=1343746483&sr=8-11&keywords=Kinect+book

It will be available on September 22, 2012 and here is the pitch:

Create rich experiences for users of Windows 7 and Windows 8 Developer Preview with this pragmatic guide to the Kinect for Windows Software Development Kit (SDK). The author, a developer evangelist for Microsoft, walks you through Kinect sensor technology and the SDK—providing hands-on insights for how to add gesture and posture recognition to your apps. If you’re skilled in C# and Windows Presentation Foundation, you’ll learn how to integrate Kinect in your applications and begin writing UIs and controls that can handle Kinect interaction.

• This book introduces the Kinect for Windows Software Development Kit to developers looking to enrich applications they build for Windows 7 and later with human motion tracking
• Teaches developers with core C# and WPF skills how to program gesture and posture recognition in Kinect
• Describes how to integrate 3D representation on top of a real scene
• Provides expert insights and code samples to get you up and running

Project Information URL: http://blogs.msdn.com/b/eternalcoding/archive/2012/07/31/book-about-kinect-and-gestures.aspx

Preorder it now! http://www.amazon.com/Programming-Kinect-Windows-Software-Development/dp/0735666814/

Contact Information:

• Blog: http://blogs.msdn.com/eternalcoding
• Twitter: @deltakosh

With touch devices, like the Windows Phone, it's important that we take advantage of gestures, making them a integral part of our applications. You're thinking, "Easy for you to say, but how?"

Here's one example...

A gesture-driven Windows Phone to-do application

This blog post describes the implementation of a gesture-based todo-list application. The simple interface is controlled entirely by drag, flick and swipe:

So far the application supports deletion and completion of tasks, but not the addition of new ones. I’ll get to this in a later blog post!

Introduction – gestures, why don’t we use them more?

I think it is fair to say that most Windows Phone applications (and mobile applications in general) have user-interfaces that are a close reflection of how we interact with a desktop computer. Mobile applications have the same buttons, checkboxes and input controls as their desktop equivalent, with the user interacting with the majority of these controls via simple clicks / taps.

The mobile multi-touch interface allows for much more control and expression than a simple mouse pointer device. Standard gestures have been developed such as pinch/stretch, flick, pan, tap-and-hold, however these are quite rarely used; one notable exception being pinch/stretch which is the standard mechanism for manipulating images.
When an application comes along that makes great use of gestures, it really stands out from the crowds. One such application is the iPhone ‘Clear’ application, a simple todo-list with not one button or checkbox in sight. You can see the app in action below:

...

Interestingly, its use of pinch to navigate the three levels of menu is similar to the Windows 8 concept of ‘semantic zoom’.

When I first saw Clear – the clean, clutter-free interface immediately spoke ‘Metro’ to me! This blog post looks at how to recreate some of the features of Clear using Silverlight for Windows Phone, in order to create a gesture-driven todo application.

NOTE: All of the work on my blog is under a Creative Commons Share-alike license. For this blog post I just want to add that I do not want someone to take this code in order to release a ‘Clear’ clone on the Windows Phone marketplace. This blog post is for fun and education, to make people think more about the possibilities of gestures.

...

The post then continues on providing specific examples and details

Handling gestures

Silverlight for Windows Phone provides manipulation events which you can handle in order to track when a user places one or more fingers on the screen and moves them around. Turning low-level manipulation events into high-level gestures is actually quite tricky. Touch devices give a much greater control when dragging objects, or flicking them, but have a much lower accuracy for the more commonplace task of trying to hit a specific spot on the screen. For this reason, gestures have a built in tolerance. As an example, a drag manipulation gesture is not initiated if the user’s finger moves by a single pixel.

Fortunately, the Silverlight Toolkit contains a GestureListener which handles manipulation events and turns them into the standard gesture events for you. Unless you need a quite fancy gesture (two-finger-swipe for example), the GestureListener probably gives you all you need. To use this class, simply attach it to the element that you want to handle gestures on, then add handlers to the events that the GestureListener provides.

While the post provides a great series of code snaps, sometimes it's easier to see it as a complete project. Well Colin's got that covered as well and provided the complete project source.

If you download the source and run into a reference issue for the Microsoft.Phone.Controls.Toolkit (aka the Silverlight for Windows Phone Toolkit) assembly (shown below), the easiest way to fix that is via NuGet. 

Just search for silverlighttoolkitwp, add it to the project and you should be good to go.

Once I did that, the project ran for me the first time. Here's a snap of it running on my machine...

Want to add natural gesture support to your Windows Phone app? You'll want to check out this project...

John O'Donnell Microsoft Dynamics ISV Architect Evangelist
Microsoft Corporation
http://blogs.msdn.com/jodonnell
http://blogs.msdn.com/usisvde
http://www.twitter.com/jodonnel

Here is the description for the session he will be doing at MIX09:

Touch and Gesture Computing, What You Haven’t Heard

by: Joe Fletcher

 

Learn about new patterns, behaviors, and design approaches for touch and gesture interfaces from a practitioners point of view.  Learn early lessons from applied knowledge of touch applications, devices, and design methods.

I’ve recently come across freeware like Mazzrick, a portable mouse gesture utility that lets you create your own custom gestures, and the full-featured gMote which, in addition to being portable, supports over 30 actions.

Given the usability of these freeware/donationware programs, I’m not sure why someone would pay for a mouse gestures app, but mgLaunch ($29.95) is hoping you will. Their app launcher is skinnable, so maybe that’s what makes it worth it? I’m not sure, but it also crossed my radar not too long ago, so it may be worth a look. 

Another gesture-based utility is the IE plugin called simply “Mouse Gestures for Internet Explorer.” With this software, you can control various tasks related to your web surfing – like opening and closing windows -  just by drawing a simple symbol on the screen. (Works with XP/Vista/64-bit,  too).

But with computers getting smaller and more portable, I would think more people are using the touchpad than the mouse. Still, the netbook phenomenon may have people starting to pack a mini mouse to go, too, since those netbook touchpads are crazy small.

Are you a fan of mouse gesture applications? If so, feel free to share which ones you like and recommend in the comments.