NUI for PowerPoint

No mouse, no pointer, no clicker, just you and PowerPoint (and a Kinect of course)...

Kinect PowerPoint Control

This project shows how to use the Kinect SDK to do basic gesture recognition to control PowerPoint.

...

Usage

1. Compile and run the application
2. Point the Kinect at you and stand at least five feet away
3. You can see yourself in the application window and the three ellipses will track your head and hands.
4. Extend your right arm to activate the "right" or "forward" gesture. Extend your left arm to active the "left" or "back" gesture. These gestures will send a right or left arrow key to the foreground application, respectively.
5. Run your PowerPoint show so PowerPoint is the foreground application, and the right and left gestures will go forward and back in your deck.

The ellipses grow and change color when your hand exceeds the threshold of 45 centimeters. The gestures will only activate once as your hand exceeds the threshold, and only one of the gestures can be active at once. You must bring your hand back closer to your body to activate the gesture a second time.
The gestures will also work for any other application. For example, open Notepad and type some text then use the gestures to move the cursor left or right one character at a time.

Project Information URL: http://kinectpowerpoint.codeplex.com/

Project Download URL: http://kinectpowerpoint.codeplex.com/releases/view/70037

Project Source URL: http://kinectpowerpoint.codeplex.com/SourceControl/list/changesets

void runtime_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
     SkeletonFrame skeletonSet = e.SkeletonFrame;

     SkeletonData data = (from s in skeletonSet.Skeletons
                          where s.TrackingState == SkeletonTrackingState.Tracked
                          select s).FirstOrDefault();

     var head = data.Joints[JointID.Head];
     var rightHand = data.Joints[JointID.HandRight];
     var leftHand = data.Joints[JointID.HandLeft];
     SetEllipsePosition(ellipseHead, head, false);
     SetEllipsePosition(ellipseLeftHand, leftHand, isBackGestureActive);
     SetEllipsePosition(ellipseRightHand, rightHand, isForwardGestureActive);

     ProcessForwardBackGesture(head, rightHand, leftHand);
}

private void ProcessForwardBackGesture(Joint head, Joint rightHand, Joint leftHand)
  {
      if (rightHand.Position.X > head.Position.X + 0.45)
      {
          if (!isBackGestureActive && !isForwardGestureActive)
          {
              isForwardGestureActive = true;
              System.Windows.Forms.SendKeys.SendWait("{Right}");
          }
      }
      else
      {
          isForwardGestureActive = false;
      }

      if (leftHand.Position.X < head.Position.X - 0.45)
      {
          if (!isBackGestureActive && !isForwardGestureActive)
          {
              isBackGestureActive = true;
              System.Windows.Forms.SendKeys.SendWait("{Left}");
          }
      }
      else
      {
          isBackGestureActive = false;
      }
  }

private void RecognizeAudio()
  {
      using (var source = new KinectAudioSource())
      {
          source.FeatureMode = true;
          source.AutomaticGainControl = false; //Important to turn this off for speech recognition
          source.SystemMode = SystemMode.OptibeamArrayOnly; //No AEC for this sample
          source.MicArrayMode = MicArrayMode.MicArrayAdaptiveBeam;
          
          RecognizerInfo ri = SpeechRecognitionEngine.InstalledRecognizers().Where(r => r.Id == RecognizerId).FirstOrDefault();

          if (ri == null)
          {
              Console.WriteLine("Could not find speech recognizer: {0}. Please refer to the sample requirements.", RecognizerId);
              return;
          }

          Console.WriteLine("Using: {0}", ri.Name);

          using (var sre = new SpeechRecognitionEngine(ri.Id))
          {
              var phrases = new Choices();
              phrases.Add("computer show window");
              phrases.Add("computer hide window");
              phrases.Add("computer show circles");
              phrases.Add("computer hide circles");

              var gb = new GrammarBuilder();
              //Specify the culture to match the recognizer in case we are running in a different culture.                                 
              gb.Culture = ri.Culture;
              gb.Append(phrases);

              // Create the actual Grammar instance, and then load it into the speech recognizer.
              var g = new Grammar(gb);

              sre.LoadGrammar(g);
              sre.SpeechRecognized += SreSpeechRecognized;
              sre.SpeechHypothesized += SreSpeechHypothesized;
              sre.SpeechRecognitionRejected += SreSpeechRecognitionRejected;

              using (Stream s = source.Start())
              {
                  sre.SetInputToAudioStream(s,
                                            new SpeechAudioFormatInfo(
                                                EncodingFormat.Pcm, 16000, 16, 1,
                                                32000, 2, null));

                  //sre.RecognizeAsync(RecognizeMode.Multiple);
                  while (shouldContinue)
                  {
                      sre.Recognize();
                  }
                  //sre.RecognizeAsyncStop();
              }
          }
      }
  }

Contact Information:

• Blog: http://nui.joshland.org/
• Twitter: @joshblake

Page image, Powerpoint + Clickers = Evil, courtesy of Steve Burt