Going MAD with Kinect. The MADExpo Kinect-enabled Session Finder

This mash-up project combines the power of Kinect, with an embedded WebBrowser in a WPF app. It's almost a Kinect enabled web browser (Actually it IS a Kinect enabled web browser, but one that's limited to a specific site and controlled functionality)

MADExpo Kinect-enabled Session Finder

Here's a video of a fun and cool project I worked on for the Mid Atlantic Developer Expo, namely a Kinect-enabled Session Finder.

The project uses the recently-released Kinect SDK for Windows beta, a custom version of the Session Sorter code from the MADExpo website, hosted in a WebBrowser control within a WPF application (the page uses the isotope.js library for sorting, filtering, and animated transitions, which would be a lot of work to replicate in WPF, so hosting in a WebBrowser control saved time). The WPF application is useful since I can use it to capture audio from Kinect and enable speech recognition, as well as to run the app in kiosk mode. Meanwhile, in the background, I'm running the Coding4Fun Mouse Cursor sample, which captures my gestures and turns them into mouse moves and clicks.

One tricky bit was getting the speech recognition commands passed through to the web page hosted inside the WPF app. Turns out that the WebBrowser control exposes a method called InvokeScript, which you can use to call a named function within the target page. So I coded up a series of javascript functions on the underlying page to do things like scroll up or down, simulate a click on the sort and filter elements (using jQuery's .click() api), etc. While the code isn't pretty, it was easy to do overall, and the result is pretty nifty. Here's the video...

Project Information URL: http://devhammer.net/blog/kinect-resources-and-news

Project Download URL: http://dl.dropbox.com/u/16993169/Presentation%20Resources/devhammer.kinect.roanoke.100611.zip

Project Source URL: http://dl.dropbox.com/u/16993169/Presentation%20Resources/devhammer.kinect.roanoke.100611.zip

private void CaptureAudio()
{
    this.source = new KinectAudioSource();
    this.source.FeatureMode = true;
    this.source.AutomaticGainControl = false;
    this.source.SystemMode = SystemMode.OptibeamArrayOnly;
    this.source.MicArrayMode = MicArrayMode.MicArrayFixedBeam;
    //this.source.MicArrayBeamAngle = 0.0;
    RecognizerInfo ri = SpeechRecognitionEngine.InstalledRecognizers()
        .Where(r => r.Id == RecognizerId).FirstOrDefault();
    if (ri == null)
    {
        return;
    }
    this.sre = new SpeechRecognitionEngine(ri.Id);
    var words = new Choices();
    words.Add("up");
    words.Add("down");
    words.Add("back");
    words.Add("reset");
    words.Add("title");
    words.Add("speaker");
    words.Add("room");
    words.Add("time");
    words.Add("thursday");
    words.Add("friday");
    words.Add("all");
    //words.Add("devhammer");
    var gb = new GrammarBuilder();
    gb.Culture = ri.Culture;
    gb.Append(words);
    var g = new Grammar(gb);
    sre.LoadGrammar(g);
    this.sre.SpeechRecognized +=
        new EventHandler<SpeechRecognizedEventArgs>(sre_SpeechRecognized);
    this.sre.SpeechHypothesized +=
        new EventHandler<SpeechHypothesizedEventArgs>(sre_SpeechHypothesized);
    this.sre.SpeechRecognitionRejected +=
        new EventHandler<SpeechRecognitionRejectedEventArgs>(sre_SpeechRecognitionRejected);
    this.stream = this.source.Start();
    this.sre.SetInputToAudioStream(this.stream, new SpeechAudioFormatInfo(EncodingFormat.Pcm, 16000, 16, 1, 32000, 2, null));
    this.sre.RecognizeAsync(RecognizeMode.Multiple);
}

private void sre_SpeechRecognitionRejected(object sender, SpeechRecognitionRejectedEventArgs e)
{
    AnimatedAlert("Speech Rejected. Please try again.");
}

private void sre_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)
{
    if (e.Result.Confidence > 0.93)
    {
        switch (e.Result.Text)
        {
            case "up":
                invokeUiCommand("ScrollUp");
                return;
            case "down":
                invokeUiCommand("ScrollDown");
                return;
            case "back":
                invokeUiCommand("back");
                return;
            case "reset":
                invokeUiCommand("reset");
                return;
            case "title":
                invokeUiCommand("title");
                return;
            case "speaker":
                invokeUiCommand("speaker");
                return;
            case "room":
                invokeUiCommand("room");
                return;
            case "time":
                invokeUiCommand("time");
                return;
            case "thursday":
                invokeUiCommand("Thursday");
                return;
            case "friday":
                invokeUiCommand("Friday");
                return;
            case "all":
                invokeUiCommand("All");
                return;
            //case "devhammer":
            //invokeUiCommand("devhammer");
            //return;
        }
    }
    else
    {
        AnimatedAlert("Word not recognized. Please try again.");
    }

}

Contact Information:

• Blog: http://devhammer.net
• Twitter: @devhammer