What Do You Want to Know About Cortana? Ask the Guy Who Led the Cortana API Team, Rob Chambers

Sign in to queue


Later this week, Microsoft Evangelist Jeremy Foster will sit down with the General Product Manager for the Conversational Understanding Platform, Rob Chambers and ask crowdsourced questions from our community on a topic we're all excited about: Cortana.

If anyone has insight into how developers can get a jumpstart on integrating Cortana into new or existing apps, it's the person that helped define and build the new capabilities available for developers for Windows Phone 8.1. As the General Product Manager on point to harness the power of Bing and bring it to life with Cortana, Rob worked closely with Dev Manager Mike Calcagno to ensure the end product would delight end users and empower developers.

What do you want to know?

Have you been curious about the CU API? Do you want to integrate it into your app? This is your chance to pick the brain of an insider and get your questions answered. Here's how:

Simply tweet your questions to #CortanaQA and we will get as many answered as possible during a taped interview segment on Channel 9 on April 25. If you're already starting to integrate Cortana into a new or existing project tweet your screenshots, Vine clips, and videos of your project using the #IDevThis hashtag where we'll be looking for the next project to showcase on Ch9.

Rob's Guidance for Developers Considering Cortana

To get you ready, we asked Rob to describe his role and give some pointers for developers chomping at the bit to build something great with Cortana.

What team do you work on at MSFT?

 I am the GPM for the Conversational Understanding (CU) Platform and API team inside Bing. The CU team is responsible for all speech and language understanding technologies that other Microsoft platforms and applications use to NUI enable their systems, like Xbox One, and most recently Cortana on WP. The Platform and API sub-team inside CU builds the API surface, and platform components at the top of the software stack that run on the phone itself. These are the APIs that developers inside WP use to build Cortana, and also the APIs that third party developers can use to extend Cortana's

At a high level (since we'll address this in the chat) what do developers need to have ready (or be prepared with, etc.) to integrate Cortana API's in their app?

 There are basically three things developers have to do to integrate with Cortana.

  1. Define what "voice commands" their application would like to support, declaratively, in a schematized XML file. These voice commands include phrase patterns, like "Skype, Call [contact_name]", or be as general as "Hulu, [whatever_the_user_might_say_here]". The voice commands also contain additional information, like how the application should be launched, and what should the user see while the application is loading (e.g. "Searching for Mr. Peabody showtimes...").

  2. Register the XML file containing the voice commands at application first launch. The voice commands can be updated after the fact, as well, but like a Background Task, you must "install" the commands when your app is run for the first time.

  3. Handle the voice activation, once Cortana recognizes that the user spoke your app name, followed by a command you've registered". You'll receive, on application activation/page navigation (WinRT app vs Silverlight app) parameters that indicate which voice command was recognized, the fill_in_the_blank_phrases, as well as the full text that was recognized (or typed) into Cortana's UX.

That's it...

What use cases should developers focus on first?

Applications that help users communicate with people, or the world, Apps that help the user remember or do something later, or applications that help the user find information, or objects, and optionally do something with those things ... All of these application patterns are good application patterns to implement when integrating applications into the Cortana experience. Primarily, that's because these patterns are the most powerful scenarios that help Cortana users the most.

What is the ultimate integration of Cortana you'd like to see?

I'd love for 100+ of the top 500 applications in the store today to integrate with Cortana, especially ones that match the patterns I just described. If I had to pick a single application that I'd personally like to see integrate with Cortana ... I guess I'd say Amazon Mobile, or maybe IMDB. If you would have asked me a couple months ago, I would have said Flixster, but they're already working on that now.

What are you working on now? Does work continue on Cortana or are you jumping to a new, awesome project?

I'm continuing to work inside CU, on the Platform and APIs, working with the same folks I worked with on WP 8.1, trying to take the next big step for NUI for our users. I've been working in or around the NUI space for 15+ of my ~19 years here at Microsoft. We're not done yet, and so ... I'm not done yet either...


[UPDATE] Check out the Q&A with Rob Chambers here: https://channel9.msdn.com/Blogs/Dev-Inspiration/What-Developers-Need-To-Know-About-the-Cortana-API


The Discussion

  • User profile image

    I would like to see a rough platform roadmap if that is OK?


    YEAR 1:

    Windows Phone -> US -> EMEA -> Rest of the World

    Windows 8.1 Integration

    XBOX One Integration

    YEAR 2:

    Cortana as a service in Azure?

    Third Party Platforms? (IOS/Android?)

  • User profile image

    @Ian2 - I agree.  I want to see Cortana pushed to Windows 8.x (or 9) and then onto Xbox One.  It would also be cool if the team created a UserVoice page for Cortana so folks could request and vote on features. 

  • User profile image

    I asked here (reddit) but just to be sure:

    One of my apps name is "Quotastic". I register it as CommandPrefix and it works perfectly on 8.0 but when this runs on 8.1 (using Cortana or not) the speech system always try to translate it to "real" words. So I say "quotastic", and the result is something like "old * kicked", "good to see", "go to seek", "old * chick" "odessa texas"(?!), "comcast". I ended up changing the CommandPrefix to "Quote" and it works, but is not ideal. So, whats up? Why 8.1 is not picking up my commandPrefix and giving priority to it when trying to find out what the user said?

  • User profile image
    Deactivated User

    As NUI evolves, the top store apps will hopefully be upset by surprises rather than existing top apps doing better with Cortana. NUI+AI will take UX to unforeseen level. Thnx. :)

  • User profile image

    It would be nice if everything was morphed into the Share contract and have that  interface made available/uniform between the Windows Phone 8 and Windows Store 8 platforms.

  • User profile image
    mahamudul haque

    I downloaded some apps hear all use my tab mobile business program.None job continues got reach.How I make it properly? Use right apps for daily life in my E - commerce.I will be happy in society.Living with lifestyle.

  • User profile image

    Cortana has irritated me for too long. How do I get it to quit popping out, taking up screen space and hanging up my keyboard while I'm trying to work? I do not use Cortana for anything except an occasional search to find something that I want to change, re: the way I use my software and a Lenovo "Yoga-710" 2-in-1 laptop. I am sick of somehow accidentally causing Cortana to popup and very tired of the increasingly annoying need to stop my workflow to do 1 or more clicks somewhere to deactivate the popup. Please help. Thanks ~ Michael

Add Your 2 Cents