"Cortana, show me how I can add you to my app's..."
- Posted: Apr 21, 2014 at 6:00 AM
- 23,640 Views
Taking a breather from Visual Studio Extensions, today our Mobile Monday project shows you how you can build Cortana enabled projects...
... You’ve likely already read about how Cortana will use the power of Bing to deliver personalized, natural experiences to users. What you may not have read is how Cortana empowers application developers to deliver robust, natural experiences of their own.
Windows Phone 8.0, our previous release, brought with it the Voice Commands feature. Voice Commands on 8.0 enable an application developer to author simple phrases and lists of words that together allow a user to quickly navigate into a specific experience within an application; for example, “Hulu, show me my queue.”
The biggest limitation of Voice Commands was that the full text of the commands themselves needed to be known ahead of time—this made them difficult to use for searching, social networking, or anything else with a large vocabulary. We knew our developers needed more and we wanted more ourselves—just check out the kinds of things that our team thought up when asked what they’d like to do:
To make these kinds of interactions possible, we’ve introduced new functionality in Windows Phone 8.1 that allows Voice Commands to leverage the functionality of both Cortana and Bing to deliver customized, large-vocabulary transcriptions of user utterances without needing to specify the details of that vocabulary ahead of time. Whereas before a developer could make a command like “Hulu, show me my queue,” now that same developer can make a command like “Hulu, watch the latest episode of Family Guy”—even if “Family Guy” wasn’t known to the application before the query!
While we’ve already worked hard with several well-known apps like Hulu, Facebook, and Twitter to ensure that our users benefit from this feature at release, we also want to provide a “how-to” guide that shows all our developers that they, too, can integrate into the Cortana experience on Windows Phone 8.1. Enter MSDN Voice Search, a freely-available Store application that demonstrates end-to-end use of Cortana-enabled Voice Commands....
MSDN Voice Search is a Store application for Windows Phone 8.1 and Windows Phone 8.0 written to demonstrate the new capabilities of Cortana-integrated Voice Commands on Windows Phone 8.1 as well as to provide examples of good practices when dealing with user input, including by "continuing the conversation" using the in-application Speech Recognition and Speech Synthesis APIs available on Windows Phone.
The application itself is a fully-functional utility designed to enable voice- and text-based natural language searches of MSDN, but it's primary purpose is to facilitate developer interest and education in the Speech and Natural Language integration features available on Windows Phone.
Building the Sample
The MSDN Voice Search project was built using the Windows Phone 8.1 SDK on Visual Studio 2013. Prior versions of Visual Studio are not supported.
MSDN Voice Search demonstrates end-to-end use of Cortana-enabled Voice Commands, Speech Recognition, and Speech Synthesis as combined to deliver a complete and natural user experience for both spoken and text input.
The included Voice Command Definition files (VCDs) are installed at application launch, which will provide integration into the system-level commands for queries like "MSDN, find Windows Phone Voice Commands" or "MSDN, go to the Windows Phone Dev Center."
The code in MainPage.xaml.cs demonstrates low- to mid-complexity management of Voice Command parsing, Speech Recognition use, and Speech Synthesis use.
Source Code Files
- MainPage.xaml - The UI definitions associated with the application
- MainPage.xaml.cs - The bulk of the programmatic logic is located in this file
- AppResources.resx - Resource strings are located here, ready for future localization
- VoiceCommandDefinition_8.1.xml - The Voice Command Definition (VCD) for Windows Phone 8.1
- VoiceCommandDefinition_8.0.xml - The fallback Voice Command Definition (VCD) for Windows Phone 8.0
To learn more about the Voice Command, Speech Recognition, and Speech Synthesis capabilities available to Store applications on Windows Phone, see the Bing Dev Center page: http://www.bing.com/dev/en-us/speech.
[GD: Project Description Copied in Full]
What's nice is that if you've not taken the jump to Update 2 RC for Visual Studio 2013 (i.e. Windows Phone 8.1 SDK), you still check this project out, assuming you have the Windows Phone 8.0 SDK installed.
But if you want to see how this really works, how you can add Cortana support in your apps, you've need Update 2, Visual Studio 2013 Update 2 RC
Users can interact with your Windows Runtime app using speech. Designed thoughtfully and implemented effectively, speech can be a robust and enjoyable way for people to interact with your app, complementing or even replacing interaction by touch, tap, and gestures. Before beginning the development stage, browse the Speech design guidelines for Windows Phone topic for helpful tips on designing a useful, engaging speech-enabled app.
There are three speech components that you can integrate with your app: voice commands (Windows Phone only), speech recognition (Windows Phone only), and text-to-speech (TTS), as demonstrated in the following image:
When a user installs your app, they can automatically use voice commands to access it by speaking "open" or "start", followed by your app name. As a developer, you can also leverage voice commands to allow users to deep link into your app, from outside of your app, by speaking a phrase such as "Start Contoso Search" or "Contoso Show Me My Favorites." You can set up your app so that the phrase links to a specific page in your app, performs a task, or initiates an action.
Discoverability is also a key aspect of voice commands. When you extend and customize voice commands, end users can find out what phrases your app is listening for through system help and the What can I say screen.
You've now should be at a good starting point. You have a sample app, doc's, quick starts and more...