Also last time we didn't look at the cool sample app that's included in the source tree...

Extended WPF Toolkit - Extended WPF Toolkit 1.5.0

Want an easier way to install the Extended WPF Toolkit?

The Extended WPF Toolkit is available on Nuget.

What's in the 1.5.0 Release?

• BusyIndicator
• ButtonSpinner
• Calculator
• CalculatorUpDown
• CheckListBox - New Control
• ChildWindow
• CollectionEditor - New Control
• CollectionEditorDialog - New Control
• ColorCanvas
• ColorPicker
• DateTimePicker
• DateTimeUpDown
• DecimalUpDown
• DoubleUpDown
• DropDownButton - New Control
• IntegerUpDown
• Magnifier
• MaskedTextBox
• MessageBox
• MultiLineTextEditor - New Control
• PrimitiveTypeCollectionEditor - New Control
• PropertyGrid - Beta
• RichTextBox
• RichTextBoxFormatBar
• SplitButton
• TimePicker
• WatermarkTextBox
• Wizard - New Control

So as you can see, v1.5 has a number of new cool things to play with.

For example the Wizard control. Many app's use them and many dev's re-invent that wheel when they do. This control looks simple, yet configurable enough, that maybe we don't need to re-invent it anymore (and even if we do, since the source is available, we don't have to start from scratch!

Then there's the some of the other controls, new and old, too;

Like one of my favorite, the calculator control

Or the watermark textbox

Or the new WPF Property Grid;

One thing about these samples... They are "live." They are not static examples or screenshots, but instead live, working code.

Speaking of the Sample app. I thought it very cool that the sample app used Prism. Prism is meant to help you build composite applications, separating the shell and components and allowing them all to evolve independently.

So what does that mean? It means that what you're seeing above is a simple shell that hosts modules that do the work. That once created, you're pretty much done with the shell and you can focus on the specific modules.

The Sample shell is a simple host for the sample modules. In this case modules are added to the shell via the app.config.

If we take a peek at the wizard sample module;

Okay, okay, I know, enough samples. Here's a snap of the Toolkit project itself;

As you can see the Toolkit project is well segmented by control, making it very easy for you to find the source for the control of interest. For example, for the wizard control;

If you're building WPF applications or re-usable controls for them, this toolkit is one you should check out...

Speaker : Christophe De Baene, RealDolmen

David Kelley, User Experience Architect for Wirestone, explains how you can get AdCenter working to monetize your Phone application. He talks with ISV Architect Evangelist Bruce Kyle about the performance considerations he did to move his application from WPF to Windows Phone.

Getting Started with AdCenter

See the AdCenter site Monetize your Windows Phone 7 Apps.

1. Download the Ad Control SDK. Download the Ad SDK to build your app and run ads in test mode that requires no on-boarding to try out the Microsoft Advertising SDK for Windows Phone 7.
2. Register your mobile apps on pubCenter. Sign up and register your Windows Phone apps using Microsoft Advertising's solution for Windows Phone 7 publishers and app developers. 
3. Implement and Submit. Set the Application Id and Ad Unit Id properties in the Ad Control and submit your ad-enabled app to the Windows Phone marketplace.

See the three training videos are now available to help you build, register, and manage your Windows Phone 7 ad-enabled apps:

• Create an Ad in a Windows Phone 7 XNA Game App.
• Register Ad-Enabled Windows Phone 7 Apps.
• Measure Ad Performance of Windows Phone 7 Apps.

Getting Started on Windows Phone 7

Tools are free. Get started on Windows Phone 7 or XBox at App Hub.

See the free developer training course for Windows Phone 7 on Channel 9.

Windows Phone 7 in 7 tutorials on MSDEV. Each short, to-the-point video in this series highlights a feature of Windows Phone 7 in less than 7 minutes. This is a series for developers who want to pick up the basics quickly with brief explanations and hands-on examples. Demonstrations and code samples are based on the beta release of the Windows Phone 7 Developer.

Free Help for Windows Phone 7

Join Microsoft Platform Ready for free assistance in developing and marketing your Windows Phone 7 applications.

Other ISV Videos

For videos on developing for Windows Phone 7, see:

• State and Local Governments for Windows Phone 7 applications.
• Talking  with Karmak.com about business applications on Windows Phone 7
• BinaryDads talk about building applications for Windows Phone 7
• Game Design and Development in Silverlight for Windows Phone 7
• Credit Card Processing for Windows Phone 7 on Windows Azure
• XNA Game Development on Windows Phone 7 in One Sitting

For other videos about independent software vendors (ISVs):

• Quark Extends SharePoint for Dynamic Publishing
• ProModel Adds Simulation, Visualization to Microsoft Project
• Thumb-Driven Workflow on Windows 7 Slates from Blue Dot Solutions
• Accumulus Makes Subscription Billing Easy for Windows Azure
• Azure Email-Enables Lists, Low-Cost Storage for SharePoint
• Crowd-Sourcing Public Sector App for Windows Phone, Azure
• Food Buster Game Achieves Scalability with Windows Azure

Up to Date News for ISVs and Software Developers

See US ISV Community blog.

Ever wonder what THAT means?! MVVM is a popular design pattern for coding the presentation layer in both Silverlight & Windows Presentation Foundation (WPF). It seems to be the rage that all the cool kids are doing these days, which means YOU should probably pay attention to it!  Get your development skills ready for MVVM (Model View View Model Design Pattern) and build some exciting applications for WPF, Silverlight or Windows Phone.

On March 21st, 2011 in New York City, Microsoft hosted a DevReady MVVM event along with DevExpress. This was a free full day training event.  We recorded the event, and now it is here on Channel 9 for your learning pleasure!

Pop on some headphones and listen & learn at your own pace!

The sessions were delivered by Microsoft MVP Miguel Castro, Microsoft Developer Evangelist Dani Diaz, and DevExpress Tech Evangelist Seth Juarez.

There are six sessions in total:

Demystifying XAML: Achieving Your Moment Of Clarity

Still find XAML intimidating? Well, throw in MVVM and certain design practices and it can get worse. But the truth is that it isn't as complicated as it appears to be. In this short session, I'll give you the basics of how XAML works. I'll cover its syntax, control usage, the visual tree, and the scariest animal of all, binding. I'll give you your moment of clarity, I promise.

Programming with MVVM - Part 1

WPF overwhelmed us with the most powerful data binding ever released in a development platform. It seemed there was so much to learn in order to use it, but it really isn't that complicated. However, to use it to its full potential, you really do need to introduce the Model-View-ViewModel pattern. The ideas and goal around this pattern seem to vary from person to person, but the one thing that they all share is that it provides a point of binding for XAML pages. In this session, I'll show you what MVVM is and how to use it and what to put in it, in order to get maximum potential out of XAML binding.

Programming with MVVM - Part 2

Now that you’ve sat through my MVVM session (what? you didn’t?), I want to take it to the next level by continuing your studies into this awesome and necessary pattern. In this session, I’ll cover more advanced topics you use when programming using Model-View-ViewModel. These include Design-Time Data, ViewModel-First, Validation, and the ever-so-popular MVVM Frameworks. There I’ll actually show and give you a small MVVM framework with which to launch your development. Though I will do a brief review on the fundamentals and goals of MVVM and its usage, this session does expect a little bit of knowledge on the pattern from you.

How Using MVVM Saved My Butt and How Laziness Came Back To Bite Me

Get a preview of upcoming Visual Studio features that will allow you to share code between Microsoft .NET platforms. You’ll learn how to structure applications to maximize the amount of shared code, how to write an app that’s used across Silverlight Desktop and Windows Phone, and how to build business logic and helper functions that work across all 3 Screens plus the cloud.

Leveraging MVVM Inside PRISM

Understanding an architecture pattern like MVVM is great but many companies invest in third-party control suites to make their application development less painful. In this session, you'll learn how to leverage some great controls from DevExpress in conjunction with MVVM to build applications in both WPF and Silverlight.

Ever wonder what THAT means?! MVVM is a popular design pattern for coding the presentation layer in both Silverlight & Windows Presentation Foundation (WPF). It seems to be the rage that all the cool kids are doing these days, which means YOU should probably pay attention to it!  Get your development skills ready for MVVM (Model View View Model Design Pattern) and build some exciting applications for WPF, Silverlight or Windows Phone.

On March 21st, 2011 in New York City, Microsoft hosted a DevReady MVVM event along with DevExpress. This was a free full day training event.  We recorded the event, and now it is here on Channel 9 for your learning pleasure!

Pop on some headphones and listen & learn at your own pace!

The sessions were delivered by Microsoft MVP Miguel Castro, Microsoft Developer Evangelist Dani Diaz, and DevExpress Tech Evangelist Seth Juarez.

There are six sessions in total:

Demystifying XAML: Achieving Your Moment Of Clarity

Still find XAML intimidating? Well, throw in MVVM and certain design practices and it can get worse. But the truth is that it isn't as complicated as it appears to be. In this short session, I'll give you the basics of how XAML works. I'll cover its syntax, control usage, the visual tree, and the scariest animal of all, binding. I'll give you your moment of clarity, I promise.

Programming with MVVM - Part 1

WPF overwhelmed us with the most powerful data binding ever released in a development platform. It seemed there was so much to learn in order to use it, but it really isn't that complicated. However, to use it to its full potential, you really do need to introduce the Model-View-ViewModel pattern. The ideas and goal around this pattern seem to vary from person to person, but the one thing that they all share is that it provides a point of binding for XAML pages. In this session, I'll show you what MVVM is and how to use it and what to put in it, in order to get maximum potential out of XAML binding.

Programming with MVVM - Part 2

Now that you’ve sat through my MVVM session (what? you didn’t?), I want to take it to the next level by continuing your studies into this awesome and necessary pattern. In this session, I’ll cover more advanced topics you use when programming using Model-View-ViewModel. These include Design-Time Data, ViewModel-First, Validation, and the ever-so-popular MVVM Frameworks. There I’ll actually show and give you a small MVVM framework with which to launch your development. Though I will do a brief review on the fundamentals and goals of MVVM and its usage, this session does expect a little bit of knowledge on the pattern from you.

How Using MVVM Saved My Butt and How Laziness Came Back To Bite Me

Get a preview of upcoming Visual Studio features that will allow you to share code between Microsoft .NET platforms. You’ll learn how to structure applications to maximize the amount of shared code, how to write an app that’s used across Silverlight Desktop and Windows Phone, and how to build business logic and helper functions that work across all 3 Screens plus the cloud.

Leveraging MVVM Inside PRISM

Understanding an architecture pattern like MVVM is great but many companies invest in third-party control suites to make their application development less painful. In this session, you'll learn how to leverage some great controls from DevExpress in conjunction with MVVM to build applications in both WPF and Silverlight.

Ever wonder what THAT means?! MVVM is a popular design pattern for coding the presentation layer in both Silverlight & Windows Presentation Foundation (WPF). It seems to be the rage that all the cool kids are doing these days, which means YOU should probably pay attention to it!  Get your development skills ready for MVVM (Model View View Model Design Pattern) and build some exciting applications for WPF, Silverlight or Windows Phone.

On March 21st, 2011 in New York City, Microsoft hosted a DevReady MVVM event along with DevExpress. This was a free full day training event.  We recorded the event, and now it is here on Channel 9 for your learning pleasure!

Pop on some headphones and listen & learn at your own pace!

The sessions were delivered by Microsoft MVP Miguel Castro, Microsoft Developer Evangelist Dani Diaz, and DevExpress Tech Evangelist Seth Juarez.

There are six sessions in total:

Demystifying XAML: Achieving Your Moment Of Clarity

Still find XAML intimidating? Well, throw in MVVM and certain design practices and it can get worse. But the truth is that it isn't as complicated as it appears to be. In this short session, I'll give you the basics of how XAML works. I'll cover its syntax, control usage, the visual tree, and the scariest animal of all, binding. I'll give you your moment of clarity, I promise.

Programming with MVVM - Part 1

WPF overwhelmed us with the most powerful data binding ever released in a development platform. It seemed there was so much to learn in order to use it, but it really isn't that complicated. However, to use it to its full potential, you really do need to introduce the Model-View-ViewModel pattern. The ideas and goal around this pattern seem to vary from person to person, but the one thing that they all share is that it provides a point of binding for XAML pages. In this session, I'll show you what MVVM is and how to use it and what to put in it, in order to get maximum potential out of XAML binding.

Programming with MVVM - Part 2

Now that you’ve sat through my MVVM session (what? you didn’t?), I want to take it to the next level by continuing your studies into this awesome and necessary pattern. In this session, I’ll cover more advanced topics you use when programming using Model-View-ViewModel. These include Design-Time Data, ViewModel-First, Validation, and the ever-so-popular MVVM Frameworks. There I’ll actually show and give you a small MVVM framework with which to launch your development. Though I will do a brief review on the fundamentals and goals of MVVM and its usage, this session does expect a little bit of knowledge on the pattern from you.

How Using MVVM Saved My Butt and How Laziness Came Back To Bite Me

Get a preview of upcoming Visual Studio features that will allow you to share code between Microsoft .NET platforms. You’ll learn how to structure applications to maximize the amount of shared code, how to write an app that’s used across Silverlight Desktop and Windows Phone, and how to build business logic and helper functions that work across all 3 Screens plus the cloud.

Leveraging MVVM Inside PRISM

Understanding an architecture pattern like MVVM is great but many companies invest in third-party control suites to make their application development less painful. In this session, you'll learn how to leverage some great controls from DevExpress in conjunction with MVVM to build applications in both WPF and Silverlight.

Ever wonder what THAT means?! MVVM is a popular design pattern for coding the presentation layer in both Silverlight & Windows Presentation Foundation (WPF). It seems to be the rage that all the cool kids are doing these days, which means YOU should probably pay attention to it!  Get your development skills ready for MVVM (Model View View Model Design Pattern) and build some exciting applications for WPF, Silverlight or Windows Phone.

On March 21st, 2011 in New York City, Microsoft hosted a DevReady MVVM event along with DevExpress. This was a free full day training event.  We recorded the event, and now it is here on Channel 9 for your learning pleasure!

Pop on some headphones and listen & learn at your own pace!

The sessions were delivered by Microsoft MVP Miguel Castro, Microsoft Developer Evangelist Dani Diaz, and DevExpress Tech Evangelist Seth Juarez.

There are six sessions in total:

Demystifying XAML: Achieving Your Moment Of Clarity

Still find XAML intimidating? Well, throw in MVVM and certain design practices and it can get worse. But the truth is that it isn't as complicated as it appears to be. In this short session, I'll give you the basics of how XAML works. I'll cover its syntax, control usage, the visual tree, and the scariest animal of all, binding. I'll give you your moment of clarity, I promise.

Programming with MVVM - Part 1

WPF overwhelmed us with the most powerful data binding ever released in a development platform. It seemed there was so much to learn in order to use it, but it really isn't that complicated. However, to use it to its full potential, you really do need to introduce the Model-View-ViewModel pattern. The ideas and goal around this pattern seem to vary from person to person, but the one thing that they all share is that it provides a point of binding for XAML pages. In this session, I'll show you what MVVM is and how to use it and what to put in it, in order to get maximum potential out of XAML binding.

Programming with MVVM - Part 2

Now that you’ve sat through my MVVM session (what? you didn’t?), I want to take it to the next level by continuing your studies into this awesome and necessary pattern. In this session, I’ll cover more advanced topics you use when programming using Model-View-ViewModel. These include Design-Time Data, ViewModel-First, Validation, and the ever-so-popular MVVM Frameworks. There I’ll actually show and give you a small MVVM framework with which to launch your development. Though I will do a brief review on the fundamentals and goals of MVVM and its usage, this session does expect a little bit of knowledge on the pattern from you.

How Using MVVM Saved My Butt and How Laziness Came Back To Bite Me

Get a preview of upcoming Visual Studio features that will allow you to share code between Microsoft .NET platforms. You’ll learn how to structure applications to maximize the amount of shared code, how to write an app that’s used across Silverlight Desktop and Windows Phone, and how to build business logic and helper functions that work across all 3 Screens plus the cloud.

Leveraging MVVM Inside PRISM

Understanding an architecture pattern like MVVM is great but many companies invest in third-party control suites to make their application development less painful. In this session, you'll learn how to leverage some great controls from DevExpress in conjunction with MVVM to build applications in both WPF and Silverlight.

Ever wonder what THAT means?! MVVM is a popular design pattern for coding the presentation layer in both Silverlight & Windows Presentation Foundation (WPF). It seems to be the rage that all the cool kids are doing these days, which means YOU should probably pay attention to it!  Get your development skills ready for MVVM (Model View View Model Design Pattern) and build some exciting applications for WPF, Silverlight or Windows Phone.

On March 21st, 2011 in New York City, Microsoft hosted a DevReady MVVM event along with DevExpress. This was a free full day training event.  We recorded the event, and now it is here on Channel 9 for your learning pleasure!

Pop on some headphones and listen & learn at your own pace! 

The sessions were delivered by Microsoft MVP Miguel Castro, Microsoft Developer Evangelist Dani Diaz, and DevExpress Tech Evangelist Seth Juarez.

There are six sessions in total:

Demystifying XAML: Achieving Your Moment Of Clarity

Still find XAML intimidating? Well, throw in MVVM and certain design practices and it can get worse. But the truth is that it isn't as complicated as it appears to be. In this short session, I'll give you the basics of how XAML works. I'll cover its syntax, control usage, the visual tree, and the scariest animal of all, binding. I'll give you your moment of clarity, I promise.

Programming with MVVM - Part 1

WPF overwhelmed us with the most powerful data binding ever released in a development platform. It seemed there was so much to learn in order to use it, but it really isn't that complicated. However, to use it to its full potential, you really do need to introduce the Model-View-ViewModel pattern. The ideas and goal around this pattern seem to vary from person to person, but the one thing that they all share is that it provides a point of binding for XAML pages. In this session, I'll show you what MVVM is and how to use it and what to put in it, in order to get maximum potential out of XAML binding.

Programming with MVVM - Part 2

Now that you’ve sat through my MVVM session (what? you didn’t?), I want to take it to the next level by continuing your studies into this awesome and necessary pattern. In this session, I’ll cover more advanced topics you use when programming using Model-View-ViewModel. These include Design-Time Data, ViewModel-First, Validation, and the ever-so-popular MVVM Frameworks. There I’ll actually show and give you a small MVVM framework with which to launch your development. Though I will do a brief review on the fundamentals and goals of MVVM and its usage, this session does expect a little bit of knowledge on the pattern from you.

How Using MVVM Saved My Butt and How Laziness Came Back To Bite Me

Get a preview of upcoming Visual Studio features that will allow you to share code between Microsoft .NET platforms. You’ll learn how to structure applications to maximize the amount of shared code, how to write an app that’s used across Silverlight Desktop and Windows Phone, and how to build business logic and helper functions that work across all 3 Screens plus the cloud.

Leveraging MVVM Inside PRISM

Understanding an architecture pattern like MVVM is great but many companies invest in third-party control suites to make their application development less painful. In this session, you'll learn how to leverage some great controls from DevExpress in conjunction with MVVM to build applications in both WPF and Silverlight.

About Ward Bell

Ward Bell is Vice President of Technology at IdeaBlade where he is responsible for the product direction of the DevForce .NET application framework, a product targeting smart client development. That product extends the ADO.NET Entity Framework with n-tier, client caching, and Silverlight support. Ward misspent much of the last 30 years programming line-of-business applications for numerous companies including several of the Fortune 100. He dispenses unsolicited advice on his blog, Never In Doubt.

About Tim Huckaby

Tim Huckaby is the Founder of InterKnowlogy, experts in Microsoft .NET and Microsoft Platforms, and has 25+ years experience including serving on a Microsoft product team as a development lead on an architecture team. Tim is a Microsoft Regional Director, an MVP and serves on multiple Microsoft councils and boards. Currently, Tim is focused on RIA & Rich Client Technologies like WPF, VSTO, Surface, Silverlight, Windows 7 Touch, and Windows Phone 7. He has been called a "Pioneer of the Smart Client Revolution" by the press. Tim has been awarded multiple times for the highest-rated keynote for Microsoft and numerous other technology conferences around the world and is consistently rated in the top 10% of all speakers at these events. Tim has also done keynote demos for numerous Microsoft executives including Bill Gates and Steve Ballmer.

Ward Bell and Tim Huckaby recommend you check out

• Silverlight
• Microsoft Silverlight 4 Download
• Silverlight Team Blog
• Silverlight How Do I? Videos
• Silverlight Wish List Forum
• IdeaBlade

This is a short video presentation of the Flashcards.Show application, and a sneak-peak to behind the scenes, looking into the code and architecture of the application.

You are more than welcome to download the source code, which also includes documentation, as well as install the application.

Follow Windows Phone announcements on Twitter at WP7DEV

Follow Yochay on Twitter

Get started with free tools, free training, and free Jump Start video course

It has gone through quite a few changes and I have expanded it to easily display MJPEG streams on a variety of platforms.  The developer just references the assembly appropriate to their platform, adds a few lines of code, and away it goes.

Usage

For those that are just interested in the usage, it's as simple as this:

1. Reference one of the following assemblies appropriate for your project:
   • MjpegProcessorSL.dll - Silverlight (Out of Browser Only!)
   • MjpegProcessorWP7.dll - Windows Phone 7 (XNA or Silverlight, performance maxes out around 320x240 @ 15fps, so set your camera settings accordingly)
   • MjpegProcessorXna4.dll - XNA 4.0 (Windows)
   • MjpegProcessor.dll - WinForms and WPF
2. Create a new MjpegDecoder object.
3. Hook up the FrameReady event.
4. In the event handler, take the Bitmap/BitmapImage and assign it to your image display control:
5. In the case of XNA, use the GetMjpegFrame method in the Update method, which will return a Texture2D you can use in your Draw method.
6. Call the ParseStream method with the Uri of the MJPEG "endpoint".

That's it!  The source code and binaries above both include projects demonstrating how to use the library on each of these platforms.  As long as you set the appropriate reference, you can just copy and paste the code in the sample to get your project running (changing the Uri, of course).

public partial class MainWindow : Window
{
    MjpegDecoder _mjpeg;

    public MainWindow()
    {
        InitializeComponent();
        _mjpeg = new MjpegDecoder();
        _mjpeg.FrameReady += mjpeg_FrameReady;
    }

    private void Start_Click(object sender, RoutedEventArgs e)
    {
        _mjpeg.ParseStream(new Uri("http://192.168.2.200/img/video.mjpeg"));
    }

    private void mjpeg_FrameReady(object sender, FrameReadyEventArgs e)
    {
        image.Source = e.BitmapImage;
    }
}

If that doesn't fit your needs, you can also access the Bitmap/BitmapImage properties directly from the MjpegDecoder object, or the CurrentFrame property, which will contain the raw JPEG data prior to being decoded.

A Word About Network/IP Cameras

I have tested this against several different cameras.  Each device has its own quirks, but all of them seem to work with this library with one exception:  several cameras will respond differently when an Internet Explorer user agent header is sent with the HTTP request.  Instead of sending down an MJPEG stream, it will send a single JPEG image as Internet Explorer does not properly support MJPEG streams.  Unfortunately, this causes the Silverlight processor to not work properly as the header cannot be changed from the Internet Explorer default.  When this happens, only a single frame will be sent, and the decoding will fail.  The only fix I have found is to use a different camera that doesn't work in this way.

What is MJPEG?

Pretty simply, it's a video format where each frame of video is sent as a separate, compressed JPEG image.  A standard HTTP request is made to a specific URL, and a multipart response is sent.  Parsing this multipart stream into separate images as they are sent results in a series of JPEG images.  The viewer displays those JPEG images as quickly as they are sent and that creates the video.  It's not a well documented format, nor is it perfectly standardized, but it does work.  For more information, see the MJPEG article on Wikipedia.

How Do I Find the MJPEG URL of My Camera?

Excellent question.  Not an excellent answer.  The user manual may mention the URL.  A quick internet search with the model number should get you a result.  Or, you can also try this company's lookup tool.

How Does It Work?

Glad you asked.  If you take a look at the project, you'll notice there isn't much code.  One single file is used with a variety of compiler directives to compile certain portions based on the platform assembly being generated.  The MjpegDecoder.cs/.vb contains the entire implementation.

First, an asynchronous request is made to the provided MJPEG URL inside the ParseStream method.  If we are in a Silverlight environment, the AllowReadStreamBuffering property must be set to false so the response returns immediately instead of being buffered.  Additionally, we need to register the http:// prefix to use the client http stack vs. the browser stack.  Finally, the request is made using the BeginGetResponse method, specifying the OnGetResponse method as the callback.  This will be called as soon as data is sent from the camera in response to our request.

public void ParseStream(Uri uri)
{
#if SILVERLIGHT
    HttpWebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);
#endif
    HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);

#if SILVERLIGHT
    // start the stream immediately
    request.AllowReadStreamBuffering = false;
#endif
    // asynchronously get a response
    request.BeginGetResponse(OnGetResponse, request);
}

OnGetResponse grabs the response headers and uses the Content-Type header to determine the boundary marker that will be sent between JPEG frames.

private void OnGetResponse(IAsyncResult asyncResult)
{
    HttpWebResponse resp;
    byte[] buff;
    byte[] imageBuffer = new byte[1024 * 1024];
    Stream s;

    // get the response
    HttpWebRequest req = (HttpWebRequest)asyncResult.AsyncState;
    resp = (HttpWebResponse)req.EndGetResponse(asyncResult);

    // find our magic boundary value
    string contentType = resp.Headers["Content-Type"];
    if(!string.IsNullOrEmpty(contentType) && !contentType.Contains("="))
        throw new Exception("Invalid content-type header.  The camera is likely not returning a proper MJPEG stream.");
    string boundary = resp.Headers["Content-Type"].Split('=')[1];
    byte[] boundaryBytes = Encoding.UTF8.GetBytes(boundary.StartsWith("--") ? boundary : "--" + boundary);
...

It then streams the response data, looks for a JPEG header marker, then reads until it finds the boundary marker, copies the data into a buffer, decodes it, passes it on to whoever wants it via an event, and then starts over.

...
    s = resp.GetResponseStream();
    BinaryReader br = new BinaryReader(s);

    _streamActive = true;

    buff = br.ReadBytes(ChunkSize);

    while (_streamActive)
    {
        int size;

        // find the JPEG header
        int imageStart = buff.Find(JpegHeader);

        if(imageStart != -1)
        {
            // copy the start of the JPEG image to the imageBuffer
            size = buff.Length - imageStart;
            Array.Copy(buff, imageStart, imageBuffer, 0, size);

            while(true)
            {
                buff = br.ReadBytes(ChunkSize);

                // find the boundary text
                int imageEnd = buff.Find(boundaryBytes);
                if(imageEnd != -1)
                {
                    // copy the remainder of the JPEG to the imageBuffer
                    Array.Copy(buff, 0, imageBuffer, size, imageEnd);
                    size += imageEnd;

                    // create a single JPEG frame
                    CurrentFrame = new byte[size];
                    Array.Copy(imageBuffer, 0, CurrentFrame, 0, size);
#if !XNA
                    ProcessFrame(CurrentFrame);
#endif
                    // copy the leftover data to the start
                    Array.Copy(buff, imageEnd, buff, 0, buff.Length - imageEnd);

                    // fill the remainder of the buffer with new data and start over
                    byte[] temp = br.ReadBytes(imageEnd);

                    Array.Copy(temp, 0, buff, buff.Length - imageEnd, temp.Length);
                    break;
                }

                // copy all of the data to the imageBuffer
                Array.Copy(buff, 0, imageBuffer, size, buff.Length);
                size += buff.Length;
            }
        }
    }
    resp.Close();
}

The ProcessFrame method seen above takes the raw byte buffer, which contains an undecoded JPEG image, and decodes it based on the environment.  However this isn't called in the case of XNA which we'll see in a moment:

private void ProcessFrame(byte[] frameBuffer)
{
#if SILVERLIGHT
    // need to get this back on the UI thread
    Deployment.Current.Dispatcher.BeginInvoke((Action)(() =>
    {
        // resets the BitmapImage to the new frame
        BitmapImage.SetSource(new MemoryStream(frameBuffer, 0, frameBuffer.Length));

        // tell whoever's listening that we have a frame to draw
        if(FrameReady != null)
            FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, BitmapImage = BitmapImage });
    }));
#endif

#if !SILVERLIGHT && !XNA
    // I assume if there's an Application.Current then we're in WPF, not WinForms
    if(Application.Current != null)
    {
        // get it on the UI thread
        Application.Current.Dispatcher.BeginInvoke((Action)(() =>
        {
            // create a new BitmapImage from the JPEG bytes
            BitmapImage = new BitmapImage();
            BitmapImage.BeginInit();
            BitmapImage.StreamSource = new MemoryStream(frameBuffer);
            BitmapImage.EndInit();

            // tell whoever's listening that we have a frame to draw
            if(FrameReady != null)
                FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, Bitmap = Bitmap, BitmapImage = BitmapImage });
        }));
    }
    else
    {
        // create a simple GDI+ happy Bitmap
        Bitmap = new Bitmap(new MemoryStream(frameBuffer));

        // tell whoever's listening that we have a frame to draw
        if(FrameReady != null)
            FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, Bitmap = Bitmap, BitmapImage = BitmapImage });
    }
#endif
}

In the case of Silverlight, the BitmapImage object has a SetSource method which takes a stream to be decoded and turned into the image.  In WPF, BitmapImage works differently.  In this case, BeginInit is called, then the StreamSource property is set to the stream of bytes, and finally EndInit is called.  In WinForms, the library will return a Bitmap object which can be initialized with the stream right in the constructor.

In the code above, I look at the Application.Current property to determine if the library is being used by a WPF project.  If that property is not null, it is assumed the library is being called from a WPF project.

When compiled as an XNA library, we have no use for a BitmapImage or a Bitmap…we need a Texture2D object.  The GetMjpegFrame method seen below is what is called by an XNA application during the Update method to pull the current frame:

public Texture2D GetMjpegFrame(GraphicsDevice graphicsDevice)
{
    // create a Texture2D from the current byte buffer
    if(CurrentFrame != null)
        return Texture2D.FromStream(graphicsDevice, new MemoryStream(CurrentFrame, 0, CurrentFrame.Length));
    return null;
}

Conclusion

And that's about it for the library.  Give it a try and please contact me with feedback or if you run into any issues.  Enjoy!

For a deeper written explanation of this approach, and to download the source code for the project seen in the video, check out my blog post on Slickthought.net.

So what takes care of a bad-pitch day? Autotune—an effect that corrects the pitch of your voice so you'll never again sing out of tune. And now, with the power of modern microprocessors, autotune is possible in real-time, allowing singers to benefit from its almost magical powers during live concerts.

The company most famous for its autotune effect is Antares. Antares Auto-Tune currently retails for $249, and a stripped down version is available for $100. In addition to simply improving the pitch of a dodgy singer, autotune can be used to create unique robotic sounding vocal effects, a technique massively popular in recent years thanks to its use by artists such as T-Pain and the group behind the “Auto-Tune the News” YouTube videos. In 1998, when the effect was first used on Cher's “Believe” single, the producer used such extreme settings that instead of subtly adjusting the pitch, autotune “snapped” instantaneously to the nearest “correct” note.

Here is a nerdy example of what Autotune can do.

How does Autotune work?

An autotune effect has two parts. The first is pitch detection, which calculates the dominant frequency of the incoming signal, and is the reason autotune is normally used on monophonic audio sources (i.e. playing one note at a time, not whole chords). So, if your guitar is out of tune, you're out of luck (Celemony's Melodyne product, however, features some incredible capabilities for pitch-shifting polyphonic audio).

The second stage is pitch shifting, or “correcting” a given note. However, the bigger the pitch shift required, the more artificial the end result will be, and it is worth noting that absolutely perfect pitch is not always desirable. Sometimes the blended notes resulting from vibrato, for example, are an important part of the performance, and eliminating them would be detrimental.

Creating a .NET Autotune Algorithm

For this project, we will be creating an autotune effect for .NET. Serious recording enthusiasts should just go out and buy a decent autotune effect, but to have a little fun, we'll see if we can make an autotune that can give us a poor man's Cher effect (or T-Pain effect, if you prefer).

To get started, I searched to see if there were some pre-existing open source autotune implementations, which brought me to awesomebox, a project created by Ravi Parikh and Keegan Poppen while they were students at Stanford University. They kindly gave me permission to make use of their code, which uses an auto-correlator for pitch detection and an open source pitch-shifting algorithm from audio DSP expert Stephan M. Bernsee.

Porting C++ to C#

Although porting from C/C++ to C# is not exactly fun, there is enough similarity in the syntax that it is possible to complete without too many changes. You need to remember, however, that a long in C is an int in C# (i.e. 32 bits long not 64).

Additionally, the C# compiler is fussier than C/C++ when it comes to casting between floats, doubles, and ints. Putting the “f” suffix on numeric literals sorts out most of these compiler errors.

Pointers can be a pain. I tend to replace them with integer variables used to index into an array. You can of course use unsafe code, but that limits your options if you plan to port to Silverlight or Windows Phone 7 at a later date, neither of which allow unsafe code or interop into unmanaged code. The necessary mathematical functions are available in the System.Math class.

To see an example, compare this pitch shifting C++ source file with my C# conversion of it.

Capturing Audio with NAudio

Interop wrappers for the NAudio Windows WaveIn APIs capture the audio. Here is the code used to start recording:

c#:

waveIn = new WaveIn();
waveIn.DeviceNumber = recordingDevice;
waveIn.DataAvailable += waveIn_DataAvailable;
waveIn.RecordingStopped += new EventHandler(waveIn_RecordingStopped);
waveIn.WaveFormat = recordingFormat;
waveIn.StartRecording();

VB.Net:

waveIn = New WaveIn
waveIn.DeviceNumber = recordingDevice
AddHandler waveIn.DataAvailable, AddressOf waveIn_DataAvailable
AddHandler waveIn.RecordingStopped, AddressOf waveIn_RecordingStopped
waveIn.WaveFormat = _recordingFormat
waveIn.StartRecording()

The steps are:

1. Create a new WaveIn device
2. [Optional] set up the device number (0 for default recording device)
3. Add a handler to the DataAvailable event—this is where we will receive the raw audio data
4. Add a handler for the RecordingStopped event. This allows us to close the temporary WAV file we created
5. Set up the recording format. For this project we are going to record in mono (i.e. one channel), 16 bit, 44.1kHz audio—the default setting for most microphones
6. Call the StartRecording method

Whenever the soundcard reports a new buffer of recorded audio, we receive it in the DataAvailable event handler:

c#:

void waveIn_DataAvailable(object sender, WaveInEventArgs e)
{
    byte[] buffer = e.Buffer;
    int bytesRecorded = e.BytesRecorded;
    WriteToFile(buffer, bytesRecorded);

    for (int index = 0; index < e.BytesRecorded; index += 2)
    {
        short sample = (short)((buffer[index + 1] << 8) |
                                buffer[index + 0]);
        float sample32 = sample / 32768f;
        sampleAggregator.Add(sample32);
    }
}

VB.Net

Private Sub waveIn_DataAvailable(ByVal sender As Object, ByVal e As WaveInEventArgs)
    Dim buffer() = e.Buffer
    Dim bytesRecorded = e.BytesRecorded
    WriteToFile(buffer, bytesRecorded)

    For index = 0 To e.BytesRecorded - 1 Step 2
        Dim sample = CShort(buffer(index + 1)) << 8 Or CShort(buffer(index + 0))
        Dim sample32 = sample / 32768.0F
        _sampleAggregator.Add(sample32)
    Next index
End Sub

The WaveInEventArgs contains the number of bytes recorded (e.BytesRecorded) and a pointer to the buffer containing those bytes (e.Buffer). The handler does two things with the recorded data. First, it calls WriteToFile, which uses the WaveFileWriter class from NAudio to write the data to disk:

c#:

// before we start recording, set up a WaveFileWriter...
writer = new WaveFileWriter(waveFileName, recordingFormat);

// ... every block we receive we write it to the WaveFileWriter:
writer.WriteData(buffer, 0, bytesRecorded);

// ... and when recording stops we must call Dispose to finalize the
// .WAV file properly
writer.Dispose()

VB.Net:

writer = New WaveFileWriter(waveFileName, _recordingFormat)
writer.WriteData(buffer, 0, bytesRecorded)
writer.Dispose()

Converting Audio to Floating Point and Back Again

Once recording has completed, we have a WAV file on which to perform our autotune effect. However, our WAV file consists of 16 bit samples (i.e. System.Int16 aka short). In other words, we have a sequence of byte pairs, each of which represent a number in the range -32768 to 32767. For the digital signal processing we will be performing, it's best to have a sequence of floating point numbers (System.Single or float) in the range -1.0f to 1.0f. This is a common requirement, so NAudio provides a utility class to convert audio from short to float called Wave16ToFloatProvider. Here's the code that takes a WAV file and implements the autotune algorithm on it:

c#:

public static void ApplyAutoTune(string fileToProcess, string tempFile, AutoTuneSettings autotuneSettings)
{
    using (WaveFileReader reader = new WaveFileReader(fileToProcess))
    {
        IWaveProvider stream32 = new Wave16toFloatProvider(reader);
        IWaveProvider streamEffect = new AutoTuneWaveProvider(stream32, autotuneSettings);
        IWaveProvider stream16 = new WaveFloatTo16Provider(streamEffect);
        using (WaveFileWriter converted = new WaveFileWriter(tempFile, stream16.WaveFormat))
        {
            // buffer length needs to be a power of 2 for FFT to work nicely
            // however, make the buffer too long and pitches aren't detected fast enough
            // successful buffer sizes: 8192, 4096, 2048, 1024
            // (some pitch detection algorithms need at least 2048)
            byte[] buffer = new byte[8192]; 
            int bytesRead;
            do
            {
                bytesRead = stream16.Read(buffer, 0, buffer.Length);
                converted.WriteData(buffer, 0, bytesRead);
            } while (bytesRead != 0 && converted.Length < reader.Length);
        }
    }
}

VB.Net

Public Shared Sub ApplyAutoTune(ByVal fileToProcess As String,
    ByVal tempFile As String,
    ByVal autotuneSettings As AutoTuneSettings)
    Using reader As New WaveFileReader(fileToProcess)
        Dim stream32 As IWaveProvider = New Wave16ToFloatProvider(reader)
        Dim streamEffect As IWaveProvider = New AutoTuneWaveProvider(stream32, autotuneSettings)
        Dim stream16 As IWaveProvider = New WaveFloatTo16Provider(streamEffect)
        Using converted As New WaveFileWriter(tempFile, stream16.WaveFormat)
            ' buffer length needs to be a power of 2 for FFT to work nicely
            ' however, make the buffer too long and pitches aren't detected fast enough
            ' successful buffer sizes: 8192, 4096, 2048, 1024
            ' (some pitch detection algorithms need at least 2048)
            Dim buffer(8191) As Byte
            Dim bytesRead As Integer
            Do
                bytesRead = stream16.Read(buffer, 0, buffer.Length)
                converted.WriteData(buffer, 0, bytesRead)
            Loop While bytesRead <> 0 AndAlso converted.Length < reader.Length
        End Using
    End Using
End Sub

Here's how it works:

1. First we use a WaveFileReader to open the file that we've just created, which contains 16 bit audio samples
2. Then we use the Wave16ToFloatProvider to perform the conversion to floating point samples
3. Next we pass it through our autotune effect (the AutotuneWaveProvider). We'll explain how this works this later
4. Then we use the WaveFloatTo16Provider to convert back to 16 bit samples ready for saving to WAV (we could save a 32 bit WAV, but it would be rather wasteful of disk space)
5. Having set up the audio pipeline, we can read from the WaveFloatTo16Provider and pull audio right through from the WAV file. We need to read in block sizes that are a power of 2, since we are passing the data through FFTs. If we want to read arbitrary block sizes, we need to introduce another element into our pipeline to buffer up enough data to pass through FFTs
6. Finally, we use the WaveFileWriter to write the data we have read into a WAV file

The AutoTuneWaveProvider

As we saw in the last code snippet, the AutoTuneWaveProvider is the piece in our audio pipeline that actually performs the autotune effect. It implements the NAudio IWaveProvider interface, which allows it to be used in the pipeline for real-time playback if necessary, even though our example code is not doing this (see the section on performance later). Here's the AutoTuneWaveProvider constructor:

c#:

public AutoTuneWaveProvider(IWaveProvider source, AutoTuneSettings autoTuneSettings)
{
    this.autoTuneSettings = autoTuneSettings;
    if (source.WaveFormat.SampleRate != 44100)
        throw new ArgumentException("AutoTune only works at 44.1kHz");
    if (source.WaveFormat.Encoding != WaveFormatEncoding.IeeeFloat)
        throw new ArgumentException("AutoTune only works on IEEE floating point audio data");
    if (source.WaveFormat.Channels != 1)
        throw new ArgumentException("AutoTune only works on mono input sources");

    this.source = source;
    this.pitchDetector = new AutoCorrelator(source.WaveFormat.SampleRate);
    this.pitchShifter = new SmbPitchShifter(Settings);
    this.waveBuffer = new WaveBuffer(8192);
}

VB.Net

Public Sub New(ByVal source As IWaveProvider, ByVal autoTuneSettings As AutoTuneSettings)
    Me.autoTuneSettings = autoTuneSettings
    If source.WaveFormat.SampleRate <> 44100 Then
        Throw New ArgumentException("AutoTune only works at 44.1kHz")
    End If
    If source.WaveFormat.Encoding <> WaveFormatEncoding.IeeeFloat Then
        Throw New ArgumentException("AutoTune only works on IEEE floating point audio data")
    End If
    If source.WaveFormat.Channels <> 1 Then
        Throw New ArgumentException("AutoTune only works on mono input sources")
    End If

    Me.source = source
    Me.pitchDetector = New AutoCorrelator(source.WaveFormat.SampleRate)
    ' alternative pitch detector:
    ' Me.pitchDetector = New FftPitchDetector(source.WaveFormat.SampleRate)
    Me.pitchShifter = New SmbPitchShifter(Settings, source.WaveFormat.SampleRate)
    Me.waveBuffer = New WaveBuffer(8192)
End Sub

Some points to notice:

1. We pass in a source IWaveProvider—this is where the data will be coming from
2. We check that the source is in the right format—floating point mono input.
3. We also pass in an AutoTuneSettings object. This not only encapsulates the settings for autotune, it is important if you want to adjust the settings in real-time while the effect is running
4. We then create the two key components of our autotune effect: a pitch detector (which uses an autocorrelator), and a pitch shifter
5. Finally we create a buffer to use for audio processing. This can be a byte[] array, but we use WaveBuffer from NAudio because it uses a clever trick that allows us to cast a byte[] into a float[] without using unsafe code or having to copy all of the data

The key method on any implementation of IWaveProvider is its Read method. This is where the audio consumer, usually the sound card or a WaveFileWriter, asks for data. The data must be supplied as a byte array, and if at all possible you should return exactly the number of bytes you were asked for (if you can't, an extra layer of buffering is usually required, or audio playback will be choppy). Here's our implementation of the Read method:

c#:

public int Read(byte[] buffer, int offset, int count)
{
    if (waveBuffer == null || waveBuffer.MaxSize < count)
    {
        waveBuffer = new WaveBuffer(count);
    }

    int bytesRead = source.Read(waveBuffer, 0, count);

    // the last bit sometimes needs to be rounded up:
    if (bytesRead > 0) bytesRead = count;

    int frames = bytesRead / sizeof(float); 
    float pitch = pitchDetector.DetectPitch(waveBuffer.FloatBuffer, frames);
        
    // an attempt to make it less "warbly" by holding onto the pitch 
    // for at least one more buffer
    if (pitch == 0f && release < maxHold)
    {
        pitch = previousPitch;
        release++;
    }
    else
    {
        this.previousPitch = pitch;
        release = 0;
    }
    
    WaveBuffer outBuffer = new WaveBuffer(buffer);

    pitchShifter.ShiftPitch(waveBuffer.FloatBuffer, pitch, 0.0f, outBuffer.FloatBuffer, frames);

    return frames * 4;
}

VB.Net:

Public Function Read(ByVal buffer() As Byte, ByVal offset As Integer,
                        ByVal count As Integer) As Integer Implements NAudio.Wave.IWaveProvider.Read
    If waveBuffer Is Nothing OrElse waveBuffer.MaxSize < count Then
        waveBuffer = New WaveBuffer(count)
    End If

    Dim bytesRead = source.Read(waveBuffer, 0, count)
    'Debug.Assert(bytesRead = count)

    ' the last bit sometimes needs to be rounded up:
    If bytesRead > 0 Then
        bytesRead = count
    End If

    'pitchsource->getPitches()
    Dim frames = bytesRead \ Len(New Single) ' MRH: was count
    Dim pitch = pitchDetector.DetectPitch(waveBuffer.FloatBuffer, frames)

    ' MRH: an attempt to make it less "warbly" by holding onto the pitch for at least one more buffer
    If pitch = 0.0F AndAlso release < maxHold Then
        pitch = previousPitch
        release += 1
    Else
        Me.previousPitch = pitch
        release = 0
    End If

    Dim midiNoteNumber = 40
    Dim targetPitch = CSng(8.175 * Math.Pow(1.05946309, midiNoteNumber))

    Dim outBuffer As New WaveBuffer(buffer)

    pitchShifter.ShiftPitch(waveBuffer.FloatBuffer, pitch, targetPitch, outBuffer.FloatBuffer, frames)

    Return frames * 4
End Function

Here's what's going on

1. First we need to read from our source (in our case, a WAV file converted to floating point samples)
2. If we get less data than we were expecting, we know that means we're at the end of the file, so we'll just pretend we got a full buffer
3. We then work out how many audio ‘frames' are present (which is the same as the number of samples since this is mono audio). It's floating point audio, so frames equal bytes divided by four
4. We then pass the data through our pitch detector algorithm (see below)
5. Next we use some experimental code to stabilize pitch detection by reporting the previous frequency when no pitch is picked up
6. Finally we pass the data into our pitch shifter, including details of the detected pitch

Now that we've seen the big picture of the AutotuneWaveProvider, let's drill down into its two main components—the pitch detector and pitch shifter.

Pitch Detection with Autocorrelation

The pitch detection part of autotune is vital to getting good results. If it can't accurately detect the input pitch, it will incorrectly calculate how much the pitch needs to be adjusted. However, high quality pitch detection is quite difficult to get right. First of all, the microphone may well pick up background noise. Second, when you sing a into a microphone, the signal consists not only of a single frequency, but also “harmonics” at different frequencies.

The good news is that we need to detect only the primary pitch.

The awesomebox algorithm makes use of “autocorrelation” for its pitch detection, but I made a few small tweaks to how the algorithm is implemented in an attempt to improve its accuracy. Autocorrelation has the advantage of being a relatively quick process. The basic principle is that if a signal is periodic, it will “correlate” well with itself when shifted forward (or backwards) one cycle.

Let's say we are looking to see if the note “Middle C” is being sung. The frequency of Middle C is around 262Hz. If we are sampling at 44.1kHz (which is standard for CD quality audio), then we will expect the signal to repeat at approximately every 168 samples (44100/262). Accordingly, for every sample in the buffer, we calculate the sum of squares of that sample and the sample 168 samples previous. We do this for every possible offset that measures a frequency in the range we want to detect (I am using 85Hz to 300Hz, which is adequate for pitch detecting vocals). The offset with the highest score is the most likely frequency.

Let's have a look at the code for an autocorrelation algorithm, starting with the constructor for the AutoCorrelator class:

c#:

public AutoCorrelator(int sampleRate)
{
    this.sampleRate = (float)sampleRate;
    int minFreq = 85;
    int maxFreq = 255;

    this.maxOffset = sampleRate / minFreq;
    this.minOffset = sampleRate / maxFreq;
}

VB.Net

Public Sub New(ByVal sampleRate As Integer)
    Me.sampleRate = CSng(sampleRate)
    Dim minFreq = 85
    Dim maxFreq = 255

    Me.maxOffset = sampleRate \ minFreq
    Me.minOffset = sampleRate \ maxFreq
End Sub

First of all, we pre-calculate some values based on the minimum and maximum frequencies we are looking for. Remember that lower frequencies are harder to detect than higher frequencies, so don't set minFreq too low. MaxOffset and MinOffset are the maximum and minimum backwards distances we will be seeking while looking for a match.

c#:

public float DetectPitch(float[] buffer, int frames)
{
    if (prevBuffer == null)
    {
        prevBuffer = new float[frames];
    }

    float maxCorr = 0;
    int maxLag = 0;

    // starting with low frequencies, working to higher
    for (int lag = maxOffset; lag >= minOffset; lag--)
    {
        float corr = 0; //  sum of squares
        for (int i = 0; i < frames; i++)
        {
            int oldIndex = i - lag;
            float sample = ((oldIndex < 0) ? prevBuffer[frames + 
            corr += (sample * buffer[i]);
        }
        if (corr > maxCorr)
        {
            maxCorr = corr;
            maxLag = lag;
        }

    }
    for (int n = 0; n < frames; n++)
    { 
        prevBuffer[n] = buffer[n]; 
    }
    float noiseThreshold = frames / 1000f;

    if (maxCorr < noiseThreshold || maxLag == 0) return 0.0f;
    return this.sampleRate / maxLag;
}

VB.Net

Public Function DetectPitch(ByVal buffer() As Single, ByVal frames As Integer) As Single Implements IPitchDetector.DetectPitch
    If prevBuffer Is Nothing Then
        prevBuffer = New Single(frames - 1){}
    End If
    Dim secCor As Single = 0
    Dim secLag = 0

    Dim maxCorr As Single = 0
    Dim maxLag = 0

    ' starting with low frequencies, working to higher
    For lag = maxOffset To minOffset Step -1
        Dim corr As Single = 0 ' this is calculated as the sum of squares
        For i = 0 To frames - 1
            Dim oldIndex = i - lag
            Dim sample = (If(oldIndex < 0, prevBuffer(frames + oldIndex), buffer(oldIndex)))
            corr += (sample * buffer(i))
        Next i
        If corr > maxCorr Then
            maxCorr = corr
            maxLag = lag
        End If
        If corr >= 0.9 * maxCorr Then
            secCor = corr
            secLag = lag
        End If
    Next lag
    For n = 0 To frames - 1
        prevBuffer(n) = buffer(n)
    Next n
    Dim noiseThreshold = frames / 1000.0F
    'Debug.WriteLine(String.Format("Max Corr: {0} ({1}), Sec Corr: {2} ({3})", Me.sampleRate / maxLag, maxCorr, Me.sampleRate / secLag, secCor))
    If maxCorr < noiseThreshold OrElse maxLag = 0 Then
        Return 0.0F
    End If
    'Return 44100.0f / secLag '--works better for singing
    Return Me.sampleRate / maxLag
End Function

A few things to notice:

1. Notice that the audio comes in as an array of floating point numbers. NAudio performs this conversion from 16 bit audio for us by using Wave16ToFloatProvider
2. We store the previous buffer. This allows us to look backwards for correlation
3. We then work through each and every possible integer offset within our range, and calculate a correlation value
4. The correlation is calculated as the sum of squares
5. If it is the largest so far, we store the “lag” value (i.e. number of samples back that we correlated with)
6. Notice that we return 0 (i.e. no frequency detected) if we don't find a strong frequency. This noise threshold may need to be tweaked depending on your input
7. Finally, we convert into a frequency with the formula sampleRate / maxLag

I wrote some unit tests to measure the accuracy of detection with sine waves (which admittedly are the easiest to detect). Here are the results for audio sampled at 44.1kHz:

Test Frequency	Detected Pitch
109.99Hz	108.35Hz
116.53Hz	118.23Hz
123.46Hz	123.18Hz
130.80Hz	129.71Hz
138.58Hz	140.00Hz
146.82Hz	148.48Hz
155.55Hz	154.74Hz
164.80Hz	163.33Hz
174.60Hz	172.94Hz
184.98Hz	183.75Hz
195.98Hz	194.27Hz
207.63Hz	206.07Hz
219.98Hz	219.40Hz
233.06Hz	234.57Hz
246.92Hz	247.75Hz
261.60Hz	256.40Hz
277.16Hz	139.56Hz
293.64Hz	146.03Hz

Notice that the detected frequencies from the final two tests are actually half the correct amount. This doesn't actually matter for our purposes, since this just means the frequency has been detected as one octave below the correct note.

To improve on the accuracy of the autocorrelator's results, there are a couple of things you can do:

• Run a band-pass filter before-hand, to remove any frequencies outside of the desired range
• Combine the results with those obtained from a different technique, such as counting zero-crossings

Pitch Detection with the Fast Fourier Transform

I decided to implement an alternative pitch detection algorithm to see if I could get better results. A different approach is to use the Fast Fourier Transform, which converts signals from the “time domain” into the “frequency domain.”

The basic approach is to take a block of samples (which must be a power of 2 – e.g. 1024), and run the FFT on them. The FFT takes complex numbers as inputs, which for audio signals are entirely real. The implementation I am using expects real and complex parts interleaved for the input buffer. Here's our code setting up fftBuffer with interleaved samples:

c#:

private float[] fftBuffer;
private float[] prevBuffer;

public float DetectPitch(float[] buffer, int inFrames)
{
    Func<int, int, float> window = HammingWindow;
    if (prevBuffer == null)
    {
        prevBuffer = new float[inFrames];
    }
 
    // double frames since we are combining present and previous buffers
    int frames = inFrames * 2;
    if (fftBuffer == null)
    {
        fftBuffer = new float[frames * 2]; // times 2 because it is complex input
    }
 
    for (int n = 0; n < frames; n++)
    {
        if (n < inFrames)
        {
            fftBuffer[n * 2] = prevBuffer[n] * window(n, frames);
            fftBuffer[n * 2 + 1] = 0; // need to clear out as fft modifies buffer
        }
        else
        {
            fftBuffer[n * 2] = buffer[n-inFrames] * window(n, frames);
            fftBuffer[n * 2 + 1] = 0; // need to clear out as fft modifies buffer
        }
    }

VB.Net

Private fftBuffer() As Single
Private prevBuffer() As Single

Public Function DetectPitch(ByVal buffer() As Single,
                            ByVal inFrames As Integer) As Single Implements IPitchDetector.DetectPitch
    Dim window As Func(Of Integer, Integer, Single) = AddressOf HammingWindow
    If prevBuffer Is Nothing Then
        prevBuffer = New Single(inFrames - 1) {}
    End If

    ' double frames since we are combining present and previous buffers
    Dim frames = inFrames * 2
    If fftBuffer Is Nothing Then
        fftBuffer = New Single(frames * 2 - 1) {} ' times 2 because it is complex input
    End If

    For n = 0 To frames - 1
        If n < inFrames Then
            fftBuffer(n * 2) = prevBuffer(n) * window(n, frames)
            fftBuffer(n * 2 + 1) = 0 ' need to clear out as fft modifies buffer
        Else
            fftBuffer(n * 2) = buffer(n - inFrames) * window(n, frames)
            fftBuffer(n * 2 + 1) = 0 ' need to clear out as fft modifies buffer
        End If
    Next n

Notice that we prepend the previous buffer we were passed. This is a common way of increasing the accuracy and resolution of an FFT by using overlapping windows, and can be further extended to store three previous buffers, allowing us to have 75% overlapping windows instead of just the 50% that we have in this example.

For better peak frequency detection, the signal that is passed into the FFT is best pre-processed with a “windowing” function. There are several to choose from, each with its own strengths and weaknesses. I used the Hamming window, which is a fairly common choice:

c#:

private float HammingWindow(int n, int N) 
{
    return 0.54f - 0.46f * (float)Math.Cos((2 * Math.PI * n) / (N - 1));
}

VB.Net

Private Function HammingWindow(ByVal n As Integer, ByVal _N As Integer) As Single
    Return 0.54F - 0.46F * CSng(Math.Cos((2 * Math.PI * n) / (_N - 1)))
End Function

The next step is to pass on our interleaved buffer to the FFT algorithm. I am using Stephan Bernsee's here, though there is an alternative implementation in NAudio that I could have used. Since the same function can be used for an inverse FFT, the -1 parameter means (rather counter-intuitively), do a forwards FFT. It processes the data in place, which is fine since we don't need to keep the contents of the input buffer:

c#:

// assuming frames is a power of 2
SmbPitchShift.smbFft(fftBuffer, frames, -1);

VB.Net

' assuming frames is a power of 2
SmbPitchShift.smbFft(fftBuffer, frames, -1)

Once we have completed the FFT, we are ready to interpret its output. The output of the FFT consists of complex numbers (again real followed by imaginary in our buffer), which represent frequency “bins.”

We start off by calculating the bin size and working out which bins correspond to the range of frequencies we are interested in detecting:

c#:

float binSize = sampleRate / frames;
int minBin = (int)(85 / binSize);
int maxBin = (int)(300 / binSize);

VB.Net

Dim binSize = sampleRate / frames
Dim minBin = CInt(Fix(85 / binSize))
Dim maxBin = CInt(Fix(300 / binSize))

For example, if our sample rate is 44.1kHz and we analyse a block of 1024 samples, then each bin represents 43Hz, which is hardly the granularity we are looking for. To increase resolution, our options are to either sample at a higher rate or analyse a bigger chunk. Our approach is to use overlapping blocks of 8192 samples, as we read 4096 samples each time. This means we have a resolution of around 5Hz, which is much more acceptable.

Now we can calculate the magnitude or “intensity” for each frequency by calculating the sum of squares (strictly we should then take the square root, but we don't need to since we are just looking for the largest value):

c#:

float maxIntensity = 0f;
int maxBinIndex = 0;

for (int bin = minBin; bin <= maxBin; bin++)
{
    float real = fftBuffer[bin * 2];
    float imaginary = fftBuffer[bin * 2 + 1];
    float intensity = real * real + imaginary * imaginary;
    if (intensity > maxIntensity)
    {
        maxIntensity = intensity;
        maxBinIndex = bin;
    }
}

VB.Net

Dim maxIntensity = 0.0F
Dim maxBinIndex = 0
For bin = minBin To maxBin
    Dim real = fftBuffer(bin * 2)
    Dim imaginary = fftBuffer(bin * 2 + 1)
    Dim intensity = real * real + imaginary * imaginary
    If intensity > maxIntensity Then
        maxIntensity = intensity
        maxBinIndex = bin
    End If
Next bin

Since we have identified the bin with the maximum intensity, we can calculate the detected frequency:

c#:

return binSize * maxBinIndex;

VB.Net

Return binSize * maxBinIndex

I don't currently specify a minimum threshold for maxIntensity, but perhaps if it were very low, the FFT pitch detector would return zero to indicate no pitch detected instead of returning an answer that is probably not accurate.

Let's have a look at how the FFT pitch detector does:

Test Frequency	Detected Pitch
109.99Hz	107.67Hz
116.53Hz	118.43Hz
123.46Hz	123.82Hz
130.80Hz	129.20Hz
138.58Hz	139.97Hz
146.82Hz	145.35Hz
155.55Hz	156.12Hz
164.80Hz	166.88Hz
174.60Hz	172.27Hz
184.98Hz	183.03Hz
195.98Hz	193.80Hz
207.63Hz	209.95Hz
219.98Hz	220.72Hz
233.06Hz	231.48Hz
246.92Hz	247.63Hz
261.60Hz	263.78Hz
277.16Hz	274.55Hz
293.64Hz	296.08Hz

As can be seen, it correctly picks out the primary frequencies of the higher notes, but overall it doesn't get that much closer than the autocorrelator, so I've left that as the default algorithm. You, however, can swap in the FFT detector in the code if it works better for the material you are auto-tuning.

There are ways of using the phase information from the FFT output to increase the accuracy of pitch detection even further, but I have left that as an exercise for the reader!

Pitch Shifting

The next step is to determine how much we will shift the pitch. The simplest way to do this is to look for the musical pitch that is closest to the detected pitch. Then, the amount of the shift by is simply the ratio of those two notes.

There are, however, some additional considerations. First, we may want to select a subset of musical notes that are acceptable. For example, only notes in the key of C#, or maybe F# minor pentatonic. This may require a slightly more radical adjustment.

Second, depending on the effect we are after, we may not want to instantaneously jump to the new frequency. The code I am using utilizes a fairly rudimentary “attack” time parameter, allowing you to gradually move to the new frequency.

The actual DSP for the pitch-shifting effect is more or less untouched from Stephan Bernsee's code, and this is because it works really well. The Bernsee's code makes use of the Fast Fourier Transform, plus a bunch of clever mathematics, which I almost understand, but not quite well enough to try and explain here! You're better off reading an article in which the man himself explains how it works.

The class that manages the pitch-shifting algorithm is called SmbPitchShifer and inherits from a PitchShifter base class. This does the bulk of its work in the ShiftPitch function:

c#:

public void ShiftPitch(float[] inputBuff, float inputPitch,
                       float targetPitch, float[] outputBuff, int nFrames)
{
     UpdateSettings();
     detectedPitch = inputPitch;

VB.Net

Public Sub ShiftPitch(ByVal inputBuff() As Single, ByVal inputPitch As Single,
                    ByVal targetPitch As Single, ByVal outputBuff() As Single, ByVal nFrames As Integer)
    UpdateSettings()
    detectedPitch = inputPitch

The inputPitch parameter is set to the frequency detected by the PitchDetector. The targetPitch parameter is currently unused, but will be used to specify the target pitch in real-time when accepting input from, say, a MIDI keyboard. In any case, we call UpdateSettings in order to see if any of the autotune algorithm settings have changed since last time.

Next we calculate the amount we need to shift the pitch shift. A shift factor of 1 means no change. We don't allow the shift factor to go above 2 or below 0.5, since these figures represent a whole octave change:

c#:

float shiftFactor = 1.0f;

if (inputPitch > 0)
{
    shiftFactor = snapFactor(inputPitch);
}

if (shiftFactor > 2.0) shiftFactor = 2.0f;
if (shiftFactor < 0.5) shiftFactor = 0.5f;

VB.Net

Dim shiftFactor = 1.0F

If inputPitch > 0 Then
   shiftFactor = snapFactor(inputPitch)
   shiftFactor += addVibrato(nFrames)
End If

If shiftFactor > 2.0 Then
   shiftFactor = 2.0F
End If
If shiftFactor < 0.5 Then
   shiftFactor = 0.5F
End If

The decision of what the target note is takes place in the snapFactor function:

c#:

protected float snapFactor(float freq)
{
    float previousFrequency = 0.0f;
    float correctedFrequency = 0.0f;
    int previousNote = 0;
    int correctedNote = 0;
    for (int i = 1; i < 120; i++)
    {
        bool endLoop = false;
        foreach (int note in this.settings.AutoPitches)
        {
            if (i % 12 == note)
            {
                previousFrequency = correctedFrequency;
                previousNote = correctedNote;
                correctedFrequency = (float)(8.175 * Math.Pow(1.05946309, (float)i));
                correctedNote = i;
                if (correctedFrequency > freq) { endLoop = true; }
                break;
            }
        }
        if (endLoop)
        {
            break;
        }
    }
    if (correctedFrequency == 0.0) { return 1.0f; }
    int destinationNote = 0;
    double destinationFrequency = 0.0;
    // decide whether we are shifting up or down
    if (correctedFrequency - freq > freq - previousFrequency)
    {
        destinationNote = previousNote;
        destinationFrequency = previousFrequency;
    }
    else
    {
        destinationNote = correctedNote;
        destinationFrequency = correctedFrequency;
    }
    if (destinationNote != currPitch)
    {
        numElapsed = 0;
        currPitch = destinationNote;
    }
    if (attack > numElapsed)
    {
        double n = (destinationFrequency - freq) / attack * numElapsed;
        destinationFrequency = freq + n;
    }
    numElapsed++;
    return (float)(destinationFrequency / freq);
}

VB.Net:

Protected Function snapFactor(ByVal freq As Single) As Single
    Dim previousFrequency = 0.0F
    Dim correctedFrequency = 0.0F
    Dim previousNote = 0
    Dim correctedNote = 0
    For i = 1 To 119
        Dim endLoop = False
        For Each note As Integer In Me.settings.AutoPitches
            If i Mod 12 = note Then
                previousFrequency = correctedFrequency
                previousNote = correctedNote
                correctedFrequency = CSng(8.175 * Math.Pow(1.05946309, CSng(i)))
                correctedNote = i
                If correctedFrequency > freq Then
                    endLoop = True
                End If
                Exit For
            End If
        Next note
        If endLoop Then
            Exit For
        End If
    Next i
    If correctedFrequency = 0.0 Then
        Return 1.0f
    End If
    Dim destinationNote = 0
    Dim destinationFrequency = 0.0
    ' decide whether we are shifting up or down
    If correctedFrequency - freq > freq - previousFrequency Then
        destinationNote = previousNote
        destinationFrequency = previousFrequency
    Else
        destinationNote = correctedNote
        destinationFrequency = correctedFrequency
    End If
    If destinationNote <> currPitch Then
        numElapsed = 0
        currPitch = destinationNote
    End If
    If attack > numElapsed Then
        Dim n = (destinationFrequency - freq) / attack * numElapsed
        destinationFrequency = freq + n
    End If
    numElapsed += 1
    Return CSng(destinationFrequency / freq)
End Function

The way this function works is that it runs through the MIDI notes 0-120 and, if that note is selected as one of the valid pitches we support, we remember the “corrected frequency,” which can be calculated from the MIDI note number with the following formula:

c#:

correctedFrequency = (float)(8.175 * Math.Pow(1.05946309, (float)midiNoteNumber));

VB.Net

correctedFrequency = CSng(8.175 * Math.Pow(1.05946309, CSng(i)))

Obviously, a pitch is likely to fall somewhere in between two valid notes, so we choose the which pitch to correct by determining which one is closest to the detected frequency.

The snapFactor function is also responsible for implementing the attack time parameter. This allows the destinationFrequency to be slowly moved to the target note over the duration of the attack period. Having calculated our shift factor, we are now ready to pass our data on to the actual pitch-shifting algorithm:

c#:

int fftFrameSize = 2048;
int osamp = 8; // 32 is best quality
SmbPitchShift.smbPitchShift(shiftFactor, nFrames, fftFrameSize, osamp, this.sampleRate, inputBuff, outputBuff);

VB.Net

Dim fftFrameSize = 2048
Dim osamp = 8 ' 32 is best quality
SmbPitchShift.smbPitchShift(shiftFactor, nFrames, fftFrameSize, osamp, Me.sampleRate, inputBuff, outputBuff)

The final thing we do in the ShiftPitch function is keep a record of the pitch shifts we have made. These are stored in a queue (maximum of 5000 entries) and are very useful for diagnosing what is going on if you are not getting the results you wanted from the algorithm:

c#:

shiftedPitch = inputPitch * shiftFactor;
updateShifts(detectedPitch, shiftedPitch, this.currPitch);

VB.Net

shiftedPitch = inputPitch * shiftFactor
updateShifts(detectedPitch, shiftedPitch, Me.currPitch)

Performance

Performance, as you might expect in a managed application that has not been extensively optimized, was not good. Using my laptop, I could autotune one minute of audio in about 90 seconds. Obviously, that rules out real-time autotuning. I decided to profile the application to see if there were any quick ways I could improve things.

The profiling tools in Visual Studio revealed that 20% of the time was spent on pitch detection and 80% pitch shifting. Unfortunately, there were not too many options available for optimisation, since further investigation pointed to calls to Math.Sin taking the bulk of the time. Possibly creating lookup tables could save a bit more time.

Fortunately, we have another option for speeding things up. The pitch-shifting algorithm takes an “oversampling” parameter, which by default is set to 32, the highest value. However, we can trade off speed for quality. Setting it to 16 meant that I could autotune a minute of audio in 55ms (on my 2.4GHz Core2Duo laptop) – realtime but only just. Setting it to 8 reduced that down to 36s. The results still sounded reasonable, so I have left it set at 8 in the code.

An alternative way of speeding it up though would be to swap in a different pitch-shifting algorithm. You could start by trying out one I created as part of the Skype Voice Changer project previously featured on Coding4Fun, which is also able to operate in real-time (although I haven't done any quality comparisons).

Creating a Test GUI

Rather than starting from scratch, I decided to build upon .NET Voice Recorder, a WPF application I created for a previous Coding4Fun article. This takes advantage of the NAudio .NET audio library for audio recording and playback capabilities. The GUI has three screens. On the first is the input device used for recording. The second records a short voice clip. And the third allows you to edit a small portion of saved audio.

Here's a screenshot of the second screen showing a recording in progress:

And here's the screen that allows you to trim the recording, preview it, and save it as WAV:

As you can see, I have added a new button allowing access to the autotune effect settings. On this screen, you can select which notes are valid, and you can also adjust the “attack time” if you prefer to not go for the robotic effect. I've included a drop-down menu that automatically selects the appropriate notes from various keys.

When you click “Apply,” the autotune effect is applied (while you wait on a background thread) and then you are returned to the screen, allowing you to play back your recording and see how it sounds. If you'd like, you can then go back and change the autotune settings (or turn it off).

MVVM Light

The original VoiceRecorder application used a MVVM (model-view-viewmodel) architecture for binding data to each view. I have updated it to make use of Laurent Bugnion's excellent MVVM Light library. This removes the need for my own RelayCommand and ViewModelBase classes, and also enables me to replace my ViewManager with a more extensible framework using the event aggregator (“Messenger”) that is included with MVVM light. This allows me to quickly navigate from one view to another by sending out a message on the event aggregator:

c#:

private void NavigateToSaveView()
{
   Messenger.Default.Send(new NavigateMessage(SaveViewModel.ViewName, this.voiceRecorderState));
}

VB.Net

Private Sub NavigateToSaveView()
    Messenger.Default.Send(New NavigateMessage(SaveViewModel.ViewName, Me.voiceRecorderState))
End Sub

Getting the Best out of Autotune

Unfortunately, Autotune is an effect that doesn't always produce the desired result . Obviously, if you want great autotune, you're best off buying a commercial implementation, but here are a few tips for getting the most out of an autotune algorithm:

• Get a good quality recording. Avoid background noise, hum, and too quiet or too loud (distorted) recordings. If you need to sing against a backing track, play it in headphones so the microphone doesn't pick it up.
• Know what key you are singing in. This is where the test application won't help you out, since if you don't know what key you are singing in, you can hardly expect to be able to select the appropriate key from the list. You might even be singing in between two keys! If you have an in-tune musical instrument at hand, play a note to give yourself a starting pitch.
• Choose your scale. The easiest option is to just go with a chromatic scale, which means all 12 notes are valid. However, you can try to make the autotune force you into a monotone. Pentatonic scales are useful for instant gratification. They have five notes in them, and so long as you stick to those five, almost anything you sing will fit in with a backing track in your chosen key.
• Adjust the attack time. An attack time of zero is great for the robot effect. A longer attack time will smooth out transitions.
• Why is it “warbling”? With this autotune algorithm, it is quite common to get a “warbling” effect. This is because it is either changing its mind about what note to pitch shift to (because the pitch detector is not providing stable pitch detection), or because the pitch detector couldn't determine a pitch at all, so your voice isn't being pitch shifted. You can play around with the release slider (and may need to modify the release algorithm) if you want to eliminate warbling.

Taking it further

.NET Voice Recorder is open source and hosted on CodePlex in a Mercurial repository. So what you waiting for? Make a fork and have a go at improving it:

• Improve the pitch detector algorithm. I have already given some suggestions for how this can be done. One idea that might be worth experimenting with is making it “hold” the detected frequency for a short period until a strong, new frequency is detected.
• Display the detected pitches. The autotune effect stores data of the detected pitches as well as the pitches it attempts to convert to. You might display this information to the user, perhaps underneath the waveform, so they can see what it is detecting. (It currently outputs this information with Debug.WriteLine, which is useful for debugging purposes).
• Suggest an appropriate scale? Instead of leaving it to the user to select what key to snap the notes to, how about auto-selecting the detected notes?
• Allow direct input of desired pitch. Instead of letting the autotune effect try to work out what note it should be targeting at any given time, it is far more effective to let the user input what note they would like to shift to. This could be done by entering the note using a MIDI keyboard in real-time, or by drawing the notes in, perhaps on a “piano-roll” control. Or, users could implement a simple, domain-specific language to specify the desired note. For example:
  • 0:00.0 C#
  • 0:01.5 E
  • 0:02.7 G#
• Port it to Windows Phone 7. This would be quite a cool effect to have on your phone. Of course, you might need to optimize performance a bit more to save battery life. And you'll want to put your graphic designer hat on to give it a more beautiful look than it currently has.

About the Author

Mark Heath is the author of several open source .NET applications and libraries, including NAudio and the Skype Voice Changer. He works for NICE Systems, developing applications that search, display, and play back vast amounts of multimedia data. He has a blog, Sound Code, and you can follow him on his sporadically updated Twitter account.

Links:

• Glo Bible
• Windows Client
• BizSpark Home Page
• Ignition Showcase
  

When Joel first told me about the application, I was really excited about how he leveraged WPF and the power of desktop integration to create something that would have been a lot more work just a few years ago. Watch the video for more info.

• SpectraCal Home Page
• Pete's Site and Blog

Demo code

Click here for Visual Studio Downloads

About Billy

Billy Hollis authored many of the earliest books on .NET, and has been working with WPF and Silverlight for over three years. He is a regular speaker at conferences such as Tech·Ed, PDC, VSLive, and DevConnections, and his two sessions at Tech·Ed 2009 were both rated in the top twenty sessions for the conference overall. He has a consulting practice in Nashville, TN, focusing on healthcare applications, advanced rule-driven architectures, and advanced user interface prototyping and development. He also offers introductory and advanced training classes on WPF and Silverlight. He is the author of a forthcoming book on WPF.

Stuff Billy recommends you check out this summer:

• WPF videos
• Silverlight videos
• LINQ
• Azure
• Jesse Liberty's Silverlight Blog
• Beatrice Costa on Silverlight and WPF
• A Whole New Mind by Daniel Pink
• .NET Rocks TV
• Developers Intro to XAML – Part 1
• Developers Intro to XAML – Part 2
• MSDN Blogs
• Billy's blog: http://www.dotnetmasters.com/

Below is the basic hardware setup of the Laser Graffiti System.

Building a Laser Tracker Engine

Giving Sight to the Blind

The very first thing that needed to be proven was the ability to track a laser point. The most obvious solution was to give our application some vision with a webcam. Since this was going to be a WPF application, I decided to use my open-source project, WPF MediaKit, which comes with a webcam control called the VideoCaptureElement.

At the time, VideoCaptureElement was not very robust, and it occurred to me that my control wasn't very useful beyond looking at your webcam in a WPF app! I needed a way to get high-performance access to every pixel of every frame the webcam spat out. This was a good time to add that ability and increase value in my project. Without getting into the gory details of DirectShow and p/invoke, I was able to add a hook to pass me the pixel buffer, where I would wrap it in a Bitmap class and raise an event for each frame.

The XAML required to get each video sample from the web camera.

<MediaKit:VideoCaptureElement LoadedBehavior="Play"
    RenderOptions.BitmapScalingMode="LowQuality"
    UnloadedBehavior="Manual"
    DesiredPixelWidth="{Binding DesiredPixelWidth}"
    DesiredPixelHeight="{Binding DesiredPixelHeight}"
    VideoCaptureDevice="{Binding SelectedItem, ElementName=videoCapDevices}"
    FPS="30"
    EnableSampleGrabbing="True"
    NewVideoSample="VideoCaptureElement_NewVideoSample"
    Margin="0"
    x:Name="videoElement"
    Stretch="Fill"
    MinHeight="380"
    MinWidth="500" />

Did I Mention Lasers?

Now that I had the ability to look at every pixel sent to me, I needed to make sense of it all. In order to complete the project, we have to be able to find a laser point shined on a wall. The difficulty lies in the fact that a laser can be of varying size and color. For example, if you take two consecutive video frames with the same green laser point, you will find the point may slightly differ between the two. This is because we are dealing with analog data and slightly changing lighting conditions. Whatever algorithm we choose to find the laser must take this into account.

There are many advanced ways to use video analytics to find the laser pointer, but this project did have a deadline, and I wanted to keep it fun, so I went with a simple method, which allows a user to filter the video based off ranges of hue, saturation, and luminance. To do this, we make use of the AForge image processing library. AForge is an open-source library that comes with tons of useful utilities for just what we want to do.

Filtering pixels and counting blobs…

As previously mentioned, we need to first filter the image based on hue, saturation, and luminance. Luckily, the AForge library comes with such a filter, so there is not much work to be done here:

/* This AForge class helps us filter out the pixels we do not want */
var hsl = new HSLFiltering
{
    Hue = new IntRange(HueMinimum, HueMaximum),
    Saturation = new DoubleRange(SaturationMinimum, SaturationMaximum),
    Luminance = new DoubleRange(LuminanceMinimum, LuminanceMaximum)
};

var bitmapData = bitmap.LockBits(
    new Rectangle((int)targetSearchArea.X, 
        (int)targetSearchArea.Y, 
        (int)targetSearchArea.Width, 
        (int)targetSearchArea.Height),
    System.Drawing.Imaging.ImageLockMode.ReadWrite, 
    m_lastBitmap.PixelFormat);


/* Apply the AForge filter.  Doing it "in place" is more efficient as
 * new bitmap does not have to be allocated and copied */
hsl.ApplyInPlace(bitmapData);

So, what needs to be done now that we have a filtered image showing only a laser dot? The answer: Blob counting! A blob, in the context of video analytics, is a set of pixels that are all touching, and AForge comes with a class ready to tackle that too!

The AForge blob counter takes in a gray-scale image for processing, and then returns all blobs found in the image. The information included on a blob includes the rectangular pixel area where the laser was found in the video frame. The center of that area should be the center of our laser. It's important to note that if the HSL filter is improperly calibrated, we will get an overabundance of blobs and there will accordingly be no way we can tell if it is the laser point or not.

/* Create an initialize our blob counter */
var blobsCounter = new BlobCounter
    {
       FilterBlobs = true,
       ObjectsOrder = ObjectsOrder.Size,
       MinHeight = BlobMinimumHeight,
       MinWidth = BlobMinimumWidth,
       MaxWidth = 25,
       MaxHeight = 25
    };

/* We first let the blob counter process our image */
blobsCounter.ProcessImage(grayImage);

/* Retrieve a list of blobs that were found */
Blob[] blobs = blobsCounter.GetObjects(grayImage);

The early laser tracking prototype. We can literally have it track anything, based off color.

Building the Laser Graffiti Application

Our Rocket Engine Needs a Rocket

At this point, I had a working proof of concept and a laser tracker. What I needed now was an application to use it. You know, the laser-graffiti application.

There are many ways to construct a WPF application and one pattern that's been getting a little bit of traction is MVVM. I chose to make a composite application using Prism and MVVM. Prism is a great guidance library pack for making composite WPF applications with tons of tools to help in common scenarios.

Composite What? Huh?

When developing applications, usually of the larger variety, things can become complex. Complexity turns into a mess. A mess turns into time investment. Time investment turns into money. The more money spent, the fewer bonuses you will receive. To make sure we get that end-of-year pay raise, we want to control complexity from the get-go by making our code and our internal systems loosely coupled. A composite application is just a collection of loosely coupled systems that is composited at runtime.

The advantages of a composite application are:

· Components can be developed independently

· Components can be swapped out more easily than if they were tightly coupled

· The application remains extendable, meaning you can more easily extend your application

· Maintenance is much clearer with less chance of breaking other components

· Unit testing is more feasible

Composite Application Entrails

There are many, many excellent resources about Prism available on the internet, so I'd rather not compete with them. I did, however, want to cover the basics, and at least how I've used it within the Laser Graffiti application.

The Shell Assembly – This contains the main entry point of the application and the initial or main UI container for the application. Before anything else, this assembly will initialize something called a bootstrapper. The bootstrapper simply does any pre-initialization that needs to be done before initializing the rest of the application.

Module Assemblies – Each module assembly can be considered a component. The bootstrapper initializes each component and then the module spins up any application services and adds any user interface to the shell. For instance, the MediaKit module handles the capture of the video and the AForgeModule handles the vision routines.

The Infrastructure Assembly – This contains any base classes, well known/shared interfaces or general infrastructure code. Simply put, most if not all modules will have reference to this.

Rendering the Graffiti

I needed a way to draw the graffiti to a projector. Overall, this will be accomplished by drawing the graffiti to a window. The window will be set to maximized with no border. I also needed to write the code to draw the graffiti. Initially, I just used the WPF InkCanvas control, but I found it wasn't well tuned for what I wanted to do with it. So, I decided XNA would give the most performance and flexibility with the level of effects desired for drawing graffiti.

I had never really done any XNA development, so this project is certainly not built using best practices. The important part to notice is that the project is more or less a regular XNA application with the exception that the executable is added as a library reference. This is because we instantiate the XNA from our WPF application's process. The following snippet of code accomplishes this:

private void ShowGraffitiWall(bool show)
{
    if (m_game != null)
        m_game.Exit();

    if (!show) 
        return;

    var t = new Thread((ThreadStart)delegate
    {
        m_game = new XnaGraffitiGame();
        m_game.SetScreen(m_lastScreen);
        m_game.Effect = m_lastGraffitiEffect;
        m_game.Run();
        m_game.Dispose();
    });

    t.SetApartmentState(ApartmentState.STA);
    t.IsBackground = true;
    t.Start();
}

We instantiate the XNA window this way because it allows our WPF code to send the XNA messages of where the laser pointer was detected.

Drawing with the GPU

A blanket statement about XNA is that it is a drawing API. XNA applications can get quite complex when you add in things like 3D or shaders, but for this article, I'm only covering the required simple stuff. That said, I do make the assumption that the reader has had at least a few hours of XNA experience.

Our primary goal is to render virtual graffiti using XNA. This means we need to be able to draw a line. I started by creating a PNG file that would act as a “brush.” In XNA, these 2D graphics are also known as sprites. The brush sprite looks like this:

Now we can use XNA's SpriteBatch class to draw our brush. The first problem I had, though, was that most XNA tutorials show drawing the sprite, continue, clear the screen, draw the sprite, continue, etc. I needed to retain previous brush strokes, which means drawing onto an intermediate drawing surface (RenderTarget2D), then drawing that to the screen.

The next issue I had was that the line I was trying to draw was very broken. See, the laser detection code runs at a maximum 30 FPS (the limit of my web camera). If I moved the laser fast enough, the screen would just look like a bunch of random points were being drawn. I had to move the laser very slowly to make it appear like a line. I remember reading an excellent article detailing a project by Rick Barraza, which led me to a link to some popular algorithms for creating a line where your input is two, 2D points.

So, drawing a line required only a simple, two-step process of A) getting a new point from laser detection, and B) drawing a line from last_laser_coordinate to current_laser_coordinate. After putting that together, we can now produce some useful looking graffiti!

Adding Crazy Graffiti Effects

The excitement of drawing simple lines in XNA wore off pretty fast. After all, the GPU is capable of some pretty amazing things. What about drawing with fire? Fire isn't boring:

The fire effect is done using a method slightly similar to the solid color sprite method described earlier. However, since it uses a particle system, the fire effect is much more complex. Each particle has its own shader to control position and color. With the time constraints of this project, I had modified code from this XNA tutorial. There weren't many changes from the tutorial's code, but a big hurdle was that the particles took 3D coordinates instead of the 2D coordinates I needed to draw the graffiti. I found a small snippet to handle that, though I was informed by other XNA gurus that there is an easier way to accomplish this. If you are curious, this is what it looks like (and it's not for the faint of heart):

protected Vector3 ScreenPointToVector3(Point coords)
{
    Matrix viewMatrix = 
        Matrix.CreateTranslation(0, -25, 0) *
        Matrix.CreateRotationY(
            MathHelper.ToRadians(CameraRotation)) *
        Matrix.CreateRotationX(
            MathHelper.ToRadians(CameraArc)) *
        Matrix.CreateLookAt(
            new Vector3(0, 0, -CameraDistance),
            new Vector3(0, 0, 0), 
            Vector3.Up);

    float aspectRatio = 
        (float)m_game.GraphicsDevice.Viewport.Width /
            (float)m_game.GraphicsDevice.Viewport.Height;

    Matrix projectionMatrix = 
        Matrix.CreatePerspectiveFieldOfView(
            MathHelper.PiOver4,
            aspectRatio,
            1, 10000);

    Vector3 nearScreenPoint = new Vector3(coords.X, coords.Y, 0);
    Vector3 farScreenPoint = new Vector3(coords.X, coords.Y, 1);
    Vector3 nearWorldPoint = 
        m_game.GraphicsDevice.Viewport.Unproject(
            nearScreenPoint, 
            projectionMatrix, 
            viewMatrix, 
            Matrix.Identity);
    Vector3 farWorldPoint = 
        m_game.GraphicsDevice.Viewport.Unproject(
            farScreenPoint, 
            projectionMatrix, 
            viewMatrix, 
            Matrix.Identity);

    Vector3 direction = farWorldPoint - nearWorldPoint;

    float zFactor = -nearWorldPoint.Z / direction.Z;
    Vector3 zeroWorldPoint = nearWorldPoint + direction * zFactor;

    return zeroWorldPoint;
}

The End Result

Web-camera device selection:

Fine tuning the laser detection:

About Jeremiah

Jeremiah Morrill is a software developer and 2010 MVP living in Las Vegas, Nevada. He owns a software company called HJT, http://www.hjtcentral.com/, with a couple partners. There he focuses on multimedia and rich user interfaces with WPF and Silverlight. Jeremiah spends a lot of his free time learning, listening and helping others in the online development communities. 

Mark and the team would love to hear your feedback on the WPF and Silverlight Designer on their MSDN Forum. You can also find a ton of great content about the designer on the team’s Blog. You can follow Mark on twitter @MarkPavWT

Host and Guest: Mike Benkovich, Senior Developer Evangelist, and Mark Wilson-Thomas, Program Manager, Microsoft Corporation

Mike Benkovich delivers technical presentations around the U.S. as a developer tools evangelist on the MSDN team at Microsoft. He has worked in a variety of professional roles, including architect, project manager, developer, and technical writer.

Mark is a Program Manager on the "Cider" team, which builds the WPF and Silverlight Designers in Visual Studio. He is responsible for the layout and visual designer features of the designer. Prior to joining developer division Mark worked in Microsoft Services as a Development Lead, building large scale web-based UIs for the UK Government, the BBC and others.

John O'Donnell Microsoft Dynamics ISV Architect Evangelist
Microsoft Corporation
http://blogs.msdn.com/jodonnell
http://blogs.msdn.com/usisvde
http://www.twitter.com/jodonnel