Introduction

Almost 10 years ago, Microsoft announced DirectX 8.0, including the huge real-time computer graphics milestone, Direct3D 8.0. Direct3D 8.0 introduced programmable shaders and gave the developers a chance to create never-before-seen effects and experiences apart from the fixed-function graphics pipeline. With Direct3D 8.0's new shader capabilities, it became possible to compute neat custom-rendering effects on mainstream graphics hardware. Today's graphics hardware is capable of running hundreds of shaders in parallel and modern games make heavy use of this technology t to achieve stunning effects.

A shader is a rather small program, a so-called kernel function, typically executed in parallel for each data element. Pixel shaders, for example, are executed for each pixel of a bitmap, and therefore used to implement per-pixel effects.

This introductory article will explain how to write pixel shaders for Silverlight and WPF, what tools should be used, and how to work with the tools. Furthermore, it will show how to build an extensible Silverlight shader application.

Demo Application

The demo application makes it possible to apply different shaders to an image or to the live stream from the webcam. The application not only comes with the two shaders that will be implemented in this article, it also contains three other shaders I've written before. The complete source code is licensed under the Ms-PL and can be downloaded from the CodePlex site.

You need at least the Silverlight 4 runtime installed to run the sample and a webcam is needed in order to exploit the full functionality. The runtime is available for Windows and Mac.

Open the sample

Figure 1: Screenshot of the demo application

How To Use?

You can start and stop the webcam with the Button, or you can load an image from disk with the Button. Use the ComboBox to change the pixel shader that is applied to the source. Each shader has its own controls to change the used parameters. The controls should be pretty much self-explaining. Just try them out.

When you click the Button for the first time, you'll need to give permission for the capturing. This application uses the default Silverlight capture device. You can specify default video and audio devices with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu, and select the "Webcam / Mic" tab to set them.

Where do start?

What do we have in Silverlight and WPF?

Pixel shaders were introduced with WPF 3.5 SP1, and later with Silverlight 3, as so-called ShaderEffects. ShaderEffects can be applied to any control to create both nice visual effects and new user experiences. WPF 3.5 SP1 and Silverlight support the Shader Model 2, which is limited to a total of 96 instructions (64 arithmetic and 32 texture instructions). Modern DirectX 11 graphics cards already support Shader Model 5, which doesn't have such limits. Silverlight pixel shaders, however, are executed on the CPU and not on the specialized GPU. Since the software rendering pipelines make use of modern CPU capabilities like SSE and multi-core execution, they still run pretty fast and are the right way to implement effects in Silverlight.

WPF renders the shaders slightly different. If the shader is applied to an element which is rendered on the graphics card, the shader will also be executed on the GPU. But if the element is being rendered for printing, certain TileBrushs are used or any other reason prevents hardware acceleration, the element and the shader will be rendered in software. This rendering is done on WPF's render thread and the software shader unit also uses fast SSE instructions like Silverlight's renderer, but does not take advantage of multiple CPU cores.

WPF 4 supports Shader Model 3 with a much higher instruction count limit and these pixel shaders are only executed on the GPU. Due to the more complex computation there's no software rendering fallback. The shader will simply be ignored if it's applied to an element being rendered in software or the graphics hardware doesn't support Shader Model 3.

This article targets Silverlight and WPF and therefore stays within the bounds of the Shader Model 2.

How to program shaders?

There are several ways to write and compile shader programs. Nowadays, the most common method used in the Direct3D and Windows world is a language called High Level Shading Language (HLSL). The Direct3D shader compiler fxc.exe compiles the HLSL code into byte-code, which is then executed by the runtime.

HLSL is a C-style language with some special data types and intrinsic functions, but without pointers. If you know how to write code in a C-style language like C#, you will quickly learn how to write a shader with HLSL. By the way, if you know HLSL, you also know NVIDIA's shading language Cg. Cg and HLSL have the same root and are very similar.

HLSL defines scalar and various vector / matrix data types for integer and floating point operations. The built-in intrinsic functions support scalar and vector data types. Flow-control statements—such as if, switch, for, and while—are also possible. Of course, the curly brace is used for code blocks and most of the C operators are also supported. The compiler uses semantics to both determine the intended usage of a parameter and provide the right data for it. The register keyword is typically used to pass a parameter into the shader program. The elements of the vector types can be accessed through various aliases, such as x, y, z, w; r, g, b, a; u, v; etc. It's also possible to combine these and write nice and short statements by using swizzling:

HLSL

// Create 3D float vector a and b with different syntax
float3 a = float3(1, 2, 3);
float3 b = {5, 6, 7};

// Calculate cross product of vector a and b using swizzling
float3 crossProduct = a.yzx * b.zxy - a.zxy * b.yzx;

The MSDN is a great resource for HLSL and provides detailed explanations of the syntax and functions. The following examples will make it clearer and the explanations should help to get you started.

What tools to use?

Of course, it's possible to write a pixel shader with a simple text editor and compile it with the command line tool fxc.exe, but there's a great tool available that makes the process a lot easier. The Shazzam Tool by Walt Ritscher is THE utility for Silverlight and WPF shader development. It comes with an HLSL editor, which includes syntax highlighting, that compiles the shader and applies it right away to a sample input. It also generates controls for each parameter, which may be used to change the shader settings on the fly, and it creates the needed C# or VB source code file with a class that is derived from ShaderEffect.

Here's what you need to get started:

1. Download the DirectX SDK and install it.
2. Download the Shazzam Tool and install it.
3. After the Shazzam Tool is started, verify that the path to the DirectX FX compiler is set (Figure 2). The fxc.exe is normally located in the DirectX SDK installation folder under Utilities\bin\x86. Also make sure the right Target framework is selected and a Generated namespace is set.
4. To see if everything works, open a Sample Shader with the Shader Loader, select a sample tab page, and try the controls on the Change Shader Settings page (Figure 3).

Figure 2: Shazzam Tool Settings

Figure 3: Shazzam Tool Overview

How to write a pixel shader?

Now that we have the right tools installed and configured, we are ready to write the first pixel shader and the Silverlight application that will use it.

Example 1: The Tint Shader

The first pixel shader we're writing is a rather simple tint shader that converts the pixel into gray and tints it with a parameterized color.

In the Shazzam Tool, select File à New Shader File, choose a location for the HLSL FX file, and name it TintShader. Shazzam will automatically create the basic pixel shader code, including a float parameter SampleI. Hit the F5 key to compile and apply the shader to the selected sample image.

HLSL

sampler2D input : register(s0);

/// <summary>Explain the purpose of this variable.</summary>
/// <minValue>05/minValue>
/// <maxValue>10</maxValue>
/// <defaultValue>3.5</defaultValue>
float SampleI : register(C0);

float4 main(float2 uv : TEXCOORD) : COLOR 
{
    float4 Color; 
    Color = tex2D(input, uv.xy);
    return Color; 
}

The input register is the actual bitmap / texture that holds the pixels and is sampled inside the pixel shader. This pixel shader main function is the entry point and is executed for each pixel of the input bitmap. The coordinate of the current pixel that is processed is passed as the float2 parameter uv. This coordinate is normalized to the range [0, 1]. The color of the pixel at the passed uv coordinate is sampled as float4 with the built-in tex2D intrinsic function. A float4 COLOR value is expected as the return value of the pixel shader.

Figure 4: Output of the initial shader code (original image)

The initial pixel shader code returns the original color for each pixel and we use this as the starter for our gray scale conversion.

Gray Conversion

HLSL

sampler2D input : register(s0);

float4 main(float2 uv : TEXCOORD) : COLOR 
{
   // Sample the original color at the coordinate
   float4 color = tex2D(input, uv);
    
   // Convert the color to gray
  float gray = dot(color.rgb, float3(0.2126, 0.7152, 0.0722));
    
   // Return gray with the original alpha value
   return float4(gray, gray, gray, color.a); 
}

The original color is sampled and then converted to gray using the dot product of the red, green, and blue values with a constant float3 vector. The result actually represents the luminance of the pixel. The dot product multiplies the elements of the color vector with the elements of constant vector and adds the three products, thus resulting in a scalar float value. The return value of the pixel shader is a new color made up of the gray value for RGB and the original alpha (transparency) of the sampled pixel.

Figure 5: Output of the gray conversion shader

Parametric Tinting

The gray conversion shader can now be extended to tint the output in a color that is passed as a parameter through a shader register.

HLSL

/// <summary>The tint color.</summary>
/// <type>Color</type>
/// <defaultValue>0.9,0.7,0.3,1</defaultValue>
float4 TintColor : register(C0);

sampler2D Input : register(s0);

float4 main(float2 uv : TEXCOORD) : COLOR 
{
   // Sample the original color at the coordinate
   float4 color = tex2D(Input, uv);
    
   // Convert the color to gray
   float gray = dot(color.rgb, float3(0.2126, 0.7152, 0.0722)); 
    
   // Create the gray color with the original alpha value
   float4 grayColor = float4(gray, gray, gray, color.a); 
   
   // Return the tinted pixel
   return grayColor * TintColor;
}

The color used to tint every pixel is passed as a parameter and therefore defined as the first register C0 (the next parameter should then be in the register C1). The XML comment is used by Shazzam to both create convenient controls and initialize the generated code. Shazzam creates the appropriate controls for the data type and uses the defaultValue, minValue and maxValue (Figure 6). The changed value of the control is applied directly to the Sample image, which allows a quick and easy shader development.

Each element (RGBA) of the passed TintColor parameter is then multiplied with the float4 gray color and returned. The result of the default TintColor values is a sepia-toned image (Figure 7).

Figure 6: Shazzam Tool Shader Settings

Figure 7: Output of the tint shader

Wiring it together with Silverlight

Now it's time to use the shader in a Silverlight application and apply it to an Image, MediaElement, or whatever UIElement you like.

1. Start Visual Studio, create a new Silverlight Application project, and select at least Silverlight 3 as the target framework.
2. In the Shazzam Tool, click on Compile Shader in the Tools menu and then on Explore Compiled Shaders. Make sure the Generated namespace setting (Figure 2) matches the assembly name of the Silverlight application.
3. Copy the compiled shader file TintShader.ps and the corresponding C# or VB TintShaderEffect.cs|vb file from the GeneratedShaders folder and the CS|VB subfolder into the Silverlight project directory.
4. In Visual Studio, add the TintShaderEffect.cs|vb and the TintShader.ps file to the project. The property Build Action of the TintShader.ps file must be set to Resource. Rebuild the solution.
5. Open the MainPage.xaml file and add the namespace declaration and a Button or any other control that has the TintShaderEffect class applied (see below).
6. Hit the F5 key and see your Silverlight shader application in action (Figure 8).

XAML

<UserControl x:Class="ShaderDemoApp.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d"
    d:DesignHeight="300" d:DesignWidth="400"
    xmlns:local="clr-namespace:ShaderDemoApp">

    <Grid x:Name="LayoutRoot" Background="Gray">
        <Button Content="Big Tinted Button" Width="200" 
                Height="200" Background="Blue">
            <Button.Effect>
                <local:TintShaderEffect />
            </Button.Effect>
        </Button>
    </Grid>
</UserControl>

Figure 8: The TintShader applied to the Button

That's all that's needed to get a pixel shader working inside a Silverlight application. Please note that no manual C# or VB code-behind was written.

Example 2: The Mosaic Shader aka The Donut Shader

The second post process shader effect we'll write is a bit more advanced. It starts with pixelating the image before rounding the blocks until we've the final Mosaic-like result.

For the development of this shader, we can use a different image. To do so, click the Open Image File menu item in Shazzam's File menu. I used a famous test picture for image processing algorithms: Lenna. By the way, there's an interesting story behind this picture of Lena Söderberg.

Figure 9: Original Lenna test image

Pixelating the Input

HLSL

/// <summary>The number of pixel blocks.</summary>
/// <type>Single</type>
/// <defaultValue>25</defaultValue>
float BlockCount : register(C0);

sampler2D input : register(S0);

// Static computed vars for optimization
static float BlockSize = 1.0f / BlockCount; 

float4 main(float2 uv : TEXCOORD) : COLOR
{
   // Calculate block center
   float2 blockPos = floor(uv * BlockCount);
   float2 blockCenter = blockPos * BlockSize + BlockSize * 0.5;
            
   // Sample color at the calculated coordinate
   return tex2D(input, blockCenter);
}

The float BlockCount parameter defines into the number of blocks (large pixels) into which the resulting image will be divided (pixelated). The size of a block (BlockSize) is the inverse of the BlockCount and calculated as static float to save some clock cycles. The coordinate of the current pixel (uv) is then used to determine the block to which it belongs. This determination depends on the BlockCount and is a result of the built-in floor function. To get the color of the output pixel, the center coordinate of each block is sampled for all the pixels that are part of the block.

Figure 10: Output of the pixelation step

Rounding the Pixels

HLSL

/// <summary>The number of pixel blocks.</summary>
/// <type>Single</type>
/// <defaultValue>25</defaultValue>
float BlockCount : register(C0);

/// <summary>The rounding of a pixel block.</summary>
/// <type>Single</type>
/// <defaultValue>0.45</defaultValue>
float Max : register(C2);

sampler2D input : register(S0);

// Static computed vars for optimization
static float BlockSize = 1.0f / BlockCount; 

float4 main(float2 uv : TEXCOORD) : COLOR
{
   // Calculate block center
   float2 blockPos = floor(uv * BlockCount);
   float2 blockCenter = blockPos * BlockSize + BlockSize * 0.5;
        
   // Round the block by testing the distance 
   // of the pixel coordinate to the center
   float dist = length(uv - blockCenter) * BlockCount;
   if(dist > Max)
   {
      return 0;
   }
    
   // Sample color at the calculated coordinate
   return tex2D(input, blockCenter);
}

The Max parameter defines the maximum distance of a pixel to its block center and, therefore, the rounding of a pixel block. If the length of the vector between the current pixel coordinate (uv) and its block center is greater than the Max parameter, a transparent pixel (0) is returned. The built-in length function is used to calculate the scalar length of the distance vector.

Figure 11: Output of the rounding step (Max = 0.45)

Figure 12: Output of the rounding step (Max = 0.60)

Baking Donuts

HLSL

/// <summary>The number of pixel blocks.</summary>
/// <type>Single</type>
/// <defaultValue>25</defaultValue>
float BlockCount : register(C0);

/// <summary>The rounding of a pixel block.</summary>
/// <type>Single</type>
/// <defaultValue>0.2</defaultValue>
float Min : register(C1);

/// <summary>The rounding of a pixel block.</summary>
/// <type>Single</type>
/// <defaultValue>0.45</defaultValue>
float Max : register(C2);

sampler2D input : register(S0);

// Static computed vars for optimization
static float BlockSize = 1.0f / BlockCount; 

float4 main(float2 uv : TEXCOORD) : COLOR
{
   // Calculate block center
   float2 blockPos = floor(uv * BlockCount);
   float2 blockCenter = blockPos * BlockSize + BlockSize * 0.5;
        
   // Round the block by testing the distance 
   // of the pixel coordinate to the center
   float dist = length(uv - blockCenter) * BlockCount;
   if(dist < Min || dist > Max)
   {
      return 0;
   }
    
   // Sample color at the calculated coordinate
   return tex2D(input, blockCenter);
} 

The last thing left to do in order to get some nice rings (donuts) is to add a test for the minimum distance. This is pretty easy and done with the additional Min parameter.

Figure 13: Output of the ring step (Min = 0.20, Max = 0.45)

Please note that all of the parameters can be animated, which could result in a nice (transition) effect when two images are overlaid. The Shazzam Tool also supports animation with the generated Shader Settings controls.

How does the Demo Application work?

The Silverlight demo application is quite flexible and can be used for many different shader effects without touching the core functionality. This last part of the article will show how this extensibility was achieved. Explanations of how to use the webcam with Silverlight and how to load an image from disk were detailed in my last Silverlight Face Detection article.

The application's extensibility was mainly accomplished by using the Managed Extensibility Framework (MEF) and a View-ViewModel approach for the shader parameters. MEF is a great way to make decoupled and flexible applications and has been part of the Silverlight framework since version 4. MEF is like Meth for .Net and Silverlight developers, but without all the undesirable side effects.

The beauty of MEF is best illustrated using source code. As you can see in the demo application (Figure 1), it's possible to select a pixel shader with a ComboBox. The items in the MainPage's shader ComboBox are populated through data binding an ObservableCollection <T> and MEF is used to build this collection.

C#

/// <summary>
/// The main Page of the application.
/// </summary>
public partial class MainPage : UserControl
{
   [ImportMany(AllowRecomposition = true)]
   public ObservableCollection<IShaderViewModel> Shaders;

   public MainPage()
   {
      InitializeComponent();
   }

   private void Initialize()
   {
      // Compose the parts with MEF
      var container = new CompositionContainer(
            new AssemblyCatalog(GetType().Assembly));
      container.ComposeParts(this);

      // Fill ComboBox
      CmbShaders.ItemsSource = Shaders;
      CmbShaders.DisplayMemberPath = "Name";
      CmbShaders.SelectedIndex = 0;

      // ...
   }

   // ...
}

/// <summary>
/// Interface of a ViewModel for a shader effect.
/// </summary>
[InheritedExport]
public interface IShaderViewModel
{
   string Name { get; }
   ShaderEffect Shader { get; }
   UserControl View { get; }
}

The MainPage has a Shaders collection property containing items that implement the IShaderViewModel interface. This collection is initialized by using MEF's CompositionContainer with an AssemblyCatalog and the ComposeParts method. ComposeParts analyses all types in the provided Catalog (here, the assembly), checks if they have certain attributes attached, and wires these so-called parts together. To put it simply, an instance of a type decorated with an Export attribute is created and assigned to each field / property / parameter of the type that has an Import attribute attached.

The IShaderViewModel interface has the InheritedExport attribute attached, which means that implementations of this interface will automatically provide that export. The Shaders collection in turn uses the ImportMany attribute, telling MEF to populate the list with all matching exports (here, all classes that implement the IShaderViewModel interface).

After MEF fills the collection, it is data bound to the ItemsSource property of the ComboBox. The Name property of the IShaderViewModel interface is used as DisplayMember.

The advantage of MEF should be pretty obvious here: you have to implement an IShaderViewModel and maybe a View UserControl for the shader's parameters, but you don't need to add an instance of the shader ViewModel to the Shaders collection manually. Since the IShaderViewModel has the InheritedExport attribute attached, it's not even necessary to add a special MEF attribute to the new shader ViewModel type. Additionally, it's possible to load an extra assembly asynchronously and let MEF compose the parts and update the collection afterwards. By setting the AllowRecomposition parameter of the ImportMany attribute to true we're asking MEF to allow dynamic updates.

As you can see, MEF is pretty easy but also very powerful. But we have just scratched the surface—this blog post and the MSDN are good starters if you want to learn more about MEF.

Further Resources

There are many great HLSL shader development resources available on the web. The following list contains some sites focused on Silverlight / WPF pixel shaders. If I forgot a great resource, please post it in a comment.

• The open source WPF Pixel Shader Effects Library at CodePlex contains a lot of common pixel shader effects.
• Nikola Mihaylov (aka Nokola) wrote a great online image editor in Silverlight that uses pixel shaders for the effects. He released the pixel shaders that are used in EasyPainter as open source. And he also released the custom controls from his tool. The Slider and the ColorPicker used in the demo application are from there. I just changed some properties to Dependency Properties to make the controls bindable. So Kudos to Nikola!
• Walt Ritscher's amazing Shazzam Tool also comes with many cool pixel shaders as samples. Kudos to Walt for making such a great tool!
• Additionally, from time to time I write some shaders and put them up on my blog. Now that we have an extensible Silverlight shader demo application, I will surely integrate my upcoming shaders directly into it.

While I was writing this article, an open source project called WPF Meta-Effects was released on CodePlex. The WPF Meta-Effects framework makes it possible to write Shaders for WPF in C# by using attributes, delegates, and dynamic HLSL compilation. It's a neat idea, but due to its dynamic compilation of shaders it's limited to WPF and can't be used with Silverlight. Additionally, I much prefer the simplicity of HLSL to C# when writing shaders. There's a reason why HLSL was invented: see the examples yourself.

Conclusion

This article explained what shaders and HLSL are, as well as how to write pixel shaders for Silverlight and WPF. It also showed what tools / frameworks to use for the best developer experience and gave an introduction on how to write an extensible application with MEF.

I hope it diminished any fear of the HLSL language and the shader development. I also hope it both inspires and helps you to write your own shaders. I'm eager to see what you come up with. Have fun!

The links to the live demo app and source code are at the top of the article.

About The Author

René Schulte is a .Net / Silverlight developer and Microsoft Silverlight MVP passionate about real-time computer graphics, physics, AI, and algorithms. He loves C#, Shaders, Augmented Reality, and computer vision. He started the SLARToolkit, the WriteableBitmapEx, and the Matrix3DEx Silverlight open source projects, and he has a Silverlight website powered by real time soft body physics. He is also a regular author for Microsoft's Coding4Fun. Contact information can be found on his Silverlight website, his blog, or via Twitter.

Wanting to have fun with the new feature, I thought of a fun sample to create – smashing bugs with motion detection. First step was to get bugs to crawl across the screen. Next I added the WebCam with frame diff calculation, which was intensive so took advantage of the multiple thread support. And finally I added a little Mantis Boy avatar to visualize the smashing of the bugs.

Learn how the BugCamSmash Demo was created

Drum loop from Zajo, Karate Shout from Robinhood76

The camera also features USB 2.0 connectivity, auto focus, a glass lens, a 4x digital zoom, a noise-canceling microphone, and a technology called “ClearFrame” which aims to deliver smooth and detailed video.

The new webcam is compatible with Windows 7, Windows Vista, and Windows XP out-of-the-box and also integrates with the Windows Live suite of tools including Live Messenger, Live Photo Gallery, and Live Movie Maker.

However, before you run out to purchase this hardware (which becomes available on September 9th), you’ll need to first check the hardware requirements. Since this camera does some heavy lifting, you’ll need at least a dual core 1.6 GHz processor in order to use it. However, a 3 GHz processor is recommended. You’ll also need 2 GB of memory.

The camera will retail for $79.99 here in the U.S.

By combining these cameras with Windows Live Messenger, you can can engage in live video chats with your friends and family at any time. And if your friends aren’t available online (or if they live on the other side of the globe and you don’t want to wake them!), you can always record a video message using the new Video Messages site at http://videomessages.live.com. 

Want to buy one now? Both are available from the new Microsoft Store.

April 7, 2006

Summary: In the belated seventh installment of the "Some Assembly Required" column, Scott Hanselman extends, with permission, Andrew Kirillov's Motion Detector to .NET 2.0 and ClickOnce. He also works around some of the quirks in the AirLink AIC250 Network Camera by tracing network traffic and using a little intuition.

The AIC250 Network Camera

Some Assembly Required has been out of circulation for a while, but I'm back. I hope you all didn't forget about me. My wife and I had a baby boy, Zenzo, just 12 weeks ago and he's been a handful. He's mostly sleeping through the night now and I'm back to work, so I'll get back into the rhythm of writing this column on a regular basic. Please do continue to send in suggestions on what kinds of gadgets I can hook up to .NET.

I wanted to be able to see Zenzo while I'm at work, so I evaluated a number of solutions. I could use Video Chat built into MSN Messenger or other chat tools, but it's kind of inconvenient to bring the baby into the computer room and I often have trouble getting the video to work through the firewall. Using video chat would also mean that both sides would have to negotiate the conversation. If I wanted to make things simpler for my wife (and increase the WAF—Wife Acceptance Factor—of the solution), I'd want to just call her at home and start the camera remotely with no effort on her part.

I wanted a camera that would have its own network connection and wouldn't require an attached computer. This means that the camera would have some kind of integrated internal Web Server.

Additionally, my parents live two hours out of town and might want to occasionally peek in on the baby, and I want to make the process as easy as possible on them. Finally, I might want to hook up some kind of motion monitor if the camera detects that the baby is thrashing in his sleep.

I looked around at a number of Network Cameras and since price was my background priority, I settled on the AIC250 Network Camera. I picked up the Wired Ethernet version for US$99, but I understand that a wireless version of the same camera can be found around the net for $99 as well.

Seting up the camera is trivial: plug it in and it gets an IP address via DHCP. It has an integrated Web Server that lets you manage the camera's settings via your browser. The way the web-based interface displays the video stream is interesting, though. It includes two options, a Java Applet or an ActiveX control. These widgets are built into the firmware of the camera and, while the camera's firmware can be updated via software direct from the manufacturer, the video can officially only be viewed in the web interface in one of these two ways.

Being the hacker-type that I am, I wanted to get access to the underlying video stream. It's great that the camera includes all the software you need built-in, but if I could get access to the stream I could do all sorts of fun stuff.

MotionJPEG

I used Simon Fell's TCPTrace to peek at the network traffic as I logged into the web interface and looked at the video using the built-in Java applet. I noticed that the traffic was a series of JPEGs, one after the other. It was an HTTP GET that never ends. The connection stayed open and spit out JPEGs as fast as I could take it, around 10-15 frames a second. I had never seen traffic like this. I'd assumed that the camera would produce AVIs or MPEG files. Then I realized that spitting out JPEGs would be the cheapest possible way to produce video because there'd be no licensing of Video Codecs (Compressor/Decompressor). A little Googling showed me that this technique was called MotionJPEG or MJPEG.

I think of MotionJPEG as a clever hack, with the emphasis on clever. It's not the MPEG (Motion Picture Experts Group) video that you may be familiar with. Instead, MotionJPEG is a stream of JPEG images, one after the other. There doesn't appear to be a formal specification, just varying implementations.

There is really no such standard as "motion JPEG" or "MJPEG" for video. Various vendors have applied JPEG to individual frames of a video sequence, and have called the renkinfgsult "M-JPEG". JPEG is designed for compressing either full-color or gray-scale images of natural, real-world scenes. [Review of Video Streaming]

MotionJPEG over HTTP uses the Content-Type header "multipart/x-mixed-replace" along with a configurable boundary. This means that the stream is made up of Multiple Parts (hence multipart) and each new frame should replace the previous frame (hence x-mixed-replace). This particular camera sends an HTTP Header like this:

Content-Type: multipart/x-mixed-replace;boundary=--video boundary--

The boundary can be anything the server chooses, in this case "--video boundary--" which is a particularly intuitive choice. After the boundary there are additional headers including the actual Content-Type of the part, in this case, image/jpeg.

I started writing a parser/renderer from scratch in .NET 2.0 and the built-in WebClient libraries. It would be fairly easy to read the data into a buffer and watch for the boundary to go by, and then use the System.Drawing libraries to read the JPEGs in and render them as came in. However, I figured that there might be a library out there that would already do the work for me. That's when I found Andrew Kirillov's CodeProject article on Motion Detection.

The MJPEG handling in Andrew's is actually secondary to his code. His .NET 1.1 project is a showcase for his motion detection algorithms that we'll talk about in a moment, but I was interested in his pluggable VideoSource implementation.

Andrew's Implementation

Andrew created an interface called IVideoSource. He includes a number of concrete implementations, one being MJPEGSourcem the one I needed.

Visual C#

public interface IVideoSource
{
  event CameraEventHandler NewFrame;
  string VideoSource{get; set;}
  string Login{get; set;}
  string Password{get; set;}
  int BytesReceived{get;}
  object UserData{get; set;}
  bool Running{get;}
  void Start();
  void SignalToStop();
  void WaitForStop();
  void Stop();
  
}

The most inspired aspect of Andrew's implementation, the one I wouldn't have thought of myself, is the CameraEventHandler.

Visual C#

public delegate void CameraEventHandler(object sender, CameraEventArgs e);
public class CameraEventArgs : EventArgs
{
  private System.Drawing.Bitmap bmp;
  public CameraEventArgs(System.Drawing.Bitmap bmp)
  {
    this.bmp = bmp;
  }
  public System.Drawing.Bitmap Bitmap
  {
    get { return bmp; }
  }

}

The VideoSource throws each frame it finds as bitmaps packaged within an event. This might seem inefficient at first glance, but remember that this event would likely not fire more than 30 times a second, more likely around 5-10 times a second. Additionally, he's throwing a reference to a bitmap, so it's not like copies of bitmaps are flying around the system. Using eventing in this manner not only makes for a clean interface and separation between an IVideoSource implementation and the form that chooses to render it, but it also allows Andrew to insert his post-processing filters and motion detection algorithm.

Working Around Quirks of the AIC250

I had hoped that I'd be able to plug the Motion application directly into the AIC250 camera's MJPEG source. That turned out to be wishful thinking. There weren't any problems with the MJPEG implementation in Andrew's code, but there were a few quirks in the AIC250's embedded Web Server's handling of authentication.

(Note the misspelled "Auther" (sic) HTTPHeader that tells us that Steven Wu wrote this Web Server that reports itself as Camera Web Server/1.0.)

The AIC250 Camera supports HTTP Basic Authorization. When you visit the administration home page for the first time, you're prompted for a name and password using HTTP Basic Auth, then the ActiveX control or Java Applet is on the next page. My TCP sniffing session earlier told me that the stream of JPEGs was the result of an HTTP GET to an endpoint called /mjpeg.cgi.

(click image to zoom)

I assumed that I could make a call to /mjpeg.cgi with .NET, passing in the name and password via HTTP Basic Auth and things would work great. It didn't work, though. However, if I "visited" the home page programmatically first, THEN requested /mjpeg.cgi, the MJPEG stream began. Apparently the Camera sets an internal flag after a user hits the home page that allows the next request to hit /mjpeg.cgi. I suspect this is because the /mjpeg.cgi endpoint is coded to simply return JPEGs, nothing else. They (Steven Wu?) likely decided that getting /mjpeg.cgi to support authentication and meet the HTTP spec exactly would have been a hassle and wouldn't have provided value. Their use case never suspected that some fool (me) would try to get to the video stream directly.

The next thing I ran into was that HTTP Basic Authentication is supposed to wait for an Authentication Challenge first before sending credentials. This web server seems to want the credentials sent right away. Fortunately the WebRequest class supports a PreAuthenticate property that sends the authentication credentials first, before the challenge.

I changed the MJPEGSource to support both "PreAuthentication" and "Authentication with the Home Page" options and I was off and running.

More Quirks - UnsafeHeaderParsing

That wasn't the end of the quirks I'd have to fight. After upgrading the application from .NET 1.1 to .NET 2.0, I started getting exceptions while calling HttpWebRequest.GetResponse(). However, the exceptions weren't very informative. The only thing that had changed was the upgrade from 1.1 to 2.0. A little Googling, MSDNing, and digging led me to Mike Flasko's Blog:

"By default, the .NET Framework strictly enforces RFC 2616 for URI parsing. Some server responses may include control characters in prohibited fields, which will cause the System.Net.HttpWebRequest.GetResponse method to throw a WebException. If useUnsafeHeaderParsing is set to true, System.Net.HttpWebRequest.GetResponse will not throw in this case; however, your application will be vulnerable to several forms of URI parsing attacks. The best solution is to change the server so that the response does not include control characters."

Of course, Steven Wu's built-in Web Server in this Camera isn't going to change, so I had little choice but to include an app.config like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.net>
    <settings>
      <httpWebRequest useUnsafeHeaderParsing="true" />
    </settings>
  </system.net>
</configuration>

Motion Detection

Andrew includes four different motion detection algorithms that he describes in detail in his CodeProject Article. His algorithms are focused on image processing, rather than video processing, which is why he throws each frame as a bitmap, saving the last frame and comparing it to the current frame. Not only can he plug in different video sources, since he "normalizes" all video sources to a series of bitmaps he can apply any motion detection algorithm to any video source.

In the image above, the teddy bear on the right has just been moved with a string. The red boxes around the bear were added by the motion detection algorithm indicating what portion of the image changed.

ClickOnce

At this point I had a WinForms 2.0 application that let me view the camera in my son's room. I forwarded a port from my cable modem's external IP address to the internal IP address of the camera so I could connect to it from outside or at work. The next step was to make the application a ClickOnce application so my parents or I could run it from anywhere.

Visual Studio makes creating a ClickOnce application very straightforward. Right-click on the project within the Visual Studio Solution Explorer and select Properties. First visit the Signing tab and create a Test Certificate (unless you have your own Signing Certificate). The certificate will be used to sign the ClickOnce manifest that will be published to the web site. Next visit the Security tab and select "This is a Full Trust application." If you create your own ClickOnce application, you can specific the exact permissions that your application needs to run. Finally, from the Publish tab you can indicate the publishing location and ultimate URL. I chose to publish to a "publish" folder, then upload the contents to my website manually because I'm a control freak.

Once the project is configured, you can also build and publish a ClickOnce application by running msbuild /target:publish from a Visual Studio command line.

As an interesting sidenote, I originally blogged about problems with ClickOnce and FireFox but a member of the ClickOnce team blogged an explanation. Fortunately this will be fixed in the next version of the Framework.

Now, anyone in my family can visit our private website, ClickOnce and run this .NET 2.0 WinForms application and connect with a name and password to the AIC250 Network Camera. Andrew's motion detection is an added bonus.

Conclusion

There are a number of fun things that could be extended, added, and improved on with this project. Here are some ideas to get you started:

• Add your own video source that reads Animated GIFs, DivX or other formats.
  
  
• Build a video source implementation that connects to WIA (Windows Image Acquisition) devices.
  
  
• Hook up to other Network Cameras.
  
  
• Create your own Motion Detection algorithm that compares to frames and see how it stacks up against Andrew's.
  
  
• Make an event that fires on motion and plugins to react to that motion. Hook up lights, send emails, or whatever you like!
  
  
• Extend the application to support multiple cameras (maybe up to 4) via a "split screen" feature.

Have fun and have no fear when faced with the words: Some Assembly Required!

If you do extend this application, be sure to release the source. Thanks again to Andrew Kirillov for allowing me to extend his wonderful source. My family thanks you.

Scott Hanselman is the Chief Architect at the Corillian Corporation, an eFinance enabler. He has thirteen years experience developing software in C, C++, VB, COM, and most recently in VB.NET and C#. Scott is proud to be both a Microsoft RD and Architecture MVP. He is co-author of Professional ASP.NET 2.0 with Bill Evjen, available on BookPool.com and Amazon. His thoughts on the Zen of .NET, Programming and Web Services can be found on his blog at http://www.computerzen.com/.