If you are a fan of Twentieth Century Fox shows or movies, you should check out the Fox Now and FX Networks Windows 8 Apps in the Windows Store today.

Because this video contains copyright material, it is not available for download.

Their latest streamer, the A-210 is due for release next week, and I’ve been lucky enough to get hold of a review unit a little ahead of time. Like other products in the range, the A-210 boasts an insane level of support for a heap of audio, video and photo file formats – throw what you like at it, and chances are it’ll gobble up the file and stream out your desired entertainment, no questions asked.

The really cool think about this new model is that it takes a look at your music and video files, and as long as you have a wired or wireless internet connection, the Popcorn Hour will go and hunt out cover artwork and other metadata for movies including cast and crew information, ratings, synopses and more.

It’s all displayed in a stunning new user interface which allows you to browse your media collection by cover, file or category – sweet! Dig down into the menu and you’ll find a host of online services supported, including SHOUTcast Radio, BBC Podcasts, Flickr, ABC News, CNN and a whole lot more.

If you’re looking for a streamer during the holidays, make sure you check out the new Popcorn Hour alongside other streamers.

Introduction

Almost 10 years ago, Microsoft announced DirectX 8.0, including the huge real-time computer graphics milestone, Direct3D 8.0. Direct3D 8.0 introduced programmable shaders and gave the developers a chance to create never-before-seen effects and experiences apart from the fixed-function graphics pipeline. With Direct3D 8.0's new shader capabilities, it became possible to compute neat custom-rendering effects on mainstream graphics hardware. Today's graphics hardware is capable of running hundreds of shaders in parallel and modern games make heavy use of this technology t to achieve stunning effects.

A shader is a rather small program, a so-called kernel function, typically executed in parallel for each data element. Pixel shaders, for example, are executed for each pixel of a bitmap, and therefore used to implement per-pixel effects.

This introductory article will explain how to write pixel shaders for Silverlight and WPF, what tools should be used, and how to work with the tools. Furthermore, it will show how to build an extensible Silverlight shader application.

Demo Application

The demo application makes it possible to apply different shaders to an image or to the live stream from the webcam. The application not only comes with the two shaders that will be implemented in this article, it also contains three other shaders I've written before. The complete source code is licensed under the Ms-PL and can be downloaded from the CodePlex site.

You need at least the Silverlight 4 runtime installed to run the sample and a webcam is needed in order to exploit the full functionality. The runtime is available for Windows and Mac.

Open the sample

Figure 1: Screenshot of the demo application

How To Use?

You can start and stop the webcam with the Button, or you can load an image from disk with the Button. Use the ComboBox to change the pixel shader that is applied to the source. Each shader has its own controls to change the used parameters. The controls should be pretty much self-explaining. Just try them out.

When you click the Button for the first time, you'll need to give permission for the capturing. This application uses the default Silverlight capture device. You can specify default video and audio devices with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu, and select the "Webcam / Mic" tab to set them.

Where do start?

What do we have in Silverlight and WPF?

Pixel shaders were introduced with WPF 3.5 SP1, and later with Silverlight 3, as so-called ShaderEffects. ShaderEffects can be applied to any control to create both nice visual effects and new user experiences. WPF 3.5 SP1 and Silverlight support the Shader Model 2, which is limited to a total of 96 instructions (64 arithmetic and 32 texture instructions). Modern DirectX 11 graphics cards already support Shader Model 5, which doesn't have such limits. Silverlight pixel shaders, however, are executed on the CPU and not on the specialized GPU. Since the software rendering pipelines make use of modern CPU capabilities like SSE and multi-core execution, they still run pretty fast and are the right way to implement effects in Silverlight.

WPF renders the shaders slightly different. If the shader is applied to an element which is rendered on the graphics card, the shader will also be executed on the GPU. But if the element is being rendered for printing, certain TileBrushs are used or any other reason prevents hardware acceleration, the element and the shader will be rendered in software. This rendering is done on WPF's render thread and the software shader unit also uses fast SSE instructions like Silverlight's renderer, but does not take advantage of multiple CPU cores.

WPF 4 supports Shader Model 3 with a much higher instruction count limit and these pixel shaders are only executed on the GPU. Due to the more complex computation there's no software rendering fallback. The shader will simply be ignored if it's applied to an element being rendered in software or the graphics hardware doesn't support Shader Model 3.

This article targets Silverlight and WPF and therefore stays within the bounds of the Shader Model 2.

How to program shaders?

There are several ways to write and compile shader programs. Nowadays, the most common method used in the Direct3D and Windows world is a language called High Level Shading Language (HLSL). The Direct3D shader compiler fxc.exe compiles the HLSL code into byte-code, which is then executed by the runtime.

HLSL is a C-style language with some special data types and intrinsic functions, but without pointers. If you know how to write code in a C-style language like C#, you will quickly learn how to write a shader with HLSL. By the way, if you know HLSL, you also know NVIDIA's shading language Cg. Cg and HLSL have the same root and are very similar.

HLSL defines scalar and various vector / matrix data types for integer and floating point operations. The built-in intrinsic functions support scalar and vector data types. Flow-control statements—such as if, switch, for, and while—are also possible. Of course, the curly brace is used for code blocks and most of the C operators are also supported. The compiler uses semantics to both determine the intended usage of a parameter and provide the right data for it. The register keyword is typically used to pass a parameter into the shader program. The elements of the vector types can be accessed through various aliases, such as x, y, z, w; r, g, b, a; u, v; etc. It's also possible to combine these and write nice and short statements by using swizzling:

HLSL

// Create 3D float vector a and b with different syntax
float3 a = float3(1, 2, 3);
float3 b = {5, 6, 7};

// Calculate cross product of vector a and b using swizzling
float3 crossProduct = a.yzx * b.zxy - a.zxy * b.yzx;

The MSDN is a great resource for HLSL and provides detailed explanations of the syntax and functions. The following examples will make it clearer and the explanations should help to get you started.

What tools to use?

Of course, it's possible to write a pixel shader with a simple text editor and compile it with the command line tool fxc.exe, but there's a great tool available that makes the process a lot easier. The Shazzam Tool by Walt Ritscher is THE utility for Silverlight and WPF shader development. It comes with an HLSL editor, which includes syntax highlighting, that compiles the shader and applies it right away to a sample input. It also generates controls for each parameter, which may be used to change the shader settings on the fly, and it creates the needed C# or VB source code file with a class that is derived from ShaderEffect.

Here's what you need to get started:

1. Download the DirectX SDK and install it.
2. Download the Shazzam Tool and install it.
3. After the Shazzam Tool is started, verify that the path to the DirectX FX compiler is set (Figure 2). The fxc.exe is normally located in the DirectX SDK installation folder under Utilities\bin\x86. Also make sure the right Target framework is selected and a Generated namespace is set.
4. To see if everything works, open a Sample Shader with the Shader Loader, select a sample tab page, and try the controls on the Change Shader Settings page (Figure 3).

Figure 2: Shazzam Tool Settings

Figure 3: Shazzam Tool Overview

How to write a pixel shader?

Now that we have the right tools installed and configured, we are ready to write the first pixel shader and the Silverlight application that will use it.

Example 1: The Tint Shader

The first pixel shader we're writing is a rather simple tint shader that converts the pixel into gray and tints it with a parameterized color.

In the Shazzam Tool, select File à New Shader File, choose a location for the HLSL FX file, and name it TintShader. Shazzam will automatically create the basic pixel shader code, including a float parameter SampleI. Hit the F5 key to compile and apply the shader to the selected sample image.

HLSL

sampler2D input : register(s0);

/// <summary>Explain the purpose of this variable.</summary>
/// <minValue>05/minValue>
/// <maxValue>10</maxValue>
/// <defaultValue>3.5</defaultValue>
float SampleI : register(C0);

float4 main(float2 uv : TEXCOORD) : COLOR 
{
    float4 Color; 
    Color = tex2D(input, uv.xy);
    return Color; 
}

The input register is the actual bitmap / texture that holds the pixels and is sampled inside the pixel shader. This pixel shader main function is the entry point and is executed for each pixel of the input bitmap. The coordinate of the current pixel that is processed is passed as the float2 parameter uv. This coordinate is normalized to the range [0, 1]. The color of the pixel at the passed uv coordinate is sampled as float4 with the built-in tex2D intrinsic function. A float4 COLOR value is expected as the return value of the pixel shader.

Figure 4: Output of the initial shader code (original image)

The initial pixel shader code returns the original color for each pixel and we use this as the starter for our gray scale conversion.

Gray Conversion

HLSL

sampler2D input : register(s0);

float4 main(float2 uv : TEXCOORD) : COLOR 
{
   // Sample the original color at the coordinate
   float4 color = tex2D(input, uv);
    
   // Convert the color to gray
  float gray = dot(color.rgb, float3(0.2126, 0.7152, 0.0722));
    
   // Return gray with the original alpha value
   return float4(gray, gray, gray, color.a); 
}

The original color is sampled and then converted to gray using the dot product of the red, green, and blue values with a constant float3 vector. The result actually represents the luminance of the pixel. The dot product multiplies the elements of the color vector with the elements of constant vector and adds the three products, thus resulting in a scalar float value. The return value of the pixel shader is a new color made up of the gray value for RGB and the original alpha (transparency) of the sampled pixel.

Figure 5: Output of the gray conversion shader

Parametric Tinting

The gray conversion shader can now be extended to tint the output in a color that is passed as a parameter through a shader register.

HLSL

/// <summary>The tint color.</summary>
/// <type>Color</type>
/// <defaultValue>0.9,0.7,0.3,1</defaultValue>
float4 TintColor : register(C0);

sampler2D Input : register(s0);

float4 main(float2 uv : TEXCOORD) : COLOR 
{
   // Sample the original color at the coordinate
   float4 color = tex2D(Input, uv);
    
   // Convert the color to gray
   float gray = dot(color.rgb, float3(0.2126, 0.7152, 0.0722)); 
    
   // Create the gray color with the original alpha value
   float4 grayColor = float4(gray, gray, gray, color.a); 
   
   // Return the tinted pixel
   return grayColor * TintColor;
}

The color used to tint every pixel is passed as a parameter and therefore defined as the first register C0 (the next parameter should then be in the register C1). The XML comment is used by Shazzam to both create convenient controls and initialize the generated code. Shazzam creates the appropriate controls for the data type and uses the defaultValue, minValue and maxValue (Figure 6). The changed value of the control is applied directly to the Sample image, which allows a quick and easy shader development.

Each element (RGBA) of the passed TintColor parameter is then multiplied with the float4 gray color and returned. The result of the default TintColor values is a sepia-toned image (Figure 7).

Figure 6: Shazzam Tool Shader Settings

Figure 7: Output of the tint shader

Wiring it together with Silverlight

Now it's time to use the shader in a Silverlight application and apply it to an Image, MediaElement, or whatever UIElement you like.

1. Start Visual Studio, create a new Silverlight Application project, and select at least Silverlight 3 as the target framework.
2. In the Shazzam Tool, click on Compile Shader in the Tools menu and then on Explore Compiled Shaders. Make sure the Generated namespace setting (Figure 2) matches the assembly name of the Silverlight application.
3. Copy the compiled shader file TintShader.ps and the corresponding C# or VB TintShaderEffect.cs|vb file from the GeneratedShaders folder and the CS|VB subfolder into the Silverlight project directory.
4. In Visual Studio, add the TintShaderEffect.cs|vb and the TintShader.ps file to the project. The property Build Action of the TintShader.ps file must be set to Resource. Rebuild the solution.
5. Open the MainPage.xaml file and add the namespace declaration and a Button or any other control that has the TintShaderEffect class applied (see below).
6. Hit the F5 key and see your Silverlight shader application in action (Figure 8).

XAML

<UserControl x:Class="ShaderDemoApp.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d"
    d:DesignHeight="300" d:DesignWidth="400"
    xmlns:local="clr-namespace:ShaderDemoApp">

    <Grid x:Name="LayoutRoot" Background="Gray">
        <Button Content="Big Tinted Button" Width="200" 
                Height="200" Background="Blue">
            <Button.Effect>
                <local:TintShaderEffect />
            </Button.Effect>
        </Button>
    </Grid>
</UserControl>

Figure 8: The TintShader applied to the Button

That's all that's needed to get a pixel shader working inside a Silverlight application. Please note that no manual C# or VB code-behind was written.

Example 2: The Mosaic Shader aka The Donut Shader

The second post process shader effect we'll write is a bit more advanced. It starts with pixelating the image before rounding the blocks until we've the final Mosaic-like result.

For the development of this shader, we can use a different image. To do so, click the Open Image File menu item in Shazzam's File menu. I used a famous test picture for image processing algorithms: Lenna. By the way, there's an interesting story behind this picture of Lena Söderberg.

Figure 9: Original Lenna test image

Pixelating the Input

HLSL

/// <summary>The number of pixel blocks.</summary>
/// <type>Single</type>
/// <defaultValue>25</defaultValue>
float BlockCount : register(C0);

sampler2D input : register(S0);

// Static computed vars for optimization
static float BlockSize = 1.0f / BlockCount; 

float4 main(float2 uv : TEXCOORD) : COLOR
{
   // Calculate block center
   float2 blockPos = floor(uv * BlockCount);
   float2 blockCenter = blockPos * BlockSize + BlockSize * 0.5;
            
   // Sample color at the calculated coordinate
   return tex2D(input, blockCenter);
}

The float BlockCount parameter defines into the number of blocks (large pixels) into which the resulting image will be divided (pixelated). The size of a block (BlockSize) is the inverse of the BlockCount and calculated as static float to save some clock cycles. The coordinate of the current pixel (uv) is then used to determine the block to which it belongs. This determination depends on the BlockCount and is a result of the built-in floor function. To get the color of the output pixel, the center coordinate of each block is sampled for all the pixels that are part of the block.

Figure 10: Output of the pixelation step

Rounding the Pixels

HLSL

/// <summary>The number of pixel blocks.</summary>
/// <type>Single</type>
/// <defaultValue>25</defaultValue>
float BlockCount : register(C0);

/// <summary>The rounding of a pixel block.</summary>
/// <type>Single</type>
/// <defaultValue>0.45</defaultValue>
float Max : register(C2);

sampler2D input : register(S0);

// Static computed vars for optimization
static float BlockSize = 1.0f / BlockCount; 

float4 main(float2 uv : TEXCOORD) : COLOR
{
   // Calculate block center
   float2 blockPos = floor(uv * BlockCount);
   float2 blockCenter = blockPos * BlockSize + BlockSize * 0.5;
        
   // Round the block by testing the distance 
   // of the pixel coordinate to the center
   float dist = length(uv - blockCenter) * BlockCount;
   if(dist > Max)
   {
      return 0;
   }
    
   // Sample color at the calculated coordinate
   return tex2D(input, blockCenter);
}

The Max parameter defines the maximum distance of a pixel to its block center and, therefore, the rounding of a pixel block. If the length of the vector between the current pixel coordinate (uv) and its block center is greater than the Max parameter, a transparent pixel (0) is returned. The built-in length function is used to calculate the scalar length of the distance vector.

Figure 11: Output of the rounding step (Max = 0.45)

Figure 12: Output of the rounding step (Max = 0.60)

Baking Donuts

HLSL

/// <summary>The number of pixel blocks.</summary>
/// <type>Single</type>
/// <defaultValue>25</defaultValue>
float BlockCount : register(C0);

/// <summary>The rounding of a pixel block.</summary>
/// <type>Single</type>
/// <defaultValue>0.2</defaultValue>
float Min : register(C1);

/// <summary>The rounding of a pixel block.</summary>
/// <type>Single</type>
/// <defaultValue>0.45</defaultValue>
float Max : register(C2);

sampler2D input : register(S0);

// Static computed vars for optimization
static float BlockSize = 1.0f / BlockCount; 

float4 main(float2 uv : TEXCOORD) : COLOR
{
   // Calculate block center
   float2 blockPos = floor(uv * BlockCount);
   float2 blockCenter = blockPos * BlockSize + BlockSize * 0.5;
        
   // Round the block by testing the distance 
   // of the pixel coordinate to the center
   float dist = length(uv - blockCenter) * BlockCount;
   if(dist < Min || dist > Max)
   {
      return 0;
   }
    
   // Sample color at the calculated coordinate
   return tex2D(input, blockCenter);
} 

The last thing left to do in order to get some nice rings (donuts) is to add a test for the minimum distance. This is pretty easy and done with the additional Min parameter.

Figure 13: Output of the ring step (Min = 0.20, Max = 0.45)

Please note that all of the parameters can be animated, which could result in a nice (transition) effect when two images are overlaid. The Shazzam Tool also supports animation with the generated Shader Settings controls.

How does the Demo Application work?

The Silverlight demo application is quite flexible and can be used for many different shader effects without touching the core functionality. This last part of the article will show how this extensibility was achieved. Explanations of how to use the webcam with Silverlight and how to load an image from disk were detailed in my last Silverlight Face Detection article.

The application's extensibility was mainly accomplished by using the Managed Extensibility Framework (MEF) and a View-ViewModel approach for the shader parameters. MEF is a great way to make decoupled and flexible applications and has been part of the Silverlight framework since version 4. MEF is like Meth for .Net and Silverlight developers, but without all the undesirable side effects.

The beauty of MEF is best illustrated using source code. As you can see in the demo application (Figure 1), it's possible to select a pixel shader with a ComboBox. The items in the MainPage's shader ComboBox are populated through data binding an ObservableCollection <T> and MEF is used to build this collection.

C#

/// <summary>
/// The main Page of the application.
/// </summary>
public partial class MainPage : UserControl
{
   [ImportMany(AllowRecomposition = true)]
   public ObservableCollection<IShaderViewModel> Shaders;

   public MainPage()
   {
      InitializeComponent();
   }

   private void Initialize()
   {
      // Compose the parts with MEF
      var container = new CompositionContainer(
            new AssemblyCatalog(GetType().Assembly));
      container.ComposeParts(this);

      // Fill ComboBox
      CmbShaders.ItemsSource = Shaders;
      CmbShaders.DisplayMemberPath = "Name";
      CmbShaders.SelectedIndex = 0;

      // ...
   }

   // ...
}

/// <summary>
/// Interface of a ViewModel for a shader effect.
/// </summary>
[InheritedExport]
public interface IShaderViewModel
{
   string Name { get; }
   ShaderEffect Shader { get; }
   UserControl View { get; }
}

The MainPage has a Shaders collection property containing items that implement the IShaderViewModel interface. This collection is initialized by using MEF's CompositionContainer with an AssemblyCatalog and the ComposeParts method. ComposeParts analyses all types in the provided Catalog (here, the assembly), checks if they have certain attributes attached, and wires these so-called parts together. To put it simply, an instance of a type decorated with an Export attribute is created and assigned to each field / property / parameter of the type that has an Import attribute attached.

The IShaderViewModel interface has the InheritedExport attribute attached, which means that implementations of this interface will automatically provide that export. The Shaders collection in turn uses the ImportMany attribute, telling MEF to populate the list with all matching exports (here, all classes that implement the IShaderViewModel interface).

After MEF fills the collection, it is data bound to the ItemsSource property of the ComboBox. The Name property of the IShaderViewModel interface is used as DisplayMember.

The advantage of MEF should be pretty obvious here: you have to implement an IShaderViewModel and maybe a View UserControl for the shader's parameters, but you don't need to add an instance of the shader ViewModel to the Shaders collection manually. Since the IShaderViewModel has the InheritedExport attribute attached, it's not even necessary to add a special MEF attribute to the new shader ViewModel type. Additionally, it's possible to load an extra assembly asynchronously and let MEF compose the parts and update the collection afterwards. By setting the AllowRecomposition parameter of the ImportMany attribute to true we're asking MEF to allow dynamic updates.

As you can see, MEF is pretty easy but also very powerful. But we have just scratched the surface—this blog post and the MSDN are good starters if you want to learn more about MEF.

Further Resources

There are many great HLSL shader development resources available on the web. The following list contains some sites focused on Silverlight / WPF pixel shaders. If I forgot a great resource, please post it in a comment.

• The open source WPF Pixel Shader Effects Library at CodePlex contains a lot of common pixel shader effects.
• Nikola Mihaylov (aka Nokola) wrote a great online image editor in Silverlight that uses pixel shaders for the effects. He released the pixel shaders that are used in EasyPainter as open source. And he also released the custom controls from his tool. The Slider and the ColorPicker used in the demo application are from there. I just changed some properties to Dependency Properties to make the controls bindable. So Kudos to Nikola!
• Walt Ritscher's amazing Shazzam Tool also comes with many cool pixel shaders as samples. Kudos to Walt for making such a great tool!
• Additionally, from time to time I write some shaders and put them up on my blog. Now that we have an extensible Silverlight shader demo application, I will surely integrate my upcoming shaders directly into it.

While I was writing this article, an open source project called WPF Meta-Effects was released on CodePlex. The WPF Meta-Effects framework makes it possible to write Shaders for WPF in C# by using attributes, delegates, and dynamic HLSL compilation. It's a neat idea, but due to its dynamic compilation of shaders it's limited to WPF and can't be used with Silverlight. Additionally, I much prefer the simplicity of HLSL to C# when writing shaders. There's a reason why HLSL was invented: see the examples yourself.

Conclusion

This article explained what shaders and HLSL are, as well as how to write pixel shaders for Silverlight and WPF. It also showed what tools / frameworks to use for the best developer experience and gave an introduction on how to write an extensible application with MEF.

I hope it diminished any fear of the HLSL language and the shader development. I also hope it both inspires and helps you to write your own shaders. I'm eager to see what you come up with. Have fun!

The links to the live demo app and source code are at the top of the article.

About The Author

René Schulte is a .Net / Silverlight developer and Microsoft Silverlight MVP passionate about real-time computer graphics, physics, AI, and algorithms. He loves C#, Shaders, Augmented Reality, and computer vision. He started the SLARToolkit, the WriteableBitmapEx, and the Matrix3DEx Silverlight open source projects, and he has a Silverlight website powered by real time soft body physics. He is also a regular author for Microsoft's Coding4Fun. Contact information can be found on his Silverlight website, his blog, or via Twitter.

Optional Reading

I cover the basics of this article in a multi-part blog series, which you should check out if you have trouble:

Part 1 - How Audio Data is Represented
Part 2 - Demystifying the WAV Format
Part 3 - Synthesizing Simple WAV Audio Using C#
Part 4 - Algorithms for Different Sound Waves in C#

What's An Oscillator?

An oscillator is a device or application that generates a waveform. In electrical engineering terms, it's a device that outputs an electrical current with varying voltage. If you plot the voltage over time, you get a regular wave in a particular form, such as a sine, square, triangle or sawtooth.

An oscillator is the most basic type of synthesizer. Analog synths use electrical circuits to output a sound wave. Digital synthesizers do the same thing, but with software.

You can create a pretty neat sounding instrument by combining the outputs of multiple oscillators. For example, if you have three oscillators oscillating at a frequency of 440Hz (concert A pitch), but each of them has a different waveform (saw, square, sine) you get a very interesting, layered sound.

But before we get too deep into this subject, let's briefly explore the physics of sound.

The Physics of Sound

Sound happens when air pressure changes on your ear drum. When you clap in an empty room, pressure waves bounce all over the place and dance on your eardrum. The changes in pressure are detected continuously by your ear.

Digitally, “pressure” is referred to by a scalar value called amplitude. The amplitude (loudness) of the wave is measured thousands of times per second (44,100 times per second on CDs). Every measurement of pressure (aka amplitude) is called a sample—CDs are recorded with 44,100 samples per second, each with a value between the minimum and maximum amplitude for the bit depth.

Think about 44,100 samples per second. That's a lot of stuff for your ear to detect. That's how we're able to hear so much stuff going on in the mix of a song, especially in stereo tracks where you have 44,100 samples per second, per ear.

It turns out that there is a horribly intense mathematical theorem which basically tells us that 44,100 samples per second is enough to accurately represent a pitch as high as 22 KHz. The human ear can really hear only up to 20KHz, so a 44.1KHz sampling rate is a more than high-enough sampling rate.

This whole section is expanded in detail on my blog:

Part 1 - How Audio Data is Represented

Terminology

So now you have a rather glancing overview of how sound works, and perhaps some clues as to how we should go about representing it in computers. Let's go over all this new terminology (plus some even newer terms) in delicious, bulleted format:

· Sample: A measurement of a sound wave at a very small point in time. 44,100 of these measurements in a row form a single channel of CD-quality audio.

· Amplitude: The value of a sample. Max and min values are dependent upon the bit depth.

· Bit depth: The number of bits used to represent a sample. 16-bit, 32-bit, etc. Max amplitude is (2^depth) / 2 – 1.

· Sample rate (aka sampling rate, aka bit rate): The number of samples per second of audio. 44,100 is standard for CD-quality audio.

How sound is represented

By now, you've probably surmised that a second of audio data is somehow represented by an array of some integer data type, which has a length of 44,100. You would be correct in that assumption. However, if you want sound to play from a computer's sound card, that data has to be accompanied with a bunch of format information. WAV is probably the easiest format to deal with.

See more in the following article:

Part 2 - Demystifying the WAV Format

You can also see how to build out a WAV file, old school and binary style, in the 3^rd part of that series:

Part 3 - Synthesizing Simple WAV Audio Using C#

However, we are taking a slightly easier route, by using DirectSound. DirectSound gives us a lot of nice classes for all the format information, abstracting all that stuff away and allowing us to pump a stream of data into a DirectSound object and play it. Perfect for a synthesizer app!

So, let's get started!

Building the app

I learned some Blend while working with this app, since it's built on WPF. The image buttons are just radio buttons. I had to differentiate the group number per instance of the user control at runtime (in the constructor of the Oscillator class).

I'm a terrible UI designer for the most part, so this is about as sexy as I'm willing to make this application. But feel free to make it look and act better!

Designing the UI

There's a dirty little secret in this application. It says it can oscillate 3 waves, but in truth, there's a constant (set to 3) that you can modify. You could have six if you wanted. How did I accomplish this? Each synth that you see is an instance of a WPF user control called Oscillator.xaml:

I have a StackPanel called Oscs in the main window. In the Window_Loaded event handler of the main window, I use this bit of code to add instances of the usercontrol:

C#

// Add 3 oscillators
Oscillator tmp;
for (int i = 0; i < NUM_GENERATORS; i++)
{
    tmp = new Oscillator();
    Oscs.Children.Add(tmp);
    mixer.Oscillators.Add(tmp);
}

VB

' Add 3 oscillators
Dim tmp As Oscillator
Dim i As Integer = 0
While i < NUM_GENERATORS
    tmp = New Oscillator()
    Oscs.Children.Add(tmp)
    mixer.Oscillators.Add(tmp)
    System.Math.Max(System.Threading.Interlocked.Increment(i),i - 1)
End While

The long rectangular canvas is used to plot the values of the generated wave, so you can visualize the wave as it's played. It is scaled along the X axis so you can see the general shape of the wave, which would be impossible without scaling it with 44,100 samples per second.

Earlier in the article, I noted that a sound file is basically a really, really long array of 16- or 32-bit floating point numbers between -1 and 1. We use this data to plot the graph as well. More on that later.

Now that we have the UI figured out (dynamic addition of oscillators), let's take a look at exactly how the sound is produced.

Bzzzzt! Making Sounds and the Mixer

One of the many cool things about DirectSound is that it basically wraps the WAV format for you. You set the buffering/format options and then shove a bunch of data into it, and it will play. Magic.

The way I've architected the solution is a little more modular. None of the oscillators has the ability to play itself—rather, uses its UI to control some values such as frequency, amplitude and wave type. These values are tied to public properties. The Oscillator component does virtually no audio work at all.

The generation of audio data is handled by the custom Mixer class, which takes a collection of Oscillators and, based on their properties, creates a composite of all the generators. This is done by averaging the samples in every oscillator and putting them into a new array of data.

The Mixer class looks like this:

One of the workhorses of the Mixer class is the method GenerateOscillatorSampleData. This takes an Oscillator as an argument to give access to the public properties set in the UI. From there, the algorithm generates 1 second of sample data (specified by the member bufferDurationSeconds) based on the wave type that has been selected in the UI. This is where the mathy stuff comes in to play. Check out this method and the different cases in the switch statement that determine what kind of wave to create below.

C#

public short[] GenerateOscillatorSampleData(Oscillator osc)
{
    // Creates a looping buffer based on the params given
    // Fill the buffer with whatever waveform at the specified frequency            
    int numSamples = Convert.ToInt32(bufferDurationSeconds * 
        waveFormat.SamplesPerSecond);
    short[] sampleData = new short[numSamples];
    double frequency = osc.Frequency;
    int amplitude = osc.Amplitude;
    double angle = (Math.PI * 2 * frequency) / 
        (waveFormat.SamplesPerSecond * waveFormat.Channels);            

    switch (osc.WaveType)
    {
        case WaveType.Sine:
            {
                for (int i = 0; i < numSamples; i++)
                    // Generate a sine wave in both channels.
                    sampleData[i] = Convert.ToInt16(amplitude * 
        Math.Sin(angle * i));
            }
            break;
        case WaveType.Square:
            {
                for (int i = 0; i < numSamples; i++)
                {
                    // Generate a square wave in both channels.
                    if (Math.Sin(angle * i) > 0)
                        sampleData[i] = Convert.ToInt16(amplitude);
                    else
                        sampleData[i] = Convert.ToInt16(-amplitude);
                }
            }
            break;
        case WaveType.Sawtooth:
            {
                int samplesPerPeriod = Convert.ToInt32(
        waveFormat.SamplesPerSecond / 
        (frequency / waveFormat.Channels));
                short sampleStep = Convert.ToInt16(
        (amplitude * 2) / samplesPerPeriod);
                short tempSample = 0;

                int i = 0;
                int totalSamplesWritten = 0;
                while (totalSamplesWritten < numSamples)
                {
                    tempSample = (short)-amplitude;
                    for (i = 0; i < samplesPerPeriod && 
        totalSamplesWritten < numSamples; i++)
                    {
                        tempSample += sampleStep;
                        sampleData[totalSamplesWritten] = tempSample;

                        totalSamplesWritten++;
                    }
                }
            }
            break;
        case WaveType.Noise:
            {
                Random rnd = new Random();
                for (int i = 0; i < numSamples; i++)
                {
                    sampleData[i] = Convert.ToInt16(
        rnd.Next(-amplitude, amplitude));
                }
            }
            break;
    }
    return sampleData;
}

VB.Net

Public Function GenerateOscillatorSampleData(ByVal osc As Oscillator) As Short()
    ' Creates a looping buffer based on the params given
    ' Fill the buffer with whatever waveform at the specified frequency 
    Dim numSamples As Integer = Convert.ToInt32(
     bufferDurationSeconds * waveFormat.SamplesPerSecond)
    Dim sampleData As Short() = New Short(numSamples - 1) {}
    Dim frequency As Double = osc.Frequency
    Dim amplitude As Integer = osc.Amplitude
    Dim angle As Double = (Math.PI * 2 * frequency) /
     (waveFormat.SamplesPerSecond * waveFormat.Channels)

    Select Case osc.WaveType
        Case WaveType.Sine
            If True Then
                For i As Integer = 0 To numSamples - 1
                    ' Generate a sine wave in both channels.
                    sampleData(i) =
                     Convert.ToInt16(amplitude * Math.Sin(angle * i))
                Next
            End If
            Exit Select
        Case WaveType.Square
            If True Then
                For i As Integer = 0 To numSamples - 1
                    ' Generate a square wave in both channels.
                    If Math.Sin(angle * i) > 0 Then
                        sampleData(i) = Convert.ToInt16(amplitude)
                    Else
                        sampleData(i) = Convert.ToInt16(-amplitude)
                    End If
                Next
            End If
            Exit Select
        Case WaveType.Sawtooth
            If True Then
                Dim samplesPerPeriod As Integer =
                 Convert.ToInt32(waveFormat.SamplesPerSecond /
                      (frequency / waveFormat.Channels))
                Dim sampleStep As Short =
                 Convert.ToInt16((amplitude * 2) / samplesPerPeriod)
                Dim tempSample As Short = 0

                Dim i As Integer = 0
                Dim totalSamplesWritten As Integer = 0
                While totalSamplesWritten < numSamples
                    tempSample = CShort(-amplitude)
                    i = 0
                    While i < samplesPerPeriod AndAlso totalSamplesWritten < numSamples
                        tempSample += sampleStep
                        sampleData(totalSamplesWritten) = tempSample

                        totalSamplesWritten += 1
                        i += 1
                    End While
                End While
            End If
            Exit Select
        Case WaveType.Noise
            If True Then
                Dim rnd As New Random()
                For i As Integer = 0 To numSamples - 1
                    sampleData(i) = Convert.ToInt16(
                     rnd.[Next](-amplitude, amplitude))
                Next
            End If
            Exit Select
    End Select
    Return sampleData
End Function

The Mixer is the heart of the app, and it's a beautiful example of object orientation and cohesion. Give it three things (oscillators) and it spits out a new thing you can use (an array of sample data).

Now that we have the sample data, all we have to do is play it back using DirectSound.

Sound Playback with DirectSound

As I mentioned, DirectSound provides a wrapper over the WAV format. You set up your buffer and format information and then feed it a bunch of data in the form of an array of shorts (arrays of trousers are known to cause errors).

First, we initialize the format information and buffer in the Window_Loaded event handler of the main form. The values below are not really arbitrary; there is an explanation of them in the Optional Reading section above (see Demystifying the WAV Format). This code also contains the code to add the oscillators, as shown earlier in the article.

C#

private void Window_Loaded(object sender, System.Windows.RoutedEventArgs e)
{
    WindowInteropHelper helper = 
        new WindowInteropHelper(Application.Current.MainWindow);
    device.SetCooperativeLevel(helper.Handle, CooperativeLevel.Normal);

    waveFormat = new Microsoft.DirectX.DirectSound.WaveFormat();
    waveFormat.SamplesPerSecond = 44100;
    waveFormat.Channels = 2;
    waveFormat.FormatTag = WaveFormatTag.Pcm;
    waveFormat.BitsPerSample = 16;
    waveFormat.BlockAlign = 4;
    waveFormat.AverageBytesPerSecond = 176400;

    bufferDesc = new BufferDescription(waveFormat);
    bufferDesc.DeferLocation = true;
    bufferDesc.BufferBytes = Convert.ToInt32(
        bufferDurationSeconds * waveFormat.AverageBytesPerSecond / waveFormat.Channels);

    // Add 3 oscillators
    Oscillator tmp;
    for (int i = 0; i < NUM_GENERATORS; i++)
    {
        tmp = new Oscillator();
        Oscs.Children.Add(tmp);
        mixer.Oscillators.Add(tmp);
    }            
}

VB

Private Sub Window_Loaded(ByVal sender As Object, ByVal e As System.Windows.RoutedEventArgs)
    Dim helper As New WindowInteropHelper(Application.Current.MainWindow)
    device.SetCooperativeLevel(helper.Handle, CooperativeLevel.Normal)

    waveFormat = New Microsoft.DirectX.DirectSound.WaveFormat()
    waveFormat.SamplesPerSecond = 44100
    waveFormat.Channels = 2
    waveFormat.FormatTag = WaveFormatTag.Pcm
    waveFormat.BitsPerSample = 16
    waveFormat.BlockAlign = 4
    waveFormat.AverageBytesPerSecond = 176400

    bufferDesc = New BufferDescription(waveFormat)
    bufferDesc.DeferLocation = True
    bufferDesc.BufferBytes = Convert.ToInt32(
     bufferDurationSeconds * waveFormat.AverageBytesPerSecond / waveFormat.Channels)

    ' Add 3 oscillators
    Dim tmp As Oscillator
    For i As Integer = 0 To NUM_GENERATORS - 1
        tmp = New Oscillator()
        Oscs.Children.Add(tmp)
        mixer.Oscillators.Add(tmp)
    Next
End Sub

When you click the Play button, the application takes its collection of oscillators and passes the values of the UI controls to the Mixer (which is initialized on each click with a reference to the main form window, so it can grab the Oscillator user controls).

The mixer outputs an array of shorts, which we write to a DirectSound buffer.

Here is the code for the Play button's click event handler:

C#

private void btnPlay_Click(object sender, System.Windows.RoutedEventArgs e)
{                                    
    mixer.Initialize(Application.Current.MainWindow);

    short[] sampleData = mixer.MixToStream();
    buffer = new SecondaryBuffer(bufferDesc, device);            
    buffer.Write(0, sampleData, LockFlag.EntireBuffer);
    buffer.Play(0, BufferPlayFlags.Default);            

    GraphWaveform(sampleData);
}

VB

Private Sub btnPlay_Click(sender As Object, e As System.Windows.RoutedEventArgs)
    mixer.Initialize(Application.Current.MainWindow)

    Dim sampleData As Short() = mixer.MixToStream()
    buffer = New SecondaryBuffer(bufferDesc, device)
    buffer.Write(0, sampleData, LockFlag.EntireBuffer)
    buffer.Play(0, BufferPlayFlags.[Default])

    GraphWaveform(sampleData)
End Sub

Drawing Pretty Graphs

All that's left is to draw the graph of the waveform on the canvas. Below is the GraphWaveform method. This method could graph anything it wanted to, as long as it was an array of shorts (not trousers). It's reminiscent of trying to graph things using Flash back in the day, when you had to actually figure out points and lines (most likely on paper), but WPF's Polyline object makes this rather trivial.

C#

private void GraphWaveform(short[] data)
{
    cvDrawingArea.Children.Clear();

    double canvasHeight = cvDrawingArea.Height;
    double canvasWidth = cvDrawingArea.Width;

    int observablePoints = 1800;
            
    double xScale = canvasWidth / observablePoints;
    double yScale = (canvasHeight / 
        (double)(amplitude * 2)) * ((double)amplitude / MAX_AMPLITUDE);            

    Polyline graphLine = new Polyline();
    graphLine.Stroke = Brushes.Black;
    graphLine.StrokeThickness = 1;

    for (int i = 0; i < observablePoints; i++)
    {
        graphLine.Points.Add(
            new Point(i * xScale, (canvasHeight / 2) - (data[i] * yScale) ));
    }

    cvDrawingArea.Children.Add(graphLine);            
}

VB

Private Sub GraphWaveform(ByVal data As Short())
    cvDrawingArea.Children.Clear()

    Dim canvasHeight As Double = cvDrawingArea.Height
    Dim canvasWidth As Double = cvDrawingArea.Width

    Dim observablePoints As Integer = 1800

    Dim xScale As Double = canvasWidth / observablePoints
    Dim yScale As Double = (canvasHeight / CDbl((amplitude * 2))) * (CDbl(amplitude) / MAX_AMPLITUDE)

    Dim graphLine As New Polyline()
    graphLine.Stroke = Brushes.Black
    graphLine.StrokeThickness = 1

    For i As Integer = 0 To observablePoints - 1
        graphLine.Points.Add(
         New Point(i * xScale, (canvasHeight / 2) - (data(i) * yScale)))
    Next

    cvDrawingArea.Children.Add(graphLine)
End Sub

Conclusion

This was a really fun little project that took way less time to code than it does to explain. It's a great exercise because it requires you to think about an ancillary field of science before you can sit down and code, which is really what coding for fun's all about, anyway!

If you want to try this out, the download link for the source code is at the top of the article.

About The Author

Dan Waters is an Academic Evangelist at Microsoft, covering schools in the Pacific Northwest, Alaska, and Hawaii. He is based in Bellevue, WA. Dan has way too many guitars at home and tries to entice both of his young daughters to learn how to play them. Music, technology, and music+technology are among his favorite hobbies, along with snowboarding and trying to maintain cool dad status. You can find his blog at www.danwaters.com or follow him on Twitter at www.twitter.com/danwaters.

Introduction

Back in November of 2009, I was met with an interesting challenge. Those of you who frequent the XNA Community Forums have probably heard of Nick Gravelyn. He was an XNA MVP, and now he works for the XNA team. He started a little contest called xna7day, which was designed to challenge developers to make a game in 7 days using a pre-defined theme.

The theme in November was “Text Based” and its rule was that you were only allowed to use text in the visuals of your game. I saw some of the cool stuff other developers were working on, including one that looked like it had text characters that could walk and talk, like people. But my mind immediately went to a different solution: I wanted to make a game that was fully 3D (effectively breaking the rule of rendering text), but then post-process the render into ASCII art, so it would look like it was text.

In this article, I'm going to show you how to do just that:

Setup

Alright, let's dive right in. Open up the base project. It should compile and run as is. It's a simple, if ugly, first-person shooter. Play through the first level. Go ahead, I'll wait. Bonus points to anyone who recognizes the level design!

Okay, got a playthrough? Let's start! I separated this into three projects so the ASCII stuff can be extracted into other games. We'll be doing most of our work in the ASCII_Renderer project, so let's open that up and add a new class to the Source\ascii folder called “Renderer.” I use the namespace “ASCII3D” for all files in this project, so get rid of the extra folder sub-namespaces. Add these using statements, too:

C#

using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Content;
using Microsoft.Xna.Framework.Graphics; 

This is going to be a post-processing effect, so we'll try to make it portable. Add the following empty function declarations:

C#

public Renderer(ContentManager content, int xRes, int yRes)
{ }

public void StartScene()
{ }

public void EndScene()
{ }

The constructor takes three arguments: The ContentManager, which we will use to load in our font and post-processing shader, and the resolution variables, which will basically tell us how many lines of text to draw and how many characters in each line. Since we have these prototyped, we'll make our only edits to the ASCIIFPS project in the Source\Game1.cs file. But first, we need to define and initialize a new Renderer object. Add a new class variable:

C#

/// <summary>
/// This is what handles the ASCII conversion
/// </summary>
private Renderer renderer; 

And at the end of the LoadContent function:

C#

// and initialize our ASCII renderer
// the variables we pass make each character
// appx 10x14 which seems to be the smallest
// readable size
renderer = new Renderer(Content, Global.ScreenWidth/10,Global.ScreenHeight/14);

Finally, call the BeginScene and EndScene functions. Modify Game1's Draw function to look like this:

C#

protected override void Draw(GameTime gameTime)
{
    // notice we only wrap StartScene and
    // EndScene if we are UsingASCII
    if (UsingASCII)
        renderer.StartScene();

    currentState.Draw(gameTime);

    if (UsingASCII)
        renderer.EndScene();

    base.Draw(gameTime);
}

The Boolean variable UsingASCII is defined for us. It defaults to false and can be toggled any time in-game by pressing the [M] key. If you want to default it to true for testing purposes, just change its value in Game1's Initialize function. At this point, the Renderer is all hooked up, so changes will go right in.

I skimmed over the other classes earlier in the article, but I need to touch on a few of them. Almost everything in GlobalModules is fairly standard utility stuff. We'll be using it, so you might want to glance at the Global class. The ASCIIFPS has the FPS engine in it and is almost entirely irrelevant to this article, except that it offers a fun test-bed for our effect. Feel free to go through the code, but I'll warn you: I wrote it for a 7-day contest, so some of it looks like gibberish and is only incidentally functional. Finally, we have the ASCII_Renderer project. This already has two files in it: BitmapFontGenerator and Letter. The latter are for generating a bitmap font map that we'll use in a bit.

The functionality isn't specific to this article, since any artist could easily make this. But I'm no artist, which is why I had the computer do it for me. Basically, BitmapFontGenerator has a static function that will take a texture of letters (see the Content project in ASCII_Renderer), trim them, sort them by how much they fill their space, and print them into a new Texture for our functions. Feel free to read through the code for this process; I commented it pretty well.

Post-Processing Basics

We need to cover the basics of post-processing before we start coding the next part. When post-processing a scene, you first render the entire scene off-screen on a RenderTarget. Then, you can apply any effects you want to the resolved texture before putting it on the screen. That's it. Here's how in XNA:

C#

public class Renderer
{
    private RenderTarget2D SrcImage;

    public Renderer(ContentManager content, int xRes, int yRes)
    {
        SrcImage = new RenderTarget2D(Global.Graphics, 
            xRes, yRes, 1, SurfaceFormat.Color);
    }

    public void StartScene()
    {
        Global.Graphics.SetRenderTarget(0, SrcImage);
        Global.Graphics.Clear(Color.Black);
    }

    public void EndScene()
    {
        Global.Graphics.SetRenderTarget(0, null);
        Global.Graphics.Clear(Color.Black);

        // TODO: draw the texture from the RenderTarget
    }
}

We create the RenderTarget2D we need in the constructor. Notice that the size of the RenderTarget is the same as the number of ASCII characters we plan to draw on screen. I'll explain that in the next section. In the StartScene function, we set the RenderTarget on the device so that all draw calls will actually draw to it. Then we clear the buffer, because RenderTargets' content from frame to frame is undefined. In EndScene, the RenderTarget is set to null, which is DirectX's way of saying “use the screen's framebuffer.” Now we need to actually draw the rendered image. XNA requires shaders for all draw calls, even if they're veiled by BasicEffect and SpriteBatch. We need to create a very simple shader to draw the texture to screen. Create a new Effect file in ASCII_Renderer's Content project and name it TextEffect. We'll start it out with this:

HLSL

texture2D sourceTex;
sampler2D SourceTextureSampler = sampler_state
{
    Texture = <sourceTex>;
    MinFilter = point;
    MagFilter = point;
    MipFilter = point;
    AddressU = wrap;
    AddressV = wrap;
};

struct VertexShaderInput
{
    float4 Position : POSITION0;
    float2 TexCoords : TEXCOORD0;
};

struct VertexShaderOutput
{
    float4 Position : POSITION0;
    float2 TexCoords : TEXCOORD0;
};

VertexShaderOutput VertexShaderFunction(VertexShaderInput input)
{
    VertexShaderOutput output;
    
    output.Position = input.Position;
    output.TexCoords = input.TexCoords;

    return output;
}

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    float4 sourceColor = tex2D(SourceTextureSampler,input.TexCoords);
    
    return sourceColor;
}

technique Technique1
{
    pass Pass1
    {
        VertexShader = compile vs_2_0 VertexShaderFunction();
        PixelShader = compile ps_2_0 PixelShaderFunction();
    }
}

There are a couple things I want to note about the base shader. First, notice that we use point filtering. I'll explain why in the next section, but it's important to see where we did it. Next, notice that we took out all the default transformation matrices. We're going to be drawing a single full screen quad, so we can just pass in the coordinates in screen-space—no processing required. Now we need to make a few changes to the Renderer class to use this Effect and draw the rendered game. First, we need to add some new class variables:

C#

private Effect effect;

private VertexPositionTexture[] verts;
private VertexDeclaration vd;

The vertices are going to be for a simple quad that takes up the whole screen. They should be initialized in the LoadContent function like this:

C#

effect = content.Load<Effect>("TextEffect");

verts = new VertexPositionTexture[6];
verts[0] = new VertexPositionTexture(
    new Vector3(-1, -1, 0), new Vector2(0, 1));
verts[1] = new VertexPositionTexture(
    new Vector3(1, -1, 0), new Vector2(1, 1));
verts[2] = new VertexPositionTexture(
    new Vector3(-1, 1, 0), new Vector2(0, 0));
verts[3] = verts[1];
verts[4] = verts[2];
verts[5] = new VertexPositionTexture(
    new Vector3(1, 1, 0), new Vector2(1, 0));

vd = new VertexDeclaration(
    Global.Graphics, VertexPositionTexture.VertexElements);

And finally, we add our draw code to the Draw function:

C#

effect.Begin();
effect.CurrentTechnique.Passes[0].Begin();

effect.Parameters["sourceTex"].SetValue(SrcImage.GetTexture());
effect.CommitChanges();
Global.Graphics.RenderState.CullMode = CullMode.None;

Global.Graphics.VertexDeclaration = vd;
Global.Graphics.DrawUserPrimitives<VertexPositionTexture>
     (PrimitiveType.TriangleList, verts, 0, 2);

effect.CurrentTechnique.Passes[0].End();
effect.End();

We set the texture we got from the RenderTarget, turn off culling (what would be culled in this geometry?), and draw it. Run the project. You should be able to toggle a difference in-game with the [M] key.

Drawing Some Text

“Okay,” you say, “it runs at a lower resolution with the [M] key. So what?” Remember what we set the resolution to in the RenderTarget? Yes, every pixel on this low-resolution version will be a character in the final image. How do we turn these big pixels into characters? First we need to define a texture bitmap font. Let's add a Texture2D class variable to Renderer and a const int for the BitmapFontGenerator:

C#

public Texture2D text; 
private const int NUM_SLOTS = 256;

Then we'll use the BitmapFontGenerator to construct this texture:

C#

text = BitmapFontGenerator.GenerateTextTexture(content, NUM_SLOTS);

Okay, that's all for now. Let's go back to the TextEffect.fx. We need to add a few variables:

HLSL

float2 dstLetterSize;

int numLetters;

texture2D textTex;
sampler2D TextTextureSampler = sampler_state
{
    Texture = <textTex>;
    MinFilter = point;
    MagFilter = point;
    MipFilter = point;
    AddressU = wrap;
    AddressV = wrap;
};

The variable “dstLetterSize” will tell us how big the letters on the screen are. The “numLetters” variable will tell us how many letters are in the font texture. And then there's the font texture itself. Here's how we draw actual characters in the Pixel Shader:

HLSL

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
     float4 sourceColor = tex2D(SourceTextureSampler,input.TexCoords);
    
     float2 texCoords = float2(
          input.TexCoords.x-((int)
               (input.TexCoords.x/dstLetterSize.x)*dstLetterSize.x),
          input.TexCoords.y-((int)
               (input.TexCoords.y/dstLetterSize.y)*dstLetterSize.y));
     texCoords /= dstLetterSize;
     texCoords.x /= numLetters;
    
     float4 letterColor = tex2D(TextTextureSampler,texCoords);
    
     return letterColor;
}

We've already seen the source color line. The next line will get the offset into the letter in letter size by subtracting the rounded-down (or integer-casted) texture coordinates scaled down. The line after that scales it back up to a 0-1 range. So now we have the new variable texCoords that is filled with the texture coordinates of the letter in 0-1 terms. We divide the x by numLetters because our texture is one long horizontal line of letters. Finally we get the letter color with these new texture coordinates. This letter color is grayscale, and when multiplied by the source color, we've assured that each letter is rendered in its own slot with the color from the original 3D render. The last thing we have to do is set those effect variables we defined from Renderer's EndScene function. Where we once just set the source texture, we now set these:

C#

effect.Parameters["dstLetterSize"].SetValue(
     new Vector2(1.0f/(float)SrcImage.Width, 1.0f/(float)SrcImage.Height));
effect.Parameters["numLetters"].SetValue(NUM_SLOTS);
effect.Parameters["sourceTex"].SetValue(SrcImage.GetTexture());
effect.Parameters["textTex"].SetValue(text);
effect.CommitChanges();

Most of these are self explanatory, but the dstLetterSize is a bit more complicated. To get the size of the letter in the 0-1 texture space, you take the total size (0 to 1) divided by the number of letters in that axis which, in our case, is the RenderTarget size.

Wheel of Fortune

‘M's aren't very interesting. In traditional ASCII art, the whole point is that you use characters that take more space or less space to handle shading. Let's add this code after the multiplication of the texCoords.x:

HLSL

float lum = (sourceColor.r+sourceColor.g+sourceColor.b)/3.0; 
float val = max(fullValueColor.r,
     max(fullValueColor.g,fullValueColor.b));
    
int ind = ((numLetters-1)-
     max(0,min((numLetters-1),(int)(lum*numLetters))));

texCoords.x += ind*(1.0/numLetters);

The first line gets the luminosity of the source color. The next gets the value. The next line gets the letter index from the luminosity. Subtracting it from (numLetters-1) reverses the index, because we want luminosity=1 to equate to index=0, because that has the letter that takes up the most space. Run the game. You'll see it now shows value, but it's still pretty hard to tell what's on the screen:

The last thing we can do is colorize it. You could just multiply it by the source color and get something like this…

…but everything looks too dark because the source color still has its value. Plus we're already practically multiplying it by choosing the letter. So we need to get the full value color first:

HLSL

float4 fullValueColor = sourceColor * (1.0/val);

We just multiply the original color by the inverse of the value. And then multiply that fullValueColor by the letter color in the return:

HLSL

return fullValueColor * letterColor;

That should produce a nice result:

Conclusion

And there it is! We have a real-time 3D render transformed into ASCII art. It's difficult to see and it's not very useful, but it's sure neat, eh? Why not add it as an unlockable easter egg in your next XBL Indie Game? I'm looking forward to seeing how people use this effect! If you use it, please send me an email with a screenshot!

About The Author

Louis Ingenthron is a Game Developer in Orlando, FL. He works on commercial Console and PC titles, but runs his own Indie Games company, FV Productions, on the side. He is best known for his open source XNA rhythm game Unsigned and specializes in real-time Graphics programming. He has been working with XNA since the 2.0 Beta and with .NET C# for just as long. He is also familiar with several other development languages, such as C, C++, and Java. Occasionally, he can even be found doing web development and Flash. Louis also writes for MSDN's Coding4Fun website, contributing articles on a monthly basis.

Archetype saw the trend around video and media related solutions moving to the cloud and took the initiative to build out their end-to-end solution.  They had a couple of projects where they had to be able to handle large scale very quickly.  In general they had trouble scaling up quickly and then back down once the project was over. Windows Azure gave them that capability in both their services and product businesses.

One of their products is a media content management system called Archetype Media Platform (AMP) that allows enterprises to control all their media assets.  Danny and Luigi spent some time here showing off some of the video content management and editing capabilities of the AMP solution running on Windows Azure.

Towards the end of the video Luigi shares some of his experiences porting to Windows Azure (first cut working in less than a week).  The solution architecture use web roles for the front end and web services as well as worker roles for various activities in the background (e.g. encoding or analyzing media).  SQL Azure is used for content metadata and Windows Azure blob storage for the video files.

Ahura based his MediaStreamSource off a MSDN example called the ManagedMediaHelpers.

Blitting is the act of blending multiple images together.  More about the BLIT operation can be found at Wikipedia.

Adam used the Blit method with different blend modes to achieve the effect from the standard image.  With masking, alpha blending, and the multiply mode combined with a few masks and textures, he was able start with a standard image and give it that “pulled out of my pocket” effect.

The WriteableBitmapEx team over at http://writeablebitmapex.codeplex.com/ really did a great job with their library too!

Tesco, one of the world’s leading retailers, and Microsoft Corp., worldwide leader in software, today announced a collaboration to launch the next generation of home video viewing. The new service, built on Microsoft Silverlight technology, will deliver a similar level of quality as consumers have come to expect from DVD and Blu-ray, but with advanced Web-based interactivity and a viewing experience that goes beyond other digital playback products in the marketplace.

Starting in the autumn, Tesco will allow customers in the U.K. who purchase certain home video titles from Tesco to download “digital copy” versions of movies to their Windows-based or Apple Macintosh computers in a “virtual DVD” experience. The digital copy versions will include a similar level of video quality, interactivity and bonus content available on the physical products. In addition, the digital copy versions will provide consumers with extra network-connected features such as auto-updated trailers, exclusive bonus content, movie viewing parties with online chat, related music offerings such as MP3s and ring tones, and networked games.

Dave Cook

He currently has 4 articles from the simple to the complex.

• API Basics
• Accessing the Title Catalog
• Accessing Subscriber Account Information
• Managing Account Information

If you are interested in doing a netflix project, you have to sign up for a developer account at the Netflix Developer Network home page.

But I've got a lot of topics in the queue I hope to get posted before IBC (and yes, I'll be in Amsterdam for the whole show).

First up, the very cool Project Tuva, a Silverlight presentation of the seven classic Messenger Lectures by famed physicist Richard Feynman Microsoft Research and Stimulant. It’s named after the small Central Asian republic of the Russian Federation to which Feynman had a long yearning to travel. Project Tuva was sponsored by Microsoft’s Tony Hey, Rick Rashid, and Bill Gates.

As I've mentioned before, I've been in this digital media game for quite a while now. And since long before we had visions of HD web video, or even DVD, multimedia education has been one of the big goals for the technology.

My school years spanned the filmstrip/16mm projector and early VHS eras. And while an in-class movie was always a treat, the linear nature of the experience could be frustrating. The really interesting parts didn't last longer than the dull parts, and there wasn't any good way to ask a question or dive deeper. And with a dull part, I could easily tune out thinking about Space: 1999 and never come back. The classic lecture format has the same problem, although the teacher could at least read body language of the class to get a sense of where to focus.

So even back in the protean CD-ROM and even laserdisc eras of multimedia, there were many efforts to add interactivity to linear video educational content. The goal was greater engagement, with students able to skim, review, and dive deep when and where something grabs them.

But while we've had a lot of great examples of the genre, the cost of creating all that rich interactive content was a real barrier to making it part of everyday education.

But the combination of the web (lots of existing content ready to be accessed) Silverlight (nice portable runtime to deliver rich experiences), and Expression Studio (highly efficient authoring), we're able to do bigger, deeper projects with a lower authoring cost than ever before.

So, check out the Project Tuva player. The content itself was quite compelling even on celluloid, but they've really done some great things leveraging Silverlight and the web. And it was a delightful surprise; I hadn’t even heard it was in progress before launched.

Let me take a tour through some of my favorite features (going roughly counterclockwise from the top):

Context sensitive extras The right-hand side shows available extras, supplementary information about what's currently being discussed. Clicking on one pauses the video (important!) and takes the user to a graphic, web page, or embedded Silverlight app like World Wide Telescope Silverlight-based web client preview. When they're done, video playback starts right where it left off.
Search Typing into the search box yields a list of the matches in any of the seven videos. Clicking on any particular video shows all matches and their context in the video.
Smooth Streaming And it uses Smooth Streaming for delivery, of course, up to 2.4 Mbps. At the top rates it does a good job of retaining that crazy old-school-movie-on-16mm texture. I haven’t got my hands on the source yet, but I’d be curious to see if HD could be extracted with some high-quality preprocessing.
Transcript All lectures have full transcripts, and automatically show the current line as a caption below the video window. A full transcript mode is also available, and can be used for navigation; just click on a line to immediately switch to playing back the video there.
Timeline The timeline has some great user interface felicities. A quick click started playback at the start of the chapter. But holding down the mouse button a moment or grabbing-and-dragging the playhead allows scrubbing within a chapter. If the timeline view is expanded, the location and type of all the extras are shown, as are the location of user created notes (described below).
Notes The user can add time synched notes that are saved on the local machine. This allows students to bookmark places for followup, or educators to set up a queue of particular topics for classrom use.
Simple Player Or someone who wants to just watch the video can leave everything minimized to cut back on visual distraction. The full-screen mode is cleaner yet. Compare the interface with all the interactive elements minimized and maximized.

Learn how to deliver web video with the quality and reliability of HDTV.  Dramatically increase viewing times and repeat visits using this scalable online video delivery technology.

 

Standard video on the Web is plagued by 2 major issues: the quality of the video (either too small or too blurry) and the reliability of the playback (constant rebuffering or stuttering). These two variables are interrelated: increasing the quality requires higher bandwidth, and using less bandwidth to minimize rebuffering sacrifices video quality.

 

Smooth Streaming solves these issues by dynamically modifying the bandwidth of the video stream based on available bandwidth and individual PC performance. This process is invisible to viewers; they just connect to the live event. If bandwidth drops, the video seamlessly transitions to the next lower stream. When the bandwidth recovers, the stream will automatically use the higher bandwidth. This enables the viewer to watch the live event in the best possible quality at all times, without pauses or interruptions in the video stream.

During this webcast, you will learn about Smooth Streaming from two industry experts, who will teach you:

• What Smooth Streaming is and why it is the future of online video delivery
• How Smooth Streaming delivers HDTV-quality experiences
• How it has been successfully implemented for major live events
• How to start delivering your content with Smooth Streaming

On Tuesday, June 22nd at the Cannes Lions International Advertising Festival 2009, Microsoft announced it will bring Interactive Advertising Bureau (IAB) recognized rich media technologies including Silverlight to Xbox LIVE within the next year.  By extending support to technologies like Silverlight, Xbox LIVE will offer advertisers and creative designers scale and reach with their campaign creative assets, enabling them to build and deliver interactive, interconnected experiences that are unlike anything else on television, and extend these experiences across multiple screens – the TV, the PC, and Mobile.

Introduction

As a fan of High Definition movies and television, I often find it difficult to explain the benefits of upgrading from DVD to blu-ray to family and friends.  Many people believe that DVDs look good enough and/or that you need a really big television to notice the difference, and, even then, feel the improvement is not good enough to justify the upgrade.  The best way to convince someone is to show him/her the same movie playing side by side on blu-ray and dvd using the same sized screens.  Since it's difficult to do, the next best thing is to look at still images from both the DVD and the blu-ray and compare them.  The images should be from the exact same scene.  Fortunately there are places on the web such as AVSForum where members take the time to do this.  They usually do this to help people determine what kind of an upgrade they can expect when viewing the blu-ray compared to the DVD.  Sometimes the differences are easy to see, and other times they are not as clear.  I built this comparison tool as a way to see what differences exist, and provided a few ways to make those differences easier to see. 

This utility will allow the user to compare two different images using 3 different modes: dual view, swap view, and Rectangle View. Dual view will allow a side by side comparison where you see part of the first image, and the remaining part of the second image. Moving the mouse left or right will show more of one image, and less of the other. The swap mode will allow the user to see one image in its entirety, and then, when placing the mouse over the image, the application will swap the first image with the second one. The last mode will allow the user to specify a region that he or she wishes to compare. They can draw a rectangle using the mouse, and the application will show the first image with a chunk cut out, replacing the rectangle area with the equivalent area on the second image. The image below shows an example of the rectangle view.

Notes

The application is set to handle images that are 1920x1080 in resolution (the standard resolution of blu-ray movies).  DVDs have a resolution of 720x480, but are typically up-converted to 1920x1080 in order to do a direct comparison.  This upconversion is similar to what happens when you play a DVD on a 1080p high definition television.  While it wouldn't be too difficult to handle other image sizes, to keep things simple, the application assumes the user will be providing images at the proper resolution.  Using different images with different resolutions will result in incorrect behavior.

At the time of this writing, Silverlight 3 beta has just recently been released, but has not been tested with this application.

Setup

Before beginning, you must have Visual Studio 2008 SP1 or Visual Web Developer Express SP1.  Be sure to install Silverlight 2 Tools in order to create and run the project.  Once you have that, start a new Silverlight Application and call it BlurayDVDCompare, or whatever you desire.  As shown in figure 1, choose the option to host Silverlight with an ASP.NET web application. This will create a web project and a separate Silverlight project.    

Creating the UI

Since we will be displaying HD images (1920x1080), our next step is to ensure that the browser adds scrollbars so that the entire Silverlight control can be seen.  To do this, we need to open and edit the hosting web page (BlurayDVDCompareTestPage.aspx) and set the Width and Height properties of the Silverlight control to 1920 x 1200 as shown below.  The 1200 is extra room for other controls that we will add.

<asp:Silverlight ID="Xaml1" runat="server"
Source="~/ClientBin/BlurayDVDCompare.xap"
MinimumVersion="2.0.31005.0" Width="1920" Height="1200" />

The main part of our UI, will be the canvas that holds the two images that we are comparing.  Each image has a clip region which will be used to display parts of each image or to hide/show an image.  For example, in DualView mode, we want to display part of one image, and the remaining part of the second image.  If no clip regions were set, the default behavior would force the second image to cover the first.  The final thing we add is a border which we will be used to draw a line as a separator between the two images in DualView.  In Rectangle view, the border is used to draw the rectangle.

<StackPanel>
    <Canvas x:Name="canvas" Background="White" Width="1920" Height="1080" Margin="0,5,0,0">    
        <Image x:Name="img1" Stretch="None" >        
            <Image.Clip>
                <RectangleGeometry  Rect="0,0,960,1080" x:Name="rcImg1" />
            </Image.Clip>
        </Image>

        <Image x:Name="img2" Stretch="None" >
            <Image.Clip >
                <RectangleGeometry Rect="960,0,1920,1080" x:Name="rcImg2"  />
            </Image.Clip>
        </Image>    
            
        <Border x:Name="border" Width="2" Height="1920" Margin="0,0,0,0" BorderBrush="Bisque" BorderThickness="0" />
    </Canvas>
</StackPanel>

Implementation

private void LoadImages(string s1, string s2)
{
    brLoading.Visibility = Visibility.Visible;
    spLoadingError.Visibility = Visibility.Collapsed;

    if (string.IsNullOrEmpty(s1) || string.IsNullOrEmpty(s2))
    {
        txtLoadingError.Text = "Invalid image location given";
        brLoading.Visibility = Visibility.Collapsed;
        spLoadingError.Visibility = Visibility.Visible;
        return;
    }

    prgLoading1.Value = 0;
    prgLoading2.Value = 0;
    
    _bmi1 = new BitmapImage();
    _bmi2 = new BitmapImage();
    _bmi1.DownloadProgress += new EventHandler<DownloadProgressEventArgs>(bmi1_DownloadProgress);
    _bmi2.DownloadProgress += new EventHandler<DownloadProgressEventArgs>(bmi2_DownloadProgress);

    _bmi1.UriSource = new Uri(s1, UriKind.Absolute);
    _bmi2.UriSource = new Uri(s2, UriKind.Absolute);

    img1.Source = _bmi1;
    img2.Source = _bmi2;           
}

Private Sub LoadImages(ByVal s1 As String, ByVal s2 As String)
    brLoading.Visibility = Visibility.Visible
    spLoadingError.Visibility = Visibility.Collapsed

    If String.IsNullOrEmpty(s1) OrElse String.IsNullOrEmpty(s2) Then
        txtLoadingError.Text = "Invalid image location given"
        brLoading.Visibility = Visibility.Collapsed
        spLoadingError.Visibility = Visibility.Visible
        Return
    End If

    prgLoading1.Value = 0
    prgLoading2.Value = 0

    _bmi1 = New BitmapImage()
    _bmi2 = New BitmapImage()
    AddHandler _bmi1.DownloadProgress, AddressOf bmi1_DownloadProgress
    AddHandler _bmi2.DownloadProgress, AddressOf bmi2_DownloadProgress

    _bmi1.UriSource = New Uri(s1, UriKind.Absolute)
    _bmi2.UriSource = New Uri(s2, UriKind.Absolute)

    img1.Source = _bmi1
    img2.Source = _bmi2
End Sub

void canvas_MouseMove(object sender, MouseEventArgs e)
{
    double x = e.GetPosition(canvas).X;
    double y = e.GetPosition(canvas).Y;
    
    if (_comparisonMode == ComparisonMode.DualView)
        DisplayDualView(x);
    else if (_comparisonMode == ComparisonMode.Rectangle)
    {
        // ...
    }
}

Private Sub canvas_MouseMove(ByVal sender As Object, ByVal e As MouseEventArgs)
    Dim x As Double = e.GetPosition(canvas).X
    Dim y As Double = e.GetPosition(canvas).Y

    If _comparisonMode = ComparisonMode.DualView Then
        DisplayDualView(x)
    ElseIf _comparisonMode = ComparisonMode.Rectangle Then            ' .....
    End If
End Sub

private void DisplayDualView(double x)
{
    rcImg1.Rect = new Rect(0, 0, x, IMGHEIGHT);         // Show the first half (left side), up to xCoord
    rcImg2.Rect = new Rect(x, 0, IMGWIDTH, IMGHEIGHT);  // Show the second-half (right side), starting from xCoord

    Canvas.SetLeft(border, x);                          // make the separator start at x
    Canvas.SetTop(border, 0);                           // set the separator to the top
}

Private Sub DisplayDualView(ByVal x As Double)
    rcImg1.Rect = New Rect(0, 0, x, IMGHEIGHT)         ' Show the first half (left side), up to xCoord
    rcImg2.Rect = New Rect(x, 0, IMGWIDTH, IMGHEIGHT)  ' Show the second-half (right side), starting from xCoord

    Canvas.SetLeft(border, x) ' make the separator start at x
    Canvas.SetTop(border, 0)  ' set the separator to the top
End Sub

private void DisplaySwapView(bool mouseOver)
{
    if (!mouseOver)
    {
        rcImg1.Rect = new Rect(0, 0, IMGWIDTH, IMGHEIGHT);
        rcImg2.Rect = new Rect(0, 0, 0, 0);
    }
    else
    {
        rcImg1.Rect = new Rect(0, 0, 0, 0);
        rcImg2.Rect = new Rect(0, 0, IMGWIDTH, IMGHEIGHT);
    }
}

Private Sub DisplaySwapView(ByVal mouseOver As Boolean)
    If (Not mouseOver) Then
        rcImg1.Rect = New Rect(0, 0, IMGWIDTH, IMGHEIGHT)
        rcImg2.Rect = New Rect(0, 0, 0, 0)
    Else
        rcImg1.Rect = New Rect(0, 0, 0, 0)
        rcImg2.Rect = New Rect(0, 0, IMGWIDTH, IMGHEIGHT)
    End If
End Sub

The rectangle view is the last and most involved.  Since the user will be drawing a rectangle with the mouse, we must keep track of mouse movement, the state of the mouse button, and the coordinates for the rectangle.  We also need to display the rectangle as the user is drawing it and update clipped area of the image in real time in.  In order to make this easier, we create a class called MouseRectangle will contain the important information about the mouse state and rectangle size and location. 

public class MouseRectangle
{
    public bool MouseLeftButtonDown { get; set;}
    private Rect _rtg;

    private double? _startX;
    private double? _startY;

    public bool RectangleIsDrawn { get; set; }
    public Rect Rectangle { get { return _rtg;} }

    public MouseRectangle()
    {
        RectangleIsDrawn = false;
    }

    public void Reset()
    {
        _startX = null;
        _startY = null;
        RectangleIsDrawn = false;
    }

    public void CalculateRectangle(double x, double y)
    {
        if (_startX == null)
            _startX = x;
        if (_startY == null)
            _startY = y;

        // pick the smaller of the two
        double left = Math.Min(_startX.Value, x);
        double top = Math.Min(_startY.Value, y);

        _rtg = new Rect(left, top, Math.Abs(_startX.Value - x), Math.Abs(_startY.Value - y));             
    }

    public bool InBoundsOfRect(double x, double y)
    {
        if (x > _rtg.Left && x < _rtg.Right &&
            y > _rtg.Top && y < _rtg.Bottom)
            return true;
        return false;
    }

}

Public Class MouseRectangle
    Private privateMouseLeftButtonDown As Boolean
    Public Property MouseLeftButtonDown() As Boolean
        Get
            Return privateMouseLeftButtonDown
        End Get
        Set(ByVal value As Boolean)
            privateMouseLeftButtonDown = value
        End Set
    End Property
    Private _rtg As Rect

    Private _startX? As Double
    Private _startY? As Double

    Private privateRectangleIsDrawn As Boolean
    Public Property RectangleIsDrawn() As Boolean
        Get
            Return privateRectangleIsDrawn
        End Get
        Set(ByVal value As Boolean)
            privateRectangleIsDrawn = value
        End Set
    End Property
    Public ReadOnly Property Rectangle() As Rect
        Get
            Return _rtg
        End Get
    End Property

    Public Sub New()
        RectangleIsDrawn = False
    End Sub

    Public Sub Reset()
        _startX = Nothing
        _startY = Nothing
        RectangleIsDrawn = False
    End Sub

    Public Sub CalculateRectangle(ByVal x As Double, ByVal y As Double)
        If Not _startX.HasValue Then
            _startX = x
        End If
        If Not _startY.HasValue Then
            _startY = y
        End If

        ' pick the smaller of the two
        Dim left As Double = Math.Min(_startX.Value, x)
        Dim top As Double = Math.Min(_startY.Value, y)

        _rtg = New Rect(left, top, Math.Abs(_startX.Value - x), Math.Abs(_startY.Value - y))
    End Sub

    Public Function InBoundsOfRect(ByVal x As Double, ByVal y As Double) As Boolean
        If x > _rtg.Left AndAlso x < _rtg.Right AndAlso y > _rtg.Top AndAlso y < _rtg.Bottom Then
            Return True
        End If
        Return False
    End Function

End Class

void canvas_MouseLeftButtonDown(object sender, MouseButtonEventArgs e)
{
    _mouseRectangle.Reset();
    _mouseRectangle.MouseLeftButtonDown = true;
}

void canvas_MouseLeftButtonUp(object sender, MouseButtonEventArgs e)
{
    _mouseRectangle.MouseLeftButtonDown = false;
    _mouseRectangle.RectangleIsDrawn = true;
}

Private Sub canvas_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseButtonEventArgs)
    _mouseRectangle.Reset()
    _mouseRectangle.MouseLeftButtonDown = True
End Sub

Private Sub canvas_MouseLeftButtonUp(ByVal sender As Object, ByVal e As MouseButtonEventArgs)
    _mouseRectangle.MouseLeftButtonDown = False
    _mouseRectangle.RectangleIsDrawn = True
End Sub

...
else if (_comparisonMode == ComparisonMode.Rectangle)
{
    if (_mouseRectangle.MouseLeftButtonDown)
        DrawRectangle(x,y);
    else if (_mouseRectangle.RectangleIsDrawn)
        DisplayRectangleView(x,y);
}
...

...
ElseIf _comparisonMode = ComparisonMode.Rectangle Then
    If _mouseRectangle.MouseLeftButtonDown Then
        DrawRectangle(x,y)
    ElseIf _mouseRectangle.RectangleIsDrawn Then
        DisplayRectangleView(x,y)
    End If
End If
...

C#

private void DrawRectangle(double x, double y)
{
    _mouseRectangle.CalculateRectangle(x,y);

    Canvas.SetLeft(border, _mouseRectangle.Rectangle.Left);
    Canvas.SetTop(border, _mouseRectangle.Rectangle.Top);
    border.Width = _mouseRectangle.Rectangle.Width;
    border.Height = _mouseRectangle.Rectangle.Height;

    rcImg1.Rect = new Rect(0, 0, IMGWIDTH, IMGHEIGHT);
    rcImg2.Rect = _mouseRectangle.Rectangle;
}

Private Sub DrawRectangle(ByVal x As Double, ByVal y As Double)
    _mouseRectangle.CalculateRectangle(x, y)

    Canvas.SetLeft(border, _mouseRectangle.Rectangle.Left)
    Canvas.SetTop(border, _mouseRectangle.Rectangle.Top)
    border.Width = _mouseRectangle.Rectangle.Width
    border.Height = _mouseRectangle.Rectangle.Height

    rcImg1.Rect = New Rect(0, 0, IMGWIDTH, IMGHEIGHT)
    rcImg2.Rect = _mouseRectangle.Rectangle
End Sub

C#

private void DisplayRectangleView(double x, double y)
{
    rcImg1.Rect = new Rect(0, 0, IMGWIDTH, IMGHEIGHT);

    if (_mouseRectangle.InBoundsOfRect(x,y))
        rcImg2.Rect = new Rect(0, 0, 0, 0);
    else
        rcImg2.Rect = _mouseRectangle.Rectangle;

}

Private Sub DisplayRectangleView(ByVal x As Double, ByVal y As Double)
    rcImg1.Rect = New Rect(0, 0, IMGWIDTH, IMGHEIGHT)

    If _mouseRectangle.InBoundsOfRect(x, y) Then
        rcImg2.Rect = New Rect(0, 0, 0, 0)
    Else
        rcImg2.Rect = _mouseRectangle.Rectangle
    End If

End Sub

Conclusion & Future Work

Overall, the blu-ray DVD comparison utility is a pretty straight-forward Silverlight application that you can use to see the differences between two images.  These images don't even need to be from blu-ray or DVD sources, but as long as they are the right dimension, a user can use it to see subtle differences between the two by allowing three different comparison modes.  For blu-ray purchases when you already own the DVD, I think it helps to see what kind of benefit you will get by moving to blu-ray.  It has been my experience that almost every blu-ray offers some type of picture upgrade over its DVD counterpart (and sound upgrade as well).  Sometimes the differences are subtle, sometimes they are incredibly better.  There are a lot of factors involved, but 

Some future work that can be done to this is adding the ability to zoom in and out on the images in order to be able to see differences in a closer view.   Soon there will be more movies which will be re-released on blu-ray and you can use the tool to compare the old release to the new one.  This will require zooming to see some of the more finer differences.  Other work can be done by creating a database to hold links, and have a control that allows a user to navigate to different movies and view different comparisons.  I have started doing both of these, but they were not ready for this article.

One definite area that needs work is the UI.  I am not good at laying out user interfaces or creating new xaml styles, so I left them default. 

What would be really useful is a full motion comparison where you can use similar modes to see how two movies compare while in motion, but the biggest problem would be finding or creating the comparison material.  Playback would be limiting as well since it requires higher end CPUs/video cards.  

Comparisons

Here are a few URLs you can use to play with the comparison utility:

Note: Images were copied from the blu-ray threads of A/V Science Forum: www.avsforum.com .

Thanks

I would like to thank Brian Peek for introducing me to Coding4Fun, and taking the time to review my article and test the code.