Introduction

Back in November of 2009, I was met with an interesting challenge. Those of you who frequent the XNA Community Forums have probably heard of Nick Gravelyn. He was an XNA MVP, and now he works for the XNA team. He started a little contest called xna7day, which was designed to challenge developers to make a game in 7 days using a pre-defined theme.

The theme in November was “Text Based” and its rule was that you were only allowed to use text in the visuals of your game. I saw some of the cool stuff other developers were working on, including one that looked like it had text characters that could walk and talk, like people. But my mind immediately went to a different solution: I wanted to make a game that was fully 3D (effectively breaking the rule of rendering text), but then post-process the render into ASCII art, so it would look like it was text.

In this article, I'm going to show you how to do just that:

Setup

Alright, let's dive right in. Open up the base project. It should compile and run as is. It's a simple, if ugly, first-person shooter. Play through the first level. Go ahead, I'll wait. Bonus points to anyone who recognizes the level design!

Okay, got a playthrough? Let's start! I separated this into three projects so the ASCII stuff can be extracted into other games. We'll be doing most of our work in the ASCII_Renderer project, so let's open that up and add a new class to the Source\ascii folder called “Renderer.” I use the namespace “ASCII3D” for all files in this project, so get rid of the extra folder sub-namespaces. Add these using statements, too:

C#

using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Content;
using Microsoft.Xna.Framework.Graphics; 

This is going to be a post-processing effect, so we'll try to make it portable. Add the following empty function declarations:

C#

public Renderer(ContentManager content, int xRes, int yRes)
{ }

public void StartScene()
{ }

public void EndScene()
{ }

The constructor takes three arguments: The ContentManager, which we will use to load in our font and post-processing shader, and the resolution variables, which will basically tell us how many lines of text to draw and how many characters in each line. Since we have these prototyped, we'll make our only edits to the ASCIIFPS project in the Source\Game1.cs file. But first, we need to define and initialize a new Renderer object. Add a new class variable:

C#

/// <summary>
/// This is what handles the ASCII conversion
/// </summary>
private Renderer renderer; 

And at the end of the LoadContent function:

C#

// and initialize our ASCII renderer
// the variables we pass make each character
// appx 10x14 which seems to be the smallest
// readable size
renderer = new Renderer(Content, Global.ScreenWidth/10,Global.ScreenHeight/14);

Finally, call the BeginScene and EndScene functions. Modify Game1's Draw function to look like this:

C#

protected override void Draw(GameTime gameTime)
{
    // notice we only wrap StartScene and
    // EndScene if we are UsingASCII
    if (UsingASCII)
        renderer.StartScene();

    currentState.Draw(gameTime);

    if (UsingASCII)
        renderer.EndScene();

    base.Draw(gameTime);
}

The Boolean variable UsingASCII is defined for us. It defaults to false and can be toggled any time in-game by pressing the [M] key. If you want to default it to true for testing purposes, just change its value in Game1's Initialize function. At this point, the Renderer is all hooked up, so changes will go right in.

I skimmed over the other classes earlier in the article, but I need to touch on a few of them. Almost everything in GlobalModules is fairly standard utility stuff. We'll be using it, so you might want to glance at the Global class. The ASCIIFPS has the FPS engine in it and is almost entirely irrelevant to this article, except that it offers a fun test-bed for our effect. Feel free to go through the code, but I'll warn you: I wrote it for a 7-day contest, so some of it looks like gibberish and is only incidentally functional. Finally, we have the ASCII_Renderer project. This already has two files in it: BitmapFontGenerator and Letter. The latter are for generating a bitmap font map that we'll use in a bit.

The functionality isn't specific to this article, since any artist could easily make this. But I'm no artist, which is why I had the computer do it for me. Basically, BitmapFontGenerator has a static function that will take a texture of letters (see the Content project in ASCII_Renderer), trim them, sort them by how much they fill their space, and print them into a new Texture for our functions. Feel free to read through the code for this process; I commented it pretty well.

Post-Processing Basics

We need to cover the basics of post-processing before we start coding the next part. When post-processing a scene, you first render the entire scene off-screen on a RenderTarget. Then, you can apply any effects you want to the resolved texture before putting it on the screen. That's it. Here's how in XNA:

C#

public class Renderer
{
    private RenderTarget2D SrcImage;

    public Renderer(ContentManager content, int xRes, int yRes)
    {
        SrcImage = new RenderTarget2D(Global.Graphics, 
            xRes, yRes, 1, SurfaceFormat.Color);
    }

    public void StartScene()
    {
        Global.Graphics.SetRenderTarget(0, SrcImage);
        Global.Graphics.Clear(Color.Black);
    }

    public void EndScene()
    {
        Global.Graphics.SetRenderTarget(0, null);
        Global.Graphics.Clear(Color.Black);

        // TODO: draw the texture from the RenderTarget
    }
}

We create the RenderTarget2D we need in the constructor. Notice that the size of the RenderTarget is the same as the number of ASCII characters we plan to draw on screen. I'll explain that in the next section. In the StartScene function, we set the RenderTarget on the device so that all draw calls will actually draw to it. Then we clear the buffer, because RenderTargets' content from frame to frame is undefined. In EndScene, the RenderTarget is set to null, which is DirectX's way of saying “use the screen's framebuffer.” Now we need to actually draw the rendered image. XNA requires shaders for all draw calls, even if they're veiled by BasicEffect and SpriteBatch. We need to create a very simple shader to draw the texture to screen. Create a new Effect file in ASCII_Renderer's Content project and name it TextEffect. We'll start it out with this:

HLSL

texture2D sourceTex;
sampler2D SourceTextureSampler = sampler_state
{
    Texture = <sourceTex>;
    MinFilter = point;
    MagFilter = point;
    MipFilter = point;
    AddressU = wrap;
    AddressV = wrap;
};

struct VertexShaderInput
{
    float4 Position : POSITION0;
    float2 TexCoords : TEXCOORD0;
};

struct VertexShaderOutput
{
    float4 Position : POSITION0;
    float2 TexCoords : TEXCOORD0;
};

VertexShaderOutput VertexShaderFunction(VertexShaderInput input)
{
    VertexShaderOutput output;
    
    output.Position = input.Position;
    output.TexCoords = input.TexCoords;

    return output;
}

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    float4 sourceColor = tex2D(SourceTextureSampler,input.TexCoords);
    
    return sourceColor;
}

technique Technique1
{
    pass Pass1
    {
        VertexShader = compile vs_2_0 VertexShaderFunction();
        PixelShader = compile ps_2_0 PixelShaderFunction();
    }
}

There are a couple things I want to note about the base shader. First, notice that we use point filtering. I'll explain why in the next section, but it's important to see where we did it. Next, notice that we took out all the default transformation matrices. We're going to be drawing a single full screen quad, so we can just pass in the coordinates in screen-space—no processing required. Now we need to make a few changes to the Renderer class to use this Effect and draw the rendered game. First, we need to add some new class variables:

C#

private Effect effect;

private VertexPositionTexture[] verts;
private VertexDeclaration vd;

The vertices are going to be for a simple quad that takes up the whole screen. They should be initialized in the LoadContent function like this:

C#

effect = content.Load<Effect>("TextEffect");

verts = new VertexPositionTexture[6];
verts[0] = new VertexPositionTexture(
    new Vector3(-1, -1, 0), new Vector2(0, 1));
verts[1] = new VertexPositionTexture(
    new Vector3(1, -1, 0), new Vector2(1, 1));
verts[2] = new VertexPositionTexture(
    new Vector3(-1, 1, 0), new Vector2(0, 0));
verts[3] = verts[1];
verts[4] = verts[2];
verts[5] = new VertexPositionTexture(
    new Vector3(1, 1, 0), new Vector2(1, 0));

vd = new VertexDeclaration(
    Global.Graphics, VertexPositionTexture.VertexElements);

And finally, we add our draw code to the Draw function:

C#

effect.Begin();
effect.CurrentTechnique.Passes[0].Begin();

effect.Parameters["sourceTex"].SetValue(SrcImage.GetTexture());
effect.CommitChanges();
Global.Graphics.RenderState.CullMode = CullMode.None;

Global.Graphics.VertexDeclaration = vd;
Global.Graphics.DrawUserPrimitives<VertexPositionTexture>
     (PrimitiveType.TriangleList, verts, 0, 2);

effect.CurrentTechnique.Passes[0].End();
effect.End();

We set the texture we got from the RenderTarget, turn off culling (what would be culled in this geometry?), and draw it. Run the project. You should be able to toggle a difference in-game with the [M] key.

Drawing Some Text

“Okay,” you say, “it runs at a lower resolution with the [M] key. So what?” Remember what we set the resolution to in the RenderTarget? Yes, every pixel on this low-resolution version will be a character in the final image. How do we turn these big pixels into characters? First we need to define a texture bitmap font. Let's add a Texture2D class variable to Renderer and a const int for the BitmapFontGenerator:

C#

public Texture2D text; 
private const int NUM_SLOTS = 256;

Then we'll use the BitmapFontGenerator to construct this texture:

C#

text = BitmapFontGenerator.GenerateTextTexture(content, NUM_SLOTS);

Okay, that's all for now. Let's go back to the TextEffect.fx. We need to add a few variables:

HLSL

float2 dstLetterSize;

int numLetters;

texture2D textTex;
sampler2D TextTextureSampler = sampler_state
{
    Texture = <textTex>;
    MinFilter = point;
    MagFilter = point;
    MipFilter = point;
    AddressU = wrap;
    AddressV = wrap;
};

The variable “dstLetterSize” will tell us how big the letters on the screen are. The “numLetters” variable will tell us how many letters are in the font texture. And then there's the font texture itself. Here's how we draw actual characters in the Pixel Shader:

HLSL

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
     float4 sourceColor = tex2D(SourceTextureSampler,input.TexCoords);
    
     float2 texCoords = float2(
          input.TexCoords.x-((int)
               (input.TexCoords.x/dstLetterSize.x)*dstLetterSize.x),
          input.TexCoords.y-((int)
               (input.TexCoords.y/dstLetterSize.y)*dstLetterSize.y));
     texCoords /= dstLetterSize;
     texCoords.x /= numLetters;
    
     float4 letterColor = tex2D(TextTextureSampler,texCoords);
    
     return letterColor;
}

We've already seen the source color line. The next line will get the offset into the letter in letter size by subtracting the rounded-down (or integer-casted) texture coordinates scaled down. The line after that scales it back up to a 0-1 range. So now we have the new variable texCoords that is filled with the texture coordinates of the letter in 0-1 terms. We divide the x by numLetters because our texture is one long horizontal line of letters. Finally we get the letter color with these new texture coordinates. This letter color is grayscale, and when multiplied by the source color, we've assured that each letter is rendered in its own slot with the color from the original 3D render. The last thing we have to do is set those effect variables we defined from Renderer's EndScene function. Where we once just set the source texture, we now set these:

C#

effect.Parameters["dstLetterSize"].SetValue(
     new Vector2(1.0f/(float)SrcImage.Width, 1.0f/(float)SrcImage.Height));
effect.Parameters["numLetters"].SetValue(NUM_SLOTS);
effect.Parameters["sourceTex"].SetValue(SrcImage.GetTexture());
effect.Parameters["textTex"].SetValue(text);
effect.CommitChanges();

Most of these are self explanatory, but the dstLetterSize is a bit more complicated. To get the size of the letter in the 0-1 texture space, you take the total size (0 to 1) divided by the number of letters in that axis which, in our case, is the RenderTarget size.

Wheel of Fortune

‘M's aren't very interesting. In traditional ASCII art, the whole point is that you use characters that take more space or less space to handle shading. Let's add this code after the multiplication of the texCoords.x:

HLSL

float lum = (sourceColor.r+sourceColor.g+sourceColor.b)/3.0; 
float val = max(fullValueColor.r,
     max(fullValueColor.g,fullValueColor.b));
    
int ind = ((numLetters-1)-
     max(0,min((numLetters-1),(int)(lum*numLetters))));

texCoords.x += ind*(1.0/numLetters);

The first line gets the luminosity of the source color. The next gets the value. The next line gets the letter index from the luminosity. Subtracting it from (numLetters-1) reverses the index, because we want luminosity=1 to equate to index=0, because that has the letter that takes up the most space. Run the game. You'll see it now shows value, but it's still pretty hard to tell what's on the screen:

The last thing we can do is colorize it. You could just multiply it by the source color and get something like this…

…but everything looks too dark because the source color still has its value. Plus we're already practically multiplying it by choosing the letter. So we need to get the full value color first:

HLSL

float4 fullValueColor = sourceColor * (1.0/val);

We just multiply the original color by the inverse of the value. And then multiply that fullValueColor by the letter color in the return:

HLSL

return fullValueColor * letterColor;

That should produce a nice result:

Conclusion

And there it is! We have a real-time 3D render transformed into ASCII art. It's difficult to see and it's not very useful, but it's sure neat, eh? Why not add it as an unlockable easter egg in your next XBL Indie Game? I'm looking forward to seeing how people use this effect! If you use it, please send me an email with a screenshot!

About The Author

Louis Ingenthron is a Game Developer in Orlando, FL. He works on commercial Console and PC titles, but runs his own Indie Games company, FV Productions, on the side. He is best known for his open source XNA rhythm game Unsigned and specializes in real-time Graphics programming. He has been working with XNA since the 2.0 Beta and with .NET C# for just as long. He is also familiar with several other development languages, such as C, C++, and Java. Occasionally, he can even be found doing web development and Flash. Louis also writes for MSDN's Coding4Fun website, contributing articles on a monthly basis.

Check it out over at Codeplex.

[via Ben Anderson]

TweetCraft is a World of Warcraft add-on that enables you to send and receive tweets using Twitter without leaving the game, automatically upload and post screenshots using TwitPic and automatically tweet certain in-game events such as achievements and moving around in the world of Azeroth. TweetCraft customizes your World of Warcraft user interface by adding its own frame similar to the built-in ones (such as the Quest Log) and provides easy access to it through a button near the minimap, slash commands or your favorite launcher (e.g. ChocolateBar).

How TweetCraft works

Let’s quickly walk through the different things that TweetCraft does to bring Twitter into World of Warcraft.

Overview

World of Warcraft’s user interface is highly customizable and enables add-on developers to do almost anything in the game they can dream of. Add-ons can interact with the world, listen to events that happen in the game, alter the look of the default user interface or create brand new ones, but add-ons are seriously limited when it comes to talking to the world outside World of Warcraft. They cannot read or write files, call web services or do anything that could be potentially harmful. This is why TweetCraft comes with a Windows application that sits in the Notification Area (also known as the tray) and does the bulk of the work, giving the TweetCraft add-on only the data that it needs to display, such as tweets, replies and user pictures. All the Twitter web service calls, uploading of screenshots and housekeeping is done by this application. Let’s see how!

The TweetCraft Tray application

The TweetCraft Tray application is a WPF application that periodically checks for new tweets, dowloads and converts the user pictures of the tweets’ authors and writes it out to a file that World of Warcraft picks up and the add-on can use. It also picks up all outgoing tweets queued up for sending and sends them using Twitter one-by-one. It watches for screenshots taken by World of Warcraft and uploads them using TwitPic immediately.

To do all this, it has a couple of settings that need to be configured before it can work. For starters, it needs your Twitter credentials to be able to send tweets and post pictures on your behalf. It also needs to know which World of Warcraft account you are going to use to play and send and receive tweets with. Finally, you can opt out of automatically uploading screenshots you take during game. Figure 1 shows the TweetCraft Settings window that shows up when you first start TweetCraft after installation and is also available by right clicking the tray icon (Figure 2).

Figure 1. TweetCraft Settings Window

Figure 2. TweetCraft tray icon in the notification area

Getting data from Twitter

TweetCraft uses an open-source library called TwitterLib to send and receive tweets using Twitter’s API. TwitterLib is part of the open-source WPF Twitter Client WittyTwitter and wraps the web services provided by Twitter into an easy to use .NET class library. TweetCraft wraps this library into a simple class called TwitterClient that all the services in TweetCraft rely on. One of those services is the TwitterBackgroundService which polls Twitter periodically to check for new tweets. By default, this is set to five minutes. If new tweets or replies are available, an event is raised that other components of TweetCraft can handle and work with the new tweets, download user pictures, etc. Figure 3 shows the three classes that take part in downloading new tweets and replies and notifying other classes.

Figure 3. TwitterBackgroundService, TwitterClient and TwitterNet (TwitterLib)

Dynamically building TGA images

When it comes to visualizing images (or textures, in WoW parlance), World of Warcraft poses quite a few restrictions on what files can be used and where does files should be. First of all, World of Warcraft only supports the TGA and BLP file formats. TGA stands for the Truevision Targa format, whereas BLP is Blizzard’s own image format. The images must also have dimensions of two (32×32, 64×64 pixels and so on…), otherwise they won’t show up. On the other hand, Twitter user pictures are available as 48×48 or 75×75 pixels and in various well-known formats, such as JPG, PNG or even GIF. TweetCraft needs to convert these pictures to 64×64 TGA files if it wants to play nice with World of Warcraft. This is done for each tweet’s authors user picture that will show up in World of Warcraft. The converted pictures must be placed in the Interface folder of World of Warcraft where the add-ons themselves also reside. The TwitterUserPictureService can retrieve the user picture and convert it to a 64×64 TGA file that contains the 48×48 user picture in the middle and puts that file into the same directory where the TweetCraft add-on is installed. The files are named after each user’s screen name and if TwitterUserPictureService sees that a file is already there, it won’t convert it again, saving some time and computing power.

Figure 4 shows TwitterUserPictureService and two helper classes that take care of downloading and the actual conversion. The GetUserPicturePath method returns the path of the resulting World of Warcraft compatible TGA file for a single user.

Figure 4. TwitterUserPictureService and its helper classes

Serializing tweets into Lua

Until now, we were quite secretive about how we can avoid World of Warcraft’s restrictions and let TweetCraft be able to read the data and show it. We’ve said that add-ons cannot read or write files and that still hasn’t changed during the last few minutes, but add-ons must have a way somehow to save their settings and other information when I log out of the game and load that when I log back in, you sincerely ask and you’re right! Although add-ons cannot read or write to files, they can ask World of Warcraft to save and load some of their global variables whenever the user reloads the UI (by entering an instance or hearthing), logs in or logs out. These files are called the SavedVariables files. When TweetCraft finishes downloading tweets, converting user pictures and everything else, it saves them using the very same format and to the very same file World of Warcraft uses on behalf of the TweetCraft add-on. Neat, huh? Just wait a little bit more…

Those variables that an add-on asks to be saved between sessions are saved into a Lua file, which is the same programming language add-on developers can use to write their add-ons. So basically, World of Warcraft creates a file that contains code full of assignment statements (think a = 5) that it will just run whenever those values need to be set again. More complex data structures can also be expressed this way using Lua tables, the swiss army-knife of Lua.

It sounds like a great idea to put all the data we would like for TweetCraft to use, but all we have is .NET objects, arrays and dictionaries. Fortunately, all these data structures can be expressed with simple Lua types and tables and TweetCraft comes with a class called LuaSerializer that produces Lua code from any .NET object (with a few restrictions, of course), very similar to how XmlSerializer works.

To make this even more simple, TweetCraft contains the concept of a SavedVariablesChannel that you can just drop objects in, flush it and those object show up on the other side. In our case, as Lua variables. What’s even better, it works the other way around as well. The TweetCraft add-on stores all the tweets queued up and ready to be sent in a variable and triggers World of Warcraft to save them to the SavedVariables file. The SavedVariablesChannel detects this by watching for changes made to the SavedVariables file using FileSystemWatcher and notifies other components of the TweetCraft application that can pull out the information and send the tweets.

Figure 5 shows SavedVariablesChannel and its base class, ValueChannelBase which keeps track of all objects thrown into it until it is flushed.

Figure 5. SavedVariablesChannel and ValueChannelBase

Refreshing the data in World of Warcraft

There’s one last problem we face when trying to achieve this two-way communication between the TweetCraft add-on and the application. If World of Warcraft is running, we cannot just overwrite the SavedVariables file and expect World of Warcraft to pick it up as it does not watch for changes made to the file. If we were to overwrite the file and subsequently the user logs out, World of Warcraft will happily overwrite the information we placed there. There’s another trick we have to resort to: the TweetCraft add-on can trigger the reloading of the UI which will unload all add-ons, save their variables and then load them again. There’s a a little bit of time between the saving and loading of a particular add-ons SavedVariables file and that’s when TweetCraft can safely read the file for queued up tweets and overwrite it with the new data. This is what happens when the user clicks the Refresh button in TweetCraft’s frame.

Sending tweets and uploading screenshots

Whenever the user queues up tweets for sending, they are not immediately sent but kept in a data structure that will be saved out to the SavedVariables file by World of Warcraft. We’ve seen that his happens when either the user logs out or presses the Refresh button. The updated SavedVariables file gets detected by SavedVariablesChannel and the ChannelUpdated event is raised. When the TweetCraft application handles this event, it cannot immediately send the tweets as that would take a couple seconds and if you remember from the last section, we only have a tiny little window of time to process the values and save back all the data we would like to send back. For this reason, the tweets that are picked up from the SavedVariables file using the SavedVariablesChannel get quickly queued up in the TwitterDispatcherService that then goes ahead and starts sending them one by one in the background. To make sure that all the values we prepared to save into the file get flushed, the SavedVariablesChannel has a property called AutoFlush that is enabled by default.

One thing we have not mentioned is that TwitterDispatcherService can also upload and post pictures using TwitPic. It watches the Screenshots folder of World of Warcraft for new files and if AutoTweeting of screenshots is enabled, immediately queues up the picture for uploading and posting it. This also means that you don’t need to press the Refresh button or wait until you log off for screenshots to be posted. They will show up immediately in your Twitter feed.

The TwitterDispatcherService uses TwitterClient (and TwitterLib) to send the tweets but it does not interfere with the quick reading and writing of the SavedVariables file we discussed previously. It also uses TwitPicClient to upload and post pictures. Figure 6 shows TwitterDispatcherService, TwitterClient and TwitPicClient.

Figure 6. TwitterDispatcherService, TwitterClient and TwitPicClient

Bringing it all together

So far, we’ve looked at components of TweetCraft that implemented a particular part of the application. Retrieving and sending tweets, converting user pictures and making all this information available for the TweetCraft add-on, but there’s an important class that brings all this together and contains all the logic of how these components interact with each other. We also need to be able to detect World of Warcraft installations, figure out what accounts are available and where to look for files like the SavedVariables files and put the converted user pictures. The TweetCraft class implements the logic required and relies on the WorldOfWarcraft class for everything World of Warcraft related.

Figure 7 shows the TweetCraft and WorldOfWarcraft side by side.

Figure 7. TweetCraft and WorldOfWarcraft

Summary

While this is a high-level overview of TweetCraft only, it gives you a good idea of how it enables World of Warcraft players to use Twitter to send and receive tweets, upload the screenshots and share their achievements right inside the game. Make sure you try out TweetCraft if you have World of Warcraft installed and if you’re curious, download the source code and take a deeper look.

Gabor Ratky (Blog, Twitter), Senior Software Engineer and Coding4Fun Ninja at EPAM Systems 

Gabor is the developer behind TweetCraft and has been working on Coding4Fun projects for almost two years. He also leads the development of AddOn Studio for World of Warcraft, a Visual Studio-based IDE for building World of Warcraft add-ons and he co-authored the chapter in the Coding4Fun book about it. He is a VSX Insider and was invited to speak at various events about Visual Studio Extensibility. When not working on Coding4Fun projects, Gabor works on large scale, enterprise solutions using exciting Microsoft technologies. He lives in Budapest, Hungary and loves traveling, good wine and Channel9 videos.

Audio and the .NET Framework

Playing back audio in a .NET application is not quite as easy as you might hope it would be. The .NET 2.0 Framework introduced the SoundPlayer component, which allows you to play back an existing WAV file. While this may be fine for many scenarios, as soon as you want to do things even slightly more advanced, such as changing the volume, or playing back from a different file-format, or pausing and repositioning, you must resort to writing P/Invoke wrappers for various Windows APIs.

Back in 2002, as I was getting started learning .NET, I create some audio-related classes of my own to compensate for the lack of audio support in the .NET Framework. I focused initially on reading and writing WAV and MIDI files, as well as playing back audio in a way that allowed real-time mixing and manipulation of the audio at a sample level. As time went by, I made this growing collection of audio classes available as an open source project, called NAudio, now hosted at CodePlex.

Audio Playback in NAudio

NAudio works by constructing an audio playback graph. Audio comes in “streams” which can be connected together and modified before eventually they go to a renderer. This might be your soundcard if you are listening to the audio, or it might be to a file on the hard disk.

In NAudio, all streams derive from WaveStream. NAudio comes with a collection of useful WaveStream derived classes such as WaveFileReader to read from WAV files or WaveStreamMixer to sum together multiple audio streams.

Audio mixing and effects are almost always performed with floating point numbers (32 bit is most common), so one of the first steps after reading audio out of a WAV file is to convert it from 16 bit to 32 bit. NAudio includes the Wave16To32ConversionStream class to do this. If the audio wasn't in PCM in the first place, for example it is MP3, then we make use of a combination of the Mp3FileReaderStream, WaveFormatConversionStream and BlockAlignmentReductionStream to read the audio out and get it into just the format we want. Here's an example showing how to create a WaveStream ready for playback.

WaveStream outStream;
if (fileName.EndsWith(".mp3"))
{
   outStream = new Mp3FileReader(fileName);
}
else if(fileName.EndsWith(".wav"))
{
   outStream = new WaveFileReader(fileName);
}
else
{
   throw new InvalidOperationException("Can't open this type of file");
}                
if (outStream.WaveFormat.Encoding != WaveFormatEncoding.Pcm)
{
   outStream = WaveFormatConversionStream.CreatePcmStream(outStream);
   outStream = new BlockAlignReductionStream(outStream); // reduces choppiness
}

If we simply want to play back the audio without any extra processing, we would use one of the audio output classes provided by NAudio to create an object that implements IWavePlayer. The options are WaveOut, DirectSoundOut, AsioOut and WasapiOut, each representing a different technology for audio playback in Windows. We will use WaveOut, which is the most universally supported. Here we are opening the default output device with a latency of 300ms, and instructing it to use windowed callbacks.

IWavePlayer player = new WaveOut(0, 300, true);
player.Init(outStream);
player.Play();

WaveOut will now repeatedly call the Read method of the output stream to get the next batch of audio samples to play. All we need to do now is to insert our audio effects into the playback chain.

An Audio Effects Framework

To allow us to process sample level audio more simply, I have create a new WaveStream derived class called EffectStream. EffectStream will simply pass each audio sample to one or more audio effects before returning the modified audio in its Read method. The reason I have chosen to host all the effects in a single EffectSteam rather than creating one WaveStream derived class is that I want to avoid the performance penalty of converting between arrays of bytes and arrays of floating point numbers at every step. Some types of effect, particularly those involving Fourier transforms can be processor intensive, so anything we can do to speed up performance will help.

As well as the EffectStream class, we need a base Effect class, from which all of our effects can derive. Here's a simplified version of the base Effect class (minus a whole load of helper mathematical functions):

public abstract class Effect
{
    private List<Slider> sliders;
    public float SampleRate { get; set; }
    public float Tempo { get; set; }
    public bool Enabled { get; set; }

    public Effect()
    {
        sliders = new List<Slider>();
        Enabled = true;
        Tempo = 120;
        SampleRate = 44100;
    }

    public IList<Slider> Sliders { get { return sliders; } }

    public Slider AddSlider(float defaultValue, float minimum, 
            float maximum, float increment, string description)
    {
        Slider slider = new Slider(defaultValue, minimum, 
            maximum, increment, description);
        sliders.Add(slider);
        return slider;
    }

    /// <summary>
    /// Should be called on effect load, 
    /// sample rate changes, and start of playback
    /// </summary>
    public virtual void Init()
    {}

    /// <summary>
    /// will be called when a slider value has been changed
    /// </summary>
    public abstract void Slider();

    /// <summary>
    /// called before each block is processed
    /// </summary>
    public virtual void Block()
    { }

    /// <summary>
    /// called for each sample
    /// </summary>
    public abstract void Sample(ref float spl0, ref float spl1);
}

The bulk of the work of the effect should be done in the overridden Sample method. The spl0 and spl1 parameters contain the current sample values to be modified for the left and right channels respectively. For example, to lower the volume we could halve the amplitude of every sample with the following code:

public override void Sample(ref float spl0, ref float spl1)
{
    spl0 *= 0.5f;
    spl1 *= 0.5f;
}

The Effect class contains Tempo and SampleRate values which are useful for certain types of effect. It also contains a concept of ‘Sliders' for each effect. These are the effect parameters, which allow real-time modification of the effect. So if we wanted to control the volume using a slider, we could write the following code (although bear in mind that normally volume sliders should be logarithmic not linear – see the Volume effect in the sample code for an example of how to do this):

public override void Sample(ref float spl0, ref float spl1)
{
    spl0 *= slider1;
    spl1 *= slider1;
}

To simplify the task of adding, removing and re-ordering effects, I created an EffectChain class, which is a simple wrapper around a List<Effect>. The EffectStream class has an EffectChain that contains all the effects it needs to run. Here is the code for the EffectStream:

public class EffectStream : WaveStream
{
    private EffectChain effects;
    public WaveStream source;
    private object effectLock = new object();
    private object sourceLock = new object();

    public EffectStream(EffectChain effects, WaveStream sourceStream)
    {
        this.effects = effects;
        this.source = sourceStream;
        foreach (Effect effect in effects)
        {
            InitialiseEffect(effect);
        }

    }

    public EffectStream(WaveStream sourceStream)
        : this(new EffectChain(), sourceStream)
    {        
    }

    public EffectStream(Effect effect, WaveStream sourceStream)
        : this(sourceStream)
    {
        AddEffect(effect);
    }

    public override WaveFormat WaveFormat
    {
        get { return source.WaveFormat; }
    }

    public override long Length
    {
        get { return source.Length; }
    }

    public override long Position
    {
        get { return source.Position; }
        set { lock (sourceLock) { source.Position = value; } }
    }        

    public override int Read(byte[] buffer, int offset, int count)
    {
        int read;
        lock(sourceLock)
        {
            read = source.Read(buffer, offset, count);
        }
        if (WaveFormat.BitsPerSample == 16)
        {
            lock (effectLock)
            {
                Process16Bit(buffer, offset, read);
            }
        }
        return read;
    }

    private void Process16Bit(byte[] buffer, int offset, int count)
    {
        foreach (Effect effect in effects)
        {
            if (effect.Enabled)
            {
                effect.Block();
            }
        }

        for(int sample = 0; sample < count/2; sample++)
        {
            // get the sample(s)
            int x = offset + sample * 2;
            short sample16Left = BitConverter.ToInt16(buffer, x);
            short sample16Right = sample16Left;
            if(WaveFormat.Channels == 2)
            {                    
                sample16Right = BitConverter.ToInt16(buffer, x + 2);
                sample++;
            }
           
            // run these samples through the effects
            float sample64Left = sample16Left / 32768.0f;
            float sample64Right = sample16Right / 32768.0f;
            foreach (Effect effect in effects)
            {
                if (effect.Enabled)
                {
                    effect.Sample(ref sample64Left, ref sample64Right);
                }
            }

            sample16Left = (short)(sample64Left * 32768.0f);
            sample16Right = (short)(sample64Right * 32768.0f);

            // put them back
            buffer[x] = (byte)(sample16Left & 0xFF);
            buffer[x + 1] = (byte)((sample16Left >> 8) & 0xFF); 

            if(WaveFormat.Channels == 2)    
            {
                buffer[x + 2] = (byte)(sample16Right & 0xFF);
                buffer[x + 3] = (byte)((sample16Right >> 8) & 0xFF);
            }
        }
    }


    public bool MoveUp(Effect effect)
    {
        lock (effectLock)
        {
            return effects.MoveUp(effect);
        }
    }

    public bool MoveDown(Effect effect)
    {
        lock (effectLock)
        {
            return effects.MoveDown(effect);
        }
    }

    public void AddEffect(Effect effect)
    {
        InitialiseEffect(effect);
        lock (effectLock)
        {
            this.effects.Add(effect);
        }
    }

    private void InitialiseEffect(Effect effect)
    {
        effect.SampleRate = WaveFormat.SampleRate;
        effect.Init();
        effect.Slider();
    }

    public bool RemoveEffect(Effect effect)
    {
        lock (effectLock)
        {
            return this.effects.Remove(effect);
        }
    }
}

When the Read method on EffectStream is called, we first read the requested number of bytes from our source WaveStream. This might be from a WAV or MP3 file, or from a microphone. Then, we convert it from 16 bit to 32 bit floating point audio. 16 bit audio is stored as integers going from -32,768 to 32,767, and 32 bit audio uses the range -1.0 to 1.0 to represent this range. This means we have plenty of headroom to mix together multiple signals without distorting. It is important though to remember that no samples should be greater than 1.0 before converting back to 16 bit.

Porting Effects to .NET

Now we have a basic effect framework, it is time to create some real effects to use. There are many sources of algorithms for digital signal processing (DSP) (try musicdsp.org for a good starting point), but I have chosen to base my effects model on that provided by the REAPER digital audio workstation (DAW). This impressive application, masterminded by legendary software developer Justin Frankel, includes a text-based effects framework. These effects, known as JS effects, allow the use of a C-like syntax to quickly write your own effects. I have modelled my Effect class on the JS syntax, allowing me to quickly port effects across.

public class Tremolo : Effect
{
    public Tremolo()
    {
        AddSlider(4,0,100,1,"frequency (Hz)");
        AddSlider(-6,-60,0,1,"amount (dB)");
        AddSlider(0, 0, 1, 0.1f, "stereo separation (0..1)");
    }

    float adv, sep, amount, sc, pos;

    public override void Slider()
    {
        adv=PI*2*slider1/SampleRate;
        sep=slider3*PI;
        amount=pow(2,slider2/6);
        sc=0.5f*amount; amount=1-amount;
    }

    public override void Sample(ref float spl0, ref float spl1)
    {
        spl0 = spl0 * ((cos(pos) + 1) * sc + amount);
        spl1 = spl1 * ((cos(pos + sep) + 1) * sc + amount);
        pos += adv;
    }
}

Some members in the Effect base class such as cos and slider1 allow me to keep the ported syntax as similar to the original JS script as possible.

REAPER ships with well over 100 of these JS Effects, so I chose about 15 of them and ported them to .NET. They are available in the download that accompanies this article. With some of them, for example pitch shifting effects, you will immediately notice the effect on the sound, while others, such as compressors, require some knowledge of how to adjust the parameters to get good results.

The Test Harness

Obviously we need a way to pass audio through our effects, so our next task is to create a test harness that will allow us to load in audio files and listen to them with the effects applied. For this purpose I created a simple Windows Forms application that allows you to select a WAV or MP3 file to play back. After being converted into PCM using various classes from the NAudio library, the resulting WaveStream is passed through an EffectStream before being passed to the soundcard for playback. The use of an EffectChain allows us to modify the effects loaded, and their order during playback.

To make the loading of effects simpler, I used the Managed Extensibility Framework (MEF), to [DF1] make each effect a “plugin” to the test harness. Each effect is decorated with an Export attribute to indicate that it is a plugin:

[Export(typeof(Effect))]
public class SuperPitch : Effect

Then I can request that MEF auto-populates a property with all the exported effects it can find:

[Import]
public ICollection<Effect> Effects { get; set; }

When the user selects an effect from the list of available effects, we create a new instance of it. This is because you might want to put the same effect into the effect chain more than once:

EffectSelectorForm effectSelectorForm = new EffectSelectorForm(Effects);
if (effectSelectorForm.ShowDialog(this) == DialogResult.OK)
{
    // create a new instance of the selected effect 
    // as we may want multiple copies of one effect
    Effect effect = (Effect)Activator.CreateInstance(
       effectSelectorForm.SelectedEffect.GetType());
    audioGraph.AddEffect(effect);
    checkedListBox1.Items.Add(effect, true);
}

To allow real-time modification of the effect parameters, I created two user controls. The first, EffectSliderPanel allows you to hook up a Windows Forms TrackBar to one of our effect's sliders and manages the minimum, maximum and granularity settings. The second user control, EffectPanel takes an Effect and creates one EffectSliderPanel for each slider in that effect. It also is responsible for calling the Slider method on the Effect whenever the user moves one of the sliders. Here's an example of what it looks like:

Now we are able to test our effects by listening to WAV files and playing them with real-time control over their parameters.

Intercepting Skype Audio

There are many interesting uses for audio effects, but it was suggested to me that I create a “voice changer” for Skype as the example program for this article. At first I didn't think that this would be possible, as you would need access to the audio samples from the microphone before Skype transmitted them over the network.

However, it turns out that Skype has a full featured SDK to allow all kinds of third-party add-ons and enhancements. The Skype API can be used in .NET via a COM object, called Skype4Com. Skype plugins are not loaded directly by the Skype application but attach to it via network sockets. The Skype4Com COM object hides much of this complexity from the user.

Having added Skype4Com as a reference to our application, we then need to connect to Skype. This is achieved by using the following code:

const int Protocol = 8;
skype = new Skype();
_ISkypeEvents_Event events = (_ISkypeEvents_Event)skype;
events.AttachmentStatus += OnSkypeAttachmentStatus;            
skype.CallStatus += OnSkypeCallStatus;
skype.Attach(Protocol, false);

In the CallStatus event handler, we tell Skype that we wish to ‘capture' the microphone. This will cause it to send us the raw audio data from the microphone via a TCP socket. Then we tell it that we will send the audio to be transmitted using another TCP socket.

void OnSkypeCallStatus(Call call, TCallStatus status)
{
    log.Info("SkypeCallStatus: {0}", status);
    if (status == TCallStatus.clsInProgress)
    {
        this.call = call;                  
        call.set_CaptureMicDevice(
        TCallIoDeviceType.callIoDeviceTypePort, MicPort.ToString());
        call.set_InputDevice(
        TCallIoDeviceType.callIoDeviceTypeSoundcard, "");
        call.set_InputDevice(
        TCallIoDeviceType.callIoDeviceTypePort, OutPort.ToString());
    }
    else if (status == TCallStatus.clsFinished)
    {
        call = null;
        packetSize = 0;
    }
}

I found an example Delphi application on the Skype developer website which shows how to intercept the microphone signal to boost the signal level. I used this sample as the starting point for creating my own application to intercept audio samples in Skype. The Delphi application made use of an object called TIdTCPServer, which is a multi-threaded socket server. I created a very simple .NET implementation of this class (without the multi-threading as we will only have one connection at a time):

class TcpServer : IDisposable
{
    TcpListener listener;
    public event EventHandler<ConnectedEventArgs> Connect;
    public event EventHandler Disconnect;
    public event EventHandler<DataReceivedEventArgs> DataReceived;
    
    public TcpServer(int port)
    {
        listener = new TcpListener(IPAddress.Loopback, port);
        listener.Start();
        ThreadPool.QueueUserWorkItem(Listen);
    }

    private void Listen(object state)
    {
        while (true)
        {
            using (TcpClient client = listener.AcceptTcpClient())
            {
                AcceptClient(client);
            }
        }
    }

    private void AcceptClient(TcpClient client)
    {
        using (NetworkStream inStream = client.GetStream())
        {
            OnConnect(inStream);
            while (client.Connected)
            {
                int available = client.Available;
                if (available > 0)
                {
                    byte[] buffer = new byte[available];
                    int read = inStream.Read(buffer, 0, available);
                    Debug.Assert(read == available);
                    OnDataReceived(buffer);
                }
                else
                {
                    Thread.Sleep(50);
                }
            }
        }
        OnDisconnect();
    }

    private void OnConnect(NetworkStream stream)
    {
        var connect = Connect;
        if (connect != null)
        {
            connect(this, new ConnectedEventArgs() { Stream = stream });
        }
    }

    private void OnDisconnect()
    {
        var disconnect = Disconnect;
        if (disconnect != null)
        {
            disconnect(this, EventArgs.Empty);
        }
    }

    private void OnDataReceived(byte[] buffer)
    {
        var execute = DataReceived;
        if (execute != null)
        {
            execute(this, new DataReceivedEventArgs() { Buffer = buffer });
        }
    }

    #region IDisposable Members

    public void Dispose()
    {
        listener.Stop();
    }

    #endregion
}

public class DataReceivedEventArgs : EventArgs
{
    public byte[] Buffer { get; set; }
}

public class ConnectedEventArgs : EventArgs
{
    public NetworkStream Stream { get; set; }
}

Once Skype has been told the port numbers on which to connect, it will attempt to open sockets to our TcpListener classes (one for audio in, and one for audio out). We now simply need to pass the audio through our effect chain. But EffectStream needs a WaveStream derived class for its input, so I created SkypeBufferStream to which we pass the raw data received on the microphone in socket, and it returns it in its Read method. One difficulty I encountered was that Skype offers no way of querying what the sample rate of the incoming data is. On my PC it seems to be 44.1kHz, but I do not know if this is guaranteed on all computers.

class SkypeBufferStream : WaveStream
{
    byte[] latestInBuffer;
    WaveFormat waveFormat;

    public SkypeBufferStream(int sampleRate)
    {
        waveFormat = new WaveFormat(sampleRate, 16, 1);
    }

    public override WaveFormat WaveFormat
    {
        get { return waveFormat; }
    }

    public override long Length
    {
        get { return 0; }
    }

    public override long Position
    {
        get
        {
            return 0;
        }
        set
        {
            throw new NotImplementedException();
        }
    }

    public void SetLatestInBuffer(byte[] buffer)
    {
        latestInBuffer = buffer;
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        if (offset != 0)
            throw new ArgumentOutOfRangeException("offset");
        if (buffer != latestInBuffer)
            Array.Copy(latestInBuffer, buffer, count);
        return count;
    }
}

Now when we receive any data from the microphone socket, we pass it through the SkypeBufferStream which in turn passes it through the EffectStream and finally out on the output socket's data stream. Here's the relevant code (found in the MicInterceptor class):

NetworkStream outStream;
SkypeBufferStream bufferStream;
WaveStream outputStream;

void OnOutServerConnect(object sender, ConnectedEventArgs e)
{
    log.Info("OutServer Connected");
    outStream = e.Stream;
}

void OnMicServerExecute(object sender, DataReceivedEventArgs args)
{
    // log.Info("Got {0} bytes", args.Buffer.Length);
    if (outStream != null)
    {
        // give the input audio to the beginning of our audio graph
        bufferStream.SetLatestInBuffer(args.Buffer);
        // process it out through the effects
        outputStream.Read(args.Buffer, 0, args.Buffer.Length);
        // play it back
        outStream.Write(args.Buffer, 0, args.Buffer.Length);
    }
}

When you run your application for the first time, you will need to grant it permission from within Skype:

To test that the effects are working in Skype is a little tricky as you will not hear the effected sound on your end of the conversation. One good way of checking the effects are working as expected is to use the Skype test call service. This is a number you can dial and it will record what you say and play it back to you.

It is a good idea to test your effect first using a local audio file, as you will not easily be able to determine whether glitches and other audio artifacts were caused by your effect, or simply due to poor network conditions.

There are a few things you should be aware of when selecting effects for use with Skype. First, the audio is mono, so there is no point using any effects such as stereo delay. Second, the audio is almost certainly down-sampled to a much lower sample rate before being transmitted to save on network bandwidth. This means any high frequency components of your sound will be lost. Third, internet telephony applications often have built in echo-suppression, so using delay-based effects might not work quite as well as you were hoping.

For silly voice effects, the most effective is pitch shifting (try the SuperPitch effect and shift either up or down about five semitones). FlangeBaby or Chorus can be used for more subtle voice changing effects. Or if you just want to be annoying, load up a Delay. Feel free to experiment with the other included effects, but bear in mind that many of them are designed with more musical uses in mind, so may not be relevant for internet telephony.

The Sample Code

The source code for all the effects, the EffectStream and the Skype connection code described in this article is available in the provided download. It uses a recent unreleased build of NAudio, so you will also need to visit the NAudio CodePlex site if you want to get access to the full source code. The EffectStream and Effect classes will eventually be made part of the NAudio framework, once I have refined their design a bit.

How to use Effect Tester sample app:

	Click this icon to attach to Skype and await a call. Click it again to disconnect, allowing you to test your effects using audio files instead. The textbox in the top right corner keeps you updated with the status of the connection to Skype
	Click this icon to load a WAV or MP3 file for playback. Files at 44.1kHz work best.
	Rewind, Play, Pause or Stop the current WAV or MP3 file.
	Brings up the effect selector dialog to add a new instance of an effect to the current Effect chain. Loaded effects appear in the CheckedListBox on the left.
	Removes the currently selected effect from the effect chain
	Move the currently selected effects up or down in the signal chain
	Use the checkboxes to enable or disable effects in the effect chain on the fly.
	Use the sliders to adjust the effect parameters in real-time

About the Author

Mark Heath is a .NET developer based in Southampton, UK. When he's not writing .NET audio applications for fun, he enjoys home studio recording, playing football, reading theology books and sword fighting with his four small children. His development blog can be found at http://mark-dot-net.blogspot.com

Introduction

Virtual Earth is the 3D interface to Microsoft's Live Maps service.  Normally this control is loaded via the web browser and allows interaction with a keyboard, mouse, and Xbox 360 controller.  In this article, we will take the Virtual Earth 3D control out of the web browser, use it in a WinForms application, and control it with a Nintendo Wii Remote (Wiimote) and a pair of Vuzix VR920 glasses, while also providing a stereoscopic 3D image to the glasses, creating the illusion of a fully three dimensional environment.  Note that use of the Virtual Earth 3D control in this way is undocumented and unsupported at the moment.  Because of this, some of the descriptions in this article are educated guesses and may not be 100% accurate…

Originally, this project started as a simple Wiimote interface to Virtual Earth 3D as shown in the video below.  Since I wrote that application, I learned of the VR920 glasses and the Wii Fit Balance Board was released, so I've decided to create a more immersive experience using all of these controls which was demonstrated at PDC2008, shown here:

Setup

Before we get started, you will need to install the Virtual Earth 3D control.  If you haven't done this already, browse to http://maps.live.com/ and click on the 3D link to install the control and supporting software.

Additionally, if you haven't already, please review my Managed Library for Nintendo's Wiimote article on this site.  We will be using the library in this article, but I will not repeat the basic information that is located in the original article.  You will also need to have the Vuzix VR920 glasses installed and setup according to its own user manual.  That will also not be covered here.

Implementation

The Virtual Earth 3D Control

The Virtual Earth 3D (VE3D) control is intended to be used through a well documented JavaScript interface from a web page, however we would not be able to access the Wiimote or VR920 glasses from JavaScript.  Therefore, we will be using the VE3D control through its native, but wholly undocumented interface.

Start by creating a new Windows Forms application named WiiEarthVR in C# or VB.  As with all controls and 3rd party libraries, a reference needs to be set to the Virtual Earth 3D libraries.  Add references to the following items: 

• Microsoft.MapPoint.Data
• Microsoft.MapPoint.Data.VirtualEarthTileDataSource
• Microsoft.MapPoint.Geometry
• Microsoft.MapPoint.Rendering3D
• Microsoft.MapPoint.Rendering3D.Utility

If they do not show up in the .NET references, they can be found by selecting the Browse tab and navigating to C:\Program Files\Virtual Earth 3D\ or C:\Program Files (x86)\Virtual Earth 3D\ .  With the references in place, the project file can now be opened and the references will be seen in the References folder in the Solution Explorer as usual.

Creating an instance of the control can be done in code just like any other control.  Used in the constructor or load event of the form, the following code will create a VE3D control and add it to the form as fully docked:

C#

private GlobeControl _globeControl;

public MainForm()

{

    InitializeComponent();

    _globeControl = new GlobeControl();

    SuspendLayout();

    _globeControl.Location = new System.Drawing.Point(0, 0);

    _globeControl.Name = "_globeControl";

    _globeControl.Size = ClientSize;

    _globeControl.Anchor = AnchorStyles.Left | AnchorStyles.Right | AnchorStyles.Top | AnchorStyles.Bottom;

    _globeControl.TabIndex = 0;

    _globeControl.SendToBack(); // we want the button to be on top

    pnlGlobe.Controls.Add(_globeControl);

    ResumeLayout(false);

}

VB

Private _globeControl As GlobeControl

Public Sub New()

    InitializeComponent()

    _globeControl = New GlobeControl()

    SuspendLayout()

    _globeControl.Location = New System.Drawing.Point(0, 0)

    _globeControl.Name = "_globeControl"

    _globeControl.Size = ClientSize

    _globeControl.Anchor = AnchorStyles.Left Or AnchorStyles.Right Or AnchorStyles.Top Or AnchorStyles.Bottom

    _globeControl.TabIndex = 0

    _globeControl.SendToBack() ' we want the button to be on top

    pnlGlobe.Controls.Add(_globeControl)

    ResumeLayout(False)

End Sub

This sets up the VE3D control in its default state.  If you were to run an application with only this code, you would see nothing but the earth.  The navigation controls and other extras would be missing.

We can start adding items to the VE3D control by listening for the FirstFrameRendered event of the GlobeControl and then setting the appropriate properties.  Setting these properties prior to this point can lead to some unexpected results.

In the FirstFrameRendered event handler, if you wish to add the default navigation controls to the screen, the PlugInLoader object is used.  The PlugInLoader is created by using the CreateLoader static method, passing in an instance of the GlobeControl's Host object.  Then, the NavigationPlugIn can be loaded and activated as shown:

C#

// load all the spiffy UI navigation goodies

PlugInLoader loader = PlugInLoader.CreateLoader(this.globeControl.Host);

loader.LoadPlugIn(typeof(NavigationPlugIn));

loader.ActivatePlugIn(typeof(NavigationPlugIn).GUID, null);

VB

' load all the spiffy UI navigation goodies

Dim loader As PlugInLoader = PlugInLoader.CreateLoader(Me.globeControl.Host)

loader.LoadPlugIn(GetType(NavigationPlugIn))

loader.ActivatePlugIn(GetType(NavigationPlugIn).GUID, Nothing)

The last thing to be added for basic functionality is the data.  As it stands, the only data that will appear on the globe is the image of the continents.  Zooming in only produces a blurry representation of that base image.

Data layers are created from specially formatted data sources provided by maps.live.com known as content manifests.  These are XML files which tell the VE3D control how to load the data required for any view.  Content layers can be added by adding them to the DataSources object of the GlobeControl.  We can add any of the following layers (note that there may be other content manifests provided by maps.live.com, but these are the only 5 that I am aware of):

For the best display, add the ElevationMap, Model and Aerial TextureMap layers as shown:

C#

// set various data sources, here for elevation data, terrain data, and model data.

_globeControl.Host.DataSources.Add(new DataSourceLayerData("Elevation", "Elevation", @"http://maps.live.com//Manifests/HD.xml", DataSourceUsage.ElevationMap));

_globeControl.Host.DataSources.Add(new DataSourceLayerData("Texture", "Texture", @"http://maps.live.com//Manifests/AT.xml", DataSourceUsage.TextureMap));

_globeControl.Host.DataSources.Add(new DataSourceLayerData("Models", "Models", @"http://maps.live.com//Manifests/MO.xml", DataSourceUsage.Model));

VB

' set various data sources, here for elevation data, terrain data, and model data.

_globeControl.Host.DataSources.Add(New DataSourceLayerData("Elevation", "Elevation", "http://maps.live.com//Manifests/HD.xml", DataSourceUsage.ElevationMap))

_globeControl.Host.DataSources.Add(New DataSourceLayerData("Texture", "Texture", "http://maps.live.com//Manifests/AT.xml", DataSourceUsage.TextureMap))

_globeControl.Host.DataSources.Add(New DataSourceLayerData("Models", "Models", "http://maps.live.com//Manifests/MO.xml", DataSourceUsage.Model))

By passing the URL of the content manifest, a name for the layer, and what the manifest represents, a new DataSource is created, which is in turn used to create a DataSourceLayerData object which is then given to the VE3D control to consume.  This should be done in the FirstFrameRendered event handler as well.

We also need to setup VE3D to turn off any UI elements, turn on the atmosphere effects, and ensure we have a full unobstructed view. Again, in the FirstFrameRendered event handler, we can use the following code to achieve this:

C#

// turn on the nice atmosphere

_globeControl.Host.WorldEngine.Environment.AtmosphereDisplay = Microsoft.MapPoint.Rendering3D.Atmospherics.EnvironmentManager.AtmosphereStyle.Scattering;

// default to all off

_globeControl.Host.WorldEngine.Display3DCursor = false;

_globeControl.Host.WorldEngine.SetWindowsCursor(null);

_globeControl.Host.WorldEngine.ShowNavigationControl = false;

_globeControl.Host.WorldEngine.ShowCursorLocationInformation = false;

_globeControl.Host.WorldEngine.ShowScale = false;

_globeControl.Host.WorldEngine.ShowUI = false;

_globeControl.Host.WorldEngine.Environment.SunPosition = null;

_globeControl.Host.WorldEngine.Environment.LocalWeatherEnabled = true;

_globeControl.Host.WorldEngine.BaseCopyrightText = " "; // workaround for a display issue

VB

' turn on the nice atmosphere

_globeControl.Host.WorldEngine.Environment.AtmosphereDisplay = Microsoft.MapPoint.Rendering3D.Atmospherics.EnvironmentManager.AtmosphereStyle.Scattering

' default to all off

_globeControl.Host.WorldEngine.Display3DCursor = False

_globeControl.Host.WorldEngine.SetWindowsCursor(Nothing)

_globeControl.Host.WorldEngine.ShowNavigationControl = False

_globeControl.Host.WorldEngine.ShowCursorLocationInformation = False

_globeControl.Host.WorldEngine.ShowScale = False

_globeControl.Host.WorldEngine.ShowUI = False

_globeControl.Host.WorldEngine.Environment.SunPosition = Nothing

_globeControl.Host.WorldEngine.Environment.LocalWeatherEnabled = True

_globeControl.Host.WorldEngine.BaseCopyrightText = " " ' workaround for a display issue

If you were to run the application at this point, you would see a fully functioning Virtual Earth 3D control with proper data and navigation.

Control Scheme and Bindings

The user will control VE3D with the Wiimote by holding the nunchuk in the left hand, which will move the user forward/back/left/right using the joystick.  The C and Z buttons on the front of the nunchuk will be used to raise and lower the altitude of the camera.  The Wiimote, held in the right hand, will be used to toggle various things on or off and interact with menus.  The user will also stand on the Balance Board which will use their center of gravity to turn them in the environment.

VE3D bindings allow you to change or create new control schemes for VE3D by placing an XML file in a specific directory as follows:

• Vista: C:\Users\<username>\AppData\LocalLow\Microsoft\Virtual Earth 3D
• XP: C:\Documents and Settings\<username>\Local Settings\Microsoft\Virtual Earth 3D

In this directory you will find a Bindings.xml file.  This XML schema defines the default keyboard, mouse, Gamepad and other input device properties.  Open the file to see the schema used to define events and parameters.

By default, VE3D will load any file named Bindings*.xml from this directory.  For the Wiimote control scheme, create a new file named BindingsWiiEarthVR.xml in this directory.  Set the contents of the file to the following:

<?xml version="1.0" encoding="utf-8" ?>

<Bindings>

    <BindingSet Name="WiiEarthVRBindings" AutoUse="True" Cursor="Drag">

        <!-- Nunchuk joystick -->

        <Bind Event="Wiimote.NunchukX" Factor="0.5"><Action Name="Strafe"/></Bind>

        <Bind Event="Wiimote.NunchukY" Factor="1"><Action Name="Move"/></Bind>

        <!-- Nunchuk buttons -->

        <Bind Event="Wiimote.NunchukC" Factor="0.20"><Action Name="Ascend"/></Bind>

        <Bind Event="Wiimote.NunchukZ" Factor="-0.20"><Action Name="Ascend"/></Bind>

        <!-- Balance Board -->

        <Bind Event="Wiimote.BalanceBoardX" Factor="-0.0009"><Action Name="Turn"/></Bind>

        <Bind Event="Wiimote.BalanceBoardY" Factor="0.0009"><Action Name="Ascend"/></Bind>

        <!-- Wiimote buttons -->

        <Bind Event="Wiimote.Home"><Action Name="ResetOnCenter"/></Bind>

        <Bind Event="Wiimote.A" Factor="1"><Action Name="Locations, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Left" Factor="-1"><Action Name="Locations, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Up" Factor="-1"><Action Name="LocationsMove, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Down" Factor="1"><Action Name="LocationsMove, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <!-- FPS-style keyboard controls in case we don't have a nunchuk -->

        <Bind Event="Key.W" Factor="22"><Action Name="Move" /></Bind>

        <Bind Event="Key.S" Factor="-22"><Action Name="Move" /></Bind>

        <Bind Event="Key.D" Factor="22"><Action Name="Strafe" /></Bind>

        <Bind Event="Key.A" Factor="-22"><Action Name="Strafe" /></Bind>

        <Bind Event="Key.Space" Factor="20"><Action Name="Ascend" /></Bind>

        <Bind Event="Key.C" Factor="-20"><Action Name="Ascend" /></Bind>

        <!-- Other keys -->

        <Bind Event="Key.F1"><Action Name="VR920SetZero, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Key.F2"><Action Name="BalanceBoardSetZero, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Key.F3"><Action Name="ToggleVR920Stereo, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Minus" Factor="-0.1"><Action Name="VR920SetEyeDistance, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Plus" Factor="0.1"><Action Name="VR920SetEyeDistance, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.One"><Action Name="ToggleBalanceBoard, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Two"><Action Name="ToggleVR920, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Key.B"><Action Name="ToggleBalanceBoard, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Key.V"><Action Name="ToggleVR920, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Key.F"><Action Name="FullScreen, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

    </BindingSet>

</Bindings>

The <BindingSet> tags wrap groups of control bindings.  It requires a Name and optionally a Cursor.  If the binding set is to be used automatically, as it would be in most cases, set the AutoUse parameter to True.  Inside of that are <Bind> tags.  The tag requires the Event parameter and optionally the Factor parameter.  The Event parameter will be used to match the binding to its handler which will be written later.  The syntax is <Handler Name>.<Event Name>.  The Factor parameter is optional and can be used to scale the data value up or down to increase or decrease sensitivity of the input method.  The Action tag inside the Bind tag is used to map the specific binding to a particular method.  Once the handler is written, these will make more sense.

The bindings above create the control scheme described above:  NunchukX/Y describe what happens when the analog joystick is moved, NunchukC/Z describe what happens with the C/Z buttons are pressed, and so on.

The bindings also allow for several variations.  Bindings are defined for both the IR position (IRX, IRY) and accelerometer values (AX, AY).  If an IR sensor bar is not available, the accelerometer values of the Wiimote can be used instead.  Additionally, keyboard bindings are created in the style of a first person shooter using WASD.  These can be used if a Nunchuk is not available.

Note that some bindings append two Events together with a + sign.  This allows for button combinations.  In this case, for the accelerometer and/or IR sensor, we only want to register the action if a button is pressed down.  So, those events which require the button to be held down contain Wiimote.B+ and the event it is combined with.

For those events which require a custom action that will be written separately and not part of the VE3D control, the Action parameter must contain the action name, followed by a comma, and then the full assembly name:

<Bind Event="Wiimote.One">
    <Action Name="ToggleBalanceBoard, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/>
</Bind>

Event Source

An EventSource is needed which will grab data from the Wiimote and pass it along to VE3D as defined by the bindings file above.  Create a new class named WiimoteEventSource which derives from Microsoft.MapPoint.Binding.EventSource  as follows:

Next, add an enumeration named WiimoteEvent (the name isn't important) which contains all of the Name items from the bindings XML file above.  It should look like this:

C#

// all events handled by this event source from XML file
public enum WiimoteEvent
{
    IRX,        // IR X position
    IRY,        // IR Y position
    NunchukX,    // Nunchuk joystick X position
    NunchukY,    // Nunchuk joystick Y position
    NunchukC,    // Nunchuk C button
    NunchukZ,    // Nunchuk Z button
    AX,            // Wiimote accelerometer X
    AY,            // Wiimote accelerometer Y
    Up,            // Dpad up
    Down,        // Dpad down
    Left,        // Dpad left
    Right,        // Dpad right
    A,            // A button
    B,            // B button
    Minus,        // Minus button
    Home,        // Wiimote Home button
    Plus,        // Plus button
    One,        // Wiimote One button
    Two,        // Wiimote Two button
    BalanceBoardX,    // Balance Board COG X
    BalanceBoardY    // Balance Board COG Y
}

VB

' all events handled by this event source from XML file
Public Enum WiimoteEvent
    IRX ' IR X position
    IRY ' IR Y position
    NunchukX ' Nunchuk joystick X position
    NunchukY ' Nunchuk joystick Y position
    NunchukC ' Nunchuk C button
    NunchukZ ' Nunchuk Z button
    AX ' Wiimote accelerometer X
    AY ' Wiimote accelerometer Y
    Up ' Dpad up
    Down ' Dpad down
    Left ' Dpad left
    Right ' Dpad right
    A ' A button
    B ' B button
    Minus ' Minus button
    Home ' Wiimote Home button
    Plus ' Plus button
    One ' Wiimote One button
    Two ' Wiimote Two button
    BalanceBoardX ' Balance Board COG X
    BalanceBoardY ' Balance Board COG Y
End Enum

Next, several methods from the EventSource object need to be overridden:  GetEventData, IsModifier, CanModify, TryGetEventId, TryGetEventName, Name.  The methods do the following:

The code for these methods is presented below:

C#

// return out value of the passed enum

public override bool TryGetEventId(string eventName, out int eventId)

{

    eventId = (int)Enum.Parse(typeof(WiimoteEvent), eventName);

    return true;

}

// return out the string name of the passed in enum value

public override bool TryGetEventName(int eventId, out string eventName)

{

    eventName = ((WiimoteEvent)eventId).ToString();

    return true;

}

// unknown

public override EventData GetEventData(int eventId, EventActivateState state)

{

    throw new NotImplementedException();

}

// can the event be used as a modifier?

public override bool IsModifier(int eventId)

{

    // yes to all for now

    return true;

}

// can the supplied event be used as a modifier?

public override bool CanModify(int eventId, EventKey other)

{

    // only if it's from us

    return (other.Source == this);

}

// this must match the Source name in the bindings XML file

public override string Name

{

    get { return "Wiimote"; }

}

VB

' return out value of the passed enum

Public Overrides Function TryGetEventId(ByVal eventName As String, <System.Runtime.InteropServices.Out()> ByRef eventId As Integer) As Boolean

    eventId = CInt(Fix(System.Enum.Parse(GetType(WiimoteEvent), eventName)))

    Return True

End Function

' return out the string name of the passed in enum value

Public Overrides Function TryGetEventName(ByVal eventId As Integer, <System.Runtime.InteropServices.Out()> ByRef eventName As String) As Boolean

    eventName = (CType(eventId, WiimoteEvent)).ToString()

    Return True

End Function

' unknown

Public Overrides Function GetEventData(ByVal eventId As Integer, ByVal state As EventActivateState) As EventData

    Throw New NotImplementedException()

End Function

' can the event be used as a modifier?

Public Overrides Function IsModifier(ByVal eventId As Integer) As Boolean

    ' yes to all for now

    Return True

End Function

' can the supplied event be used as a modifier?

Public Overrides Function CanModify(ByVal eventId As Integer, ByVal other As EventKey) As Boolean

    ' only if it's from us

    Return (other.Source Is Me)

End Function

' this must match the Source name in the bindings XML file

Public Overrides ReadOnly Property Name() As String

    Get

        Return "Wiimote"

    End Get

End Property

With that in place, the constructor can be implemented which will call the base constructor and connect to the Wiimote.  It is assumed you read the Wiimote article above and know how the library works.

The constructor must take one argument passed from the main from:  an instance of the GlobeControl's ActionSystem.  This just gets passed directly to the parent object's constructor untouched.  The constructor code looks like the following:

C#

public WiimoteEventSource(ActionSystem actionSystem) : base(actionSystem)
{
    // get all connected Wiimotes
    WiimoteCollection wc = new WiimoteCollection();
    wc.FindAllWiimotes();

    // setup wiimotes and event handlers
    foreach(Wiimote wm in wc)
    {
        wm.WiimoteChanged += new EventHandler<WiimoteChangedEventArgs>(OnWiimoteChanged);
        wm.WiimoteExtensionChanged += new EventHandler<WiimoteExtensionChangedEventArgs>(OnWiimoteExtensionChanged);
        wm.Connect();

        // if we don't have an extension, set the report type to IR and accel's only
        if(!wm.WiimoteState.Extension && wm.WiimoteState.ExtensionType != ExtensionType.BalanceBoard)
            wm.SetReportType(InputReport.IRAccel, true);

        if(wm.WiimoteState.ExtensionType == ExtensionType.BalanceBoard)
            _bb = wm;
        else
            _wm = wm;

        // turn off all LEDs
        wm.SetLEDs(0x00);
    }
}

VB

Public Sub New(ByVal actionSystem As ActionSystem)
    MyBase.New(actionSystem)

    ' get all connected Wiimotes
    Dim wc As New WiimoteCollection()
    wc.FindAllWiimotes()

    ' setup wiimotes and event handlers
    For Each wm As Wiimote In wc
        AddHandler wm.WiimoteChanged, AddressOf OnWiimoteChanged
        AddHandler wm.WiimoteExtensionChanged, AddressOf OnWiimoteExtensionChanged
        wm.Connect()

        ' if we don't have an extension, set the report type to IR and accel's only
        If (Not wm.WiimoteState.Extension) AndAlso wm.WiimoteState.ExtensionType <> ExtensionType.BalanceBoard Then
            wm.SetReportType(InputReport.IRAccel, True)
        End If

        If wm.WiimoteState.ExtensionType = ExtensionType.BalanceBoard Then
            _bb = wm
        Else
            _wm = wm
        End If

        ' turn off all LEDs
        wm.SetLEDs(&H00)
    Next wm
End Sub

The OnWiimoteExtensionChanged method simply sets the report mode for the Wiimote based on whether or not a Nunchuk is inserted as shown:

C#

private void OnWiimoteExtensionChanged(object sender, WiimoteExtensionChangedEventArgs args)
{
    if(_wm == null)
        return;

    // if nunchuk inserted, set the report type to return extension data
    if(args.ExtensionType == ExtensionType.Nunchuk && args.Inserted)
        _wm.SetReportType(InputReport.IRExtensionAccel, true);
    else // in all other cases, set it to the default IR and accel's
        _wm.SetReportType(InputReport.IRAccel, true);
}

VB

Private Sub OnWiimoteExtensionChanged(ByVal sender As Object, ByVal args As WiimoteExtensionChangedEventArgs)
    If _wm Is Nothing Then
        Return
    End If

    ' if nunchuk inserted, set the report type to return extension data
    If args.ExtensionType = ExtensionType.Nunchuk AndAlso args.Inserted Then
        _wm.SetReportType(InputReport.IRExtensionAccel, True)
    Else ' in all other cases, set it to the default IR and accel's
        _wm.SetReportType(InputReport.IRAccel, True)
    End If
End Sub

The OnWiimoteChanged event handler is where the Wiimote data is handled and sent off to the VE3D control to reflect the changes.  First, we handle the Balance Board.  If the Balance Board is the controller reporting data, we take the center of gravity values and pass them to VE3D.  This code looks at the appropriate values, determines if they are beyond the specified thresholds for the dead zones, and, if they are, activates the event for that value using the Execute method.  Execute is a method in the base EventSource class.  This method will activate the event specified from the enumeration (which, remember, is contained in the bindings XML file) with the value associated with that event.  An EventData object of some type must be created and passed to the Execute method.  There are two EventData types to know about:  AxisEventData and ButtonEventData.  AxisEventData should be used when an event is activated that will modify the map position in some way.  That is, if the map is being turned, elevation is changing, etc.  ButtonEventData should be used if the event is a simple toggle like pressing a button down and releasing it.

The Balance Board code can be seen below:

C#

if(ws.ExtensionType == ExtensionType.BalanceBoard && this.BalanceBoardEnabled)

{

    float x1 = ws.BalanceBoardState.CenterOfGravity.X;

    if(x1 > Properties.Settings.Default.BBDeadX || x1 < -Properties.Settings.Default.BBDeadX)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.BalanceBoardX), x1 - _zero.X));

    float y1 = ws.BalanceBoardState.CenterOfGravity.Y;

    if(y1 > Properties.Settings.Default.BBDeadY || y1 < -Properties.Settings.Default.BBDeadY)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.BalanceBoardY), y1 - _zero.Y));

}

VB

If ws.ExtensionType = ExtensionType.BalanceBoard AndAlso Me.BalanceBoardEnabled Then

    Dim x1 As Single = ws.BalanceBoardState.CenterOfGravity.X

    If x1 > My.Settings.Default.BBDeadX OrElse x1 < -My.Settings.Default.BBDeadX Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.BalanceBoardX))), x1 - _zero.X))

    End If

    Dim y1 As Single = ws.BalanceBoardState.CenterOfGravity.Y

    If y1 > My.Settings.Default.BBDeadY OrElse y1 < -My.Settings.Default.BBDeadY Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.BalanceBoardY))), y1 - _zero.Y))

    End If

Next, let's handle the IR and accelerometer data.  The IR midpoint of the X and Y axes will be used from the WiimoteState object to activate the IRX and IRY events we defined above in the bindings XML file.  The accelerometer X and Y values will be used to activate the AX and AY events.

This snippet assumes that there is a boolean property named UseIR created in the project to determine whether IR or motion values are used.  Additionally, it assumes there are property settings created which contain values for the X/Y "dead zones" for the IR and accelerometers.  These dead zones are used as a way to only activate the event when the values are pushed beyond the thresholds.  This allows there to be a margin where the user's hand will not be read as movement, allowing the user to not have to worry about keeping a steady hand.

The application linked above uses the following default values for dead zones:

• NunchukDeadX/Y -> 0.025
• WiimoteDeadX/Y -> 0.15
• IRDeadX/Y -> 0.1
• BBDeadX/Y –> 3

C#

// if we're using the IR

if(Properties.Settings.Default.UseIR)

{

    // and both LEDs are found

    if(ws.IRState.Found1 && ws.IRState.Found2)

    {

        // normalize the midpoints to -0.5 to 0.5 (from 0 to 1.0)

        float x = ws.IRState.MidX - 0.5f;

        float y = ws.IRState.MidY - 0.5f;

        // if we're beyond the thresholds, activate the events

        if(x > Properties.Settings.Default.IRDeadX || x < -Properties.Settings.Default.IRDeadX)

            this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.IRX), x));

        if(y > Properties.Settings.Default.IRDeadY || y < -Properties.Settings.Default.IRDeadY)

            this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.IRY), y));

        // save the last IR settings...these get used if we go beyond the range of the IRs.

        // in that case, the last used positions will be used until the Wiimote comes back in range

        this._lastIRX = x;

        this._lastIRY = y;

    }

    else // one or both LEDs aren't seen

    {

        // activate events based on the last known positions

        if(this._lastIRX > Properties.Settings.Default.IRDeadX || this._lastIRX < -Properties.Settings.Default.IRDeadX)

            this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.IRX), this._lastIRX));

        if(this._lastIRY > Properties.Settings.Default.IRDeadY || this._lastIRY < -Properties.Settings.Default.IRDeadY)

            this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.IRY), this._lastIRY));

    }

}

else // we're using motion controls

{

    // activate the events based on the accelerometer values

    if(ws.AccelState.X > Properties.Settings.Default.WiimoteDeadX || ws.AccelState.X < -Properties.Settings.Default.WiimoteDeadX)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.AX), ws.AccelState.X));

    if(ws.AccelState.Y > Properties.Settings.Default.WiimoteDeadY || ws.AccelState.Y < -Properties.Settings.Default.WiimoteDeadY)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.AY), ws.AccelState.Y));

}

VB

' if we're using the IR

If My.Settings.Default.UseIR Then

    ' and both LEDs are found

    If ws.IRState.Found1 AndAlso ws.IRState.Found2 Then

        ' normalize the midpoints to -0.5 to 0.5 (from 0 to 1.0)

        Dim x As Single = ws.IRState.MidX - 0.5f

        Dim y As Single = ws.IRState.MidY - 0.5f

        ' if we're beyond the thresholds, activate the events

        If x > My.Settings.Default.IRDeadX OrElse x < -My.Settings.Default.IRDeadX Then

            Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.IRX))), x))

        End If

        If y > My.Settings.Default.IRDeadY OrElse y < -My.Settings.Default.IRDeadY Then

            Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.IRY))), y))

        End If

        ' save the last IR settings...these get used if we go beyond the range of the IRs.

        ' in that case, the last used positions will be used until the Wiimote comes back in range

        Me._lastIRX = x

        Me._lastIRY = y

    Else ' one or both LEDs aren't seen

        ' activate events based on the last known positions

        If Me._lastIRX > My.Settings.Default.IRDeadX OrElse Me._lastIRX < -My.Settings.Default.IRDeadX Then

            Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.IRX))), Me._lastIRX))

        End If

        If Me._lastIRY > My.Settings.Default.IRDeadY OrElse Me._lastIRY < -My.Settings.Default.IRDeadY Then

            Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.IRY))), Me._lastIRY))

        End If

    End If

Else ' we're using motion controls

    ' activate the events based on the accelerometer values

    If ws.AccelState.X > My.Settings.Default.WiimoteDeadX OrElse ws.AccelState.X < -My.Settings.Default.WiimoteDeadX Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.AX))), ws.AccelState.X))

    End If

    If ws.AccelState.Y > My.Settings.Default.WiimoteDeadY OrElse ws.AccelState.Y < -My.Settings.Default.WiimoteDeadY Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.AY))), ws.AccelState.Y))

    End If

End If

Next, the nunchuk values need to be read and the associated events activated.  This is done as follows:

C#

// if the nunchuk is connected

if(ws.Extension && ws.ExtensionType == ExtensionType.Nunchuk)

{

    // activate the nunchuk-based events

    if(ws.NunchukState.X > Properties.Settings.Default.NunchukDeadX || ws.NunchukState.X < -Properties.Settings.Default.NunchukDeadX)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.NunchukX), ws.NunchukState.X));

    if(ws.NunchukState.Y > Properties.Settings.Default.NunchukDeadY || ws.NunchukState.Y < -Properties.Settings.Default.NunchukDeadY)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.NunchukY), ws.NunchukState.Y));

    if(ws.NunchukState.C)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.NunchukC), 1.0f));

    if(ws.NunchukState.Z)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.NunchukZ), 1.0f));

}

VB

' if the nunchuk is connected

If ws.Extension AndAlso ws.ExtensionType = ExtensionType.Nunchuk Then

    ' activate the nunchuk-based events

    If ws.NunchukState.X > My.Settings.Default.NunchukDeadX OrElse ws.NunchukState.X < -My.Settings.Default.NunchukDeadX Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.NunchukX))), ws.NunchukState.X))

    End If

    If ws.NunchukState.Y > My.Settings.Default.NunchukDeadY OrElse ws.NunchukState.Y < -My.Settings.Default.NunchukDeadY Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.NunchukY))), ws.NunchukState.Y))

    End If

    If ws.NunchukState.C Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.NunchukC))), 1.0f))

    End If

    If ws.NunchukState.Z Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.NunchukZ))), 1.0f))

    End If

End If

Finally, the button events need to be activated.  A helper method which will check the current button state will be used for determining which button of all the Wiimote buttons is pressed.  For those that are, the appropriate event is activated with a call to Execute.

C#

private void HandleButton(WiimoteEvent we, bool buttonState, bool lastButtonState)

{

    if(buttonState == lastButtonState)

        return;

    else

    {

        if(buttonState)

            this.Execute(new ButtonEventData(new EventKey(this, (int)we), EventActivateState.Activate));

        else

            this.Execute(new ButtonEventData(new EventKey(this, (int)we), EventActivateState.Deactivate));

    }

}

// handle all the Wiimote buttons

HandleButton(WiimoteEvent.Up, ws.ButtonState.Up, _lastBS.Up);

HandleButton(WiimoteEvent.Down, ws.ButtonState.Down, _lastBS.Down);

HandleButton(WiimoteEvent.Left, ws.ButtonState.Left, _lastBS.Left);

HandleButton(WiimoteEvent.Right, ws.ButtonState.Right, _lastBS.Right);

HandleButton(WiimoteEvent.A, ws.ButtonState.A, _lastBS.A);

HandleButton(WiimoteEvent.B, ws.ButtonState.B, _lastBS.B);

HandleButton(WiimoteEvent.Minus, ws.ButtonState.Minus, _lastBS.Minus);

HandleButton(WiimoteEvent.Home, ws.ButtonState.Home, _lastBS.Home);

HandleButton(WiimoteEvent.Plus, ws.ButtonState.Plus, _lastBS.Plus);

HandleButton(WiimoteEvent.One, ws.ButtonState.One, _lastBS.One);

HandleButton(WiimoteEvent.Two, ws.ButtonState.Two, _lastBS.Two);

...

// save off the current button state for next time

_lastBS = ws.ButtonState;

_lastNunchuk = ws.NunchukState;

VB

Private Sub HandleButton(ByVal we As WiimoteEvent, ByVal buttonState As Boolean, ByVal lastButtonState As Boolean)

    If buttonState = lastButtonState Then

        Return

    Else

        If buttonState Then

            Me.Execute(New ButtonEventData(New EventKey(Me, CInt(Fix(we))), EventActivateState.Activate))

        Else

            Me.Execute(New ButtonEventData(New EventKey(Me, CInt(Fix(we))), EventActivateState.Deactivate))

        End If

    End If

End Sub

...

' handle all the Wiimote buttons

HandleButton(WiimoteEvent.Up, ws.ButtonState.Up, _lastBS.Up)

HandleButton(WiimoteEvent.Down, ws.ButtonState.Down, _lastBS.Down)

HandleButton(WiimoteEvent.Left, ws.ButtonState.Left, _lastBS.Left)

HandleButton(WiimoteEvent.Right, ws.ButtonState.Right, _lastBS.Right)

HandleButton(WiimoteEvent.A, ws.ButtonState.A, _lastBS.A)

HandleButton(WiimoteEvent.B, ws.ButtonState.B, _lastBS.B)

HandleButton(WiimoteEvent.Minus, ws.ButtonState.Minus, _lastBS.Minus)

HandleButton(WiimoteEvent.Home, ws.ButtonState.Home, _lastBS.Home)

HandleButton(WiimoteEvent.Plus, ws.ButtonState.Plus, _lastBS.Plus)

HandleButton(WiimoteEvent.One, ws.ButtonState.One, _lastBS.One)

HandleButton(WiimoteEvent.Two, ws.ButtonState.Two, _lastBS.Two)

' save off the current button state for next time

_lastBS = ws.ButtonState

_lastNunchuk = ws.NunchukState

And, the current button values are stored away to check on the next event so button events are only fired once.

Now that the event source object is written, it needs to be hooked up to the GlobeControl so it can be used.  This can be done by creating an instance of the WiimoteEventSource object, passing in the VE3D's ActionSystem from the BindingsManager object.  Then, the event source instance is passed to the ActionSystem's EventSourceManager and registered using the RegisterEventSource method.  Event sources should re registered before the control is added to the form.

C#

// wiimote events

private WiimoteEventSource _wiimoteEventSource;

...

// create a new instance of the Wiimote event handler

_wiimoteEventSource = new WiimoteEventSource(this.globeControl.Host.BindingsManager.ActionSystem, this);

// register it in the event source list

_globeControl.Host.BindingsManager.ActionSystem.EventSourceManager.RegisterEventSource(this._wiimoteEventSource);

VB

' wiimote events

Private _wiimoteEventSource As WiimoteEventSource

...

' create a new instance of the Wiimote event handler

_wiimoteEventSource = New WiimoteEventSource(Me.globeControl.Host.BindingsManager.ActionSystem, Me)

' register it in the event source list

_globeControl.Host.BindingsManager.ActionSystem.EventSourceManager.RegisterEventSource(Me._wiimoteEventSource)

Actions and the BindingManager

Our binding list contains several action types that are not defined by the default VE3D actions.  These actions and their handlers must be registered with the VE3D control.  The actions can be registered as follows in the FirstFrameRendered event handler:

C#

BindingsSource bs = new BindingsSource(base.GetType());

_globeControl.Host.BindingsManager.RegisterAction(bs, "Locations", LocationsHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "LocationsMove", LocationsMoveHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleVR920", ToggleVR920Handler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "VR920SetZero", VR920SetZero);

_globeControl.Host.BindingsManager.RegisterAction(bs, "BalanceBoardSetZero", BalanceBoardSetZero);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleBalanceBoard", ToggleBalanceBoardHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "FullScreen", FullScreenHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleVR920Stereo", ToggleVR920Stereo);

_globeControl.Host.BindingsManager.RegisterAction(bs, "VR920SetEyeDistance", VR920SetEyeDistance);

VB

BindingsSource bs = new BindingsSource(base.GetType());

_globeControl.Host.BindingsManager.RegisterAction(bs, "Locations", LocationsHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "LocationsMove", LocationsMoveHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleVR920", ToggleVR920Handler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "VR920SetZero", VR920SetZero);

_globeControl.Host.BindingsManager.RegisterAction(bs, "BalanceBoardSetZero", BalanceBoardSetZero);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleBalanceBoard", ToggleBalanceBoardHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "FullScreen", FullScreenHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleVR920Stereo", ToggleVR920Stereo);

_globeControl.Host.BindingsManager.RegisterAction(bs, "VR920SetEyeDistance", VR920SetEyeDistance);

With the actions registered and handlers associated with them, the actual handlers need to be implemented.  All event handler methods must be of the following signature:

C#

public bool EventHandler(EventData cause)

VB

Public Function EventHandler(ByVal cause As EventData) As Boolean

Let's take a look at the FullScreen binding which simply turns the status bar at the bottom of the window on and off:

C#

private bool FullScreenHandler(EventData eventData)
{
    if(eventData.Activate)
        BeginInvoke(new UIEventHandlerDelegate(FullScreen), eventData);
    return true;
}

VB

Private Function FullScreenHandler(ByVal eventData As EventData) As Boolean
    If eventData.Activate Then
        BeginInvoke(New UIEventHandlerDelegate(AddressOf FullScreen), eventData)
    End If
    Return True
End Function

Private Sub FullScreen(ByVal eventData As EventData)
    statusStrip1.Visible = Not statusStrip1.Visible
End Sub

Because we are in the VE3D thread when this handler is called, we need to use BeginInvoke to call the real method on the UI thread.

Be sure to check the source code for the full demo linked above for the location handler methods.  I omitted them here since it is just more of the same type of code above.

VR920 Head Tracker

Once the VR920 glasses and driver are installed on your PC, getting the data required from the glasses is quite easy.  We are going to create an object named VR920Tracker which will pull data from the device and send it to VE3D.

First we need to setup a few P/Invoke signatures to talk to the glasses as follows:

C#

[DllImport("IWEARDRV.dll", SetLastError = true, CharSet = CharSet.Auto)]

private static extern int IWROpenTracker();

[DllImport("IWEARDRV.dll", SetLastError = true, CharSet = CharSet.Auto)]

private static extern void IWRZeroSet();

[DllImport("IWEARDRV.dll", SetLastError = true, CharSet = CharSet.Auto)]

private static extern int IWRGetTracking(out int yaw, out int pitch, out int roll);

private const int ERROR_SUCCESS  = 0;

VB

<DllImport("IWEARDRV.dll", SetLastError := True, CharSet := CharSet.Auto)> _

Private Shared Function IWROpenTracker() As Integer

End Function

<DllImport("IWEARDRV.dll", SetLastError := True, CharSet := CharSet.Auto)> _

Private Shared Sub IWRZeroSet()

End Sub

<DllImport("IWEARDRV.dll", SetLastError := True, CharSet := CharSet.Auto)> _

Private Shared Function IWRGetTracking(<System.Runtime.InteropServices.Out()> ByRef yaw As Integer, <System.Runtime.InteropServices.Out()> ByRef pitch As Integer, <System.Runtime.InteropServices.Out()> ByRef roll As Integer) As Integer

End Function

private const int ERROR_SUCCESS  = 0;

These 3 methods will allows to open access to the glasses, set its zero position, and get the roll, pitch and yaw information from the sensors.

In the constructor for this object, we will open a handle to the glasses and startup a timer that will poll the glasses at a regular interval to get the sensor data as shown below:

C#

private const int TimerPeriod = 1;

private MainForm _mainForm;

private Timer _timer;

public VR920Tracker(MainForm mainForm)

{

    _mainForm = mainForm;

    int openResult = IWROpenTracker();

    if(openResult != ERROR_SUCCESS)

        throw new ApplicationException("Could not connect to VR920: " + openResult);

    _timer = new Timer(VR920Poller, null, Timeout.Infinite, TimerPeriod);

}

VB

Private Const TimerPeriod As Integer = 1

Private _mainForm As MainForm

Private _timer As Timer

Public Sub New(ByVal mainForm As MainForm)

    _mainForm = mainForm

    Dim openResult As Integer = IWROpenTracker()

    If openResult <> ERROR_SUCCESS Then

        Throw New ApplicationException("Could not connect to VR920: " & openResult)

    End If

    _timer = New Timer(AddressOf VR920Poller, Nothing, Timeout.Infinite, TimerPeriod)

End Sub

The VR920Poller method referenced above will be called at a specific interval and read the values from the glasses as shown below:

C#

// collection of values for averaging

private List<double> _yawValues = new List<double>();

private List<double> _rollValues = new List<double>();

private List<double> _pitchValues = new List<double>();

// last calculated values

private double _lastYaw;

private double _lastPitch;

private double _lastRoll;

private void VR920Poller(object state)

{

    lock(this)

    {

        int yaw, pitch, roll;

        int result = IWRGetTracking(out yaw, out pitch, out roll);

        if(result != ERROR_SUCCESS)

            throw new ApplicationException("Could not get VR920 tracking information: " + result);

        _yawValues.Add(VR920ToRadians(yaw));

        _rollValues.Add(VR920ToRadians(roll));

        _pitchValues.Add(VR920ToRadians(pitch));

        if(_yawValues.Count == 5)

        {

            double y = Average(_yawValues, _lastYaw);

            if(Math.Abs(y - _lastYaw) > 0.026)

                _lastYaw = y;

            double p = Average(_pitchValues, _lastPitch);

            if(Math.Abs(p - _lastPitch) > 0.017)

                _lastPitch = p;

            _lastRoll = Average(_rollValues, _lastRoll);

            _yawValues.Clear();

            _pitchValues.Clear();

            _rollValues.Clear();

            RollPitchYaw rollPitchYaw = new RollPitchYaw(_lastRoll, _lastPitch, _lastYaw);

            _mainForm.SetRollPitchYaw(rollPitchYaw);

            _mainForm.lblAxes.Text = VR920ToDegrees(yaw) + ", " + VR920ToDegrees(pitch) + ", " + VR920ToDegrees(roll);

        }

    }

}

private double Average(List<double> values, double last)

{

    double total = 0;

    foreach(double value in values)

        total += value;

    total += last;

    return total / (values.Count+1);

}

private double VR920ToRadians(int vr920Value)

{

    return (vr920Value * .00549) * (Math.PI/180);

}

private double VR920ToDegrees(int vr920Value)

{

    return (vr920Value * .00549);

}

VB

' collection of values for averaging

Private _yawValues As List(Of Double) = New List(Of Double)()

Private _rollValues As List(Of Double) = New List(Of Double)()

Private _pitchValues As List(Of Double) = New List(Of Double)()

' last calculated values

Private _lastYaw As Double

Private _lastPitch As Double

Private _lastRoll As Double

Private Sub VR920Poller(ByVal state As Object)

    SyncLock Me

        Dim yaw, pitch, roll As Integer

        Dim result As Integer = IWRGetTracking(yaw, pitch, roll)

        If result <> ERROR_SUCCESS Then

            Throw New ApplicationException("Could not get VR920 tracking information: " & result)

        End If

        _yawValues.Add(VR920ToRadians(yaw))

        _rollValues.Add(VR920ToRadians(roll))

        _pitchValues.Add(VR920ToRadians(pitch))

        If _yawValues.Count = 5 Then

            Dim y As Double = Average(_yawValues, _lastYaw)

            If Math.Abs(y - _lastYaw) > 0.026 Then

                _lastYaw = y

            End If

            Dim p As Double = Average(_pitchValues, _lastPitch)

            If Math.Abs(p - _lastPitch) > 0.017 Then

                _lastPitch = p

            End If

            _lastRoll = Average(_rollValues, _lastRoll)

            _yawValues.Clear()

            _pitchValues.Clear()

            _rollValues.Clear()

            Dim rollPitchYaw As New RollPitchYaw(_lastRoll, _lastPitch, _lastYaw)

            _mainForm.SetRollPitchYaw(rollPitchYaw)

            _mainForm.lblAxes.Text = VR920ToDegrees(yaw) & ", " & VR920ToDegrees(pitch) & ", " & VR920ToDegrees(roll)

        End If

    End SyncLock

End Sub

Private Function Average(ByVal values As List(Of Double), ByVal last As Double) As Double

    Dim total As Double = 0

    For Each value As Double In values

        total += value

    Next value

    total += last

    Return total / (values.Count+1)

End Function

Private Function VR920ToRadians(ByVal vr920Value As Integer) As Double

    Return (vr920Value *.00549) * (Math.PI/180)

End Function

Private Function VR920ToDegrees(ByVal vr920Value As Integer) As Double

    Return (vr920Value *.00549)

End Function

This chunk calls the IWRGetTracking method and tosses the results in 3 lists for roll, pitch and yaw.  then, when the list has 5 members in it, the values are averaged and, if that resulting values is past a certain threshold, the values are passed back to the MainForm's SetRollPitchYaw method as shown below:

C#

private double _lastYaw;

public void SetRollPitchYaw(RollPitchYaw rollPitchYaw)

{

    if(!_initialized || _globeControl.IsDisposed || (_globeControl.Host) == null || _globeControl.Host.CameraControllers.Current == null)

        return;

    double y = ((_globeControl.Host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.LocalOrientation.Yaw - _lastYaw) + rollPitchYaw.Yaw;

    RollPitchYaw rpw = new RollPitchYaw(rollPitchYaw.Roll, rollPitchYaw.Pitch, y);

    (_globeControl.Host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.LocalOrientation.RollPitchYaw = rpw;

    _lastYaw = rollPitchYaw.Yaw;

}

VB

Private _lastYaw As Double

Public Sub SetRollPitchYaw(ByVal rollPitchYaw As RollPitchYaw)

    If (Not _initialized) OrElse _globeControl.IsDisposed OrElse (_globeControl.Host) Is Nothing OrElse _globeControl.Host.CameraControllers.Current Is Nothing Then

        Return

    End If

    Dim y As Double = ((TryCast(_globeControl.Host.CameraControllers.Current, ActionCameraController)).LastReportedViewpoint.LocalOrientation.Yaw - _lastYaw) + rollPitchYaw.Yaw

    Dim rpw As New RollPitchYaw(rollPitchYaw.Roll, rollPitchYaw.Pitch, y)

    TryCast(_globeControl.Host.CameraControllers.Current, ActionCameraController).LastReportedViewpoint.LocalOrientation.RollPitchYaw = rpw

    _lastYaw = rollPitchYaw.Yaw

End Sub

This method takes the roll, pitch and yaw values provided and applies them to the current CameraController used by VE3D (which, by default, is an ActionCameraController).

So, this code ultimately takes the roll, yaw and pitch of the user's head and translates that directly to the roll, yaw and pitch of the camera in VE3D providing an accurate, real-time view into the VE3D world.

Stereoscopic Images

The final piece of the puzzle is drawing frames to the VR920 glasses in such a way that the final view to the user will be three dimensional.  This works similarly to those Magic Eye puzzles you may have seen.  Essentially, for every frame, we want to take the current VE3D camera position and move it several units to the left, render that to the left eye of the glasses, and then move the camera several units to the right and render that to the right eye of the glasses.  To handle this we,  can create an object named VR920StereoStep which derives from the Step class provided by VE3D and add this to the StepManager.  When this is done, the VR920StereoStep will be called at the end of every frame drawn by VE3D.

As with the VR920Tracker class, we will need to setup a few P/Invoke method signatures to talk to the stereo driver as shown below:

C#

[DllImport("iWrstDrv.dll", EntryPoint = "IWRSTEREO_Open", SetLastError=true)]

public static extern IntPtr OpenStereo();

[DllImport("iWrstDrv.dll", EntryPoint = "IWRSTEREO_SetStereo")]

public static extern Boolean SetStereoEnabled(IntPtr handle, Boolean enabled);

[DllImport("iWrstDrv.dll", EntryPoint = "IWRSTEREO_SetLR")]

public static extern Boolean SetStereoLR(IntPtr handle, Boolean eye);

[DllImport("iWrstDrv.dll", EntryPoint = "IWRSTEREO_WaitForAck")]

public static extern Byte WaitForOpenFrame(IntPtr handle, Boolean eye);

VB

<DllImport("iWrstDrv.dll", EntryPoint := "IWRSTEREO_Open", SetLastError:=True)> _

Public Shared Function OpenStereo() As IntPtr

End Function

<DllImport("iWrstDrv.dll", EntryPoint := "IWRSTEREO_SetLR")> _

Public Shared Function SetStereoLR(ByVal handle As IntPtr, ByVal eye As Boolean) As Boolean

End Function

<DllImport("iWrstDrv.dll", EntryPoint := "IWRSTEREO_SetStereo")> _

Public Shared Function SetStereoEnabled(ByVal handle As IntPtr, ByVal enabled As Boolean) As Boolean

End Function

<DllImport("iWrstDrv.dll", EntryPoint := "IWRSTEREO_WaitForAck")> _

Public Shared Function WaitForOpenFrame(ByVal handle As IntPtr, ByVal eye As Boolean) As Byte

End Function

In the constructor for this object, we will open a handle to the stereo driver and turn on the stereo functionality of the glasses as shown below:

C#

private static readonly IntPtr INVALID_FILE_HANDLE = (IntPtr)(-1);

private IntPtr _hStereo = INVALID_FILE_HANDLE;

private Host _host;

private bool _stereoEnabled = true;

public double EyeDistance { get; set; }

public VR920StereoStep(StepManager manager, Host host) : base(manager)

{

    EyeDistance = 10;

    _host = host;

    _hStereo = OpenStereo();

    if(_hStereo != INVALID_FILE_HANDLE)

        SetStereoEnabled(_hStereo, true);

    else

        _stereoEnabled = false;

}

VB

Private Shared ReadOnly INVALID_FILE_HANDLE As IntPtr = CType(-1, IntPtr)

Private _hStereo As IntPtr = INVALID_FILE_HANDLE

Private _host As Host

Private _stereoEnabled As Boolean = True

Private privateEyeDistance As Double

Public Property EyeDistance() As Double

    Get

        Return privateEyeDistance

    End Get

    Set(ByVal value As Double)

        privateEyeDistance = value

    End Set

End Property

Public Sub New(ByVal manager As StepManager, ByVal host As Host)

    MyBase.New(manager)

    EyeDistance = 10

    _host = host

    _hStereo = OpenStereo()

    If _hStereo <> INVALID_FILE_HANDLE Then

        SetStereoEnabled(_hStereo, True)

    Else

        _stereoEnabled = False

    End If

End Sub

Next, we need to override the OnExecute method provided by the base Step class as shown below:

C#

private bool _rightEye = true;

private Vector3D _position;

public override void OnExecute(SceneState state)

{

    if(_hStereo != INVALID_FILE_HANDLE && _stereoEnabled && (_host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint != null)

    {

        if(!_rightEye)

        {

            _position = (_host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.Position.Vector;

            (_host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.Position.Vector = _position + new Vector3D(-EyeDistance, 0, 0);

        }

        else

        {

            _position = (_host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.Position.Vector;

            (_host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.Position.Vector = _position + new Vector3D(EyeDistance, 0, 0);;

        }

        SetStereoLR(_hStereo, _rightEye);

        WaitForOpenFrame(_hStereo, _rightEye);

        _rightEye = !_rightEye;

    }

}

VB

Private _rightEye As Boolean = True

Private _position As Vector3D

Public Overrides Sub OnExecute(ByVal state As SceneState)

    If _hStereo <> INVALID_FILE_HANDLE AndAlso _stereoEnabled AndAlso (TryCast(_host.CameraControllers.Current, ActionCameraController)).LastReportedViewpoint IsNot Nothing Then

        If (Not _rightEye) Then

            _position = (TryCast(_host.CameraControllers.Current, ActionCameraController)).LastReportedViewpoint.Position.Vector

            TryCast(_host.CameraControllers.Current, ActionCameraController).LastReportedViewpoint.Position.Vector = _position + New Vector3D(-EyeDistance, 0, 0)

        Else

            _position = (TryCast(_host.CameraControllers.Current, ActionCameraController)).LastReportedViewpoint.Position.Vector

            TryCast(_host.CameraControllers.Current, ActionCameraController).LastReportedViewpoint.Position.Vector = _position + New Vector3D(EyeDistance, 0, 0)

        End If

        SetStereoLR(_hStereo, _rightEye)

        WaitForOpenFrame(_hStereo, _rightEye)

        _rightEye = Not _rightEye

    End If

End Sub

OnExecute will be called at the end of each VE3D frame rendered.  In this method, we obtain the current position of the VE3D camera and then set its position several units to the left or right (- or + EyeDistance).  SetStereoLR from the VR920 API is called, setting the appropriate eye to be active (left = false, right = true).  Then WaitForOpenFrame is called which will pause the action long enough for the glasses to draw the entire frame to eye to avoid image tearing.  Essentially, this call “sits and spins” at this location until the glasses report back that the entire frame is drawn in the proper eye and that we can continue on drawing the opposite eye for the next frame.  Finally, the _rightEye member variable is togged to the opposite value, so the next time through this method the opposite eye is drawn.

With this class in place, back in our MainForm in the FirstFrameRendered event handler, we can instantiate the object and add it to the StepManager as shown below:

C#

private VR920StereoStep _vr920StereoStep;

_vr920StereoStep = new VR920StereoStep(_globeControl.Host.RenderEngine.StepManager, _globeControl.Host);

_globeControl.Host.RenderEngine.StepManager.Add(_vr920StereoStep);

VB

Private _vr920StereoStep As VR920StereoStep

_vr920StereoStep = New VR920StereoStep(_globeControl.Host.RenderEngine.StepManager, _globeControl.Host)

_globeControl.Host.RenderEngine.StepManager.Add(_vr920StereoStep)

Running the Application

To run the demo, do the following:

1. Copy BindingsWiimote.xml to the appropriate directory listed above
2. Pair the Wiimote and Balance Board to the computer.  See the WiimoteLib article for more information on how to do that
3. Run the executable
4. Stand on the Balance Board
5. Put on the glasses
6. Zero both the Balance Board and glasses
7. Toggle everything on or off at will

Controls

• F1 – Set VR920 glasses to zero position
• F2 – Set Balance Board to zero position
• F3 – Toggle stereo mode on and off
• Wiimote 1 or B – Toggle Balance Board on or off
• Wiimote 2 or V – Toggle VR920 head tracking on or off
• F – Toggle full-screen on or off
• Nunchuk joystick X/Y – Strafe/Move
• Nunchuk C/Z buttons – Raise/lower altitude
• Wiimote A – Open location menu/Select location (note that if a new location is selected, the Balance Board is turned off and must be re-enabled after “landing”
• Wiimote Dpad Up/Down – Move through List
• Balance Board – Lean your body left or right to turn in the VE3D environment

Conclusion

With the above code, we have created a very interesting interface for Virtual Earth 3D.  The demo and source code linked above contain a few more features and bindings which enhance the application a bit more.  Be sure to give the full demo a try and check out the full source code for a few more implementation details.

Additional Information

• Vuzix VR920 Glasses
• Managed Library for Nintendo's Wiimote
• Heptazane's Blog
• Virtual Earth Blog
• Virtual Earth Developer's Blog

Bio

Brian is a Microsoft C# MVP who has been actively developing in .NET since its early betas in 2000, and who has been developing solutions using Microsoft technologies and platforms for even longer. Along with .NET, Brian is particularly skilled in the languages of C, C++ and assembly language for a variety of CPUs. He is also well-versed in a wide variety of technologies including web development, document imaging, GIS, graphics, game development, and hardware interfacing. Brian has a strong background in developing applications for the health-care industry, as well as developing solutions for portable devices, such as tablet PCs and PDAs. Additionally, Brian has co-authored the book "Debugging ASP.NET" published by New Riders, and is currently co-authoring a book titled "Coding4Fun: 10 .NET Programming Projects for Wiimote, YouTube, World of Warcraft, and More" to be published by O'Reilly in December 2008. Brian is also an author for MSDN's Coding4Fun website.  You can reach Brian via vis website at http://www.brianpeek.com/ .

Clint Rutkas - Academic Developer Evangelist - Microsoft
Monkey see, Monkey Build

Difficulty: Intermediate
Time Required: 6-10 hours
Cost: Free
Software: Visual Studio Express, Silverlight SDK and Runtime for 1.1 2.0 beta2
Download: Download c# - Download VB Updated to Silverlight 2.0 Beta - Download c#

Updates:

• Fixed the Silverlight.Js, used the new version that is included with the 1.1 Refresh SDK so the proper installer will prompt now.  Downloads are corrected also too.
• 6/20/2008 - Verified the solution works with Silverlight 2.0beta2.

Spec's, Requirements and headaches

So I had a few mental requirements for the application to "entertain" myself. 

1. Be able to rotate the video blocks
2. Be able to translate
3. Be able to increase the difficulty of the puzzle on the fly.
4. Zero interaction with the keyboard

Simple, right?  You want to see a demo too see some awesomeness?  No problem, head over to my site and see it in person.

Microsoft Surface Video Puzzle

Clint's Silverlight Video Puzzle

To the Internet Batman!

To help save time building this, I first when out and attempting to see if anything has been done anything close to this before.  I found

Also I need 2 small controls which I found out are in the Silverlight SDK.  I needed horizontal slider controls for this app.  While I know a text box control would have worked just as well, see requirement 4.

On the Silverlight community site, there is also a very nice demo showing the photo application in Surface with source code.  This is where I learned the majority of what I needed.  This application was very close to where I needed to go with mine.  However, while it was close, it wasn't perfect.  Parts of the photo application code are pretty much copied.  This isn't the best way of doing this, but it is one way of doing it.

Toolbelt: check, Keyboard: check, Band-aids: check

So on to building it.  I first started off with a new project and some XAML.

After clicking "OK", I get a very nice blank project.

We'll want to add an additional XAML object for the video blocks.  Go to "Project -> Add New Item", select Silverlight Page, name it "VideoBlock.xaml" and click "OK".

So now we have everything we need for the most part, grab any video you have, if you don't have one, go to a site like TeamXbox and grab one from there.  I used Windows Movie Maker and grabbed a snippet from The Office (best show ever).

In addition to all these files, we need to add in a reference to the Silverlight SDK to get a hold of the Slider controls. Go to "Project -> Add Reference", and add in the Silverlight.Samples.Controls.dll.

While we have a reference in the application, we still need the XAML to know about this also.  So we'll add a line to link this in.  It is the xmlns line that I bolded, XML name space, real clever, eh?  We'll also  change the background color to a gray so we can see it more easy.

<Canvas x:Name="parentCanvas"
        xmlns="http://schemas.microsoft.com/client/2007" 
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
        Loaded="Page_Loaded" 
        x:Class="c4f_SilverlightVideoDemo.VideoBlock;assembly=ClientBin/c4f_SilverlightVideoDemo.dll"
        xmlns:uicontrol="clr-namespace:Silverlight.Samples.Controls;assembly=ClientBin/Silverlight.Samples.Controls.dll"
        Background="#CCCCCC" Width="640" Height="480">
</Canvas>

Silverlight 101

So in Silverlight, you have brushes which can "painted" onto objects.  One of these brushes is called the VideoBrush which I'll talk about later on.  For playing videos, one would use the MediaElement.  Lets add this to our canvas.

<MediaElement x:Name="media" Source="theOffice.wmv" />

Now if we run the application at this point, we'll get the video playing.  Lets sit back with some popcorn and enjoy this victory for a second or two.  Now that we have this, lets hide it since this isn't what we want.  We do that by adding in an XML attribute Opacity and setting it to 0.

On top of that element, lets add on some TextBlock elements and the sliders.

<Canvas x:Name="parentCanvas"
        xmlns="http://schemas.microsoft.com/client/2007" 
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
        Loaded="Page_Loaded" 
        x:Class="VideoCreatedObjectTest.Page;assembly=ClientBin/VideoCreatedObjectTest.dll"
        xmlns:uicontrol="clr-namespace:Silverlight.Samples.Controls;assembly=ClientBin/Silverlight.Samples.Controls.dll" 
        Background="#CCCCCC" >
    <MediaElement x:Name="media" Source="theOffice.wmv" Opacity="0" />
    
    <TextBlock x:Name="sliderX" Canvas.Top="5" Canvas.Left="5" Text="X (01):" />
    <TextBlock x:Name="sliderY"  Canvas.Top="21" Canvas.Left="5" Text="Y (01):" />
    
    <uicontrol:Slider x:Name="hSliderX" Canvas.Top="8" Canvas.Left="55" />
    <uicontrol:Slider x:Name="hSliderY" Canvas.Top="25" Canvas.Left="55" />
</Canvas>

So now when we run the application, we get a blank screen but we can hear the audio from the video.

Super exciting I know.  Now if you resize the browser, you'll notice also the canvas isn't resizing automatically.  Lets fix that.

So in the code behind lets add in a new event on the BrowserHost object with the Resize event.  We'll attach the Browser_Resize event to it.

C#

private void BrowserHost_Resize(object sender, EventArgs e)
{
    // Set size to host size
    Width = BrowserHost.ActualWidth;
    Height = BrowserHost.ActualHeight;
}

VB.Net

Private Sub BrowserHost_Resize(ByVal sender As Object, ByVal e As EventArgs)
    ' Set size to host size
    Width = BrowserHost.ActualWidth
    Height = BrowserHost.ActualHeight
End Sub

Video as Paint?

So now that we have that small part done, lets get into more of the nitty gritty with video.  As I said earlier, you'll use a VideoBrush to paint on the video.

We'll use the brush as the Fill for the rectangle we'll be using to show off the clip of the video we want.

Still not moving?  Try explosives?

While doing this project, I discovered an interesting thing when using the VideoBrush and dynamically appending items on.  Having items in the XAML rather than programmatically creating everything caused it not to work properly.  Nothing would appear.

Throwback to how UI's use to be done

The XAML for this is a simplified version of the Surface demo Photo.XAML.  You have a 3 Rectangles, 1 for the video, 1 for translation and 1 for rotation.  Simple, no?  Doing all this by hand gets a bit annoying since you need to nest everything properly.

Here is the more interesting VideoBrush code.  You need the name from the MediaElement along with the how much of an offset you want the video.  Since your moving the "camera" in, this value needs to be negative.

C#

public delegate void SetActiveVideo(VideoBlock videoBlock, ActionType actionType, Point photoCenter, Point lastPosition);
public VideoBlock(string mediaName, int offsetX, int offsetY, int rectWidth, int rectHeight, SetActiveVideo functionPointer)
{
    // additional code here
    Rectangle video = new Rectangle();
    VideoBrush vb = new VideoBrush();
    TranslateTransform videoTransform = new TranslateTransform();
    videoTransform.X = -1 * offsetX;
    videoTransform.Y = -1 * offsetY;

    vb.SourceName = mediaName;
    vb.Transform = videoTransform;
    vb.Stretch = Stretch.None;
    vb.AlignmentX = AlignmentX.Left;
    vb.AlignmentY = AlignmentY.Top;

    video.Width = Width;
    video.Height = Height;
    video.Fill = vb;
    // additional code here
}

VB.Net

Public Delegate Sub SetActiveVideo(ByVal videoBlock As VideoBlock, ByVal actionType As ActionType, ByVal photoCenter As Point, ByVal lastPosition As Point)
Public Sub VideoBlock(ByVal mediaName As String, ByVal offsetX As Integer, ByVal offsetY As Integer, ByVal rectWidth As Integer, ByVal rectHeight As Integer, ByVal functionPointer As SetActiveVideo)
      ' additional code here
      Dim video As Rectangle = New Rectangle
      Dim vb As VideoBrush = New VideoBrush
      Dim videoTransform As TranslateTransform = New TranslateTransform
      videoTransform.X = ((1 * offsetX)  * -1)
      videoTransform.Y = ((1 * offsetY)  * -1)
      vb.SourceName = mediaName
      vb.Transform = videoTransform
      vb.Stretch = Stretch.None
      vb.AlignmentX = AlignmentX.Left
      vb.AlignmentY = AlignmentY.Top
      video.Width = Width
      video.Height = Height
      video.Fill = vb
      ' additional code here
End Sub

There is a mouse on my object!

So now we have our video block element prepped and ready.  There are additional events added in, but they are fairly boiler plate.  When looking at this code, you see I pass in a delegate, a function pointer if you will.  Why do I do this?  This is how the mouse position is calculated.  I bet this could be redone better, but this is how I did it.  The parent element, the canvas, has the mouse position I need to calculate proper translation and rotation, however, mouse events on the VideoBlock object will give me the mouse position only on that element.  So I may be at point 150, 150 on the screen, on the VideoBlock object, I'm actually at 10,15.

Additionally, this approach allows me to move the VideoBlock to the top of the objects.  Once again, this could be done other ways, this is how I chose to do it.

So on all my mouse related all boils down to doing the exact same function.  Since events are bubbled, a mouse movement on a VideoBlock is also a mouse movement on the root canvas.

C#

private void root_MouseLeftButtonDown(object sender, MouseEventArgs e)
{
    HandleMouseLeftButtonDown(ActionType.Selecting, e);
}

private void translateControls_MouseLeftButtonDown(object sender, MouseEventArgs e)
{
    HandleMouseLeftButtonDown(ActionType.Moving, e);
}

private void rotateScaleControls_MouseLeftButtonDown(object sender, MouseEventArgs e)
{
    HandleMouseLeftButtonDown(ActionType.RotatingScaling, e);
}

private void HandleMouseLeftButtonDown(ActionType actionType, MouseEventArgs e)
{
    if (functionPointer != null)
        functionPointer(this, actionType,
            new Point(translateTransform.X + rotateTransform.CenterX,
                  translateTransform.Y + rotateTransform.CenterY), e.GetPosition(null));
}

VB.Net

    Private Sub root_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseEventArgs)
        HandleMouseLeftButtonDown(ActionType.Selecting, e)
    End Sub
    
    Private Sub translateControls_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseEventArgs)
        HandleMouseLeftButtonDown(ActionType.Moving, e)
    End Sub
    
    Private Sub rotateScaleControls_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseEventArgs)
        HandleMouseLeftButtonDown(ActionType.RotatingScaling, e)
    End Sub
    
    Private Sub HandleMouseLeftButtonDown(ByVal actionType As ActionType, ByVal e As MouseEventArgs)
        If (Not (functionPointer) Is Nothing) Then
            functionPointer(Me, actionType, New Point((translateTransform.X + rotateTransform.CenterX), 
              (translateTransform.Y + rotateTransform.CenterY)), e.GetPosition(Nothing))
        End If
    End Sub

You built a car out of all these parts?

So we move back from VideoBlock.xaml.cs to Page.xaml.cs

We've created our primary object for painting video, lets populate it.  First, how do we know how big the media file is?  With the MediaOpened event!  And we'll restart the video with the MediaEnded event.

So here is the full Page_Load event from Page.xaml.cs

C#

public void Page_Loaded(object o, EventArgs e)
{
    // Required to initialize variables
    InitializeComponent();

    BrowserHost.Resize +=new EventHandler(BrowserHost_Resize);
    media.MediaOpened += new EventHandler(media_MediaOpened);
    media.MediaEnded += new EventHandler(media_MediaEnded);
    MouseLeftButtonUp += Page_MouseLeftButtonUpOrLeave;
    MouseLeave += Page_MouseLeftButtonUpOrLeave;
    MouseMove += Page_MouseMove;

    hSliderX.Range = new ValueRange(1, 20);
    hSliderY.Range = new ValueRange(1, 20);
    hSliderX.ValueChanged += new EventHandler(slider_ValueChanged);
    hSliderY.ValueChanged += new EventHandler(slider_ValueChanged);

    hSliderX.SetValue(ZIndexProperty, 500);
    hSliderY.SetValue(ZIndexProperty, 500);
    sliderX.SetValue(ZIndexProperty, 500);
    sliderY.SetValue(ZIndexProperty, 500);
}

VB.Net

    Public Sub Page_Loaded(ByVal o As Object, ByVal e As EventArgs)
        ' Required to initialize variables
        InitializeComponent
        AddHandler BrowserHost.Resize, AddressOf Me.BrowserHost_Resize
        AddHandler media.MediaOpened, AddressOf Me.media_MediaOpened
        AddHandler media.MediaEnded, AddressOf Me.media_MediaEnded
        MouseLeftButtonUp = (MouseLeftButtonUp + Page_MouseLeftButtonUpOrLeave)
        MouseLeave = (MouseLeave + Page_MouseLeftButtonUpOrLeave)
        MouseMove = (MouseMove + Page_MouseMove)
        hSliderX.Range = New ValueRange(1, 20)
        hSliderY.Range = New ValueRange(1, 20)
        AddHandler hSliderX.ValueChanged, AddressOf Me.slider_ValueChanged
        AddHandler hSliderY.ValueChanged, AddressOf Me.slider_ValueChanged
        hSliderX.SetValue(ZIndexProperty, 500)
        hSliderY.SetValue(ZIndexProperty, 500)
        sliderX.SetValue(ZIndexProperty, 500)
        sliderY.SetValue(ZIndexProperty, 500)
    End Sub

So now that we have the media loaded, we can use the media.NaturalVideoWidth and media.NaturalVideoHeight properties to find out exactly how big the video is!  I have a function that is fired off when the page is first loaded or a slider value has changed.  This is how we chop it up.

C#

private void prepVideoObjects()
{
    int totalPhotosX = (int)hSliderX.Value;
    int totalPhotosY = (int)hSliderY.Value;

    if (lastX != totalPhotosX || lastY != totalPhotosY)
    {
        lastX = totalPhotosX;
        lastY = totalPhotosY;

        sliderX.Text = string.Format("X ({0}):", totalPhotosX.ToString("D2"));
        sliderY.Text = string.Format("Y ({0}):", totalPhotosY.ToString("D2"));

        int rectWidth = (int)Math.Floor(media.NaturalVideoWidth / totalPhotosX);
        int rectHeight = (int)Math.Floor(media.NaturalVideoHeight / totalPhotosY);
        Random random = new Random();

        // remove all old videoblocks
        for (int i = Children.Count - 1; i > 0; i--)
        {
            if (Children[i].GetType() == typeof(VideoBlock))
                Children.RemoveAt(i);
        }

        for (int x = 0; x < totalPhotosX; x++)
        {
            for (int y = 0; y < totalPhotosY; y++)
            {
                VideoBlock vb = new VideoBlock(media.Name, x * rectWidth, y * rectHeight, rectWidth, rectHeight, SetActivePhoto);
                shuffle(vb, random);
                Children.Add(vb);
            }
        }
    }
}

VB.Net

Private Sub prepVideoObjects()
        Dim totalPhotosX As Integer = CType(hSliderX.Value,Integer)
        Dim totalPhotosY As Integer = CType(hSliderY.Value,Integer)
        If ((lastX <> totalPhotosX)  _
                    OrElse (lastY <> totalPhotosY)) Then
            lastX = totalPhotosX
            lastY = totalPhotosY
            sliderX.Text = String.Format("X ({0}):", totalPhotosX.ToString("D2"))
            sliderY.Text = String.Format("Y ({0}):", totalPhotosY.ToString("D2"))
            Dim rectWidth As Integer = CType(Math.Floor((media.NaturalVideoWidth / totalPhotosX)),Integer)
            Dim rectHeight As Integer = CType(Math.Floor((media.NaturalVideoHeight / totalPhotosY)),Integer)
            Dim random As Random = New Random
            ' remove all old videoblocks
            Dim i As Integer = (Children.Count - 1)
            Do While (i > 0)
                If (Children(i).GetType = GetType(VideoBlock)) Then
                    Children.RemoveAt(i)
                End If
                i = (i - 1)
            Loop
            Dim x As Integer = 0
            Do While (x < totalPhotosX)
                Dim y As Integer = 0
                Do While (y < totalPhotosY)
                    Dim vb As VideoBlock = New VideoBlock(media.Name, (x * rectWidth), (y * rectHeight), rectWidth, rectHeight, SetActivePhoto)
                    shuffle(vb, random)
                    Children.Add(vb)
                    y = (y + 1)
                Loop
                x = (x + 1)
            Loop
        End If
    End Sub

Last but not least, lets see how the mouse movements are handled on the page.

C#

private void Page_MouseMove(object sender, MouseEventArgs e)
{
    if (null != activeVideoBlock)
    {
        // Perform the appropriate transform on the active photo
        Point position = e.GetPosition(null);
        switch (currentActionType)
        {
            case VideoBlock.ActionType.Moving:
                // Move it by the amount of the mouse move
                activeVideoBlock.Translate(position.X - currentLastPosition.X, position.Y - currentLastPosition.Y);
                break;
            case VideoBlock.ActionType.RotatingScaling:
                // Rotate it according to the angle the mouse moved around the photo's center
                double radiansToDegrees = 360 / (2 * Math.PI);
                double lastAngle = Math.Atan2(currentLastPosition.Y - currentVideoCenter.Y, currentLastPosition.X - currentVideoCenter.X) * radiansToDegrees;
                double currentAngle = Math.Atan2(position.Y - currentVideoCenter.Y, position.X - currentVideoCenter.X) * radiansToDegrees;
                activeVideoBlock.Rotate(currentAngle - lastAngle);

                break;
        }
        currentLastPosition = position;
    }
}

VB.Net

    Private Sub Page_MouseMove(ByVal sender As Object, ByVal e As MouseEventArgs)
        If (Not (activeVideoBlock) Is Nothing) Then
            ' Perform the appropriate transform on the active photo
            Dim position As Point = e.GetPosition(Nothing)
            Select Case (currentActionType)
                Case VideoBlock.ActionType.Moving
                    ' Move it by the amount of the mouse move
                    activeVideoBlock.Translate((position.X - currentLastPosition.X), (position.Y - currentLastPosition.Y))
                Case VideoBlock.ActionType.RotatingScaling
                    ' Rotate it according to the angle the mouse moved around the photo's center
                    Dim radiansToDegrees As Double = (360 / (2 * Math.PI))
                    Dim lastAngle As Double = (Math.Atan2((currentLastPosition.Y - currentVideoCenter.Y), (currentLastPosition.X - currentVideoCenter.X)) * radiansToDegrees)
                    Dim currentAngle As Double = (Math.Atan2((position.Y - currentVideoCenter.Y), (position.X - currentVideoCenter.X)) * radiansToDegrees)
                    activeVideoBlock.Rotate((currentAngle - lastAngle))
            End Select
            currentLastPosition = position
        End If
    End Sub

Wrapping it up

So as you can see, Silverlight is pretty powerful once you figure out how to tame it.  I used example base code and modified it to suit my own needs to dynamically alter a video in real time.  There are a few code tweaks that could happen to improve this code and I'm betting with some additional effort, one could remove the delegate.  In addition, you could add in edge detection code and a timer to make this into a true video puzzle game.  This was a pet project I worked on while I was at the airport.  This demo can be downgraded to Silverlight 1.0 also.  I don't believe anything I did was really Silverlight 1.1 (now 2.0) only.

Clint's Bio:

Clint is an academic developer evangelist for Microsoft.  His two primary development languages are C# and JavaScript. He has built a Disco Dance Floor too! In his off time, he whips up other random weird projects and does twenty something activities with his friends.  His next two big projects are an automated bartender and a skateboard segway.  Clint's blog is betterthaneveryone.com and can be emailed at crutkas@microsoft.com if you have any question.

Do you have a Facebook account and want to do more with it? Start creating cool, fun Windows applications and Web sites with the Facebook Developer Toolkit. This toolkit offers you a huge amount of easy-to-use components and controls, plus ready-to-run samples and detailed documentation to get you started. With Visual Studio Express Editions and the Facebook Developer Toolkit, you'll have the tools you need to get going today!

The Facebook Developer Toolkit is a full complement of components, controls and samples for developing applications using the Facebook Platform. The kit also includes Visual C# and Visual Basic wrappers for the Facebook API to make application development simple, fun and quick. You can use the Facebook Developer Toolkit to build applications using Windows Forms, ASP.NET or WPF. You can even use the toolkit to take advantage of LINQ.

See How Easy It Is

Explore the QuickStarts to see just how easy it is to develop using the Facebook Developer Toolkit and Visual Studio Express Editions. Watch the walkthrough videos to see how easy it is to build Facebook-enabled applications. You can have an application or website up and running in only a few minutes.

The Facebook Development Toolkit allows you to create applications that you can share with your Facebook friends. Download the toolkit today!

Want to see the Facebook Developer Toolkit in action? Download this sample application that uses the Facebook Developer Toolkit to build a dynamic screensaver with Visual C# Express Edition. Explore the Facebook Developer Toolkit wiki for more details on the toolkit.

This month, we create a vanity search page. The vanity search page displays any images, news or website entries about a particular person (for example, you). To create this page, we take advantage of the Windows Live Search API. This free API enables you to retrieve search results by calling a web service that Microsoft exposes on the web.

I decided to write this article because I wanted an excuse to play with the Windows Live SDK (see http://dev.live.com). The SDK includes documentation and samples for accessing all of the Windows Live web services including the Windows Live Search Web Service, Virtual Earth, and Windows Live Alerts.

I fully admit that I was actively avoiding writing this article because I dreaded the thought of needing to learn yet another Application Programming Interface. I was pleasantly surprised to discover that the API for Windows Live Search is both logical and painlessly simple to understand. I wrote the code for this article in less than an hour.

Before you can access the Windows Live services, you need to get an Application ID. You pass this ID to the Windows Live services to identify yourself. You get an Application ID by signing an agreement at the following URL: http://search.msn.com/developer/appids.aspx.

Figure 1 - Adding a Web Reference for Windows Live Search

After you have an Application ID, you are ready to start calling Windows Live web services. Here are the steps for creating the vanity search page:

1. Launch Visual Web Developer and create a new Web Site

2. Select the menu option Website, Add Web Reference to add a Web Reference to the Windows Live Search web service (see Figure 1). Enter the following address into the URL field: http://soap.search.msn.com/webservices.asmx?wsdl. When you click OK, a new Web reference named com.msn.search.soap is created.

3. Create a new Web Form Page by selecting the menu option Website, Add New Item. When you create the new Web Form Page, give it the name Vanity.aspx. Also, make sure that the checkbox labeled Place code in separate file is not checked.

4. Replace the contents of the page with the following code:

VB.NET Code 

<%@ Page Language="VB" %>

<%@ Import Namespace="com.msn.search.soap" %>

<%@ Import Namespace="System.Text.RegularExpressions" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<script runat="server">

Const appId As String = ""

Const query As String = """Bill Gates"""

Const startMark As String = "\uE000"

Const endMark As String = "\uE001"

Private Sub Page_Load()

' Create instance of MSN Search Services

Dim search As New MSNSearchService()

' Build Source Request

Dim NewsSource As New SourceRequest()

NewsSource.Source = SourceType.News

Dim webSource As New SourceRequest()

webSource.Source = SourceType.Web

Dim imageSource As New SourceRequest()

imageSource.Source = SourceType.Image

imageSource.ResultFields = ResultFieldMask.Image Or ResultFieldMask.Url

' Build the Search Request

Dim request As New SearchRequest()

request.AppID = appId

request.Requests = New SourceRequest() {imageSource, NewsSource, webSource}

request.Query = query

request.CultureInfo = "en-US"

request.Flags = SearchFlags.MarkQueryWords

' Execute Search

Dim response As SearchResponse = search.Search(request)

lstResponses.DataSource = response.Responses

lstResponses.DataBind()

End Sub

Private Function hiLite(ByVal item As Object) As String

If item Is Nothing Then

Return String.Empty

End If

Dim text As String = item.ToString()

text = Regex.Replace(text, startMark, "<span class='hiLite'>")

text = Regex.Replace(text, endMark, "</span>")

Return text

End Function

Private Function toImage(ByVal objImage As Object) As String

If Not objImage Is Nothing Then

Dim image As com.msn.search.soap.Image = CType(objImage, com.msn.search.soap.Image)

Return image.ImageURL

End If

Return String.Empty

End Function

</script>

<html xmlns="http://www.w3.org/1999/xhtml" >

<head runat="server">

<title>Untitled Page</title>

<style type="text/css">

html 

{

font:12px Verdana;

}

a {text-decoration:none}

.hiLite {background-color:yellow}

</style>

</head>

<body>

<form id="form1" runat="server">

<div>

<asp:DataList

id="lstResponses"

Runat="server">

<ItemTemplate>

<h1><%# Eval("Source") %> (<%# Eval("Total") %> Results)</h1>

<asp:Repeater

id="grdResults"

DataSource='<%# Eval("Results") %>'

Runat="server">

<HeaderTemplate>

<ul>

</HeaderTemplate>

<ItemTemplate>

<li>

<asp:HyperLink 

id="lnkResult"

ImageUrl='<%# toImage( Eval("Image")) %>'

Text='<%# hiLite( Eval("Title") ) %>'

NavigateUrl='<%# Eval("Url") %>'

Runat="server" />

<p>

<%# hiLite( Eval("Description") ) %>

</p>

</li>

</ItemTemplate>

<FooterTemplate>

</ul>

</FooterTemplate>

</asp:Repeater>

</ItemTemplate>

</asp:DataList>

</div>

</form>

</body>

</html>

C# Code

<%@ Page Language="C#" %>

<%@ Import Namespace="com.msn.search.soap" %>

<%@ Import Namespace="System.Text.RegularExpressions" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<script runat="server">

const string appId = "";

const string query = "\"Bill Gates\"";

const string startMark = "\uE000";

const string endMark = "\uE001";

void Page_Load()

{

// Create instance of MSN Search Services

MSNSearchService search = new MSNSearchService();

// Build Source Request

SourceRequest newsSource = new SourceRequest();

newsSource.Source = SourceType.News;

SourceRequest webSource = new SourceRequest();

webSource.Source = SourceType.Web;

SourceRequest imageSource = new SourceRequest();

imageSource.Source = SourceType.Image;

imageSource.ResultFields = ResultFieldMask.Image | ResultFieldMask.Url;

// Build the Search Request

SearchRequest request = new SearchRequest();

request.AppID = appId;

request.Requests = new SourceRequest[] {imageSource, newsSource, webSource};

request.Query = query;

request.CultureInfo = "en-US";

request.Flags = SearchFlags.MarkQueryWords;

// Execute Search

SearchResponse response = search.Search(request);

lstResponses.DataSource = response.Responses;

lstResponses.DataBind();

}

string hiLite(object item)

{

if (item == null)

return String.Empty;

string text = item.ToString();

text = Regex.Replace(text, startMark, "<span class='hiLite'>");

text = Regex.Replace(text, endMark, "</span>");

return text;

}

string toImage(object objImage)

{

if (objImage != null)

{

com.msn.search.soap.Image image = (com.msn.search.soap.Image)objImage;

return image.ImageURL;

}

return String.Empty;

}

</script>

<html xmlns="http://www.w3.org/1999/xhtml" >

<head runat="server">

<title>Vanity Search</title>

<style type="text/css">

html 

{

font:12px Verdana;

}

a {text-decoration:none}

.hiLite {background-color:yellow}

</style>

</head>

<body>

<form id="form1" runat="server">

<div>

<asp:DataList

id="lstResponses"

Runat="server">

<ItemTemplate>

<h1><%# Eval("Source") %> (<%# Eval("Total") %> Results)</h1>

<asp:Repeater

id="grdResults"

DataSource='<%# Eval("Results") %>'

Runat="server">

<HeaderTemplate>

<ul>

</HeaderTemplate>

<ItemTemplate>

<li>

<asp:HyperLink 

id="lnkResult"

ImageUrl='<%# toImage( Eval("Image")) %>'

Text='<%# hiLite( Eval("Title") ) %>'

NavigateUrl='<%# Eval("Url") %>'

Runat="server" />

<p>

<%# hiLite( Eval("Description") ) %>

</p>

</li>

</ItemTemplate>

<FooterTemplate>

</ul>

</FooterTemplate>

</asp:Repeater>

</ItemTemplate>

</asp:DataList>

</div>

</form>

</body>

</html>

Before you use the vanity search page, you should modify two of the constants that appear at the top of the page: the appId and query constants. Assign the App ID that you get from visiting the following URL to the appId constant: http://search.msn.com/developer/appids.aspx. Enter anything you want for the query constant. In the page above, the name Bill Gates (enclosed in quotes) is assigned to the query constant. When you open the page in a web browser, you should see a page similar to Figure 2.

Figure 2 - Image results for Bill Gates vanity search page

The page displays three different groups of results. First, the page displays a list of images that match the search query. Next, the page displays a list of news articles. Finally, the page displays a list of websites (see Figure 3).

Figure 3 - Web results for Bill Gates vanity search page

The Windows Live Search web service currently supports the following types of searches:

• Image - Returns a list of images that match the search query.
• Inline Answers - Only available for commercial use. This type of search returns information from Encarta, Finance, Weather, and Movie show times.
• News - Returns results from online news sources.
• Phone Book - Returns results from white and yellow page phone books.
• Query Location - Returns latitude and longitude information associated with a query.
• Spelling - Returns a list of spelling suggestions.
• Web - Returns a list of websites that match the search query.

You perform a search by creating an instance of the SearchRequest class. You assign the text of your search query to the SearchRequest class's Query property. This class also includes a Requests collection. You add one or more SourceRequest objects to this collection that represent different types of searches (Image, Web, News, and so on). After you get all of the objects ready, you call the MSNSearchService.Search() method to actually perform the search.

The search results are displayed with the help of a DataList and Repeater control. The DataList displays the source of the search and the number of results for each source returned. The Repeater control is nested in the DataList and it displays the actual search results.  

Conclusion

As I mentioned before, I wrote this article as an excuse to play with the Windows Live SDK. I am very happy that I got the chance to experiment with this SDK since I discovered that it is very easy to work with. I'm looking forward to trying out some of the other Windows Live services such as the Virtual Earth and Alerts services over the coming months.

Background

Have you ever worked on multiple machines and wished you could copy the clipboard contents from one machine to the other? I've often found myself wishing that there was a quick and easy way to move text snippets, screenshots, or even files to another machine with a simple copy and paste. If this sounds interesting to you then read on.

I wanted this to work regardless of whether both machines were online at the same time, and not be halted by firewalls, NAT's, etc. so I opted for a server based rather than a peer to peer architecture. The architecture involved a client application to transfer the clipboard contents to the server via a web services call, a web service to cache the clipboard contents, and another client component to retrieve the clipboard contents from the server and place them on the local machine's clipboard.

To tackle this problem we need programmatic access to the clipboard on both the machine to copy from and also the machine we want to copy to. Luckily .NET provides a managed wrapper around the native Windows Clipboard API that gives us access to this. The relevant namespaces are Clipboard for C# and my.Computer.Clipboard. Since we're interested in moving clipboard objects from one machine to another we first need to determine what type of objects get placed onto the clipboard for our various actions (copy text, images, and files). Writing a quick code snippet with the Clipboard namespace allows us to iterate through all objects on the clipboard for various types of actions to see what we're dealing with.

C#

        IDataObject clipData = Clipboard.GetDataObject();

        //retrieve an array of strings for all the formats available on the clipboard.

        string[] formats = clipData.GetFormats();

        //iterate through the list of formats on the clipboard

        foreach (string format in formats)

        {

            //add each objeoct to an arraylist so we can inspect the object types

            object dataObject = clipData.GetData(format);

        }

VB

    Dim clipData As IDataObject = Clipboard.GetDataObject

    Dim formats() As String = clipData.GetFormats

    'iterate through the list of formats on the clipboard

    For Each format As String In formats

    'add each objeoct to an arraylist so we can inspect the object types

          Dim dataObject As Object = clipData.GetData(Format)

    Next

The screenshot below shows the results for having some text from Word copied onto the clipboard. Each format in the string array is a representation of the data on the clipboard. Since the target application (where you're going to paste to) is unknown the clipboard has multiple formats each containing the same data. The objective with this project is to replicate all these formats on the target machine's clipboard.

After inspection of the objects on the Clipboard for text, images, and file objects it was easy to determine that the primary object types we need to be concerned with. These are "System.IO.MemoryStream", “System.IO.FileStream”, "System.Drawing.Bitmap", and "System.String" . Since all this information would be transferred to the server via web services it's simple to serialize all objects to bytes for transmission. This is necessary for a number of reasons including the fact that complex objects such as MemoryStreams cannot be simply serialized and sent over the web service call as Strings can. In addition, some of the objects are larger than the web service call allows, and would need to be broken up into smaller pieces for transmission and then reassembled on the server side in the correct order. Again, when the client requests the clipboard items we'll need to disassemble each object, send it over the web service return result to the client, and then reassemble it.

The first item to build is a base function that handles the breaking up of these large streams into more manageable byte arrays for transmission to the web service. The function below performs this task by sending chunks of the MemoryStream limited in size by the “byteCount” constant. Once this limit is reached the buffer is sent over the web service call for storage and assembly on the server. Once we have either 0 bytes left to send, or a number less than “byteCount” we send the remaining elements of the buffer and signal the web service that this particular object is complete with the “isFinalTransaction” flag.

C#

        private void UploadStreamBlock(string format, string objectType, MemoryStream memStream)

        {

            //each time we enter this function we have a new transaction beginning.  

            //  A transaction represents a comlete

            //object on the clipboard and we'll use this on the server side to know 

            //  how to put the stream back together

            string transactionGuid = System.Guid.NewGuid().ToString();

            memStream.Position = 0;

            byte[] buffer = new byte[byteCount];

            bool isFinalTransaction = false;

            //while the current stream position plus our byte count is less than the 

            //  length of the stream continue sending as much as we can.

            while ((memStream.Position + byteCount) <= memStream.Length)

            {

                //if we happen to be on the last byte of the stream set the final 

                // transaction flag to true so the server will know that this is the 

                //  last bit of this transaction to expect.

                if (memStream.Position + byteCount == memStream.Length)

                {

                    isFinalTransaction = true;

                }

                //read the stream into our buffer for transmission over the web service.

                memStream.Read(buffer, 0, byteCount);

                ws.InsertMessageStream(buffer, format, objectType, transactionGuid, 

                    isFinalTransaction, clipBoardGUID);

            }

            long remainingBytes = memStream.Length - memStream.Position;

            //if we still have remaining bytes left figure out how many and transmit the 

            //  last bit of this ojbect over the web service.

            if ((int)remainingBytes > 0)

            {

                byte[] remainingBuffer = new byte[(int)remainingBytes];

                memStream.Read(remainingBuffer, 0, (int)remainingBytes);

                ws.InsertMessageStream(remainingBuffer, format, objectType, 

                    transactionGuid, true, clipBoardGUID);

            }

        }

VB

    Private Sub UploadStreamBlock(ByVal format As String, ByVal objectType As String, _

                                  ByVal memStream As MemoryStream)

        'each time we enter this function we have a new transaction beginning. 

        ' A transaction represents a comlete

        'object on the clipboard and we'll use this on the server side to know how 

        ' to put the stream back together

        Dim transactionGuid As String = System.Guid.NewGuid.ToString

        memStream.Position = 0

        Dim buffer() As Byte = New Byte((byteCount) - 1) {}

        Dim isFinalTransaction As Boolean = False

        'while the current stream position plus our byte count is less than the 

        ' length of the stream continue sending as much as we can.

        While ((memStream.Position + byteCount) _

                    <= memStream.Length)

            'if we happen to be on the last byte of the stream set the final 

            ' transaction flag to true so the server will know that this is the 

            ' last bit of this transaction to expect.

            If ((memStream.Position + byteCount) _

                        = memStream.Length) Then

                isFinalTransaction = True

            End If

            'ream the stream into our buffer for transmission over the web service.

            memStream.Read(buffer, 0, byteCount)

            clipService.InsertMessageStream(buffer, format, objectType, transactionGuid, _

                                            isFinalTransaction, clipBoardGUID)

        End While

        Dim remainingBytes As Long = (memStream.Length - memStream.Position)

        'if we still have remaining bytes left figure out how many and 

        ' transmit the last bit of this ojbect over the web service.

        If (CType(remainingBytes, Integer) > 0) Then

            Dim remainingBuffer() As Byte = New Byte((CType(remainingBytes, Integer)) - 1) {}

            memStream.Read(remainingBuffer, 0, CType(remainingBytes, Integer))

            clipService.InsertMessageStream(remainingBuffer, format, objectType, _

                                            transactionGuid, True, clipBoardGUID)

        End If

The Server side of the web service needs to put the whole clipboard back together from a number of byte arrays so it's important that all the objects, their types, and formats be preserved for the clipboard to work properly on the target machine. We use the clipBoardGuid to determine if we are on a new clipboard posting or adding objects to an already existing instance. We use the isFinalTranaction flag to know if this byte array should be part of an existing transaction, or is the first in a new transaction. All clipboard items are saved to disk for later retrieval by any client requesting them. The code for this is below.

C#

    [WebMethod]

    public void InsertMessageStream(byte[] buffer, string format, string objectType, 

        string transactionGuid, bool isFinalTransaction, string clipBoardGUID)

    {

        //always base the current directory on the clipboard that we're sending now.

        string clipBoardGUIDDirectory = 

            System.Web.HttpContext.Current.Request.PhysicalApplicationPath + clipBoardGUID;

        try

        {

            //if the directory does not exist then delete all the other directories 

            // (clipboard instances) and create a new directory if the directory already 

            // exists then this particular transaction is part of the same clipboard so 

            // don't do anything. this works because othe clipboardDirectory is based 

            // off of the GUID sent from the client.

            if (!Directory.Exists(clipBoardGUIDDirectory))

            {

                string[] dirs = Directory.GetDirectories(

                    System.Web.HttpContext.Current.Request.PhysicalApplicationPath);

                foreach (string dir in dirs)

                {

                    Directory.Delete(dir, true);

                }

                Directory.CreateDirectory(clipBoardGUIDDirectory);

            }

        }

        catch

        {

        }

        //create the filename based on the current transaction, format, and object type.  

        // We will parse this out later so we know how to add this back to the target clipboard.

        string fileName = clipBoardGUIDDirectory + "\\" + transactionGuid 

            + "_" + format + "_" + objectType;

        FileStream fs = new FileStream(fileName, FileMode.Append, FileAccess.Write);

        fs.Position = fs.Length;

        fs.Write(buffer, 0, buffer.Length);

        fs.Close();

    }

VB

    <WebMethod()> Public Sub InsertMessageStream(ByVal buffer() As Byte, ByVal _

            format As String, ByVal objectType As String, ByVal transactionGuid _

            As String, ByVal isFinalTransaction As Boolean, ByVal clipBoardGUID As String)

        'always base the current directory on the clipboard that we're sending now.

        Dim clipBoardDataDirectory As String = _

        (System.Web.HttpContext.Current.Request.PhysicalApplicationPath _

         + "\\Clipboard_Data")

        Dim clipBoardGUIDDirectory As String = (clipBoardDataDirectory + _

                                                ("\\" + clipBoardGUID))

        Try

            'if the directory does not exist then delete all the other directories (clipboard 

            ' instances) and create a new directory if the directory already exists then this 

            ' particular transaction is part of the same clipboard so don't do anything. this 

            ' works because othe clipboardDirectory is based off of the GUID sent from the client.

            If Not Directory.Exists(clipBoardGUIDDirectory) Then

                Dim dirs() As String = Directory.GetDirectories(clipBoardDataDirectory)

                For Each dir As String In dirs

                    Directory.Delete(dir, True)

                Next

                Directory.CreateDirectory(clipBoardGUIDDirectory)

            End If

        Catch

        End Try

        'create the filename based on the current transaction, format, and object type.  We 

        ' will parse this out later so we know how to add this back to the target clipboard.

        Dim fileName As String = (clipBoardGUIDDirectory + ("\\" _

                    + (transactionGuid + ("_" _

                    + (format + ("_" + objectType))))))

        Dim fs As FileStream = New FileStream(fileName, FileMode.Append, FileAccess.Write)

        fs.Position = fs.Length

        fs.Write(buffer, 0, buffer.Length)

        fs.Close()

    End Sub

Each clipgoard format object is stored on disk for later retrievel by the client. Note in the screenshot below how the filename is used to store the unique transactionID for the object, the object type and also the clipboard format. All these pieces of information are necessary to reassemble the items correctly and place onto the target clipboard.

Now that we have a representation of each clipboard format object on the server we need a way to get each item back onto the target clipboard. The following web service method provides a return result of type “ClipboardStream”. The ClipboardStream object contains all relevant information necessary to reassemble each item onto the target clipboard. Since a web service is a request-response type relationship the web service expects the client to continue to call the web service until the all the clipboard items have been successfully received. In addition, further complexity is introduced because each individual clipboard item may be split into multiple items if they exceed the maximum length set by our contstant “byteCount”, therefore the target machine must keep track of each request and tell the server where the last transaction left off via the variable named “currentByte”. The web service code is shown below.

C#

    [WebMethod]

    public ClipboardStream GetMessageStream(string transactionGUID, string[] 

        previousTransactionGUIDs, string clipBoardGUID, long currentByte)

    {

        string clipBoardDataDirectory = 

            System.Web.HttpContext.Current.Request.PhysicalApplicationPath + 

            "Clipboard_Data";

        string clipBoardGUIDDirectory = clipBoardDataDirectory + "\\" + 

            clipBoardGUID;

        string currentTransaction = "";

        bool isLastTransaction = false;

        //if the clipBoardGUID is not empty then we only need to make sure 

        // that the directory still exists.

        if (clipBoardGUID != "")

        {

            //if the directory does not exist throw an exception, it must 

            // have already been deleted.

            if (!Directory.Exists(clipBoardGUIDDirectory))

            {

                throw new Exception("Requested clipboard does not exist." +

                        "It must have been deleted.");

            }

        }

        //if the clipboardGUID is empty then this is the client's first contact 

        // with the server and we need

        //to select the available clipboard GUID to return to the user.

        else

        {

            string[] availableClipBoard = 

                Directory.GetDirectories(clipBoardDataDirectory)[0].Split('\\');

            clipBoardGUID = availableClipBoard[availableClipBoard.Length - 1];

            clipBoardGUIDDirectory += clipBoardGUID;

        }

        //we need to get the next transaction.  Each time we finish a transaction 

        // we add it to previousTransactionGUIDs at the client end so we know not 

        // to send it again.

        currentTransaction = GetCurrentTransaction(clipBoardGUIDDirectory, 

            previousTransactionGUIDs);

        //if the current transaction is null then we're done and there are no 

        // more to send to the client

        if (currentTransaction == null)

        {

            return null;

        }

        //open the filestream and set it to the position requested by the client.

        FileStream fs = new FileStream(currentTransaction, FileMode.Open);

        fs.Position = currentByte;

        //determind if this is the last transaction or not for this object 

        // so we can let the client know.

        long numBytesToRead = fs.Length - currentByte;

        if (numBytesToRead > byteCount)

        {

            numBytesToRead = byteCount;

            isLastTransaction = false;

        }

        else

        {

            isLastTransaction = true;

        }

        //read the filestream bytes to the buffer and populate the object to 

        // return to the client.

        byte[] buffer = new byte[numBytesToRead];

        fs.Read(buffer, 0, (int)numBytesToRead);

        fs.Close();

        FileInfo fi = new FileInfo(currentTransaction);

        ClipboardStream clipboardStream = new ClipboardStream();

        clipboardStream.Buffer = buffer;

        clipboardStream.ClipBoardID = clipBoardGUID;

        clipboardStream.Format = fi.Name.Split('_')[1];

        clipboardStream.ObjectType = fi.Name.Split('_')[2];

        clipboardStream.IsLastTransaction = isLastTransaction;

        clipboardStream.TransactionID = currentTransaction;

        return clipboardStream;

    }

VB

    <WebMethod()> Public Function GetMessageStream(ByVal transactionGUID As String, _

            ByVal previousTransactionGUIDs() As String, ByVal clipBoardGUID As String, _

            ByVal currentByte As Long) As ClipboardStream

        Dim clipBoardDataDirectory As String = _

            (System.Web.HttpContext.Current.Request.PhysicalApplicationPath + "Clipboard_Data")

        Dim clipBoardGUIDDirectory As String = clipBoardDataDirectory

        Dim currentTransaction As String = ""

        Dim isLastTransaction As Boolean = False

        'if the clipBoardGUID is not empty then we only need to make sure that 

        ' the directory still exists.

        If (clipBoardGUID <> "") Then

            'if the directory does not exist throw an exception, it must have 

            ' already been deleted.

            If Not Directory.Exists(clipBoardGUIDDirectory) Then

                Throw New Exception("Requested clipboard does not exist." + _

                                    "It must have been deleted.")

            End If

        End If

        'if the clipboardGUID is empty then this is the client's first contact 

        ' with the server and we need

        'to select the available clipboard GUID to return to the user.

        Dim availableClipBoard() As String = Directory.GetDirectories _

            (clipBoardDataDirectory)(0).Split(Microsoft.VisualBasic.ChrW(92))

        clipBoardGUID = availableClipBoard((availableClipBoard.Length - 1))

        clipBoardGUIDDirectory = (clipBoardGUIDDirectory + "\" + clipBoardGUID)

        'we need to get the next transaction.  Each time we finish a transaction 

        ' we add it to previousTransactionGUIDs

        'at the client end so we know not to send it again.

        currentTransaction = GetCurrentTransaction(clipBoardGUIDDirectory, _

                                                   previousTransactionGUIDs)

        'if the current transaction is null then we're done and there are no 

        ' more to send to the client

        If (currentTransaction Is Nothing) Then

            Return Nothing

        End If

        'open the filestream and set it to the position requested by the client.

        Dim fs As FileStream = New FileStream(currentTransaction, FileMode.Open)

        fs.Position = currentByte

        'determind if this is the last transaction or not for this object so 

        ' we can let the client know.

        Dim numBytesToRead As Long = (fs.Length - currentByte)

        If (numBytesToRead > byteCount) Then

            numBytesToRead = byteCount

            isLastTransaction = False

        Else

            isLastTransaction = True

        End If

        'read the filestream bytes to the buffer and populate the object to 

        ' return to the client.

        Dim buffer() As Byte = New Byte((numBytesToRead) - 1) {}

        fs.Read(buffer, 0, CType(numBytesToRead, Integer))

        fs.Close()

        Dim fi As FileInfo = New FileInfo(currentTransaction)

        Dim clipboardStream As ClipboardStream = New ClipboardStream

        clipboardStream.Buffer = buffer

        clipboardStream.ClipBoardID = clipBoardGUID

        clipboardStream.Format = fi.Name.Split(Microsoft.VisualBasic.ChrW(95))(1)

        clipboardStream.ObjectType = fi.Name.Split(Microsoft.VisualBasic.ChrW(95))(2)

        clipboardStream.IsLastTransaction = isLastTransaction

        clipboardStream.TransactionID = currentTransaction

        Return clipboardStream

    End Function

The last remaining piece for our project to function correctly is the client code that receives the clipboard contents from the server and reassembles each item in the correct order, adding it to the clipboard with the correct format as each is completed. The client code for this is shown below.  

C#

        string[] transactionGuids = null;

        ClipboardService.ClipboardStream clipBoardStream = 

            new WindowsApplication1.ClipboardService.ClipboardStream();

        DataObject dataObject = new DataObject();

        clipBoardStream.ClipBoardID = "";

        clipBoardStream.IsLastTransaction = false;

        clipBoardStream.TransactionID = "";

        long currentByte = 0;

        Clipboard.Clear();

        //while we don't get null back keep on contacting the web service 

        // to get the next ojbect.

        while (clipBoardStream != null)

        {

            MemoryStream memStream = new MemoryStream();

            //while this is not the last transaction keep on contacting 

            // the web service to get the rest of this particular object.

            while (clipBoardStream.IsLastTransaction == false)

            {

                //contact the web service to get the next transaction

                clipBoardStream = clipService.GetMessageStream(

                    clipBoardStream.TransactionID, transactionGuids, 

                    clipBoardStream.ClipBoardID, currentByte);

                if (clipBoardStream != null)

                {

                    //write the results to the memory stream

                    memStream.Write(clipBoardStream.Buffer, 0, 

                        clipBoardStream.Buffer.Length);

                    //increment the current byte so next time we contact 

                    // the webservice we'll pick up where we left off

                    currentByte = memStream.Position;

                    //if it is the last transaction then we need to place 

                    // this item onto the clipboard.

                    if (clipBoardStream.IsLastTransaction)

                    {

                        //handle the clipBoardStream appropriately and add it 

                        // to the dataObject for posting to the clipblard.

                        HandleFinalTransaction(clipBoardStream, memStream, ref dataObject);

                        //resize the transactionGuids array as necessary and 

                        // add the current transaction so next time we contact 

                        // the web service we won't get this one again.

                        if (transactionGuids == null)

                        {

                            Array.Resize(ref transactionGuids, 1);

                        }

                        else

                        {

                            Array.Resize(ref transactionGuids, 

                                transactionGuids.Length + 1);

                        }

                        transactionGuids[transactionGuids.Length - 1] = 

                            clipBoardStream.TransactionID;

                    }

                }

                else

                {

                    break;

                }

            }

            if (clipBoardStream != null)

            {

                clipBoardStream.IsLastTransaction = false;

                currentByte = 0;

            }

        }

        Clipboard.SetDataObject(dataObject, true);

VB

        Dim transactionGuids() As String = Nothing

        Dim clipBoardStream As ClipboardService.ClipboardStream = _

            New ClipboardVB.ClipboardService.ClipboardStream

        Dim dataObject As DataObject = New DataObject

        clipBoardStream.ClipBoardID = ""

        clipBoardStream.IsLastTransaction = False

        clipBoardStream.TransactionID = ""

        Dim currentByte As Long = 0

        Clipboard.Clear()

        'while we don't get null back keep on contacting the web service 

        ' to get the next ojbect.

        While (Not (clipBoardStream) Is Nothing)

            Dim memStream As MemoryStream = New MemoryStream

            'while this is not the last transaction keep on contacting the

            ' web service to get the rest of this particular object.

            While (clipBoardStream.IsLastTransaction = False)

                'contact the web service to get the next transaction

                clipBoardStream = clipService.GetMessageStream( _

                    clipBoardStream.TransactionID, transactionGuids, _

                    clipBoardStream.ClipBoardID, currentByte)

                If (Not (clipBoardStream) Is Nothing) Then

                    'write the results to the memory stream

                    memStream.Write(clipBoardStream.Buffer, 0, _

                                    clipBoardStream.Buffer.Length)

                    'increment the current byte so next time we contact 

                    ' the webservice we'll pick up where we left off

                    currentByte = memStream.Position

                    'if it is the last transaction then we need to place 

                    ' this item onto the clipboard.

                    If clipBoardStream.IsLastTransaction Then

                        'handle the clipBoardStream appropriately and add 

                        ' it to the dataObject for posting to the clipblard.

                        HandleFinalTransaction(clipBoardStream, memStream, dataObject)

                        'resize the transactionGuids array as necessary and 

                        ' add the current transaction so next time we contact

                        ' the web service we won't get this one again.

                        If (transactionGuids Is Nothing) Then

                            Array.Resize(transactionGuids, 1)

                        Else

                            Array.Resize(transactionGuids, (transactionGuids.Length + 1))

                        End If

                        transactionGuids((transactionGuids.Length - 1)) = _

                            clipBoardStream.TransactionID

                    End If

                Else

                    Exit While

                End If

            End While

            If (Not (clipBoardStream) Is Nothing) Then

                clipBoardStream.IsLastTransaction = False

                currentByte = 0

            End If

        End While

        Clipboard.SetDataObject(dataObject, True)

Conclusion

In this project we've shown how any object (string, bitmap, or file) from the Clipboard can be serialized to an array of bytes for transmission over a web service call. We've also show how to store these clipboard items on disk on the server, and then retrieve them later on for assembly onto the target clipboard. I've often thought that this would be an interesting idea for a hosted ASP (Application Service Provider) service, where a hosting company would host the web service of this project for customer use. Look out for an upcoming article on how to add ASP .NET membership class functionality providing authentication and allowing multiple users to use the same instance of the web service.

For working samples of the client application and web service in both C# and VB check out the DOWNLOAD.

Bio

As President and principal founder of Personify Design Brian oversees the operations of the design and development businesses. Brian has more than 10 years experience in the technology industry. In his current role Brian's expertise lies in developing and architecting end to end customer solutions involving web application technologies such as SQL Server and ASP .NET. When not writing code Brian enjoys sailing in the Puget Sound on Far Niente, a 36 foot Catalina MKII cutter.

This article shows how to build a .NET application using an accelerometer to measure the performance of a vehicle. An accelerometer is an instrument that is used for measuring acceleration. By using basic physics equations, we can derive a vehicle's current speed, distance traveled, horsepower, and other performance metrics from the acceleration of the vehicle. iAccelerate is solely for use in a vehicle on a closed course or on private property while under the supervision of a certified emergency care provider. As a reminder, please always wear your safety belt.

Mike Harkabus Clarity Consulting Difficulty: Intermediate Time Required: 6-10 hours Cost: $50-$100 Software: Visual Basic or Visual C# Express Editions Hardware: Analog Devices ADXL320 Download: Download

Introduction

I have had a passion for computers and cars for as long as I can remember. Ever since I was little, I have been able to name the make and model of almost any car on the road. I have been around computers since I was young, and can still remember the day I wrote (read: “copied out of a book”) my first program in BASIC for IBM's PCjr. Twenty years later, these same interests coupled with an accelerometer played a role in the development of iAccelerate. iAccelerate displays accurate vehicle performance data from calculations performed 20-30 times per second. The data that is provided can be used to check if your car actually does benefit from a higher octane fuel, or to check if a new exhaust system has increased the horsepower as much as the manufacturer claims (or just makes a lot more noise). The article will first give you some background on what an accelerometer is, and show how you can use it to calculate performance data. An explanation of the development of iAccelerate will follow, describing the calculations and formulas used in code and the factors considered during development. Finally, I will demonstrate how iAccelerate works by walking you through a typical test scenario and comparing the results of the test to the results gathered from reputable sources to determine the accuracy of the data.

Background

In the abstract, I explained that an accelerometer is an instrument that is used to measure acceleration. Accelerometers are commonly used in many different types of applications. For example, they are used to determine if an airbag should be deployed during a crash, or to detect when your laptop has been dropped to prevent hard drive damage. Also, both the Nintendo Wii and PlayStation 3 controllers use accelerometers to calculate how you are holding or moving the controller. If you took physics class in high school or college, you may remember studying kinematics or dynamics. Together, these two areas of mechanics make up the foundation for iAccelerate. By using a few simple physics equations and an accelerometer, you can determine both the velocity (mph) and distance traveled (miles) of a car, assuming it is at a complete stop when the measurements are started. This assumption is important because acceleration is the rate of change of velocity with respect to time. For example, if we are sitting at 0 mph and we measure a change in acceleration with our accelerometer then we must be moving now since acceleration cannot occur without a change in velocity. We can determine the exact value of the car's velocity and distance traveled by using derivations of the following two equations:

Velocity_final = Velocity_initial + Acceleration × Time

Distance = 1/2 (Velocity_final + Velocity_initial ) × Time

Once we obtain these values, we can use a computer to calculate virtually any metric relating to a car's performance. The following must hold true for iAccelerate to provide accurate data: the accelerometer provides both the direction and magnitude of the acceleration; we are able to calculate the time spent accelerating; we were not moving when we started timing. Using iAccelerate, we can measure 0-60 mph (acceleration), 60-0 mph (braking distance), lateral acceleration (handling), ¼-mile time (acceleration), and much more. For example, if we provide the vehicle's weight, other measurements such as horsepower and torque can be calculated. The accuracy of the measurements for horsepower and torque can be improved significantly by using the vehicle's coefficient of drag to account for the air resistance incurred on the vehicle. The accuracy of the other measurements should theoretically be identical to a measurement taken with a radar gun, or from the use of a 5th wheel. In practice, however, factors such as the earth's gravity will introduce some margin of error. You will obtain the most accurate measurements by testing on a flat surface.

Development

This application was developed using the 2005 Express Editions of VB.NET and Visual C#, which are available free of charge from the link provided at the beginning of this article. I created the project with each class being as loosely coupled from one another as possible. The project has a class called CarCalculator that is responsible for all of the necessary physics calculations and handles the conversion of the calculated results to U.S. customary units. This class is instantiated by calling the constructor with the gross weight (in pounds) of the car tested as its parameter. Most properties of this class return their value in U.S. customary units but are set internally using the metric system of units, as these are the native units the class uses in the calculations. The following code is an example of how the CarCalculator stores weight privately (in kilograms) and how the class exposes this value publically (in pounds).

The constructor:

C#

public CarCalculator(double weight)
{
     this.Weight = weight;
     this.reset();
}

VB.NET

Public Sub New(ByVal weight As Double)
    Me.Weight = weight
    Me.reset()
End Sub 

The Weight property of the class:

C#

public double Weight
{
    get     
    {
         return _weight;
    }
    set
    {
         _weight = (value / POUNDS_PER_KG);
    }
}

Public Property Weight() As Double
    Get
        Return _weight
    End Get
    Set(ByVal Value As Double)
        _weight = Value / POUNDS_PER_KG
    End Set
End Property 

The CarCalculator uses an event model to notify consumers during a test. The SpeedReachedCheckpoint events are fired when the CurrentSpeed property reaches one of the speeds (e.g. 60 mph) added by the user in the Setup form. This functionality is shown in the following code snippets.

C#

if (_checkPoints != null)
{
    foreach (double speed in _checkPoints)
    {
        if (_lastCheckpoint != Math.Floor(speed)
             &&  (Math.Floor(CurrentSpeed) == Math.Floor(speed)))
        {
            SpeedReachedCheckpoint(_totalTime, CurrentSpeed);
            _lastCheckpoint = Math.Floor(CurrentSpeed);
        }
     }
 }

VB.NET

If Not _checkPoints Is Nothing Then
    For Each speed As Double In _checkPoints
        If lastCheckpoint <> Math.Floor(speed) AndAlso Math.Floor(CurrentSpeed) = Math.Floor(speed) Then
            RaiseEvent SpeedReachedCheckpoint(_totalTime, CurrentSpeed)
            lastCheckpoint = Math.Floor(CurrentSpeed)
        End If
    Next
End If

The next class I will discuss, called Setup, is instantiated only when the application first loads. It is solely responsible for handling all of the data that is user configurable and validating input for any user-supplied settings, such as ”speed reached checkpoints”, vehicle and passenger weights, and the car's coefficient of drag (if horsepower is being measured).

Figure 1 (below) shows the Setup form.

The next class, called Main, is responsible for handling all of the input and output to and from the CarCalculator class. The accelerometer raises an event providing the current acceleration value to the wired event handler 20-30 times per second. This value, along with the amount of time that has elapsed since the last time the event was raised, is passed into a public method in the CarCalculator class. This method uses these two values to perform all of the calculations, and sets the results of the calculations to properties in the CarCalculator class. Since this event is fired so frequently, the Main class/form only displays these results on the screen four times per second. The user can also configure the application to write the results to a CSV file for further analysis. In this case, a row is added to a file every time this event is fired. Consequently, the resulting file will contain the results of every calculation made by the CarCalculator. This file can be opened with Excel to create charts and graphs so the data can be easily interpreted.

This event handler is very hard to test or debug because all of the calculations performed are based on time. For instance, if you set a breakpoint in the event handler so you can evaluate an expression, the Stopwatch object responsible for measuring the elapsed time will continue to tick away when the debugger reaches the breakpoint – rendering it nearly impossible to evaluate the expression the next time the event is fired. The elapsed time value that the CarCalculator relies on will now also include the time that it took to debug, ultimately resulting in inaccurate output.

private void accel_AccelerationChange(object sender, AccelerationChangeEventArgs e)

{

    // pseudo code

    time = SetElapsedTimeSinceLastCalculation;

    // when a breakpoint is set to check the results of the following

    // statement, the time value will be increasing, causing inaccurate

    // data the next time this code is executed

    CalculateVehiclePerformance(time, e.Acceleration);

    // example calculation (used in the above method): 

    // currentSpeed = lastSpeed + (((e.Acceleration + _lastAcceleration) * time) / 2);

    DisplayResults;

}

There is too much code to cover here, and I have already outlined the location and functionality of the methods that are pertinent to accurate data output. I highly recommend reading through the source code to fully understand the program. I will now demonstrate the application using the data from my RX-8's test run.

Final Product

To use this application, you will need a vehicle to test, a notebook computer, and an accelerometer. You can purchase a USB accelerometer from http://www.trossenrobotics.com/ for less than $70. A .NET assembly is also available for download from this site so you can start coding without having to write your own interface for the accelerometer. Most, if not all, RS232 accelerometers that I found would work just as well but would require much more development time. In addition, since you would be using the accelerometer from within the confines of a car, being able to quickly plug and unplug the device is essential. From information gathered from various forums, a tri-axis USB model is currently in development, which you could use to negate the error caused by the acceleration due to gravity that was mentioned earlier. I will now walk you through a scenario where the 0 to 60 mph time of my RX-8 is tested.

Setting the weight, output types, and checkpoints:

Figure 1 – Setting up the calculator

The frontal area of your car can be found by having a friend park behind you with their lights on (park your car's nose 5 feet or so from a wall or building) and then trace the shadow. Measure the shadow's area using a metric ruler. The current default setting is for an older RX-7 that I found data for. I am sure my car is a little bigger, so this probably has an effect on the resulting horsepower value. Since horsepower is the only calculation that uses this setting, all other data will be accurate – so in other words, you can leave this setting as the default if you do not know your car's frontal area. The other defaulted setting is the coefficient of drag, which is defaulted to the average value found for about ten sports cars. If you would like to use your car's specific value, you can find it by running your car's name through a search engine with the word “drag” next to it.

Main output screen (after start has been clicked):

Figure 2 – The application is waiting for the change of acceleration to exceed the value entered during the setup stage (.09g) before timing will begin.

Main output screen results (10 seconds elapsed – timing over):

Figure 3 – The results of a 0-60 test on a 2004 Mazda RX-8 (6-speed)

If you noticed, the 0-60 time arrived in 5.871x seconds. Two magazines quoted it right around 5.8 s while another quoted it at 6.5 s.

Excel output (every calculation from the 10 second run is in this file – much more detailed than the screen's output):

Figure 4 – A CSV file opened with Excel allowing for analysis of all of the calculated data (e.g. horsepower graph)

Conclusion

I have tried to make this project as easy to modify as possible by keeping the data calculations separate from the display of calculated data to the user. This allows for greater flexibility when developing iAccelerate on other platforms. For example, rewriting this application for use on a Windows Mobile Smartphone would be cool, as you would not have to worry about breaking your laptop when you try to beat your cornering record. The processing power required to perform the needed calculations and output the results fast enough was unknown during design, so the decision to build a standard WinForm application was predisposed. If anyone redesigns this for use on a cell phone (or redesigns it at all), please provide some feedback on how it turns out. The use or modification of this software for educational or testing purposes is strongly encouraged. The developer prohibits the use of this source code and/or assembly for monetary and/or personal gain. If errors are discovered in any of the code or the calculations that were used within iAccelerate, please feel free to contact me at the following email address: mharkabus@claritycon.com