Overall Recap of each project

Iron Bartender!

Introduction

The term Virtual Reality (VR) has been around many years by now but is still associated with very special and expensive hardware that is only available for research laboratories or big companies. Most people might think of a setup where the user wears a head-mounted-display and a data glove. Technically speaking Virtual Reality is defined by two things:

• Immersion and
• Interactivity

By immersion is meant, that the user gets the impression of being inside a computer generated world. Usually VR systems use stereoscopic output devices to give the user a real 3D impression of the virtual world. But besides visual immersion, haptic and acoustic can also play an important role.

3D images can for example be experienced in IMAX 3D cinemas – but this is still no Virtual Reality. What is missing is the interactivity. When the user is able to interact with the virtual world, by navigation and manipulation of the objects then the definition for VR is complete.

In this article I want to introduce a Virtual Reality system that can be build with two Nintendo Wii Remotes (Wiimotes) and just a little extra hardware that costs less than 10 USD. By that VR should be affordable for everyone. The demo software and source can be downloaded from my website:
http://www.vrhome.de.

Virtual Reality Desktop

A VR Desktop setup uses a monitor as 3D output display and is sometimes referred to as fishtank Virtual Reality. I use a stereo monitor that needs a pair of polarization glasses to display stereo images. If you don't own a stereo monitor you can still construct the VR Desktop by using the anaglyph or red/green stereo method. For that you just need a pair of red/green glasses and you can use any monitor. Even though you loose color information the stereo effect is still very good.

For interactivity the two Wiimotes come into play. In this setup there are two kinds of interaction possible. First we have a kind of navigation support. Navigation inside a rendered 3D scenes means the movement of the camera, as the camera represents the eyes of the user. For the Desktop setup this means we have to track the users' eyes or head position to move the 3D scene camera correctly – therefore this is also called head tracking. For the head tracking I mount one Wiimote at the top of the monitor to point at the user's face.

The second Wiimote is mounted from the top in vertical down position facing the area in front of the monitor. This will be used to interact with the 3D scene by manipulating the transformation of a 3D object. As you use the hand for this interaction I call this hand tracking. The Virtual Reality Desktop setup is shown in Picture 1.

Wiimote 6DOF Tracking

The core of this VR setup is using the Nintendo Wiimote Controller as a tracking camera. As one might already know the Wiimote is equipped with an infra-red camera that is able to recognize up to four infra-red lights. The main idea now is to build a device with four infra-red LEDs that can be recognized by the Wiimote. This device I call IR-LED beacon. Using the values of the 4 LEDs as recognized by the Wiimote I use an algorithm to reconstruct the original position and orientation of the IR-LED beacon. Getting all 3 axis translations and all 3 axis rotation angles of the device is called 6 degree of freedom tracking – or short 6DOF tracking.

IR-LED Beacon

Picture 1: Wiimote Desktop Virtual Reality Setup

Because this setup uses head tracking as well as a tracking for a hand manipulation two beacon have to be built. In pictures 2 and 3 the beacons that I built for my setup are shown.

Picture 2: Head Tracking IR-LED Beacon

Picture 3: Hand Tracking IR-LED Beacon

As can bee seen the beacons just consist of 4 IR-LEDs, a battery holder and wiring. For easy construction and less wiring I use a stripboard. When choosing the IR-LEDs it is important to look out for a very wide angle of radiation. Typical LED just have a small angle, try to find LEDs with angles greater than 120°, a good source might be digikey.com. Three of the four LEDs are aligned in a line with only slightly different height. The fourth LED is mounted above the line with more height. This special order of the lights is needed by the algorithm to be able to assign the IR-points recognized by the Wiimote to the original LEDs of the beacon. It is also important that the fourth LED has not the same height, so that the LEDs are not so planar. Please see picture 4 for a schematic layout of the beacon. For power supply I just us on AAA battery and connect all LEDs in parallel to the battery poles. For easy handling I use a battery holder which are also available at electronic components supply stores.

After soldering everything together the exact positions of the LEDs have to be measured in the correct order. It is very important to measure the 3D position of the LED lights very accurately. If the data for these coordinates is not accurate the tracking results will be poor. The values have to be in millimeters.

Picture 4: Schematic of IR-LED Beacon

For example, my LEDs have the following measurements:

<point3d value="1, 0, 8.5"/>
<point3d value="29, 0, 11"/>
<point3d value="56, 0, 6"/>
<point3d value="29, 45, 21"/>

Configuring the Software

After position the Wiimotes as shown and constructing and measuring the beacons you have to configure the software using two configuration files. You will find the files in the application's directory.

Settings.xml

<Settings> 
    <Stereo
        eyeDistance = "0.02"
        switchLeftRight = "False"
        fieldOfView = "60"
        antiAlias = "False"
        stereoMode = "lineInterlaced"
        fullscreen = "False"
        resolution = "1280,1024"
        displayDevice = "Screen"
        anaglyph = "True"
        windowPosition = "0,0" />
</Settings>

If you own a line-interlaced 3D Monitor from Zalman, you can change the anaglyph to False. If you have difficulties adjusting your eyes to the stereo image you can try changing the value for the eyeDistance. Reducing the value makes it easier to adjust but might reduce the 3D effect. The other parameters should work as you might expect from the name.

Tracker.xml

In this file the configuration for the tracking is stored. It defines the Wiimotes and its positions and the IR-LED beacons. Furthermore it defines filtering parameters. Here I will point to the places where you need to adjust the values for your setup.

First you will need to specify the exact position of your Wiimotes relative to the center of the screen. You need to measure this distance for each Wiimote in millimeter. Probably the values will be similar to the default values:

<TrackerCam 
    id="2" cameramodel="Wiimote" 
    translation="0,200,50" 
    rotation="0,0,0" scale="0.001" 
    xAxis="x" yAxis="y" zAxis="z"> 
</TrackerCam>
<TrackerCam
    id="1" cameramodel="Wiimote"
    translation="0,350,350" rotation="0,0,0" scale="0.001"
    xAxis="x" yAxis="z" zAxis="y">
</TrackerCam>

The Wiimote with id=2 is the one mounted at the top of the monitor to do the head tracking. In the translation field enter the distance of the front tip of the Wiimote to the center of the screen. In the default settings the Wiimote is 200mm above (y-value) and 50mm in front (z-value) of the center. Do the same for the second Wiimote in the following entry with id=1. You can see, that here the y-axis is assigned to the z-axis and the z-axis is assigned to the y-axis. This is necessary because the Wiimote is mounted in vertical position. If the software recognizes the Wiimotes in wrong order you can exchange the order of the definition blocks.

The second part where you need to make changes is the definition of the IR-LED-Beacons:

<MarkerBody>
    <WiiMarkerBody id="0" name="WiiMote Head Beacon" nearClip="20"
        farClip="1500" translation="0,0,0" rotation="0,0,0">
        <point3d value="0, 4, 7"/>
        <point3d value="40, 4.5, 10.5"/>
        <point3d value="83.5, 5, 7.5"/>
        <point3d value="38, 45, 18"/>
    </WiiMarkerBody>
    <WiiMarkerBody id="4" name="WiiMote Hand Beacon" nearClip="20" 
        farClip="1500" translation="0,0,0" rotation="0,0,0">
        <point3d value="1, 0, 8.5"/>
        <point3d value="29, 0, 11"/>
        <point3d value="56, 0, 6"/>
        <point3d value="29, 45, 21"/>
    </WiiMarkerBody>
</MarkerBody>

Here you need to change the values for the beacon points in millimeter according to your measurements. Please note that the correct order of the points is necessary. They have to be from LED1 to LED4 according to the schematic in Picture 4.

Furthermore you will find a description for the tracked device:

<TrackedDevice id="4" type="WiiMote" rotation="True" translation="True">
    <MarkerBodyId>0</MarkerBodyId>
    <TrackerCamId>2</TrackerCamId>
    <ReverseTranslation>False,True,True</ReverseTranslation>
    <ReverseRotation>False,False,False</ReverseRotation>
    <LocalTranslation>-45,25,0</LocalTranslation>
    <LocalRotation>0,0,0</LocalRotation>
    <WorldTranslation>0,0,0</WorldTranslation>
    <WorldRotation>0,0,0 </WorldRotation>
    <RotationFilterId>4</RotationFilterId>
    <TranslationFilterId>4</TranslationFilterId>
</TrackedDevice>

Here you assign which Wiimote (TrackedCamId) to use with which IR-LED-Beacon (MarkerBodyId). The only value that you might want to adjust from these settings is the LocalTranslation. The translation and rotation values that are calculated need to have their reference point on the beacon. By default the first LED will be this reference. Usually you want to have a different reference point, like the center of the beacon. Therefore measure the distance from the first LED to your preferred reference point in the same metric as the beacon values and write it in the LocalTranslation.

There are lot of more parameters you can tweak inside this configuration file, but to make the VR-Desktop run this shouldn't be necessary.

Run the Software

Before running the software it is necessary to connect the Wiimotes with the computer. For that the computer needs to be equipped with a compatible Bluetooth adapter. For more detailed description on how to do that please refer to Brian Peek's Wiimote Library article:

http://blogs.msdn.com/coding4fun/archive/2007/03/14/1879033.aspx

After putting in your values in the configuration files and placement and connection of the two Wiimotes you can run the binary installation by clicking on the VRDesktoDemo in the start menu. If you want to run from the source code you have to copy the OpenCV dlls from (VRDesktopSrc)\ExtLibs\OpenCV\opencvlib to the binary destination directory of the compiled project, e.g. (VRDesktopSrc)\VRDesktopDemo\bin\x86\Release before you can start the application.

Using the Library

Using the VRDesktop in your own XNA application is very easy. Here I will point out the relevant steps from the VRDesktopDemo application.

To start from scratch you will have to create a new XNA Windows Game Project. First include the references to the two libraries Tgex and Tvrx. Then open the created Game class.

At the top the namespaces for libraries have to be added:

using Tgex.Graphics;
using Tgex;
using Tvrx;

Then the parent class has to be changed from Game to VRGame:

public class VRDesktop : VRGame

VRGame is part of the Tgex library and adds the support for the stereo display. It creates a stereo camera and the application window according to the settings file. You can use the class more or less similar to the original game class. The main difference is, that for the drawing you must not override the Draw(GameTime time) function but the new DrawScene(GameTime time) because the Draw function of the VRGame class takes care about the stereo rendering.

For storing the transformation matrix of the hand tracker we define a variable:

Matrix modelTransform = Matrix.Identity;

And in this simple example we define a variable for the model:

Model model;

In the Initialize function the initialization of the TrackerManager needs to be done:

protected override void Initialize()
{
    // The tracker manager is a singleton but needs to be initialized once.
    TrackerManager.Instance.Initialize();

    // initialize base clase.
    base.Initialize();
}

In the LoadContent function the model is loaded and the tracking is started:

protected override void LoadContent()
{
    // for this demo just load the coordinate cross
    model = Content.Load<Model>("coordinate");
    modelTransforms = new Matrix[model.Bones.Count];

    // start tracking now
    TrackerManager.Instance.StartTracking();
}

The Update function is for the main game logic. First we allow the user to exit the game and the tracker to stop regularly:

// Allows the default game to exit on Xbox 360 and Windows
if ((GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed)
    || (Keyboard.GetState().IsKeyDown(Keys.Escape)))
{
    TrackerManager.Instance.StopTracking();
    this.Exit();
}

Before getting the latest tracking data we have to call an update on the TrackerManager:

TrackerManager.Instance.Update();

To get the transformation data we have to call the GetProxyTransform(indexNumber) from the manager. The proxies are defined in the tracking.xml file. In the example we call:

// tracking proxy with id 1 is the hand tracker
modelTransform = TrackerManager.Instance.GetProxyTransform(1);

// tracking proxy with id 0 is the head tracker - change eye position.
m_camera.EyePosition = 
    TrackerManager.Instance.GetProxyTransform(0).Translation;

The m_camera is defined in the VRGame parent class. The camera class will also make the necessary camera transformation adjustments for screen projected head-tracking by creating a perspective of center projection matrix.

Finally in the DrawScene function the model mesh is drawn. Here we need to pass the model transformation matrix as well as the camera matrices to the effect:

//Draw the model, a model can have multiple meshes, so loop
foreach (ModelMesh mesh in model.Meshes)
{
    //This is where the mesh orientation is set, as well 
    // as our camera and projection
    foreach (BasicEffect effect in mesh.Effects)
    {
        effect.EnableDefaultLighting();
        effect.World = modelTransforms[mesh.ParentBone.Index]
                       * Matrix.CreateScale(0.01f) // coordinate scale
                       * modelTransform;
        effect.View = m_camera.ViewMatrix;
        effect.Projection = m_camera.ProjectionMatrix;
    }

    //Draw the mesh, will use the effects set above.
    mesh.Draw();
}

That's all what has to be done to make use of the Wiimote Virtual Reality Desktop in your own application.

How does it work

For the interested reader I will explain now in more detail how the Wiimote tracking actually works. Anyhow, I will not go into the mathematical details of the core algorithm but provide you with the necessary references. I will just focus on the Wiimote tracking part and will not go into details of neither the tracking library Tvrx nor the game library Tgex.

Pose Estimation

Technically speaking, the Wiimote tracking is an optical marker based tracking. It is optical because we use the Wiimote camera and it is marker based, because we don't use the whole camera image. Instead the Wiimote returns the position of up to four infrared LEDs which represent our markers.
The four LED positions are given in 2D coordinates which represent the markers as seen by the camera. Technically this is a 2D projection of the real LEDs according to the focus of the camera lens, this projection plane is called image plane - the 2D coordinated are therefore called image points.
The main task now is to calculate the position and rotation of the IR-LED Beacon in real space according to the image points of the LEDs measured by the Wiimote. This task is called Pose Estimation and has been investigated by scientists since many years. One main application of pose estimation is computer vision for robotics. The pose estimation algorithm used for the Wiimote tracking has already been published in 1995 and still performs nice for our purpose. If you are interested in the exact details about how the pose estimation algorithm works, you can study the original paper, please see the references.
To be able to perform the pose estimation calculation you need the exact information about the image plane. Actually, the plane is nothing else then the image sensor of the camera. The picture is focused on the image sensor by the camera lens. For the calculation of the projection we need a mathematical model for the camera, in computer vision usually a pinhole camera model is used. The pinhole is the projection point origin. The image plane is at a certain distance from the origin. This distance is called focus length.
Now, what we need for the calculation is the focal length and the size of image plane, the camera sensor. These values are also called the intrinsic camera values. Unfortunately Nintendo does not publish these values for the Wiimote, so the values I use are assumptions:
focal length in pixel = 1380

// assume 1/4" cd sensor (even though it probably is not)
pixel size in mm = 0.0035
chip resolution = 1024x768

// Wiimote center (approx.)
principal point = 512x384

The resolution of the values returned by the Wiimote are 1024x768. Obviously this is not the physical resolution because cameras with this resolution would cost more than 1000 USD. The Wiimote has a PixArt Imaging Inc. (http://www.pixart.com.tw) sensor and probably has a resolution of 352x288 or 164x124. However, trying to guess the real values with the help of the PixArt Sensor data sheets did not work out satisfactory so I decided to fix the pixel size and resolution on the above values and estimate the focal length. Even thought the values are not correct, they only need to be correct relative to each other to make the pose estimation work. The principal point is the actual origin of the image plane. Ideally this value should be measured, here I just assume it is the middle of the sensor chip.

Overview of the tracking algorithm

The overall tracking algorithm can be divided in the following steps:

• Retrieve the image point values of the infrared LEDs from the Wiimote.
• Assign the image points to the LED beacon lights.
• Run the pose estimation to calculate the rotation and translation of the LED beacon.
• Filter the resulting rotation and translation values.
• Build a transformation matrix and transform the result according to configuration file.
  (Wiimote position and orientation and local LED beacon transformation)

In the following I will describe the steps in more detail and with code examples.

Retrieve image points

The connection and data retrieval with the Wiimote is done in the WiiMoteTracker class. This class implements the interface IMarkerTracker, that defines the interface for an optical marker based tracker. The Wiimote is initialized in the Initialize() function and connected when StartTracking() is called :

public void Initialize()
{
    // test static variable for first time call
    if (m_wiimoteCount == 0)
        m_wiimoteCollection.FindAllWiimotes();

    if (m_wiimoteCollection.Count <= m_wiimoteCount)
    {
        ErrorHandler.Report("Invalid WiimoteTracker count, only "
        + m_wiimoteCollection.Count.ToString() + " Wiimotes found");
        return;
    }

    wm = m_wiimoteCollection.ElementAt(m_wiimoteCount);
    m_wiimoteId = m_wiimoteCount;
    m_wiimoteCount++;

    // setup the event to handle state changes
    wm.WiimoteChanged += wm_WiimoteChanged;

    // setup the event to handle insertion/removal of extensions
    wm.WiimoteExtensionChanged += wm_WiimoteExtensionChanged;

    // create filter for accelerator values
    AverageFilterDesc filterDesc = new AverageFilterDesc();
    filterDesc.numOfValues = 1000;

    for (int i = 0; i < 3; i++)
    {
        m_acceleratorFilter[i] = new AverageFilter();
        m_acceleratorFilter[i].SetFilterDesc(filterDesc);
    }

    // create filter for image points
    filterDesc = new AverageFilterDesc();
    filterDesc.numOfValues = 5;

    for (int i = 0; i < 8; i++)
    {
        m_imagePointsFilter[i] = new AverageFilter();
        m_imagePointsFilter[i].SetFilterDesc(filterDesc);
    }
}

public void StartTracking()
{
    if (!m_isTracking)
    {
        m_isTracking = true;

        // connect to the Wiimote
        try
        {
            wm.Connect();

            // set the report type to return the IR sensor and accelerometer data 
            // (buttons always come back)
            wm.SetReportType(InputReport.IRAccel, true);
            wm.SetLEDs(m_wiimoteId);
        }
        catch
        {
            ErrorHandler.Report("Cannot connect to Wiimote");
            m_isTracking = false;
        }
    }
}

To receive the Wiimote data the wm_WiimoteChanged Callback has been registered. This function is then called whenever the Wiimote has updated values. Inside this function the infrared LED values are read and the LED beacon light assignment is done.

For that, first a list with with Vector2 for the image points is created:

// put in list
irList.Clear();

for (int i = 0; i < 4; i++)
    irList.Add(new Vector2((float)
           (ws.IRState.IRSensors[i].Position.X * m_resolution.X) - m_principalPoint.X,
           (float)(ws.IRState.IRSensors[i].Position.Y * m_resolution.Y) - m_principalPoint.Y));

Assign image points

Then the values have to be assigned to the IR-LEDs by putting them into the right order. This is done by a simple geometric pattern recognition. The idea is to have a geometric pattern that is invariant to the projection from 3D to 2D. As can be seen in picture 4, the 3 LEDs of the LED beacon are arranged in more or less a line and the 4^th LED is above of the line. In the 2D image data of the Wiimote the 3 LEDs also form more or less a line. Therefore, the first step in the assignment algorithm is to find the 3 image points which are most close to form a line.The line test is done in the following function:

void TestPoints(Vector2 lineStartPoint, Vector2 lineEndPoint,
                Vector2 onLinePoint, Vector2 freePoint)
{
    float lambda;
    float dist = onLinePoint.DistanceToLine(lineStartPoint, lineEndPoint, out lambda);

    // check if projected point is between line end and start point
    if ((lambda > 0) && (lambda < 1))
    {
        // if distance is short, make this combination the result
        if (dist < m_pointLineDist)
        {
            m_pointLineDist = dist;
            m_lineStartPoint = lineStartPoint;
            m_lineEndPoint = lineEndPoint;
            m_onLinePoint = onLinePoint;
            m_freePoint = freePoint;
        }
    }
}

The function is called with the four image points as input. It assumes the first point to be the line start point and the second point to be the line end point. Then the distance of the third point to the line is calculated. This is done by using a C# 3.0 extension function of Vector2:

public static float DistanceToLine(this Vector2 point,
                                   Vector2 startLinePoint,
                                   Vector2 endLinePoint,
                                   out float lambda)
{
    Vector2 rv = endLinePoint - startLinePoint;
    Vector2 p_ap = point - startLinePoint;
    float dot_rv = Vector2.Dot(rv, rv);
    lambda = Vector2.Dot(p_ap, (rv / dot_rv));
    Vector2 distVec = point - (startLinePoint + lambda * rv);
    return distVec.Length();
}

The line distance test is a typical algorithm as described on: http://mathenexus.zum.de/html/geometrie/abstaende/AbstandPG.htm

It returns the distance to the line and an lambda value, which defines the position of the line projected point. If the projection point is outside the line start and line end point the lambda will be below 0 or greater than 1.

If the line was valid the distance of the third point is compared to the formerly smallest distance, and if it is smaller, then this order of image points will be saved as best solution.

To find the right order of the image points this function has to be called with all possible combinations of the 4 LED image points. In my code I do all the calls explicitly after initializing the minimum distance with float max value:

m_pointLineDist = float.MaxValue;

// write all test cases explicite
// in the end the three point line should be found
TestPoints(irList[0], irList[1], irList[2], irList[3]);
TestPoints(irList[0], irList[1], irList[3], irList[2]);
TestPoints(irList[0], irList[2], irList[1], irList[3]);
TestPoints(irList[0], irList[2], irList[3], irList[1]);
TestPoints(irList[0], irList[3], irList[1], irList[2]);
TestPoints(irList[0], irList[3], irList[2], irList[1]);
TestPoints(irList[1], irList[2], irList[0], irList[3]);
TestPoints(irList[1], irList[2], irList[3], irList[0]);
TestPoints(irList[1], irList[3], irList[0], irList[2]);
TestPoints(irList[1], irList[3], irList[2], irList[0]);
TestPoints(irList[2], irList[3], irList[0], irList[1]);
TestPoints(irList[2], irList[3], irList[1], irList[0]);

Now that we have the right order of the three points that form the line and the 4^th point, it is still necessary to determine the right direction of the line. In our LED beacon the 4^th LED is above the line. If the start and end point of the line would be interchanged then the 4^th LED would be below the line. Mathematically we check if the order of the points is clockwise or counterclockwise:

// only remaing test is check if line start and 
// end point is in right order
// check start and end line point with free point
// if clockwise direction then ok, if counterclockwise then exchange start 
// and end point

Vector2 E1 = m_lineStartPoint - m_freePoint; // P1-P2 
Vector2 E2 = m_lineEndPoint - m_freePoint; // P3-P2 
bool clockwise;
if ((E1.X * E2.Y - E1.Y * E2.X) >= 0)
    clockwise = true;
else
    clockwise = false;

if (!clockwise)
{
    Vector2 tmp = m_lineEndPoint;
    m_lineEndPoint = m_lineStartPoint;
    m_lineStartPoint = tmp;
}

The algorithm for the clockwise check is taken from

http://www.geocities.com/siliconvalley/2151/math2d.html

Then the correctly ordered points are slightly filtered with a simple average filter over the last 5 values and multiplied with the pixel size the change from pixel metric to millimeter metric. Finally the points are passed to the pose estimation class.

// now write resulting order to image points
m_imagePoints[0].X = m_imagePointsFilter[0].Filter(m_lineStartPoint.X);
m_imagePoints[0].Y = m_imagePointsFilter[1].Filter(m_lineStartPoint.Y);
m_imagePoints[1].X = m_imagePointsFilter[2].Filter(m_onLinePoint.X);
m_imagePoints[1].Y = m_imagePointsFilter[3].Filter(m_onLinePoint.Y);
m_imagePoints[2].X = m_imagePointsFilter[4].Filter(m_lineEndPoint.X);
m_imagePoints[2].Y = m_imagePointsFilter[5].Filter(m_lineEndPoint.Y);
m_imagePoints[3].X = m_imagePointsFilter[6].Filter(m_freePoint.X);
m_imagePoints[3].Y = m_imagePointsFilter[7].Filter(m_freePoint.Y);

for (int i = 0; i < 4; i++)
{
    m_imagePoints[i] *= m_pixelSize;
}

// send points to estimation
m_poseEstimate.UpdateImagePoints(m_imagePoints);

Run pose estimation

The pose estimation is done by the class Posit. This class implements the IPoseEstimate interface:

public interface IPoseEstimate
{
    void InitializeCameraParameter(double focalLengthMM, bool flipImage,
                        float scale, int[] assignAxis, int[] assignAxisSign);
    void InitializeMarkerBody(Vector3[] markerPoints);
    void UpdateImagePoints(Vector2[] imagePoints);
    void GetTransform(out Vector3 position, out Vector3 rotation);
    void StartEstimation();
    void StopEstimation();
}

The pose estimation has to be initialized with the focal length of the tracking camera. The 3D positions of the real device – in our case the LEDs of the LED beacon – are passed in the InitializateMarkerBody function. The measured image points are passed with the UpdateImagePoints call and the calculated result can be taken from the GetTransform function. Because the pose estimation itself runs in an own thread asynchronously it has to be started and stopped by StartEstimation and StopEstimation. By using the interface it is easy to plug in different pose estimation algorithms.

As mentioned before the pose estimation algorithm used here is the PosIt algorithm published by D. DeMenthon. I use the implementation from the OpenCV computer vision library. As this library is C code it has to wrapped to manage code. I use the freely available wrapper EmguCV. Before the pose estimation can be done a pose estimation object has to be created. This is done when the 3D positions of the markers are passed:

public void InitializeMarkerBody(Vector3[] markerPoints)
{
    m_numOfMarker = markerPoints.Length;
    MCvPoint3D32f[] worldMarker = new MCvPoint3D32f[m_numOfMarker];
    for (int i = 0; i < m_numOfMarker; i++)
    {
        worldMarker[i].x = markerPoints[i].X;
        worldMarker[i].y = markerPoints[i].Y;
        worldMarker[i].z = markerPoints[i].Z;
    }
    m_positObject = CvInvoke.cvCreatePOSITObject(
    worldMarker, m_numOfMarker);
    m_imagePoints = new MCvPoint2D32f[m_numOfMarker];
    m_imagePointsBuffer = new Vector2[m_numOfMarker];
}

MCvPoint32f is a managed structure for the OpenCV CvPoint32f and similar to a Vector3f. The CvInvoke class of the EmguCV wrapper is a collection of static functions to invoke the original OpenCV functions. Because the pose estimation algorithm was not included in the class I had to insert the following functions:

/// <summary>
/// Create pose estimation object
/// </summary>
[DllImport(CV_LIBRARY)]
public static extern IntPtr cvCreatePOSITObject(MCvPoint3D32f[] points, 
                                                int point_count);
/// <summary>
/// Do pose estimation
/// </summary>
[DllImport(CV_LIBRARY)]
public static extern void cvPOSIT(IntPtr posit_object, 
                                  MCvPoint2D32f[] image_points,
                                  double focal_length,
                                  MCvTermCriteria criteria,
                                  float[] rotation_matrix,
                                  float[] translation_vector);
/// <summary>
/// Release pose esitmation object
/// </summary>
[DllImport(CV_LIBRARY)]
public static extern void cvReleasePOSITObject(IntPtr posit_object);

The returned object of the CvInvoke.cvCreatePOSITObject function call is a simple IntPtr and is used later for the pose estimation function.

The pose estimation itself is done in an own thread in the PoseEstimate() function. First the new image points are fetched. If no update is available we wait for new values. This is done with the Monitor.Wait and Monitor.Pulse mechanism:

// copy image points
Monitor.Enter(m_imagePointsLock);
if (!m_imagePointsUpdate)
    Monitor.Wait(m_imagePointsLock);

for (int i = 0; i < m_numOfMarker; i++)
{
    m_imagePoints[i].x = m_imagePointsBuffer[i].X;
    m_imagePoints[i].y = m_imagePointsBuffer[i].Y;
}
m_imagePointsUpdate = false;
Monitor.Exit(m_imagePointsLock);

After getting the new image values the cvPOSIT function is invoked:

MCvTermCriteria criteria;
criteria.type = TERMCRIT.CV_TERMCRIT_EPS | TERMCRIT.CV_TERMCRIT_ITER;
criteria.epsilon = 0.00001;
criteria.max_iter = 500;
CvInvoke.cvPOSIT(m_positObject,m_imagePoints, m_focalLengthMM,
                 criteria, POSITRot, POSITTrans);

Because the algorithm is an iterative algorithm the MCvTermCriteria defines when the algorithm should terminate. Here I defined that it should either terminate when either 500 iteration steps have been reached or when the difference of the values from the former iteration are smaller than 0.00001. You can play around with these value to see how the tracking accuracy reacts. Besides the termination criteria you have to pass the cvPOSIT function the IntPtr to the posit object, the image points and the camera focal length in millimeter. As result you get a 9 float array for the rotation matrix and a 3 float array for the translation result.

Because the rotation values should later be filtered they are translated to euler angels in the EulerAngles function. Euler angles define the rotation by giving the rotation about each coordinate axis. Before storing the final values there is some axis swapping and scaling according to the settings in the tracker.xml.

Filter the estimation results

Because the resolution of the Wiimote camera is not very hight and optical tracking always to some degree noisy the transformation results have quite strong jitters. To reduce the jittering the result values have to be filtered. As a side effect of strong filtering in tracking the virtual object seems not to follow the tracked objects movements directly and feels like swimming behind. A good compromise between jitter reduction and direct response is using Kalman filters. Kalman filters use a mathematical model to predict the change of the values and then uses the measured data to correct its prediction. A good introduction to Kalman Filters is the Siggraph 2001 Course from Greg Welch – see References. Anyhow, determine the best parameters for the filter for non-mathematicians is difficult. A good reference how to apply the filter in the tracking domain is the dissertation of Ronald Azuma “Predictive Tracking for Augmented Reality”. Please refer to that document if you want to learn the meaning of the parameters. For the Wiimote VR-Desktop the Kalman parameters are defined in the tracking.xml file:

<Kalman class="KalmanFilter" id="4" A="1, 0.005, 0, 1" 
    measurement_noise_cov="1.0" process_noise_cov_1="0.0000001"
    process_noise_cov_2="0.0000001"/>

The Kalman implementation used is again part of the OpenCV library. The EmguCV wrapper comes with a complete wrapper for this function. In my implementation there is an interface for data filers, the IDataFilter. The KalmanFilter class implements this interface. Beside the Kalman filter there is also a simple AvarageFilter in the library. The initialization of the Kalman filter is like this:

public void SetFilterDesc(DataFilterDesc desc)
{
    m_kalman = new Kalman(2, 1, 0);
    filterDesc = (KalmanFilterDesc)desc;
    // set A - second parameter is frames per second
    m_kalman.TransitionMatrix.Data.SetValue(filterDesc.A[0], 0, 0);
    m_kalman.TransitionMatrix.Data.SetValue(filterDesc.A[1], 0, 1);
    m_kalman.TransitionMatrix.Data.SetValue(filterDesc.A[2], 1, 0);
    m_kalman.TransitionMatrix.Data.SetValue(filterDesc.A[3], 1, 1);
    // set H
    m_kalman.MeasurementMatrix.Data.SetValue(1.0f, 0, 0);
    m_kalman.MeasurementMatrix.Data.SetValue(0.0f, 0, 1);
    // set Q
    CvInvoke.cvSetIdentity(m_kalman.ProcessNoiseCovariance.Ptr, new MCvScalar(1));
    m_kalman.ProcessNoiseCovariance.Data.SetValue(filterDesc.process_noise_cov_1, 0, 0);
    m_kalman.ProcessNoiseCovariance.Data.SetValue(filterDesc.process_noise_cov_2, 1, 0);
    // set R
    CvInvoke.cvSetIdentity(m_kalman.MeasurementNoiseCovariance.Ptr, new MCvScalar(1e-5));
    m_kalman.MeasurementNoiseCovariance.Data.SetValue(filterDesc.measurement_noise_cov, 0, 0);
    CvInvoke.cvSetIdentity(m_kalman.ErrorCovariancePost.Ptr, new 
    MCvScalar(500));
    m_kalman.ErrorCovariancePost.Data.SetValue(2, 0, 0);
}

After initializing a float value can be simply filter in the Filter function:

public float Filter(float inData)
{
    // Z measurement
    data.Data[0,0] = inData;
    m_kalman.Predict(predict);
    m_kalman.Correct(data);
    return m_kalman.CorrectedState[0, 0];
}

Because in the resulting transformation of the pose estimation there are 3 float values for the translation and 3 float values for the rotation altogether 6 separate instances of the filter are needed. In the Tvrx library the filtering is done inside the TrackedDevice class, which is a parent class for tracked devices and is derived to the TrackedWiimote:

public virtual void Filter() 
{
    m_rawTranslation.X = m_translationFilter[0].Filter(m_rawTranslation.X);
    m_rawTranslation.Y = m_translationFilter[1].Filter(m_rawTranslation.Y);
    m_rawTranslation.Z = m_translationFilter[2].Filter(m_rawTranslation.Z);
    m_rawRotation.X = m_rotationFilter[0].Filter(m_rawRotation.X);
    m_rawRotation.Y = m_rotationFilter[1].Filter(m_rawRotation.Y);
    m_rawRotation.Z = m_rotationFilter[2].Filter(m_rawRotation.Z);
}

Final transformations

Also in the TrackedDevice class the transformation of the tracking values from the camera world space coordinate system to the actual in game virtual space coordinate system takes place:

public virtual void TransformToVirtualSpace() 
{
    Matrix bodyTransformMatrix =
    Matrix.CreateFromYawPitchRoll(m_rawRotation.Y, m_rawRotation.X, m_rawRotation.Z)
                                  * Matrix.CreateTranslation(m_rawTranslation);
    Matrix result = m_TrackerWorldTransform * m_DeviceWorldTransform;
    result = bodyTransformMatrix * result;
    result = m_DeviceLocalTransform * result;
    Vector3 scale;
    result.Decompose(out scale, out m_Rotation, out m_Translation);
}

First a transformation matrix of the Euler angels and the translation vector is composed to the Body-Transform. From the tracking.xml configuration file we also have matrices for Tracker-World-Transform, Device-World-Transform and Device-Local-Transform.

In addition to the Tracker-World-Transform translation from the tracker.xml I calculate the rotation angles of the Wiimotes using the acceleration sensors. By that it is possible to rotate the Wiimotes around the x and z axis to better focus the area you want to track and still get correct tracking results automatically.

Finally, to get the correct final transformation the matrices have to be multiplied in the correct order:

Device-Local-Transform * Body-Transform * Tracker-World-Transform * Device-World-Transform

Now the pose estimation transformation result is ready to be read by the TrackerManager.

Conclusion and Limitation

As I have shown it is possible to create a low-cost Desktop Virtual Reality Setup using two Wiimotes and anaglyph stereo glasses. But because of the low resolution of the Wiimote camera the quality is not comparable to professional monocular tracking systems. However, the quality of the presented system could still be improved by correctly measuring the intrinsic parameters of the Wiimote camera. There are various known algorithms how to measure these parameters by a set of sample data of an object with regular known geometry. For standard cameras usually a checkerboard pattern is used. An algorithm for that purpose is also integrated in OpenCV, so applying it to the Wiimote shouldn't be to hard.

Another issue with intrinsic parameters that I completely ignored so far is the lens distortion. All camera lenses do distort the image to a certain degree. By assuming a circular distortion the algorithms that measure the intrinsic parameters also calculate distortion parameters. If those parameters would be measured the Wiimote image points could be easily undistorted and the pose estimation results would be improved.

References

• D. DeMenthon and L.S. Davis, "Model-Based Object Pose in 25 Lines of Code", International Journal of Computer Vision, 15, pp. 123-141, June 1995.
  Paper about the used pose estimation algorithm by Daniel DeMenthon. You can find this brilliant paper on his homepage.
• OpenCV: Open Source computer vision library. Includes implementation of the pose estimation algorithm and Kalman filter.
• EmguCV: C# wrapper for OpenCV
• WiimoteLib: Managed Library for Nintendo's Wiimote by Brian Peek
• XNAnimation: Very nice library for animations in XNA. I didn't use it for the actual Wiimote tracking, but used his animation demo application in the demonstration video.
• XNA: Game development library in C# that I use as base for my applications.
• An Introduction to the Kalman Filter. Greg Welch and Gary Bishop. Siggraph 2001 Course 8.
• Predictive Tracking for Augmented Reality. Ronald Tadao Azuma. Dissertation. University of North Carolina. February 1995.

The Micro Framework is one of the newest kids on the .NET block, but it does something really rather wonderful. It brings embedded development within the reach of any C# programmer. If you know C# and love Visual Studio, you can now get started building hardware and controlling it with your software. Moreover, it lets developers achieve one of their most cherished dreams, to control their festive lights using programs that they have written. This project shows you how to do just that and adds an extra magical feature, in that you can make all your festive lights flash red whenever I, Rob Miles, make a new post on that most famous of blogs, www.robmiles.com.

Actually, you can modify the code so that you can make your lights do most anything in response to an event that happens on the web. You could signal home that you are running late, send the weather forecast to your Christmas tree or explore any number of communication options. You might decide that this is so useful that you leave your decorations up all year round.

If you just want to play with the .NET Micro Framework and get a feel for how easy it is to create software for tiny devices you don't actually need to use any extra hardware at all. The project comes with a complete emulation of the lights display so that you can run the whole thing on your computer and learn how hardware and software can be made to work together without burning your fingers with a soldering iron.

However building the hardware will give you an understanding of how some simple electronic components can be controlled from C# and even how serial and parallel data transfer works. It is also great fun.

To get started you will need some hardware and some software. Let's take each in turn.

Hardware

Processor Hardware

The .NET Micro Framework lets you run C# programs on tiny embedded devices. There are a number of these available today, and they are getting progressively cheaper. You can base this project on any.NET Micro Framework device that has a network port and three or more output ports. The ones I'd recommend are:

• Digi Connect-ME: http://www.digi.com/products/embeddedsolutions/digiconnectme.jsp
• GHI Electronics Embedded Master: http://www.ghielectronics.com/embeddedmaster.php
• Devices Solutions Tahoe II: http://devicesolutions.net/Products/Tahoe.aspx

I built the project for Digi Connect board but the code can be customised for any of the above platforms. The beauty of the .NET Micro Framework is that you hardly need to change your program even if the underlying hardware is completely replaced. The only thing you will have to do is adjust the code to target different output pins. I'll flag this part of the program up when we get to it.

The project as supplied runs on a special emulator that runs on the PC and behaves like a .NET Micro Framework device with lights connected, so you can get started exploring the code right away.

Lights

I'm keeping things deliberately low voltage for this project. This means that kids of all ages can have a go at building the hardware without messing around with mains. The lights that I used were supplied as lines of twenty leds wired up as four strands of five leds each. All the leds in each strand were the same colour. The lights were fitted with a little battery box which held three AA batteries and a tiny controller. To get my display I simply removed the battery box and connected the strands to my hardware.

 
Figure 1: My Battery Powered Lights

You can do the same with any low power lighting kit that you wish to use. The drivers that I'm using can handle around 500 milliamps of continuous current and so you can connect a fair number of lights to each strand.

Buying the Lights

The lights I used can be bought in the UK from Lights4Fun: www.lights4fun.co.uk and are called "C-LED-4.5-M 20 Multi Coloured Battery Operated LED Fairy Lights". They are supplied with a battery box and controller that you can remove to connect to the Darlington drivers. I used an old 5 volt mobile phone charger to power the lights. If you search eBay for "led christmas lights battery" you should find plenty of suppliers.

Driver Hardware

We can't connect a .NET Micro Framework device directly to our lights. There are two reasons for this:

1. The Micro Framework device will not be able to switch the amount of current that we need to drive the lights themselves.
2. The Micro Framework device will not have enough outputs to control all the lights that we want to use.

This means that we need to make some hardware that goes between the processor and our lights. I actually don't see this as a problem. Making hardware is great fun, seeing something that you've built spring into life is very nice. The hardware we are going to use will provide a means by which we can control many hundreds of lights from a single Micro Framework board if we wish. It also has the ability to switch reasonable amounts of current, so you can use quite large numbers of lights if you wish. We are going to use two semiconductor components, a CD4094 Shift Register/Latch and a ULN2803 Octal Darlington Driver. A pair of these chips will allow us to control 8 outputs. If you want to control more outputs you simply get more chips and chain them together. For my version of the project I just used one of each chip, you can use as many as you like.The shift registers and Darlington amplifiers can be obtained in the UK from Maplin: www.maplin.co.uk. The chip numbers and part numbers as are as follows: QW54J 4094 Shift Register, QY79L ULN2803A Darlington Driver. In the US you can obtain the components and breadboard from Digi-Key: www.digikey.com.

Serial and Parallel Data

You might be wondering how we can use just three output lines to control lots of lights. We are going to do this by using the three output lines to provide a serial data stream which is converted by our hardware into parallel data that can be used to control our lights. This is a fundamental principle of digital electronics and is how, amongst many other things, computer networks transfer data.

We are going to use three signals which are called clock, data and latch. Each of these can be set high (a voltage is present) or low (no voltage is present) by the .NET Micro Framework device under the control of our software. The signals are connected to the clock, data and latch inputs of our CD4094 shift register so that the program can talk to it.

The clock line triggers the shift register to do two things:

1. Shift all the bits along to make room.
2. Sample the value of the data input and store this value in the space that was created.

To get a feel for how this works, take a look at Figure 2 below. It shows a shift register with a pattern of bits in it. The pattern is 01100001. Note that although this represents a number; it can also be regarded as a pattern of 0s and 1s in the shift register itself. The value 0 means 0 volts and the value 1 means some volts. These are the signals that will be used to control our lights. I'm going to call them 0 and 1 from now on. The Shift Register has Clock, Data and Latch signals connected and they are all set to 1. We can ignore the latch part of the chip for now.

Figure 2: A shift register and latch with some data in it

When the clock signal changes from 1 to 0 this causes the shift register to perform the two steps described above. First the data is shifted along to the right. Note that this means that there is an "empty" location at the start of the register, and that the right most bit in the register "falls off" the register and disappears. Figure 3 shows how this works.

Figure 3: Shifting along the values in the shift register

Once the shift has finished the shift register can copy the data signal into the empty bit at the left as shown in Figure 4 below.

Figure 4: Storing the new data bit

By repeating this process 8 times a program can load a new pattern of 8 bits into the shift register. Then it is time to latch the new value to control the lights. This is the point at which the lights will appear to change. When the latch value is changed from 0 to 1 this causes the chip to copy the value in the Shift Register into the latch, as shown in Figure 5 below.

Figure 5: Copying the shift register pattern into the latch.

The latch is required so that the lights will not flicker as the new patterns are shifted into position. Each of the bits in the latch is connected to an output pin on the CD4094 which is used to switch a particular light colour on or off.

We need to create some C# that will provide the appropriate sequence of signals. It turns out that using the .NET Micro Framework to achieve this is actually very easy. The displayByte method below sends an 8 bit value into a shift register and then latches it into the output. If you read through the code you can see how the clock, data and latch values are all set to true (high) or false (low) to first clock the data out and then trigger the latch to display the pattern on the lights. The input is an 8 bit byte value and the program uses a mask to pick out the value of each bit in turn and set the data output accordingly.

private static void displayByte(byte value)
{
    latchPort.Write(false);
    clockPort.Write(false);

    byte mask = 1;

    for (int i = 0; i < 8; i++)
    {
        if ((value & mask) > 0)
        {
            dataPort.Write(true);
        }
        else
        {
            dataPort.Write(false);
        }
        clockPort.Write(true);
        clockPort.Write(false);
        mask <<= 1;
    }
    latchPort.Write(true);
    latchPort.Write(false);
}

The dataPort, clockPort and latchPort variables are instances of the .NET Micro Framework class OutputPort, which provides a method called Write which can be used to control the state of the output signal. We will consider how these are created a little later in the article.

The sample code for this project comes with a software emulator of the CD4094 which shows how it works. Figure 6 below shows that a new pattern is in the process of being shifted into the shift register, while the lights retain the previous one in the latch. The Clock and Data signals are high and the next statement will drop the Clock signal to add the next bit into the new pattern.

Figure 6: The light emulator

This figure also shows that I have two bits in the pattern controlling a strand of each colour. You can single step through the displayByte method above and watch it perform this output.

You can use this technique every time you want to control a large number of outputs using a small number of output pins. The CD4094 has "daisy chain" inputs and outputs so that the devices can be connected in sequence. If I used two devices I could control 16 bits, with three I could control 24 and so on. This would require only minimal changes to the software.

Output Driver

The CD4094 device will produce a signal output, but it is not really powerful enough to drive things like lights. To do this we need an amplifier and the ULN2803 Octal Darlington Driver is perfect for this. It is packaged as a single chip which contains 8 pairs of transistors. Each transistor pair is wired in a "Darlington" configuration and can be used as a switch which is controlled by on output from the CD4094. When the transistors are turned on they allow current to pass through them and this will cause the lamps to light. The lights that I bought used a "pull down" arrangement to make them light up. All of the light emitting diode (LED) lamps had one end wired to a common line that was connected to the positive supply. To make the a chain of LEDs light the other end that controls that chain needed to be pulled down to the ground level. This is a common arrangement with lights like these. The ULN2803 driver has the transistors wired in an arrangement that allows it to pull signals low in this way. Figure 7 shows how this arrangement works. The resistor shown is actually wired into each LED in the set of lights that I used.

Figure 7: Lighting the LEDs

Complete Circuit

The complete circuit shown in Figure 8 simply links each output of the CD4094 shift register to an input on the ULN2803 Darlington Driver. Note that there is no reason to link any particular bit with any other, the diagram shown is one which will translate most easily to a prototype "breadboard".

Figure 8: The Complete Circuit

The diagram just shows four lights connected to OUT1 from pin 18 of the ULN2803, the other 7 channels are connected in exactly the same way. Some of the pins can be left unconnected. Pins 9 and 10 of the CD4094 are only required if you are connecting multiple shift registers together and the common connection on pin 10 of the ULN2803 is not required. The Latch, Data and Clock signals on the CD4094 are connected to the output signals from the Micro Framework device.

Figure 9 shows the completed circuit built up on a prototype breadboard. The chip on the left is the CD4094 and the one on the right is the ULN2803.

Figure 9: The Completed Circuit

The red wires carry the positive voltage from the power supply, which is connected to the top left of the circuit. The green wires are ground. Other coloured wires are used for signals. The Clock, Latch and Data signals are brought out to a connector which will fit a Digi-ME prototyping board.

Selecting Lights

Each bit in the value sent to the displayByte method and then into the shift register will be mapped to a particular chain of lights. The mapping of these is not particularly important, since you can use program constants to represent particular values. I wired the red signals to output pins 4 and 8. To light up just the red lights I used the value 0x88 which is the appropriate bit pattern. I then set up constants for all the other colors:

const byte GREEN = 0x11;
const byte BLUE = 0x22;
const byte YELLOW = 0x44;
const byte RED = 0x88;

Setting Up the Hardware

The .NET Micro Framework provides a set of classes that can be used to represent the hardware in a system. The program uses instances of the OutputPort class to represent the output pins. These are created in the method that sets up the hardware:

const Cpu.Pin clockPin = Cpu.Pin.GPIO_Pin0;
const Cpu.Pin dataPin = Cpu.Pin.GPIO_Pin1;
const Cpu.Pin latchPin = Cpu.Pin.GPIO_Pin2;

static OutputPort clockPort;
static OutputPort dataPort;
static OutputPort latchPort;

private static void setupOutputs()
{
    clockPort = new OutputPort(clockPin, false);
    dataPort = new OutputPort(dataPin, false);
    latchPort = new OutputPort(latchPin, false);
}

In this version of the hardware I have connected pin 0 of the processor to the clock, pin 1 to the data and pin 2 to the latch. If you use different pins you can change the settings above.

Software

Now that we have working hardware we can consider how the software is to work. Note that this version of the program works correctly but lacks exception handlers that would make it truly robust. I've left these out to simplify the explanation. To make sense of this description you will need to have a copy of the program itself available for reference.

You can develop the code using Visual Studio 2008 Express edition, which you can download from http://www.microsoft.com/express/

The software is written for the .NET Micro Framework 3.0 which you can download from http://www.microsoft.com/downloads/details.aspx?FamilyID=9356ed6f-f1f0-43ef-b21a-4644dd089b4a&displaylang=en

Reading the Blog

The program reads the RSS feed from a blog and looks for the <pubDate> value. This contains the date when the feed was last updated. Whenever this date changes the lights must flash red for a few seconds before resuming a random display. Users of the full .NET Framework can use the HTTPRequest class to build a GET command to be sent to a server. Unfortunately the .NET Micro Framework does not support this, so we have to access the web feed using socket based communication. This part of the program is heavily based on the SocketClient example supplied with the .NET Micro Framework.

// This method requests a page from the specified server.
private static String GetWebPage(String server, string webPage)
{
    const Int32 c_httpPort = 80;
    const Int32 c_microsecondsPerSecond = 1000000;

    // Create a socket connection to the specified server and port.
    using (Socket serverSocket = ConnectSocket(server, c_httpPort))
    {
        // Send request to the server.
        String request = "GET "+ webPage + 
             " HTTP/1.1\r\nHost: " + server + 
             "\r\nConnection: Close\r\n\r\n";
        Byte[] bytesToSend = Encoding.UTF8.GetBytes(request);
        serverSocket.Send(bytesToSend, bytesToSend.Length, 0);

        // Allocate a buffer to receive HTML chunks
        Byte[] buffer = new Byte[1024];

        // 'page' refers to the HTML data as it is built up.
        String page = String.Empty; 

        // Wait up to 30 seconds for initial data 
        // Will throw exception if connection closed
        DateTime timeoutAt = DateTime.Now.AddSeconds(30);
        while (serverSocket.Available == 0 && 
               DateTime.Now < timeoutAt)
        {
            System.Threading.Thread.Sleep(100);
        }

        // Poll for data until 30 second time out
        // Returns true for data and connection closed
        while (serverSocket.Poll(30 * c_microsecondsPerSecond,
                                 SelectMode.SelectRead))
        {
            // Zero all bytes in the re-usable buffer
            Array.Clear(buffer, 0, buffer.Length);

            // Read a buffer-sized HTML chunk
            Int32 bytesRead = serverSocket.Receive(buffer);

            // If 0 bytes in buffer, then connection is closed, 
            // or we have timed out
            if (bytesRead == 0)
                break;

            // Append the chunk to the string
            page += new String(Encoding.UTF8.GetChars(buffer));
        }

        return page;   // Return the complete string
    }
}

This method is called to fetch the journal RSS feed from my blog:

string address = "www.robmiles.com";
string name = "/journal/rss.xml";

string html = GetWebPage(address, name);

You can use it to download from any RSS feed or page on the web. It will throw an exception if the page cannot be read. This version of my program does not perform exception handling however.

Getting the Published Date

The date is held in the form:

<pubDate>Wed, 05 Nov 2008 22:38:52 +0000</pubDate>

It would be enough just to record this content and check for changes in the text, but I decided that I might want to use the date information in a later version of the program and so I created some small helper methods to read numbers from the input string and a larger method to read the publish date itself:

private static DateTime getRecentPubDate(string html, string startTag)
{
    int index = html.IndexOf(startTag) ;

    if (index < 0) 
        throw new Exception("Missing tag " + startTag);

    index += startTag.Length;

    // spin past the name of the day
    while (index < html.Length && html[index] != ',') index++;
    if (index == html.Length) 
        throw new Exception("Short publish date");

    int dayValue = getInt(html, ref index, ' ');

    string monthName = getString(html, ref index, ' ');
    int monthValue = getMonth(monthName);

    int yearValue = getInt(html, ref index, ' ');
    int hourValue = getInt(html, ref index, ':');
    int minuteValue = getInt(html, ref index, ':');
    int secondValue = getInt(html, ref index, ' ');

    return new DateTime(yearValue, monthValue, dayValue, 
                        hourValue, minuteValue, secondValue);
}

The main body of the program uses this method to extract the date out of the RSS feed. When a new date is found it is time to update the lights.

Lights and Threads

My first version of the program flashed the lights for a while and then checked to see if a new blog post had been made. This worked OK, but the process of loading the RSS feed from the server and checking the dates can take a few seconds, which meant that the flashing lights would freeze every now and then. This did not look very good, and so I decided to use two threads instead. One is in charge of flashing the lights in a random pattern and the other loads the RSS feed from my blog and checks the date of the most recent publication.

Note that the threading I am using is exactly the same as threading in the full .NET Framework.

Thread Communication

The two threads communicate by means of a single boolean variable which is set to true when the blog alert is to take place. The display thread reads this flag and flashes the lights red if it is time to alert.

public static bool alert = false;

public static void flasher()
{
    setupOutputs();

    while (true)
    {
        randomDisplay(400, 10);
        if (alert)
        {
            alert = false;
            flashRed(600, 20);
        }
    }
}

The methods randomDisplay and flashRed do exactly what you would expect. Each of them gets two numbers to control the lights. The first number gives the delay in milliseconds between flashes (a few hundred milliseconds giving best results). The second gives the number of times that the lights should be flashed before the method finishes. RandomDisplay displays random colors, whereas flashRed is simply flashes the red lights. The alert flag is cleared before the red lights are flashed so that the system will resume normal display after the alert.

The second thread in the system performs the blog download and date test behaviour. This all happens inside the Main method:

public static void Main()
{
    flashThread = new System.Threading.Thread(flasher);

    flashThread.Start();

    string address = "www.robmiles.com";
    string name = "/journal/rss.xml";

    String html = GetWebPage(address, name);

    DateTime lastUpdate = getRecentPubDate(html, "<pubDate>");

    Debug.Print("Initial Update Value : " + lastUpdate.ToString());

    while (true)
    {
        System.Threading.Thread.Sleep(10000);

        html = GetWebPage(address, name);

        DateTime blogUpdate = getRecentPubDate(html, "<pubDate>");

        if (!blogUpdate.Equals(lastUpdate))
        {
            Debug.Print("Updated at : " + blogUpdate.ToString());
            lastUpdate = blogUpdate;
            alert = true;
        }
    }
}

To reduce the load on the network the system only checks the blog feed every 10 seconds.

Future Work

The program itself works fine, but the error handling is not wonderful. Although some of the methods used throw exceptions these are not caught anywhere, causing the blog reading thread to terminate. This does not stop the lights flashing, but it does mean that there will be no more alerts. However, it is not too hard to create a version which contains proper error handling and even flashes the lights different colors to indicate the alarm conditions. I have created a version that flashes the lights yellow every now and then if the network connection fails. There is also considerable scope for reading other web based sources and changing the output accordingly. Feel free to do all these things and make sure that you have fun.

Thanks

Thanks go to Ian Mitchell of Ormston Technology (http://www.ormtec.co.uk/) for doing such a great job of hardware design and build.

Introduction

Virtual Earth is the 3D interface to Microsoft's Live Maps service.  Normally this control is loaded via the web browser and allows interaction with a keyboard, mouse, and Xbox 360 controller.  In this article, we will take the Virtual Earth 3D control out of the web browser, use it in a WinForms application, and control it with a Nintendo Wii Remote (Wiimote) and a pair of Vuzix VR920 glasses, while also providing a stereoscopic 3D image to the glasses, creating the illusion of a fully three dimensional environment.  Note that use of the Virtual Earth 3D control in this way is undocumented and unsupported at the moment.  Because of this, some of the descriptions in this article are educated guesses and may not be 100% accurate…

Originally, this project started as a simple Wiimote interface to Virtual Earth 3D as shown in the video below.  Since I wrote that application, I learned of the VR920 glasses and the Wii Fit Balance Board was released, so I've decided to create a more immersive experience using all of these controls which was demonstrated at PDC2008, shown here:

Setup

Before we get started, you will need to install the Virtual Earth 3D control.  If you haven't done this already, browse to http://maps.live.com/ and click on the 3D link to install the control and supporting software.

Additionally, if you haven't already, please review my Managed Library for Nintendo's Wiimote article on this site.  We will be using the library in this article, but I will not repeat the basic information that is located in the original article.  You will also need to have the Vuzix VR920 glasses installed and setup according to its own user manual.  That will also not be covered here.

Implementation

The Virtual Earth 3D Control

The Virtual Earth 3D (VE3D) control is intended to be used through a well documented JavaScript interface from a web page, however we would not be able to access the Wiimote or VR920 glasses from JavaScript.  Therefore, we will be using the VE3D control through its native, but wholly undocumented interface.

Start by creating a new Windows Forms application named WiiEarthVR in C# or VB.  As with all controls and 3rd party libraries, a reference needs to be set to the Virtual Earth 3D libraries.  Add references to the following items: 

• Microsoft.MapPoint.Data
• Microsoft.MapPoint.Data.VirtualEarthTileDataSource
• Microsoft.MapPoint.Geometry
• Microsoft.MapPoint.Rendering3D
• Microsoft.MapPoint.Rendering3D.Utility

If they do not show up in the .NET references, they can be found by selecting the Browse tab and navigating to C:\Program Files\Virtual Earth 3D\ or C:\Program Files (x86)\Virtual Earth 3D\ .  With the references in place, the project file can now be opened and the references will be seen in the References folder in the Solution Explorer as usual.

Creating an instance of the control can be done in code just like any other control.  Used in the constructor or load event of the form, the following code will create a VE3D control and add it to the form as fully docked:

C#

private GlobeControl _globeControl;

public MainForm()

{

    InitializeComponent();

    _globeControl = new GlobeControl();

    SuspendLayout();

    _globeControl.Location = new System.Drawing.Point(0, 0);

    _globeControl.Name = "_globeControl";

    _globeControl.Size = ClientSize;

    _globeControl.Anchor = AnchorStyles.Left | AnchorStyles.Right | AnchorStyles.Top | AnchorStyles.Bottom;

    _globeControl.TabIndex = 0;

    _globeControl.SendToBack(); // we want the button to be on top

    pnlGlobe.Controls.Add(_globeControl);

    ResumeLayout(false);

}

VB

Private _globeControl As GlobeControl

Public Sub New()

    InitializeComponent()

    _globeControl = New GlobeControl()

    SuspendLayout()

    _globeControl.Location = New System.Drawing.Point(0, 0)

    _globeControl.Name = "_globeControl"

    _globeControl.Size = ClientSize

    _globeControl.Anchor = AnchorStyles.Left Or AnchorStyles.Right Or AnchorStyles.Top Or AnchorStyles.Bottom

    _globeControl.TabIndex = 0

    _globeControl.SendToBack() ' we want the button to be on top

    pnlGlobe.Controls.Add(_globeControl)

    ResumeLayout(False)

End Sub

This sets up the VE3D control in its default state.  If you were to run an application with only this code, you would see nothing but the earth.  The navigation controls and other extras would be missing.

We can start adding items to the VE3D control by listening for the FirstFrameRendered event of the GlobeControl and then setting the appropriate properties.  Setting these properties prior to this point can lead to some unexpected results.

In the FirstFrameRendered event handler, if you wish to add the default navigation controls to the screen, the PlugInLoader object is used.  The PlugInLoader is created by using the CreateLoader static method, passing in an instance of the GlobeControl's Host object.  Then, the NavigationPlugIn can be loaded and activated as shown:

C#

// load all the spiffy UI navigation goodies

PlugInLoader loader = PlugInLoader.CreateLoader(this.globeControl.Host);

loader.LoadPlugIn(typeof(NavigationPlugIn));

loader.ActivatePlugIn(typeof(NavigationPlugIn).GUID, null);

VB

' load all the spiffy UI navigation goodies

Dim loader As PlugInLoader = PlugInLoader.CreateLoader(Me.globeControl.Host)

loader.LoadPlugIn(GetType(NavigationPlugIn))

loader.ActivatePlugIn(GetType(NavigationPlugIn).GUID, Nothing)

The last thing to be added for basic functionality is the data.  As it stands, the only data that will appear on the globe is the image of the continents.  Zooming in only produces a blurry representation of that base image.

Data layers are created from specially formatted data sources provided by maps.live.com known as content manifests.  These are XML files which tell the VE3D control how to load the data required for any view.  Content layers can be added by adding them to the DataSources object of the GlobeControl.  We can add any of the following layers (note that there may be other content manifests provided by maps.live.com, but these are the only 5 that I am aware of):

For the best display, add the ElevationMap, Model and Aerial TextureMap layers as shown:

C#

// set various data sources, here for elevation data, terrain data, and model data.

_globeControl.Host.DataSources.Add(new DataSourceLayerData("Elevation", "Elevation", @"http://maps.live.com//Manifests/HD.xml", DataSourceUsage.ElevationMap));

_globeControl.Host.DataSources.Add(new DataSourceLayerData("Texture", "Texture", @"http://maps.live.com//Manifests/AT.xml", DataSourceUsage.TextureMap));

_globeControl.Host.DataSources.Add(new DataSourceLayerData("Models", "Models", @"http://maps.live.com//Manifests/MO.xml", DataSourceUsage.Model));

VB

' set various data sources, here for elevation data, terrain data, and model data.

_globeControl.Host.DataSources.Add(New DataSourceLayerData("Elevation", "Elevation", "http://maps.live.com//Manifests/HD.xml", DataSourceUsage.ElevationMap))

_globeControl.Host.DataSources.Add(New DataSourceLayerData("Texture", "Texture", "http://maps.live.com//Manifests/AT.xml", DataSourceUsage.TextureMap))

_globeControl.Host.DataSources.Add(New DataSourceLayerData("Models", "Models", "http://maps.live.com//Manifests/MO.xml", DataSourceUsage.Model))

By passing the URL of the content manifest, a name for the layer, and what the manifest represents, a new DataSource is created, which is in turn used to create a DataSourceLayerData object which is then given to the VE3D control to consume.  This should be done in the FirstFrameRendered event handler as well.

We also need to setup VE3D to turn off any UI elements, turn on the atmosphere effects, and ensure we have a full unobstructed view. Again, in the FirstFrameRendered event handler, we can use the following code to achieve this:

C#

// turn on the nice atmosphere

_globeControl.Host.WorldEngine.Environment.AtmosphereDisplay = Microsoft.MapPoint.Rendering3D.Atmospherics.EnvironmentManager.AtmosphereStyle.Scattering;

// default to all off

_globeControl.Host.WorldEngine.Display3DCursor = false;

_globeControl.Host.WorldEngine.SetWindowsCursor(null);

_globeControl.Host.WorldEngine.ShowNavigationControl = false;

_globeControl.Host.WorldEngine.ShowCursorLocationInformation = false;

_globeControl.Host.WorldEngine.ShowScale = false;

_globeControl.Host.WorldEngine.ShowUI = false;

_globeControl.Host.WorldEngine.Environment.SunPosition = null;

_globeControl.Host.WorldEngine.Environment.LocalWeatherEnabled = true;

_globeControl.Host.WorldEngine.BaseCopyrightText = " "; // workaround for a display issue

VB

' turn on the nice atmosphere

_globeControl.Host.WorldEngine.Environment.AtmosphereDisplay = Microsoft.MapPoint.Rendering3D.Atmospherics.EnvironmentManager.AtmosphereStyle.Scattering

' default to all off

_globeControl.Host.WorldEngine.Display3DCursor = False

_globeControl.Host.WorldEngine.SetWindowsCursor(Nothing)

_globeControl.Host.WorldEngine.ShowNavigationControl = False

_globeControl.Host.WorldEngine.ShowCursorLocationInformation = False

_globeControl.Host.WorldEngine.ShowScale = False

_globeControl.Host.WorldEngine.ShowUI = False

_globeControl.Host.WorldEngine.Environment.SunPosition = Nothing

_globeControl.Host.WorldEngine.Environment.LocalWeatherEnabled = True

_globeControl.Host.WorldEngine.BaseCopyrightText = " " ' workaround for a display issue

If you were to run the application at this point, you would see a fully functioning Virtual Earth 3D control with proper data and navigation.

Control Scheme and Bindings

The user will control VE3D with the Wiimote by holding the nunchuk in the left hand, which will move the user forward/back/left/right using the joystick.  The C and Z buttons on the front of the nunchuk will be used to raise and lower the altitude of the camera.  The Wiimote, held in the right hand, will be used to toggle various things on or off and interact with menus.  The user will also stand on the Balance Board which will use their center of gravity to turn them in the environment.

VE3D bindings allow you to change or create new control schemes for VE3D by placing an XML file in a specific directory as follows:

• Vista: C:\Users\<username>\AppData\LocalLow\Microsoft\Virtual Earth 3D
• XP: C:\Documents and Settings\<username>\Local Settings\Microsoft\Virtual Earth 3D

In this directory you will find a Bindings.xml file.  This XML schema defines the default keyboard, mouse, Gamepad and other input device properties.  Open the file to see the schema used to define events and parameters.

By default, VE3D will load any file named Bindings*.xml from this directory.  For the Wiimote control scheme, create a new file named BindingsWiiEarthVR.xml in this directory.  Set the contents of the file to the following:

<?xml version="1.0" encoding="utf-8" ?>

<Bindings>

    <BindingSet Name="WiiEarthVRBindings" AutoUse="True" Cursor="Drag">

        <!-- Nunchuk joystick -->

        <Bind Event="Wiimote.NunchukX" Factor="0.5"><Action Name="Strafe"/></Bind>

        <Bind Event="Wiimote.NunchukY" Factor="1"><Action Name="Move"/></Bind>

        <!-- Nunchuk buttons -->

        <Bind Event="Wiimote.NunchukC" Factor="0.20"><Action Name="Ascend"/></Bind>

        <Bind Event="Wiimote.NunchukZ" Factor="-0.20"><Action Name="Ascend"/></Bind>

        <!-- Balance Board -->

        <Bind Event="Wiimote.BalanceBoardX" Factor="-0.0009"><Action Name="Turn"/></Bind>

        <Bind Event="Wiimote.BalanceBoardY" Factor="0.0009"><Action Name="Ascend"/></Bind>

        <!-- Wiimote buttons -->

        <Bind Event="Wiimote.Home"><Action Name="ResetOnCenter"/></Bind>

        <Bind Event="Wiimote.A" Factor="1"><Action Name="Locations, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Left" Factor="-1"><Action Name="Locations, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Up" Factor="-1"><Action Name="LocationsMove, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Down" Factor="1"><Action Name="LocationsMove, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <!-- FPS-style keyboard controls in case we don't have a nunchuk -->

        <Bind Event="Key.W" Factor="22"><Action Name="Move" /></Bind>

        <Bind Event="Key.S" Factor="-22"><Action Name="Move" /></Bind>

        <Bind Event="Key.D" Factor="22"><Action Name="Strafe" /></Bind>

        <Bind Event="Key.A" Factor="-22"><Action Name="Strafe" /></Bind>

        <Bind Event="Key.Space" Factor="20"><Action Name="Ascend" /></Bind>

        <Bind Event="Key.C" Factor="-20"><Action Name="Ascend" /></Bind>

        <!-- Other keys -->

        <Bind Event="Key.F1"><Action Name="VR920SetZero, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Key.F2"><Action Name="BalanceBoardSetZero, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Key.F3"><Action Name="ToggleVR920Stereo, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Minus" Factor="-0.1"><Action Name="VR920SetEyeDistance, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Plus" Factor="0.1"><Action Name="VR920SetEyeDistance, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.One"><Action Name="ToggleBalanceBoard, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Wiimote.Two"><Action Name="ToggleVR920, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Key.B"><Action Name="ToggleBalanceBoard, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Key.V"><Action Name="ToggleVR920, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

        <Bind Event="Key.F"><Action Name="FullScreen, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/></Bind>

    </BindingSet>

</Bindings>

The <BindingSet> tags wrap groups of control bindings.  It requires a Name and optionally a Cursor.  If the binding set is to be used automatically, as it would be in most cases, set the AutoUse parameter to True.  Inside of that are <Bind> tags.  The tag requires the Event parameter and optionally the Factor parameter.  The Event parameter will be used to match the binding to its handler which will be written later.  The syntax is <Handler Name>.<Event Name>.  The Factor parameter is optional and can be used to scale the data value up or down to increase or decrease sensitivity of the input method.  The Action tag inside the Bind tag is used to map the specific binding to a particular method.  Once the handler is written, these will make more sense.

The bindings above create the control scheme described above:  NunchukX/Y describe what happens when the analog joystick is moved, NunchukC/Z describe what happens with the C/Z buttons are pressed, and so on.

The bindings also allow for several variations.  Bindings are defined for both the IR position (IRX, IRY) and accelerometer values (AX, AY).  If an IR sensor bar is not available, the accelerometer values of the Wiimote can be used instead.  Additionally, keyboard bindings are created in the style of a first person shooter using WASD.  These can be used if a Nunchuk is not available.

Note that some bindings append two Events together with a + sign.  This allows for button combinations.  In this case, for the accelerometer and/or IR sensor, we only want to register the action if a button is pressed down.  So, those events which require the button to be held down contain Wiimote.B+ and the event it is combined with.

For those events which require a custom action that will be written separately and not part of the VE3D control, the Action parameter must contain the action name, followed by a comma, and then the full assembly name:

<Bind Event="Wiimote.One">
    <Action Name="ToggleBalanceBoard, WiiEarthVR, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/>
</Bind>

Event Source

An EventSource is needed which will grab data from the Wiimote and pass it along to VE3D as defined by the bindings file above.  Create a new class named WiimoteEventSource which derives from Microsoft.MapPoint.Binding.EventSource  as follows:

Next, add an enumeration named WiimoteEvent (the name isn't important) which contains all of the Name items from the bindings XML file above.  It should look like this:

C#

// all events handled by this event source from XML file
public enum WiimoteEvent
{
    IRX,        // IR X position
    IRY,        // IR Y position
    NunchukX,    // Nunchuk joystick X position
    NunchukY,    // Nunchuk joystick Y position
    NunchukC,    // Nunchuk C button
    NunchukZ,    // Nunchuk Z button
    AX,            // Wiimote accelerometer X
    AY,            // Wiimote accelerometer Y
    Up,            // Dpad up
    Down,        // Dpad down
    Left,        // Dpad left
    Right,        // Dpad right
    A,            // A button
    B,            // B button
    Minus,        // Minus button
    Home,        // Wiimote Home button
    Plus,        // Plus button
    One,        // Wiimote One button
    Two,        // Wiimote Two button
    BalanceBoardX,    // Balance Board COG X
    BalanceBoardY    // Balance Board COG Y
}

VB

' all events handled by this event source from XML file
Public Enum WiimoteEvent
    IRX ' IR X position
    IRY ' IR Y position
    NunchukX ' Nunchuk joystick X position
    NunchukY ' Nunchuk joystick Y position
    NunchukC ' Nunchuk C button
    NunchukZ ' Nunchuk Z button
    AX ' Wiimote accelerometer X
    AY ' Wiimote accelerometer Y
    Up ' Dpad up
    Down ' Dpad down
    Left ' Dpad left
    Right ' Dpad right
    A ' A button
    B ' B button
    Minus ' Minus button
    Home ' Wiimote Home button
    Plus ' Plus button
    One ' Wiimote One button
    Two ' Wiimote Two button
    BalanceBoardX ' Balance Board COG X
    BalanceBoardY ' Balance Board COG Y
End Enum

Next, several methods from the EventSource object need to be overridden:  GetEventData, IsModifier, CanModify, TryGetEventId, TryGetEventName, Name.  The methods do the following:

The code for these methods is presented below:

C#

// return out value of the passed enum

public override bool TryGetEventId(string eventName, out int eventId)

{

    eventId = (int)Enum.Parse(typeof(WiimoteEvent), eventName);

    return true;

}

// return out the string name of the passed in enum value

public override bool TryGetEventName(int eventId, out string eventName)

{

    eventName = ((WiimoteEvent)eventId).ToString();

    return true;

}

// unknown

public override EventData GetEventData(int eventId, EventActivateState state)

{

    throw new NotImplementedException();

}

// can the event be used as a modifier?

public override bool IsModifier(int eventId)

{

    // yes to all for now

    return true;

}

// can the supplied event be used as a modifier?

public override bool CanModify(int eventId, EventKey other)

{

    // only if it's from us

    return (other.Source == this);

}

// this must match the Source name in the bindings XML file

public override string Name

{

    get { return "Wiimote"; }

}

VB

' return out value of the passed enum

Public Overrides Function TryGetEventId(ByVal eventName As String, <System.Runtime.InteropServices.Out()> ByRef eventId As Integer) As Boolean

    eventId = CInt(Fix(System.Enum.Parse(GetType(WiimoteEvent), eventName)))

    Return True

End Function

' return out the string name of the passed in enum value

Public Overrides Function TryGetEventName(ByVal eventId As Integer, <System.Runtime.InteropServices.Out()> ByRef eventName As String) As Boolean

    eventName = (CType(eventId, WiimoteEvent)).ToString()

    Return True

End Function

' unknown

Public Overrides Function GetEventData(ByVal eventId As Integer, ByVal state As EventActivateState) As EventData

    Throw New NotImplementedException()

End Function

' can the event be used as a modifier?

Public Overrides Function IsModifier(ByVal eventId As Integer) As Boolean

    ' yes to all for now

    Return True

End Function

' can the supplied event be used as a modifier?

Public Overrides Function CanModify(ByVal eventId As Integer, ByVal other As EventKey) As Boolean

    ' only if it's from us

    Return (other.Source Is Me)

End Function

' this must match the Source name in the bindings XML file

Public Overrides ReadOnly Property Name() As String

    Get

        Return "Wiimote"

    End Get

End Property

With that in place, the constructor can be implemented which will call the base constructor and connect to the Wiimote.  It is assumed you read the Wiimote article above and know how the library works.

The constructor must take one argument passed from the main from:  an instance of the GlobeControl's ActionSystem.  This just gets passed directly to the parent object's constructor untouched.  The constructor code looks like the following:

C#

public WiimoteEventSource(ActionSystem actionSystem) : base(actionSystem)
{
    // get all connected Wiimotes
    WiimoteCollection wc = new WiimoteCollection();
    wc.FindAllWiimotes();

    // setup wiimotes and event handlers
    foreach(Wiimote wm in wc)
    {
        wm.WiimoteChanged += new EventHandler<WiimoteChangedEventArgs>(OnWiimoteChanged);
        wm.WiimoteExtensionChanged += new EventHandler<WiimoteExtensionChangedEventArgs>(OnWiimoteExtensionChanged);
        wm.Connect();

        // if we don't have an extension, set the report type to IR and accel's only
        if(!wm.WiimoteState.Extension && wm.WiimoteState.ExtensionType != ExtensionType.BalanceBoard)
            wm.SetReportType(InputReport.IRAccel, true);

        if(wm.WiimoteState.ExtensionType == ExtensionType.BalanceBoard)
            _bb = wm;
        else
            _wm = wm;

        // turn off all LEDs
        wm.SetLEDs(0x00);
    }
}

VB

Public Sub New(ByVal actionSystem As ActionSystem)
    MyBase.New(actionSystem)

    ' get all connected Wiimotes
    Dim wc As New WiimoteCollection()
    wc.FindAllWiimotes()

    ' setup wiimotes and event handlers
    For Each wm As Wiimote In wc
        AddHandler wm.WiimoteChanged, AddressOf OnWiimoteChanged
        AddHandler wm.WiimoteExtensionChanged, AddressOf OnWiimoteExtensionChanged
        wm.Connect()

        ' if we don't have an extension, set the report type to IR and accel's only
        If (Not wm.WiimoteState.Extension) AndAlso wm.WiimoteState.ExtensionType <> ExtensionType.BalanceBoard Then
            wm.SetReportType(InputReport.IRAccel, True)
        End If

        If wm.WiimoteState.ExtensionType = ExtensionType.BalanceBoard Then
            _bb = wm
        Else
            _wm = wm
        End If

        ' turn off all LEDs
        wm.SetLEDs(&H00)
    Next wm
End Sub

The OnWiimoteExtensionChanged method simply sets the report mode for the Wiimote based on whether or not a Nunchuk is inserted as shown:

C#

private void OnWiimoteExtensionChanged(object sender, WiimoteExtensionChangedEventArgs args)
{
    if(_wm == null)
        return;

    // if nunchuk inserted, set the report type to return extension data
    if(args.ExtensionType == ExtensionType.Nunchuk && args.Inserted)
        _wm.SetReportType(InputReport.IRExtensionAccel, true);
    else // in all other cases, set it to the default IR and accel's
        _wm.SetReportType(InputReport.IRAccel, true);
}

VB

Private Sub OnWiimoteExtensionChanged(ByVal sender As Object, ByVal args As WiimoteExtensionChangedEventArgs)
    If _wm Is Nothing Then
        Return
    End If

    ' if nunchuk inserted, set the report type to return extension data
    If args.ExtensionType = ExtensionType.Nunchuk AndAlso args.Inserted Then
        _wm.SetReportType(InputReport.IRExtensionAccel, True)
    Else ' in all other cases, set it to the default IR and accel's
        _wm.SetReportType(InputReport.IRAccel, True)
    End If
End Sub

The OnWiimoteChanged event handler is where the Wiimote data is handled and sent off to the VE3D control to reflect the changes.  First, we handle the Balance Board.  If the Balance Board is the controller reporting data, we take the center of gravity values and pass them to VE3D.  This code looks at the appropriate values, determines if they are beyond the specified thresholds for the dead zones, and, if they are, activates the event for that value using the Execute method.  Execute is a method in the base EventSource class.  This method will activate the event specified from the enumeration (which, remember, is contained in the bindings XML file) with the value associated with that event.  An EventData object of some type must be created and passed to the Execute method.  There are two EventData types to know about:  AxisEventData and ButtonEventData.  AxisEventData should be used when an event is activated that will modify the map position in some way.  That is, if the map is being turned, elevation is changing, etc.  ButtonEventData should be used if the event is a simple toggle like pressing a button down and releasing it.

The Balance Board code can be seen below:

C#

if(ws.ExtensionType == ExtensionType.BalanceBoard && this.BalanceBoardEnabled)

{

    float x1 = ws.BalanceBoardState.CenterOfGravity.X;

    if(x1 > Properties.Settings.Default.BBDeadX || x1 < -Properties.Settings.Default.BBDeadX)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.BalanceBoardX), x1 - _zero.X));

    float y1 = ws.BalanceBoardState.CenterOfGravity.Y;

    if(y1 > Properties.Settings.Default.BBDeadY || y1 < -Properties.Settings.Default.BBDeadY)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.BalanceBoardY), y1 - _zero.Y));

}

VB

If ws.ExtensionType = ExtensionType.BalanceBoard AndAlso Me.BalanceBoardEnabled Then

    Dim x1 As Single = ws.BalanceBoardState.CenterOfGravity.X

    If x1 > My.Settings.Default.BBDeadX OrElse x1 < -My.Settings.Default.BBDeadX Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.BalanceBoardX))), x1 - _zero.X))

    End If

    Dim y1 As Single = ws.BalanceBoardState.CenterOfGravity.Y

    If y1 > My.Settings.Default.BBDeadY OrElse y1 < -My.Settings.Default.BBDeadY Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.BalanceBoardY))), y1 - _zero.Y))

    End If

Next, let's handle the IR and accelerometer data.  The IR midpoint of the X and Y axes will be used from the WiimoteState object to activate the IRX and IRY events we defined above in the bindings XML file.  The accelerometer X and Y values will be used to activate the AX and AY events.

This snippet assumes that there is a boolean property named UseIR created in the project to determine whether IR or motion values are used.  Additionally, it assumes there are property settings created which contain values for the X/Y "dead zones" for the IR and accelerometers.  These dead zones are used as a way to only activate the event when the values are pushed beyond the thresholds.  This allows there to be a margin where the user's hand will not be read as movement, allowing the user to not have to worry about keeping a steady hand.

The application linked above uses the following default values for dead zones:

• NunchukDeadX/Y -> 0.025
• WiimoteDeadX/Y -> 0.15
• IRDeadX/Y -> 0.1
• BBDeadX/Y –> 3

C#

// if we're using the IR

if(Properties.Settings.Default.UseIR)

{

    // and both LEDs are found

    if(ws.IRState.Found1 && ws.IRState.Found2)

    {

        // normalize the midpoints to -0.5 to 0.5 (from 0 to 1.0)

        float x = ws.IRState.MidX - 0.5f;

        float y = ws.IRState.MidY - 0.5f;

        // if we're beyond the thresholds, activate the events

        if(x > Properties.Settings.Default.IRDeadX || x < -Properties.Settings.Default.IRDeadX)

            this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.IRX), x));

        if(y > Properties.Settings.Default.IRDeadY || y < -Properties.Settings.Default.IRDeadY)

            this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.IRY), y));

        // save the last IR settings...these get used if we go beyond the range of the IRs.

        // in that case, the last used positions will be used until the Wiimote comes back in range

        this._lastIRX = x;

        this._lastIRY = y;

    }

    else // one or both LEDs aren't seen

    {

        // activate events based on the last known positions

        if(this._lastIRX > Properties.Settings.Default.IRDeadX || this._lastIRX < -Properties.Settings.Default.IRDeadX)

            this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.IRX), this._lastIRX));

        if(this._lastIRY > Properties.Settings.Default.IRDeadY || this._lastIRY < -Properties.Settings.Default.IRDeadY)

            this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.IRY), this._lastIRY));

    }

}

else // we're using motion controls

{

    // activate the events based on the accelerometer values

    if(ws.AccelState.X > Properties.Settings.Default.WiimoteDeadX || ws.AccelState.X < -Properties.Settings.Default.WiimoteDeadX)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.AX), ws.AccelState.X));

    if(ws.AccelState.Y > Properties.Settings.Default.WiimoteDeadY || ws.AccelState.Y < -Properties.Settings.Default.WiimoteDeadY)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.AY), ws.AccelState.Y));

}

VB

' if we're using the IR

If My.Settings.Default.UseIR Then

    ' and both LEDs are found

    If ws.IRState.Found1 AndAlso ws.IRState.Found2 Then

        ' normalize the midpoints to -0.5 to 0.5 (from 0 to 1.0)

        Dim x As Single = ws.IRState.MidX - 0.5f

        Dim y As Single = ws.IRState.MidY - 0.5f

        ' if we're beyond the thresholds, activate the events

        If x > My.Settings.Default.IRDeadX OrElse x < -My.Settings.Default.IRDeadX Then

            Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.IRX))), x))

        End If

        If y > My.Settings.Default.IRDeadY OrElse y < -My.Settings.Default.IRDeadY Then

            Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.IRY))), y))

        End If

        ' save the last IR settings...these get used if we go beyond the range of the IRs.

        ' in that case, the last used positions will be used until the Wiimote comes back in range

        Me._lastIRX = x

        Me._lastIRY = y

    Else ' one or both LEDs aren't seen

        ' activate events based on the last known positions

        If Me._lastIRX > My.Settings.Default.IRDeadX OrElse Me._lastIRX < -My.Settings.Default.IRDeadX Then

            Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.IRX))), Me._lastIRX))

        End If

        If Me._lastIRY > My.Settings.Default.IRDeadY OrElse Me._lastIRY < -My.Settings.Default.IRDeadY Then

            Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.IRY))), Me._lastIRY))

        End If

    End If

Else ' we're using motion controls

    ' activate the events based on the accelerometer values

    If ws.AccelState.X > My.Settings.Default.WiimoteDeadX OrElse ws.AccelState.X < -My.Settings.Default.WiimoteDeadX Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.AX))), ws.AccelState.X))

    End If

    If ws.AccelState.Y > My.Settings.Default.WiimoteDeadY OrElse ws.AccelState.Y < -My.Settings.Default.WiimoteDeadY Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.AY))), ws.AccelState.Y))

    End If

End If

Next, the nunchuk values need to be read and the associated events activated.  This is done as follows:

C#

// if the nunchuk is connected

if(ws.Extension && ws.ExtensionType == ExtensionType.Nunchuk)

{

    // activate the nunchuk-based events

    if(ws.NunchukState.X > Properties.Settings.Default.NunchukDeadX || ws.NunchukState.X < -Properties.Settings.Default.NunchukDeadX)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.NunchukX), ws.NunchukState.X));

    if(ws.NunchukState.Y > Properties.Settings.Default.NunchukDeadY || ws.NunchukState.Y < -Properties.Settings.Default.NunchukDeadY)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.NunchukY), ws.NunchukState.Y));

    if(ws.NunchukState.C)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.NunchukC), 1.0f));

    if(ws.NunchukState.Z)

        this.Execute(new AxisEventData(new EventKey(this, (int)WiimoteEvent.NunchukZ), 1.0f));

}

VB

' if the nunchuk is connected

If ws.Extension AndAlso ws.ExtensionType = ExtensionType.Nunchuk Then

    ' activate the nunchuk-based events

    If ws.NunchukState.X > My.Settings.Default.NunchukDeadX OrElse ws.NunchukState.X < -My.Settings.Default.NunchukDeadX Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.NunchukX))), ws.NunchukState.X))

    End If

    If ws.NunchukState.Y > My.Settings.Default.NunchukDeadY OrElse ws.NunchukState.Y < -My.Settings.Default.NunchukDeadY Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.NunchukY))), ws.NunchukState.Y))

    End If

    If ws.NunchukState.C Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.NunchukC))), 1.0f))

    End If

    If ws.NunchukState.Z Then

        Me.Execute(New AxisEventData(New EventKey(Me, CInt(Fix(WiimoteEvent.NunchukZ))), 1.0f))

    End If

End If

Finally, the button events need to be activated.  A helper method which will check the current button state will be used for determining which button of all the Wiimote buttons is pressed.  For those that are, the appropriate event is activated with a call to Execute.

C#

private void HandleButton(WiimoteEvent we, bool buttonState, bool lastButtonState)

{

    if(buttonState == lastButtonState)

        return;

    else

    {

        if(buttonState)

            this.Execute(new ButtonEventData(new EventKey(this, (int)we), EventActivateState.Activate));

        else

            this.Execute(new ButtonEventData(new EventKey(this, (int)we), EventActivateState.Deactivate));

    }

}

// handle all the Wiimote buttons

HandleButton(WiimoteEvent.Up, ws.ButtonState.Up, _lastBS.Up);

HandleButton(WiimoteEvent.Down, ws.ButtonState.Down, _lastBS.Down);

HandleButton(WiimoteEvent.Left, ws.ButtonState.Left, _lastBS.Left);

HandleButton(WiimoteEvent.Right, ws.ButtonState.Right, _lastBS.Right);

HandleButton(WiimoteEvent.A, ws.ButtonState.A, _lastBS.A);

HandleButton(WiimoteEvent.B, ws.ButtonState.B, _lastBS.B);

HandleButton(WiimoteEvent.Minus, ws.ButtonState.Minus, _lastBS.Minus);

HandleButton(WiimoteEvent.Home, ws.ButtonState.Home, _lastBS.Home);

HandleButton(WiimoteEvent.Plus, ws.ButtonState.Plus, _lastBS.Plus);

HandleButton(WiimoteEvent.One, ws.ButtonState.One, _lastBS.One);

HandleButton(WiimoteEvent.Two, ws.ButtonState.Two, _lastBS.Two);

...

// save off the current button state for next time

_lastBS = ws.ButtonState;

_lastNunchuk = ws.NunchukState;

VB

Private Sub HandleButton(ByVal we As WiimoteEvent, ByVal buttonState As Boolean, ByVal lastButtonState As Boolean)

    If buttonState = lastButtonState Then

        Return

    Else

        If buttonState Then

            Me.Execute(New ButtonEventData(New EventKey(Me, CInt(Fix(we))), EventActivateState.Activate))

        Else

            Me.Execute(New ButtonEventData(New EventKey(Me, CInt(Fix(we))), EventActivateState.Deactivate))

        End If

    End If

End Sub

...

' handle all the Wiimote buttons

HandleButton(WiimoteEvent.Up, ws.ButtonState.Up, _lastBS.Up)

HandleButton(WiimoteEvent.Down, ws.ButtonState.Down, _lastBS.Down)

HandleButton(WiimoteEvent.Left, ws.ButtonState.Left, _lastBS.Left)

HandleButton(WiimoteEvent.Right, ws.ButtonState.Right, _lastBS.Right)

HandleButton(WiimoteEvent.A, ws.ButtonState.A, _lastBS.A)

HandleButton(WiimoteEvent.B, ws.ButtonState.B, _lastBS.B)

HandleButton(WiimoteEvent.Minus, ws.ButtonState.Minus, _lastBS.Minus)

HandleButton(WiimoteEvent.Home, ws.ButtonState.Home, _lastBS.Home)

HandleButton(WiimoteEvent.Plus, ws.ButtonState.Plus, _lastBS.Plus)

HandleButton(WiimoteEvent.One, ws.ButtonState.One, _lastBS.One)

HandleButton(WiimoteEvent.Two, ws.ButtonState.Two, _lastBS.Two)

' save off the current button state for next time

_lastBS = ws.ButtonState

_lastNunchuk = ws.NunchukState

And, the current button values are stored away to check on the next event so button events are only fired once.

Now that the event source object is written, it needs to be hooked up to the GlobeControl so it can be used.  This can be done by creating an instance of the WiimoteEventSource object, passing in the VE3D's ActionSystem from the BindingsManager object.  Then, the event source instance is passed to the ActionSystem's EventSourceManager and registered using the RegisterEventSource method.  Event sources should re registered before the control is added to the form.

C#

// wiimote events

private WiimoteEventSource _wiimoteEventSource;

...

// create a new instance of the Wiimote event handler

_wiimoteEventSource = new WiimoteEventSource(this.globeControl.Host.BindingsManager.ActionSystem, this);

// register it in the event source list

_globeControl.Host.BindingsManager.ActionSystem.EventSourceManager.RegisterEventSource(this._wiimoteEventSource);

VB

' wiimote events

Private _wiimoteEventSource As WiimoteEventSource

...

' create a new instance of the Wiimote event handler

_wiimoteEventSource = New WiimoteEventSource(Me.globeControl.Host.BindingsManager.ActionSystem, Me)

' register it in the event source list

_globeControl.Host.BindingsManager.ActionSystem.EventSourceManager.RegisterEventSource(Me._wiimoteEventSource)

Actions and the BindingManager

Our binding list contains several action types that are not defined by the default VE3D actions.  These actions and their handlers must be registered with the VE3D control.  The actions can be registered as follows in the FirstFrameRendered event handler:

C#

BindingsSource bs = new BindingsSource(base.GetType());

_globeControl.Host.BindingsManager.RegisterAction(bs, "Locations", LocationsHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "LocationsMove", LocationsMoveHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleVR920", ToggleVR920Handler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "VR920SetZero", VR920SetZero);

_globeControl.Host.BindingsManager.RegisterAction(bs, "BalanceBoardSetZero", BalanceBoardSetZero);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleBalanceBoard", ToggleBalanceBoardHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "FullScreen", FullScreenHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleVR920Stereo", ToggleVR920Stereo);

_globeControl.Host.BindingsManager.RegisterAction(bs, "VR920SetEyeDistance", VR920SetEyeDistance);

VB

BindingsSource bs = new BindingsSource(base.GetType());

_globeControl.Host.BindingsManager.RegisterAction(bs, "Locations", LocationsHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "LocationsMove", LocationsMoveHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleVR920", ToggleVR920Handler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "VR920SetZero", VR920SetZero);

_globeControl.Host.BindingsManager.RegisterAction(bs, "BalanceBoardSetZero", BalanceBoardSetZero);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleBalanceBoard", ToggleBalanceBoardHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "FullScreen", FullScreenHandler);

_globeControl.Host.BindingsManager.RegisterAction(bs, "ToggleVR920Stereo", ToggleVR920Stereo);

_globeControl.Host.BindingsManager.RegisterAction(bs, "VR920SetEyeDistance", VR920SetEyeDistance);

With the actions registered and handlers associated with them, the actual handlers need to be implemented.  All event handler methods must be of the following signature:

C#

public bool EventHandler(EventData cause)

VB

Public Function EventHandler(ByVal cause As EventData) As Boolean

Let's take a look at the FullScreen binding which simply turns the status bar at the bottom of the window on and off:

C#

private bool FullScreenHandler(EventData eventData)
{
    if(eventData.Activate)
        BeginInvoke(new UIEventHandlerDelegate(FullScreen), eventData);
    return true;
}

VB

Private Function FullScreenHandler(ByVal eventData As EventData) As Boolean
    If eventData.Activate Then
        BeginInvoke(New UIEventHandlerDelegate(AddressOf FullScreen), eventData)
    End If
    Return True
End Function

Private Sub FullScreen(ByVal eventData As EventData)
    statusStrip1.Visible = Not statusStrip1.Visible
End Sub

Because we are in the VE3D thread when this handler is called, we need to use BeginInvoke to call the real method on the UI thread.

Be sure to check the source code for the full demo linked above for the location handler methods.  I omitted them here since it is just more of the same type of code above.

VR920 Head Tracker

Once the VR920 glasses and driver are installed on your PC, getting the data required from the glasses is quite easy.  We are going to create an object named VR920Tracker which will pull data from the device and send it to VE3D.

First we need to setup a few P/Invoke signatures to talk to the glasses as follows:

C#

[DllImport("IWEARDRV.dll", SetLastError = true, CharSet = CharSet.Auto)]

private static extern int IWROpenTracker();

[DllImport("IWEARDRV.dll", SetLastError = true, CharSet = CharSet.Auto)]

private static extern void IWRZeroSet();

[DllImport("IWEARDRV.dll", SetLastError = true, CharSet = CharSet.Auto)]

private static extern int IWRGetTracking(out int yaw, out int pitch, out int roll);

private const int ERROR_SUCCESS  = 0;

VB

<DllImport("IWEARDRV.dll", SetLastError := True, CharSet := CharSet.Auto)> _

Private Shared Function IWROpenTracker() As Integer

End Function

<DllImport("IWEARDRV.dll", SetLastError := True, CharSet := CharSet.Auto)> _

Private Shared Sub IWRZeroSet()

End Sub

<DllImport("IWEARDRV.dll", SetLastError := True, CharSet := CharSet.Auto)> _

Private Shared Function IWRGetTracking(<System.Runtime.InteropServices.Out()> ByRef yaw As Integer, <System.Runtime.InteropServices.Out()> ByRef pitch As Integer, <System.Runtime.InteropServices.Out()> ByRef roll As Integer) As Integer

End Function

private const int ERROR_SUCCESS  = 0;

These 3 methods will allows to open access to the glasses, set its zero position, and get the roll, pitch and yaw information from the sensors.

In the constructor for this object, we will open a handle to the glasses and startup a timer that will poll the glasses at a regular interval to get the sensor data as shown below:

C#

private const int TimerPeriod = 1;

private MainForm _mainForm;

private Timer _timer;

public VR920Tracker(MainForm mainForm)

{

    _mainForm = mainForm;

    int openResult = IWROpenTracker();

    if(openResult != ERROR_SUCCESS)

        throw new ApplicationException("Could not connect to VR920: " + openResult);

    _timer = new Timer(VR920Poller, null, Timeout.Infinite, TimerPeriod);

}

VB

Private Const TimerPeriod As Integer = 1

Private _mainForm As MainForm

Private _timer As Timer

Public Sub New(ByVal mainForm As MainForm)

    _mainForm = mainForm

    Dim openResult As Integer = IWROpenTracker()

    If openResult <> ERROR_SUCCESS Then

        Throw New ApplicationException("Could not connect to VR920: " & openResult)

    End If

    _timer = New Timer(AddressOf VR920Poller, Nothing, Timeout.Infinite, TimerPeriod)

End Sub

The VR920Poller method referenced above will be called at a specific interval and read the values from the glasses as shown below:

C#

// collection of values for averaging

private List<double> _yawValues = new List<double>();

private List<double> _rollValues = new List<double>();

private List<double> _pitchValues = new List<double>();

// last calculated values

private double _lastYaw;

private double _lastPitch;

private double _lastRoll;

private void VR920Poller(object state)

{

    lock(this)

    {

        int yaw, pitch, roll;

        int result = IWRGetTracking(out yaw, out pitch, out roll);

        if(result != ERROR_SUCCESS)

            throw new ApplicationException("Could not get VR920 tracking information: " + result);

        _yawValues.Add(VR920ToRadians(yaw));

        _rollValues.Add(VR920ToRadians(roll));

        _pitchValues.Add(VR920ToRadians(pitch));

        if(_yawValues.Count == 5)

        {

            double y = Average(_yawValues, _lastYaw);

            if(Math.Abs(y - _lastYaw) > 0.026)

                _lastYaw = y;

            double p = Average(_pitchValues, _lastPitch);

            if(Math.Abs(p - _lastPitch) > 0.017)

                _lastPitch = p;

            _lastRoll = Average(_rollValues, _lastRoll);

            _yawValues.Clear();

            _pitchValues.Clear();

            _rollValues.Clear();

            RollPitchYaw rollPitchYaw = new RollPitchYaw(_lastRoll, _lastPitch, _lastYaw);

            _mainForm.SetRollPitchYaw(rollPitchYaw);

            _mainForm.lblAxes.Text = VR920ToDegrees(yaw) + ", " + VR920ToDegrees(pitch) + ", " + VR920ToDegrees(roll);

        }

    }

}

private double Average(List<double> values, double last)

{

    double total = 0;

    foreach(double value in values)

        total += value;

    total += last;

    return total / (values.Count+1);

}

private double VR920ToRadians(int vr920Value)

{

    return (vr920Value * .00549) * (Math.PI/180);

}

private double VR920ToDegrees(int vr920Value)

{

    return (vr920Value * .00549);

}

VB

' collection of values for averaging

Private _yawValues As List(Of Double) = New List(Of Double)()

Private _rollValues As List(Of Double) = New List(Of Double)()

Private _pitchValues As List(Of Double) = New List(Of Double)()

' last calculated values

Private _lastYaw As Double

Private _lastPitch As Double

Private _lastRoll As Double

Private Sub VR920Poller(ByVal state As Object)

    SyncLock Me

        Dim yaw, pitch, roll As Integer

        Dim result As Integer = IWRGetTracking(yaw, pitch, roll)

        If result <> ERROR_SUCCESS Then

            Throw New ApplicationException("Could not get VR920 tracking information: " & result)

        End If

        _yawValues.Add(VR920ToRadians(yaw))

        _rollValues.Add(VR920ToRadians(roll))

        _pitchValues.Add(VR920ToRadians(pitch))

        If _yawValues.Count = 5 Then

            Dim y As Double = Average(_yawValues, _lastYaw)

            If Math.Abs(y - _lastYaw) > 0.026 Then

                _lastYaw = y

            End If

            Dim p As Double = Average(_pitchValues, _lastPitch)

            If Math.Abs(p - _lastPitch) > 0.017 Then

                _lastPitch = p

            End If

            _lastRoll = Average(_rollValues, _lastRoll)

            _yawValues.Clear()

            _pitchValues.Clear()

            _rollValues.Clear()

            Dim rollPitchYaw As New RollPitchYaw(_lastRoll, _lastPitch, _lastYaw)

            _mainForm.SetRollPitchYaw(rollPitchYaw)

            _mainForm.lblAxes.Text = VR920ToDegrees(yaw) & ", " & VR920ToDegrees(pitch) & ", " & VR920ToDegrees(roll)

        End If

    End SyncLock

End Sub

Private Function Average(ByVal values As List(Of Double), ByVal last As Double) As Double

    Dim total As Double = 0

    For Each value As Double In values

        total += value

    Next value

    total += last

    Return total / (values.Count+1)

End Function

Private Function VR920ToRadians(ByVal vr920Value As Integer) As Double

    Return (vr920Value *.00549) * (Math.PI/180)

End Function

Private Function VR920ToDegrees(ByVal vr920Value As Integer) As Double

    Return (vr920Value *.00549)

End Function

This chunk calls the IWRGetTracking method and tosses the results in 3 lists for roll, pitch and yaw.  then, when the list has 5 members in it, the values are averaged and, if that resulting values is past a certain threshold, the values are passed back to the MainForm's SetRollPitchYaw method as shown below:

C#

private double _lastYaw;

public void SetRollPitchYaw(RollPitchYaw rollPitchYaw)

{

    if(!_initialized || _globeControl.IsDisposed || (_globeControl.Host) == null || _globeControl.Host.CameraControllers.Current == null)

        return;

    double y = ((_globeControl.Host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.LocalOrientation.Yaw - _lastYaw) + rollPitchYaw.Yaw;

    RollPitchYaw rpw = new RollPitchYaw(rollPitchYaw.Roll, rollPitchYaw.Pitch, y);

    (_globeControl.Host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.LocalOrientation.RollPitchYaw = rpw;

    _lastYaw = rollPitchYaw.Yaw;

}

VB

Private _lastYaw As Double

Public Sub SetRollPitchYaw(ByVal rollPitchYaw As RollPitchYaw)

    If (Not _initialized) OrElse _globeControl.IsDisposed OrElse (_globeControl.Host) Is Nothing OrElse _globeControl.Host.CameraControllers.Current Is Nothing Then

        Return

    End If

    Dim y As Double = ((TryCast(_globeControl.Host.CameraControllers.Current, ActionCameraController)).LastReportedViewpoint.LocalOrientation.Yaw - _lastYaw) + rollPitchYaw.Yaw

    Dim rpw As New RollPitchYaw(rollPitchYaw.Roll, rollPitchYaw.Pitch, y)

    TryCast(_globeControl.Host.CameraControllers.Current, ActionCameraController).LastReportedViewpoint.LocalOrientation.RollPitchYaw = rpw

    _lastYaw = rollPitchYaw.Yaw

End Sub

This method takes the roll, pitch and yaw values provided and applies them to the current CameraController used by VE3D (which, by default, is an ActionCameraController).

So, this code ultimately takes the roll, yaw and pitch of the user's head and translates that directly to the roll, yaw and pitch of the camera in VE3D providing an accurate, real-time view into the VE3D world.

Stereoscopic Images

The final piece of the puzzle is drawing frames to the VR920 glasses in such a way that the final view to the user will be three dimensional.  This works similarly to those Magic Eye puzzles you may have seen.  Essentially, for every frame, we want to take the current VE3D camera position and move it several units to the left, render that to the left eye of the glasses, and then move the camera several units to the right and render that to the right eye of the glasses.  To handle this we,  can create an object named VR920StereoStep which derives from the Step class provided by VE3D and add this to the StepManager.  When this is done, the VR920StereoStep will be called at the end of every frame drawn by VE3D.

As with the VR920Tracker class, we will need to setup a few P/Invoke method signatures to talk to the stereo driver as shown below:

C#

[DllImport("iWrstDrv.dll", EntryPoint = "IWRSTEREO_Open", SetLastError=true)]

public static extern IntPtr OpenStereo();

[DllImport("iWrstDrv.dll", EntryPoint = "IWRSTEREO_SetStereo")]

public static extern Boolean SetStereoEnabled(IntPtr handle, Boolean enabled);

[DllImport("iWrstDrv.dll", EntryPoint = "IWRSTEREO_SetLR")]

public static extern Boolean SetStereoLR(IntPtr handle, Boolean eye);

[DllImport("iWrstDrv.dll", EntryPoint = "IWRSTEREO_WaitForAck")]

public static extern Byte WaitForOpenFrame(IntPtr handle, Boolean eye);

VB

<DllImport("iWrstDrv.dll", EntryPoint := "IWRSTEREO_Open", SetLastError:=True)> _

Public Shared Function OpenStereo() As IntPtr

End Function

<DllImport("iWrstDrv.dll", EntryPoint := "IWRSTEREO_SetLR")> _

Public Shared Function SetStereoLR(ByVal handle As IntPtr, ByVal eye As Boolean) As Boolean

End Function

<DllImport("iWrstDrv.dll", EntryPoint := "IWRSTEREO_SetStereo")> _

Public Shared Function SetStereoEnabled(ByVal handle As IntPtr, ByVal enabled As Boolean) As Boolean

End Function

<DllImport("iWrstDrv.dll", EntryPoint := "IWRSTEREO_WaitForAck")> _

Public Shared Function WaitForOpenFrame(ByVal handle As IntPtr, ByVal eye As Boolean) As Byte

End Function

In the constructor for this object, we will open a handle to the stereo driver and turn on the stereo functionality of the glasses as shown below:

C#

private static readonly IntPtr INVALID_FILE_HANDLE = (IntPtr)(-1);

private IntPtr _hStereo = INVALID_FILE_HANDLE;

private Host _host;

private bool _stereoEnabled = true;

public double EyeDistance { get; set; }

public VR920StereoStep(StepManager manager, Host host) : base(manager)

{

    EyeDistance = 10;

    _host = host;

    _hStereo = OpenStereo();

    if(_hStereo != INVALID_FILE_HANDLE)

        SetStereoEnabled(_hStereo, true);

    else

        _stereoEnabled = false;

}

VB

Private Shared ReadOnly INVALID_FILE_HANDLE As IntPtr = CType(-1, IntPtr)

Private _hStereo As IntPtr = INVALID_FILE_HANDLE

Private _host As Host

Private _stereoEnabled As Boolean = True

Private privateEyeDistance As Double

Public Property EyeDistance() As Double

    Get

        Return privateEyeDistance

    End Get

    Set(ByVal value As Double)

        privateEyeDistance = value

    End Set

End Property

Public Sub New(ByVal manager As StepManager, ByVal host As Host)

    MyBase.New(manager)

    EyeDistance = 10

    _host = host

    _hStereo = OpenStereo()

    If _hStereo <> INVALID_FILE_HANDLE Then

        SetStereoEnabled(_hStereo, True)

    Else

        _stereoEnabled = False

    End If

End Sub

Next, we need to override the OnExecute method provided by the base Step class as shown below:

C#

private bool _rightEye = true;

private Vector3D _position;

public override void OnExecute(SceneState state)

{

    if(_hStereo != INVALID_FILE_HANDLE && _stereoEnabled && (_host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint != null)

    {

        if(!_rightEye)

        {

            _position = (_host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.Position.Vector;

            (_host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.Position.Vector = _position + new Vector3D(-EyeDistance, 0, 0);

        }

        else

        {

            _position = (_host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.Position.Vector;

            (_host.CameraControllers.Current as ActionCameraController).LastReportedViewpoint.Position.Vector = _position + new Vector3D(EyeDistance, 0, 0);;

        }

        SetStereoLR(_hStereo, _rightEye);

        WaitForOpenFrame(_hStereo, _rightEye);

        _rightEye = !_rightEye;

    }

}

VB

Private _rightEye As Boolean = True

Private _position As Vector3D

Public Overrides Sub OnExecute(ByVal state As SceneState)

    If _hStereo <> INVALID_FILE_HANDLE AndAlso _stereoEnabled AndAlso (TryCast(_host.CameraControllers.Current, ActionCameraController)).LastReportedViewpoint IsNot Nothing Then

        If (Not _rightEye) Then

            _position = (TryCast(_host.CameraControllers.Current, ActionCameraController)).LastReportedViewpoint.Position.Vector

            TryCast(_host.CameraControllers.Current, ActionCameraController).LastReportedViewpoint.Position.Vector = _position + New Vector3D(-EyeDistance, 0, 0)

        Else

            _position = (TryCast(_host.CameraControllers.Current, ActionCameraController)).LastReportedViewpoint.Position.Vector

            TryCast(_host.CameraControllers.Current, ActionCameraController).LastReportedViewpoint.Position.Vector = _position + New Vector3D(EyeDistance, 0, 0)

        End If

        SetStereoLR(_hStereo, _rightEye)

        WaitForOpenFrame(_hStereo, _rightEye)

        _rightEye = Not _rightEye

    End If

End Sub

OnExecute will be called at the end of each VE3D frame rendered.  In this method, we obtain the current position of the VE3D camera and then set its position several units to the left or right (- or + EyeDistance).  SetStereoLR from the VR920 API is called, setting the appropriate eye to be active (left = false, right = true).  Then WaitForOpenFrame is called which will pause the action long enough for the glasses to draw the entire frame to eye to avoid image tearing.  Essentially, this call “sits and spins” at this location until the glasses report back that the entire frame is drawn in the proper eye and that we can continue on drawing the opposite eye for the next frame.  Finally, the _rightEye member variable is togged to the opposite value, so the next time through this method the opposite eye is drawn.

With this class in place, back in our MainForm in the FirstFrameRendered event handler, we can instantiate the object and add it to the StepManager as shown below:

C#

private VR920StereoStep _vr920StereoStep;

_vr920StereoStep = new VR920StereoStep(_globeControl.Host.RenderEngine.StepManager, _globeControl.Host);

_globeControl.Host.RenderEngine.StepManager.Add(_vr920StereoStep);

VB

Private _vr920StereoStep As VR920StereoStep

_vr920StereoStep = New VR920StereoStep(_globeControl.Host.RenderEngine.StepManager, _globeControl.Host)

_globeControl.Host.RenderEngine.StepManager.Add(_vr920StereoStep)

Running the Application

To run the demo, do the following:

1. Copy BindingsWiimote.xml to the appropriate directory listed above
2. Pair the Wiimote and Balance Board to the computer.  See the WiimoteLib article for more information on how to do that
3. Run the executable
4. Stand on the Balance Board
5. Put on the glasses
6. Zero both the Balance Board and glasses
7. Toggle everything on or off at will

Controls

• F1 – Set VR920 glasses to zero position
• F2 – Set Balance Board to zero position
• F3 – Toggle stereo mode on and off
• Wiimote 1 or B – Toggle Balance Board on or off
• Wiimote 2 or V – Toggle VR920 head tracking on or off
• F – Toggle full-screen on or off
• Nunchuk joystick X/Y – Strafe/Move
• Nunchuk C/Z buttons – Raise/lower altitude
• Wiimote A – Open location menu/Select location (note that if a new location is selected, the Balance Board is turned off and must be re-enabled after “landing”
• Wiimote Dpad Up/Down – Move through List
• Balance Board – Lean your body left or right to turn in the VE3D environment

Conclusion

With the above code, we have created a very interesting interface for Virtual Earth 3D.  The demo and source code linked above contain a few more features and bindings which enhance the application a bit more.  Be sure to give the full demo a try and check out the full source code for a few more implementation details.

Additional Information

• Vuzix VR920 Glasses
• Managed Library for Nintendo's Wiimote
• Heptazane's Blog
• Virtual Earth Blog
• Virtual Earth Developer's Blog

Bio

Brian is a Microsoft C# MVP who has been actively developing in .NET since its early betas in 2000, and who has been developing solutions using Microsoft technologies and platforms for even longer. Along with .NET, Brian is particularly skilled in the languages of C, C++ and assembly language for a variety of CPUs. He is also well-versed in a wide variety of technologies including web development, document imaging, GIS, graphics, game development, and hardware interfacing. Brian has a strong background in developing applications for the health-care industry, as well as developing solutions for portable devices, such as tablet PCs and PDAs. Additionally, Brian has co-authored the book "Debugging ASP.NET" published by New Riders, and is currently co-authoring a book titled "Coding4Fun: 10 .NET Programming Projects for Wiimote, YouTube, World of Warcraft, and More" to be published by O'Reilly in December 2008. Brian is also an author for MSDN's Coding4Fun website.  You can reach Brian via vis website at http://www.brianpeek.com/ .

Matt Ivers

Difficulty: Intermediate

Time Required: 6-10 hours

Cost: $100-$200

Software: Visual Basic or Visual C# Express Editions, Windows Media Center SDK 5.0, Windows Mobile 5.0 Smartphone SDK

Hardware: Mobile device running Windows Mobile 5.0

Download: C# and VB.Net

Introduction

By combining the power of a Media Center add-in with a mobile device running Windows Mobile 5.0, we can extend the capabilities of a Media Center add-in even further. The add-in service I developed runs as a listener service within Windows Media Center on the host machine. It is initialized when Windows Media Center is started, runs invisibly in the background, and allows mobile devices to connect and submit requests to it. The mobile solution leverages an existing WiFi internet connection to communicate with the add-in via TCP socket connections. The mobile device runs a .NET Compact Framework 2.0 forms application designed for devices running Windows Mobile 5 or higher.

The included remote control application is capable of the following functionality:

• Retrieve metadata about currently loaded media
• Affect playback of currently loaded media – Play, Pause, Stop, Next Track, Previous Track
• Control Media Center volume
• Retrieve a list of available media – Photos, Audio, Video
• Explicitly load media for playback – Audio, Video

Creating a Windows Media Center add-in application

Creating a new project – Windows Media Center Presentation Layer Application

Figure 1 - Creating the project

Using the Windows Media Center Presentation Layer Application project template yields the following project skeleton:

Figure 2 - Solution Explorer for our new project

There are two files of interest here: AddIn.vb and App.xml. AddIn.vb contains our add-in class. Interface methods implemented in this class will be invoked by Windows Media Center when our add-in is launched. App.xml contains configuration information identifying our assemblies and add-in attributes. The setup project will use this XML to register our add-in with Windows Media Center during installation.

The AddIn class

Visual Basic .NET

Public Class Class1
    Implements IAddInModule
    Implements IAddInEntryPoint

    ' Initialize (IAddInModule)
    Public Sub Initialize(ByVal appInfo As Dictionary(Of String, Object), ByVal entryPointInfo As Dictionary(Of String, Object)) Implements IAddInModule.Initialize
        ' Initialization logic goes here
    End Sub

    ' Uninitialize (IAddInModule)
    Public Sub Uninitialize() Implements IAddInModule.Uninitialize
        ' Clean-up logic goes here
    End Sub

    ' Launch (IAddInEntryPoint)
    Public Sub Launch(ByVal host As AddInHost) Implements IAddInEntryPoint.Launch
        ' This is where normal execution begins
    End Sub
End Class

Visual C#

public class Class1 : IAddInModule, IAddInEntryPoint 
{
    //  Initialize (IAddInModule)
    public void Initialize(Dictionary<string, object> appInfo, Dictionary<string, object> entryPointInfo)
    {
        //  Initialization logic goes here
    }    
    //  Uninitialize (IAddInModule)
    public void Uninitialize() 
    {
        //  Clean-up logic goes here
    }    
    //  Launch (IAddInEntryPoint)
    public void Launch(AddInHost host) 
    {
        //  This is where normal execution begins
    }
}

The IAddInModule.Initialize and IAddInModule.Uninitialize interface methods are used by Media Center to signal the beginning and end of the add-in application's lifetime. Use Initialize to create any internal objects your application may require, and the Uninitialize to perform any clean-up. Keep in mind that since the add-in application runs within Media Center, it may decide to close your add-in application before it has a chance to exit on its own. In any case, Media Center will call the Uninitialize method and give the add-in a window of execution to exit as gracefully as possible.

The IAddInEntryPoint.Launch method is where normal execution code should begin. Media Center will pass an AddInHost object as an input parameter. Through this object, your add-in application can access all the features of the Media Center API.

Signing the assembly

Figure 3 - Signing our assembly

Explicit instruction is also available here:
How to: Sign an Assembly (Visual Studio)

Tweaking the App.xml file

<application title="DemoProject" id="{f6dd1141-f288-46fe-bcdb-2d9c5ffd986e}">
  <entrypoint id="{31a15ce9-0487-43bc-94de-a43409129641}" 
              addin="DemoProject.Class1, DemoProject,Culture=Neutral,Version=6.0.6000.0"
              title="DemoProject"
              description="DemoProject"
              ImageUrl=".\AppIcon.png">
    <category category="More Programs"/>
  </entrypoint>
</application>

The XML definition specifies the add-in applications and their points of entry within Media Center. Add-in applications can be launched on demand from various locations within Media Center, such as from an icon in the “More Programs” group. A full listing of all the available points of entry can be found here. Add-in applications can also be launched in the background when Media Center starts. To indicate this, we specify “Background” in the “category” attribute. The “addin” attribute must also be modified to point to our add-in class in the format “Assemblyname.ClassName”. The “title”, “description”, and “ImageUrl” attributes are used primarily for on-demand add-ins.

After making our changes, the Windows Media Center Remote application XML looks like this:

<application title="WMCServerAddIn" id="{7b71a7ca-b459-4023-ab12-d5cc8c56b991}">
  <entrypoint id="{5fe9982b-ca4a-459e-b62c-40399e875d66}" 
              addin="WMCServerAddIn.ServerAddIn, WMCServerAddIn,Culture=Neutral,Version=6.0.6000.0"
              title="WMCServerAddIn"
              description="WMCServerAddIn"
              ImageUrl=".\AppIcon.png">
    <category category="Background"/>
  </entrypoint>
</application>

Creating the setup project

With a basic interface implementation complete and a configuration XML created, we are ready to create the setup project. The setup project will install our assemblies into the Global Assembly Cache and register our XML-defined entry point with Media Center. A full step-by-step walkthrough can be found here:

Creating a Windows Installer File for a Windows Media Center Add-in

Once the setup project is complete, it's time for a test. By using the Host.MediaCenterEnvironment.Dialog class off of the AddInHost object, we can present a dialog box to the user to ensure the add-in launches successfully. After compiling the solution, run the installer by right-clicking the setup project and choosing “Install”. Once installation has completed, run Windows Media Center. A few seconds after startup, the add-in is launched:

Figure 4 - Running our Hello World Media Center Add-in

Creating a Windows Mobile 5.0 client application

Creating a new project – Windows Mobile CE 5.0 application

Installing the Windows Mobile Smartphone SDK 5.0 makes this step simple. Once the SDK is installed, you can create a project in Visual Studio using the Windows CE 5.0 Project template.

The only other typical setup required for developing mobile apps is to change the deployment settings to point to your WM5 device. This can be configured within the Visual Studio toolbar:

Figure 5 - changing the deployment settings to target the mobile device

Implementing the Media Center Add-in service

Implementing the socket server

The socket server uses the Socket object from the System.Networking.Sockets namespace for all of its communications. The listener service begins by binding to the host machine's local address and listening on a port of your choosing. Once the socket is instantiated and bound to a local port, it begins listening for and accepting inbound connections. When a new connection is established, a target method BeginSocketConversation will be invoked on a new thread, where the actual communication between the two parties will take place. Once the new thread has been invoked, execution will continue in the AcceptConnections method. At this time the newly launched thread notifies the listener to begin accepting new connections again. A ManualResetEvent from the System.Threading namespace can be used to communicate changes in state between two or more threads. Here is the code implementation of the socket listener:

Visual Basic .NET

Private Sub AcceptConnections()
        Dim localMachineInfo As IPHostEntry
        Dim localEndPoint As IPEndPoint
        Dim listener As New Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)
        Try
            localMachineInfo = Dns.Resolve(String.Empty)
            localEndPoint = New IPEndPoint(localMachineInfo.AddressList(0), _port)
            listener.Bind(localEndPoint)
            listener.Listen(10)
            While True
                threadEvent.Reset()
                listener.BeginAccept(New AsyncCallback(AddressOf BeginSocketConversation), listener)
                threadEvent.WaitOne()
            End While
        Catch ex As Exception
          Log.Write(ex)
    End Try
End Sub

Visual C#

Public Class Class1
    Implements IAddInModule
    Implements IAddInEntryPoint

    ' Initialize (IAddInModule)
    Public Sub Initialize(ByVal appInfo As Dictionary(Of String, Object), ByVal entryPointInfo As Dictionary(Of String, Object)) Implements IAddInModule.Initialize
        ' Initialization logic goes here
    End Sub

    ' Uninitialize (IAddInModule)
    Public Sub Uninitialize() Implements IAddInModule.Uninitialize
        ' Clean-up logic goes here
    End Sub

    ' Launch (IAddInEntryPoint)
    Public Sub Launch(ByVal host As AddInHost) Implements IAddInEntryPoint.Launch
        ' This is where normal execution begins
    End Sub
End Class

Once a connection has been established, the server begins to listen for inbound messages. When new data is received, it is parsed into a custom Message object for easy inspection by a consuming class. An event is then raised to any consuming classes, passing a reference to the newly created Message object. The server then resumes listening for new data on the same socket. Here is the code implementation:

Visual Basic .NET

Private Sub BeginSocketConversation(ByVal ar As IAsyncResult)
    threadEvent.Set()
    RaiseEvent ClientConnected()
    Dim listener As Socket = CType(ar.AsyncState, Socket)
    _socket = listener.EndAccept(ar)
    Dim buffer(PACKET_SIZE) As Byte
    Dim bytesReadCount As Integer = 0
    While True
        Dim sb As New System.Text.StringBuilder
        Do
            Try
                bytesReadCount = _socket.Receive(buffer, buffer.Length, SocketFlags.None)
            Catch ex As Exception
                _socket.Close()
                Exit While
            End Try
            sb.Append(System.Text.ASCIIEncoding.ASCII.GetString(buffer, 0, bytesReadCount))
        Loop While _socket.Available > 0
        Dim requestMessage As New RequestMessage()
        requestMessage.DeSerialize(sb.ToString())
        RaiseEvent MessageReceived(requestMessage)
    End While
    RaiseEvent ClientDisconnected()
End Sub

Visual C#

private void BeginSocketConversation(IAsyncResult ar)
{
    threadEvent.Set();
    ClientConnected();
    Socket listener = ((Socket)(ar.AsyncState));
    _socket = listener.EndAccept(ar);
    byte[,] buffer;
    int bytesReadCount = 0;
    while (true)
    {
        System.Text.StringBuilder sb = new System.Text.StringBuilder();
        for (
        ; (_socket.Available > 0);
        )
        {
            try
            {
                bytesReadCount = _socket.Receive(buffer, buffer.Length, SocketFlags.None);
            }
            catch (Exception ex)
            {
                _socket.Close();
                break;
            }
            sb.Append(System.Text.ASCIIEncoding.ASCII.GetString(buffer, 0, bytesReadCount));
        }
        RequestMessage requestMessage = new RequestMessage();
        requestMessage.DeSerialize(sb.ToString());
        MessageReceived(requestMessage);
    }
    ClientDisconnected();
}

Object serialization

Implementing the AddIn.Launch method

The custom server object created in the previous section encapsulates all the grunt work of socket-level protocol, accepting connections, serializing / deserializing message objects, and data transfer. Within the AddIn class implementation, all we have to do now is instantiate the Server class, indicating the port to listen on:

Visual Basic .NET

Public Sub Launch(ByVal host As AddInHost) Implements IAddInEntryPoint.Launch
    _wmcHost = host
    _waitForExit = New System.Threading.ManualResetEvent(False)
    _server.Start(DEFAULT_PORT)
    System.Threading.Thread.CurrentThread.Priority = Threading.ThreadPriority.Lowest
    _waitForExit.WaitOne()
End Sub

Visual C#

public void Launch(AddInHost host) 
{
    _wmcHost = host;
    _waitForExit = new System.Threading.ManualResetEvent(false);
    _server.Start(DEFAULT_PORT);
    System.Threading.Thread.CurrentThread.Priority = Threading.ThreadPriority.Lowest;
    _waitForExit.WaitOne();
}

We also perform two other critical tasks at this time. First, we assign a reference to the AddInHost to a private member variable for later use. Second, we reduce the priority of the add-in's main thread and use a ManualResetEvent to put it to sleep until Media Center is closed. The reason for this relates to the nature of an add-in's lifetime within Media Center. The Media Center add-in hosting process will only allow an add-in to execute for the duration of its Launch method. This means that as soon as the Launch method returns, the add-in and any child threads it has spawned will be terminated. For this reason, we must keep the add-in alive by blocking the Launch method from exiting.

Media Center manipulation

Using the communications layer and socket listener developed in the previous sections, we can finally get to the task of performing something useful with the input requests. By subscribing to the custom Server object's MessageReceived event, the AddIn class's OnDataReceived method will automatically be invoked when a new request message is received from the listener:

Visual Basic .NET

Private Sub OnDataReceived(ByVal message As RequestMessage) Handles _server.MessageReceived
    Select Case message.RequestAction
        Case RequestMessage.Action.PauseMedia
_wmcHost.MediaCenterEnvironment.MediaExperience.Transport.PlayRate = PlayRates.PLAYRATE_PAUSE
        Case RequestMessage.Action.VolumeUp
           _wmcHost.MediaCenterEnvironment.AudioMixer.VolumeUp()
        Case RequestMessage.Action.VolumeDown
           _wmcHost.MediaCenterEnvironment.AudioMixer.VolumeDown()
        Case RequestMessage.Action.NextMedia _wmcHost.MediaCenterEnvironment.MediaExperience.Transport.SkipForward()
        Case Else
    End Select
    SendAcknowledgement(message)
End Sub

Visual C#

private void OnDataReceived(RequestMessage message) 
{
    switch (message.RequestAction) 
    {
        case RequestMessage.Action.PauseMedia:
            _wmcHost.MediaCenterEnvironment.MediaExperience.Transport.PlayRate = PlayRates.PLAYRATE_PAUSE;
            break;
        case RequestMessage.Action.VolumeUp:
            _wmcHost.MediaCenterEnvironment.AudioMixer.VolumeUp();
            break;
        case RequestMessage.Action.VolumeDown:
            _wmcHost.MediaCenterEnvironment.AudioMixer.VolumeDown();
            break;
        case RequestMessage.Action.NextMedia:
            _wmcHost.MediaCenterEnvironment.MediaExperience.Transport.SkipForward();
            break;
    }
    SendAcknowledgement(message);
}

The RequestMessage class shown above is a custom object, originally created by Windows Mobile 5 application running on a connected mobile device. The transport object was then serialized on the device, transmitted, deserialized in the listener service, and raised to the add-in class via the MessageReceived event. The RequestAction property is an enumerable type indicating the type of action to take within Media Center. The Media Center API exposed by the AddInHost object (_wmcHost) can then be used to actually perform the desired Media Center operation. Finally, we send an acknowledgement message to the mobile device.

Implementing the Windows Mobile 5 remote client application

Implementing the socket client

The design of the socket client is much simpler than that of the socket server. All execution will be performed on the primary thread. The client begins by establishing a connection with an add-in server. Once established, the socket client sends and receives data synchronously over the socket. Consumers of the socket client need only call the Send method, which will return with the transactional response:

Visual Basic .NET

Public Function Send(ByVal request As Message) As String
    Dim responseString As String = Nothing
    Try
        If ((Not (_socket) Is Nothing) _
                    AndAlso _socket.Connected) Then
            Dim requestString As String = request.Serialize
            Send(requestString)
            responseString = RecvSynchronous
            Return responseString
        End If
    Catch ex As Exception
        Return Nothing
    End Try
    Return responseString
End Function

Visual C#

public string Send(Message request)
{
    string responseString = null;
    try
    {
        if (_socket != null && _socket.Connected)
        {
            string requestString = request.Serialize();
            Send(requestString);
            responseString = RecvSynchronous();
            return responseString;
        }
    }
    catch (Exception ex)
    {
        return null;
    }
    return responseString;
}

Adding the UI layer

Figure 6 - The configuration screen on the mobile device

Once we've specified the host and port of our running Media Center add-in service, we can click the “Connect” button to initialize the TCP/IP socket connection. Using the TCP Client class developed earlier, the code required to handle this event is simple:

Visual Basic .NET

Private Sub btnConnect_Click(ByVal sender As Object, ByVal e As EventArgs)
    Dim host As String = Me.txtHost.Text.Trim
    Dim port As String = Me.txtPort.Text.Trim
    _client.Connect(host, Convert.ToInt32(port))
End Sub

Visual C#

private void btnConnect_Click(object sender, EventArgs e)
{
     string host = this.txtHost.Text.Trim();
     string port = this.txtPort.Text.Trim();
     _client.Connect(host, Convert.ToInt32(port));
}

Once connected, the user can navigate the application using the provided tabs at the bottom of the screen. Since this application is intended to function as a remote control, there is a standard media control screen:

Figure 8: The control screen on the mobile device

Nothing exotic here, but the basic media control functionality you'd expect from a remote is available. Here's a sample of the code required to support the “Stop” button's click event:

Visual Basic .NET

Private Sub btnConnect_Click(ByVal sender As Object, ByVal e As EventArgs)
    Dim host As String = Me.txtHost.Text.Trim
    Dim port As String = Me.txtPort.Text.Trim
    _client.Connect(host, Convert.ToInt32(port))
End Sub

Visual C#

private void btnStop_Click(object sender, EventArgs e)
{
    RequestMessage message = new RequestMessage();
    message.RequestAction = RequestMessage.Action.StopMedia;
    message.Data = String.Empty;
    string responseString = client.Send(message);
    ResponseMessage response = new ResponseMessage(responseString);
}

The UI form uses the connected Client object to execute the Stop command. First, the form constructs a Request message, populating the type of action desired (StopMedia). Next, the Request message is passed to the Client class, where it is serialized for transport and sent over the connected TCP socket. From there, the running Media Center service will receive the request, perform the media stop, and send an acknowledgement. The Client then returns a Response message, which will contain a Boolean indicating if the request was successful or not.

On a Media Stop request, examining the Response object is not very informative. However, for other commands, the Media Center add-in service can be designed to return any kind of information required of the host. The Detail screen is a good example:

Figure 9 - The detail page 

When a song or video is currently playing, the above screen can be used to retrieve any metadata associated with it. The event handler for the screen's “Refresh” button looks similar in form to the Stop button's handler:

Visual Basic .NET

Private Sub btnRefreshDetail_Click(ByVal sender As Object, ByVal e As EventArgs)
    Dim request As RequestMessage = New RequestMessage
    request.RequestAction = RequestMessage.Action.GetMediaMetadata
    Dim responseString As String = _client.Send(request)
    Dim response As MediaMetadataMessage = New MediaMetadataMessage(responseString)
    For Each item As MediaMetadataMessage.MediaMetadataItem In message.Data
        If ((Not (item.Value) Is Nothing) AndAlso (item.Value.Length > 0)) Then
            Me.txtDetail.Text = (Me.txtDetail.Text + (item.Key + (": " + item.Value)))
            Me.txtDetail.Text = (Me.txtDetail.Text + "" & vbCrLf)
        End If
    Next
End Sub

Visual C#

private void btnRefreshDetail_Click(object sender, EventArgs e)
{
    RequestMessage request = new RequestMessage();
    request.RequestAction = RequestMessage.Action.GetMediaMetadata;
    string responseString = _client.Send(request);
    MediaMetadataMessage response = new MediaMetadataMessage(responseString);
    foreach (MediaMetadataMessage.MediaMetadataItem item in message.Data) {
        if (item.Value != null && item.Value.Length > 0){
            this.txtDetail.Text += item.Key + ": " + item.Value;
            this.txtDetail.Text += "\r\n";
        }
    }
}

This time, however, the Response object has a list of metadata keys and values, which are then enumerated and displayed in a textbox. This Response object was originally populated by the Media Center add-in service:

Visual Basic .NET

Private Sub SendMediaMetadata()
    Dim response As New MediaMetadataMessage()
    Dim val As String
    For Each key As String In _wmcHost.MediaCenterEnvironment.MediaExperience.MediaMetadata.Keys
    val = _wmcHost.MediaCenterEnvironment.MediaExperience.MediaMetadata.Item(key).ToString()
    response.Data.Add(New MediaMetadataMessage.MediaMetadataItem(key, val))
    Next
    response.ResponseIndicator = MediaMetadataMessage.ResponseCode.Success
    _server.Send(response)
End Sub

Visual C#

private void SendMediaMetadata() 
{
    MediaMetadataMessage response = new MediaMetadataMessage();
    string val;
    foreach (string key in _wmcHost.MediaCenterEnvironment.MediaExperience.MediaMetadata.Keys) 
    {
        val = _wmcHost.MediaCenterEnvironment.MediaExperience.MediaMetadata.Item[key].ToString();
        response.Data.Add(new MediaMetadataMessage.MediaMetadataItem(key, val));
    }
    response.ResponseIndicator = MediaMetadataMessage.ResponseCode.Success;
    _server.Send(response);
}

After receiving a request from the mobile device for the currently playing media's metadata, the Media Center add-in service constructs a list of custom objects (MediaMetadataItem). It populates this list by enumerating properties of the AddInHost object, through the namespace MediaCenterEnvironment.MediaExperience.MediaMetadata. It then sends this list on the Response object, where the mobile device will inspect and display the data, as shown above.

The last feature that the mobile remote application provides is the ability to select specific media to play based on a list of available media on the host. This functionality is provided for audio, video, and images:

Figure 10 - Music and Video list management 

The mobile application requests the list of available media in a similar fashion to the Details screen implementation. The resulting list of files is then bound and displayed in a DataGrid. The Media Center add-in service can then be instructed to play individual songs or videos using the AddInHost's MediaCenterEnvironment.PlayMedia method.

Mobile Application UI design considerations

Developing on the mobile platform introduces design challenges not found in Windows Forms applications due to the small amount of screen real estate available. Many typical form layouts found in Windows Forms applications won't work well on a mobile device's small screen. When developing for a mobile application, try to keep the user interface as simple as possible. Window containers and managers such as a tabbed control enable the developer to functionally divide an application into logical components. This alleviates the clutter created by having too many controls in a single window. Also try to keep your text and controls as simple and large as possible. This makes the application easier to view and manipulate, especially in outdoor or bright conditions.

Conclusion

Once complete, the remote application gives you a level of control simply not available on hardware-based remote controls. I really enjoyed working on this application because it gave me a chance to integrate a number of different technologies into a single implementation. Be sure to download the code and take the remote for a spin. For those not familiar with all of the components used in this article (Media Center add-in services, socket communication, installer and assembly signing, etc.), don't be intimidated. The Microsoft-provided SDK packages for both parts of the solution make creating your own add-in projects a snap. Plus, the technical challenges unique to this project have already been overcome! Feel free to modify my implementation, or reuse the included socket classes to create an add-in all your own. Finally, please post feedback or questions here or at my blog.

About the Author

Matt Ivers is a software engineer at Chicago-based Clarity Consulting. He has extensive professional experience developing .NET solutions in both Visual C# .NET and VB.NET. Check him out at his blog or through the Clarity blog site, Clarity Blogs.

References

Windows Media Center SDK 5.0
Windows Mobile 5.0 Smartphone SDK

How to: Sign an Assembly using Visual Studio 2005
How to: Create a Windows Installer File for a Windows Media Center Add-in

Windows Media Center - SDK Overview
Windows Media Center - Understanding the Basics
Windows Media Center - Development Tools
Windows Media Center - AddIn example: "Time Travel"

Using an Asynchronous Server Socket

	In this article, Brian Peek demonstrates how to connect to and use the Nintendo Wiimote from C# and VB.NET. The final output is an easy-to-use managed API for the Wiimote that can be used in any managed application.
Brian Peek ASPSOFT, Inc. Difficulty: Intermediate Time Required: 1-3 hours Cost: Less Than $50 (Free if you already own a Wiimote) Software: Visual C# or Visual Basic 2008 Express Editions Hardware: Nintendo Wii Remote (Wiimote), a compatible PC Bluetooth adapter and stack Download: CodePlex

Updates

1/19/09 - Version 1.7 now up at CodePlex.

6/15/08 – Version 1.5.2 is now up at CodePlex.  The Balance Board should really work this time….

6/12/08 - Version 1.5.1 has been removed from CodePlex...it has proven to be too buggy for most users (though it continues to work just fine for me).  Look for version 1.5.2 soon...

6/11/08 – Version 1.5.1 posted at CodePlex.  Fixed a bug with the Balance Board…  UPDATE:  It appears some people are still having issues with this build as well due to some Balance Boards being a bit finicky in their response times.  Stay tuned for build 1.5.2 soon…

6/9/08 – Version 1.5 posted at CodePlex.  The Wii Fit Balance Board is now supported.

6/3/08 – New version 1.4 posted at the download link above.  The big change is multiple Wiimotes are now finally supported.

5/27/08 - New version 1.3 posted at the download link above.  Lots of changes, so be sure to read the included docs!

1/29/08 - New version 1.2.1.0 posted at the download link above.  The only change for this release is adding support for IR3 and IR4 since I've gotten so many questions about it.

10/22/07 - New version 1.2.0.0 posted at the download link above.  There are a variety of fixes and new features added.  The source code and binary releases are now hosted at CodePlex.  A .chm help file is also included documenting the API itself.  Please note that a license is now included with the library which explicitly defines how the library can be used.  For 99% of you, this won't change anything, but I've received tons of emails asking about restrictions, so now there's something official attached.  Check out the readme.txt and license.txt in the docs directory included with the official distribution for more information.

6/12/07 - New version 1.1.0.0 posted at the download link above.  This fixes several issues, adds an alternate writing method which may help those with troubled bluetooth stacks/adapters, adds Vista/XP x64 support, and a Microsoft Robotics Studio service version of the library.  See the included readme.txt for more information.  Additionally, see my new article on using the new MSRS service to create a Wiimote-controlled car!

3/17/07 - New version 1.0.1.0 posted at the download link above.  This fixes a bug with the calibration data being stored by the API.  Thanks to James Darpinian for the catch!

Introduction

Nintendo's Wii Remote (forever known as the Wiimote) is a fantastic little controller for the Nintendo Wii system.  Because it uses Bluetooth to communicate with the Wii, it can be connected to and used by practically any Bluetooth capable device.

If you care to just use the library and are not concerned with the implementation details, skip down to the usage section.

Before we get started, there are two websites that one should peruse before reading the code contained here.  These two sites have done 99% of the work of figuring out the data that is sent and received with the Wiimote.  I am not going to repeat most of this information here since both of the these sites explain the protocol so thoroughly.  Without the work of the people posting to these sites, none of this would have been possible.

• WiiLi Wiki
• WiiBrew Wiki

Getting Connected

This will likely be the biggest sticking point.  The Wiimote will not pair and communicate successfully with every Bluetooth device and stack on the planet.   There's little I can do to help get you connected if the following steps do not work.  Either it's going to work, or it isn't.  Cross your fingers...

1. Start up your Bluetooth software and have it search for a device.
2. Hold down the 1 and 2 buttons on the Wiimote.  You should see the LEDs at the bottom start flashing.  Do not let go of these buttons until this procedure is complete.  If pairing the Wii Fit Balance Board, open the battery cover on the underside of the balance board and hold the little red sync button.
3. Wiimotes should show up in the list of devices found as Nintendo RVL-CNT-01.  Balance Boards will show up as Nintendo RVL-WBC-01.  If it's not there, start over and try again.
4. Click Next to move your way through the wizard.  If at any point you are asked to enter a security code or PIN, leave the number blank or click Skip.  Do not enter a number.
5. You may be asked which service to use from the Wiimote.  Select the keyboard/mouse/HID service if prompted (you should only see one service available).
6. Finish the wizard.

That's it.  The LEDs at the bottom should continue to flash and you should see the device listed in your list of connected Bluetooth devices.  If you run the test application included with the source code and you see the numbers change, you are all set.  If you don't see them change or you get an error, try the above again.  If it continues to not function, you are likely stuck with an incompatible device or stack.

The Exciting World of HID and P/Invoke

When the Wiimote is paired with your PC, it will be identified as a HID-compliant device.  Therefore, to connect to the device, we must use the HID and Device Management Win32 APIs.  Unfortunately there is no built-in support for these APIs in the current .NET runtime, so we must enter the realm of P/Invoke.  These APIs are defined in the Windows Driver Kit (WDK) so if you wish to see the original C header files or read the API documentation, you will need to download and install the latest WDK.

P/Invoke, as you probably know, allows one to directly call methods of the Win32 API from .NET.  The difficulty of this is finding the right method signatures and structure definitions which will properly marshal the data through to Win32.  A great resource for these signatures is the P/Invoke wiki.  Almost all of the methods used in this project were defined by this resource.  For this project, all P/Invoke'd methods live in the HIDImports class.

The process to begin communication with the Wiimote is as follows:

1. Get the GUID of the HID class defined by Windows
2. Get a handle to the list of all devices which are part of the HID class
3. Enumerate through those devices and get detailed information about each
4. Compare the Vendor ID and Product ID of each device to the known Wiimote's VID and PID
5. When found, create a FileStream to read/write to the device
6. Clean up the device list

VB

' read/write handle to the device
Private mHandle As SafeFileHandle

' a pretty .NET stream to read/write from/to
Private mStream As FileStream
Private found As Boolean = False
Private guid As Guid
Private index As UInteger = 0

' 1. get the GUID of the HID class
HIDImports.HidD_GetHidGuid(guid)

' 2. get a handle to all devices that are part of the HID class
Dim hDevInfo As IntPtr = HIDImports.SetupDiGetClassDevs(guid, Nothing, IntPtr.Zero, HIDImports.DIGCF_DEVICEINTERFACE) ' | HIDImports.DIGCF_PRESENT);

' create a new interface data struct and initialize its size
Dim diData As HIDImports.SP_DEVICE_INTERFACE_DATA = New HIDImports.SP_DEVICE_INTERFACE_DATA()
diData.cbSize = Marshal.SizeOf(diData)

' 3. get a device interface to a single device (enumerate all devices)
Do While HIDImports.SetupDiEnumDeviceInterfaces(hDevInfo, IntPtr.Zero, guid, index, diData)
    ' create a detail struct and set its size
    Dim diDetail As HIDImports.SP_DEVICE_INTERFACE_DETAIL_DATA = New HIDImports.SP_DEVICE_INTERFACE_DETAIL_DATA()
    diDetail.cbSize = 5 'should be: (uint)Marshal.SizeOf(diDetail);, but that's the wrong size

    Dim size As UInt32 = 0

    ' get the buffer size for this device detail instance (returned in the size parameter)
    HIDImports.SetupDiGetDeviceInterfaceDetail(hDevInfo, diData, IntPtr.Zero, 0, size, IntPtr.Zero)

    ' actually get the detail struct
    If HIDImports.SetupDiGetDeviceInterfaceDetail(hDevInfo, diData, diDetail, size, size, IntPtr.Zero) Then
        ' open a read/write handle to our device using the DevicePath returned
        mHandle = HIDImports.CreateFile(diDetail.DevicePath, FileAccess.ReadWrite, FileShare.ReadWrite, IntPtr.Zero, FileMode.Open, HIDImports.EFileAttributes.Overlapped, IntPtr.Zero)

        ' 4. create an attributes struct and initialize the size
        Dim attrib As HIDImports.HIDD_ATTRIBUTES = New HIDImports.HIDD_ATTRIBUTES()
        attrib.Size = Marshal.SizeOf(attrib)

        ' get the attributes of the current device
        If HIDImports.HidD_GetAttributes(mHandle.DangerousGetHandle(), attrib) Then
            ' if the vendor and product IDs match up
            If attrib.VendorID = VID AndAlso attrib.ProductID = PID Then
                ' 5. create a nice .NET FileStream wrapping the handle above
                mStream = New FileStream(mHandle, FileAccess.ReadWrite, REPORT_LENGTH, True)
            Else
                mHandle.Close()
            End If
        End If
    End If

    ' move to the next device
    index += 1
Loop

' 6. clean up our list
HIDImports.SetupDiDestroyDeviceInfoList(hDevInfo)

C#

// read/write handle to the device
private SafeFileHandle mHandle;

// a pretty .NET stream to read/write from/to
private FileStream mStream;
bool found = false;
Guid guid;
uint index = 0;

// 1. get the GUID of the HID class
HIDImports.HidD_GetHidGuid(out guid);

// 2. get a handle to all devices that are part of the HID class
IntPtr hDevInfo = HIDImports.SetupDiGetClassDevs(ref guid, null, IntPtr.Zero, HIDImports.DIGCF_DEVICEINTERFACE);// | HIDImports.DIGCF_PRESENT);

// create a new interface data struct and initialize its size
HIDImports.SP_DEVICE_INTERFACE_DATA diData = new HIDImports.SP_DEVICE_INTERFACE_DATA();
diData.cbSize = Marshal.SizeOf(diData);

// 3. get a device interface to a single device (enumerate all devices)
while(HIDImports.SetupDiEnumDeviceInterfaces(hDevInfo, IntPtr.Zero, ref guid, index, ref diData))
{
    // create a detail struct and set its size
    HIDImports.SP_DEVICE_INTERFACE_DETAIL_DATA diDetail = new HIDImports.SP_DEVICE_INTERFACE_DETAIL_DATA();
    diDetail.cbSize = 5; //should be: (uint)Marshal.SizeOf(diDetail);, but that's the wrong size

    UInt32 size = 0;

    // get the buffer size for this device detail instance (returned in the size parameter)
    HIDImports.SetupDiGetDeviceInterfaceDetail(hDevInfo, ref diData, IntPtr.Zero, 0, out size, IntPtr.Zero);

    // actually get the detail struct
    if(HIDImports.SetupDiGetDeviceInterfaceDetail(hDevInfo, ref diData, ref diDetail, size, out size, IntPtr.Zero))
    {
        // open a read/write handle to our device using the DevicePath returned
        mHandle = HIDImports.CreateFile(diDetail.DevicePath, FileAccess.ReadWrite, FileShare.ReadWrite, IntPtr.Zero, FileMode.Open, HIDImports.EFileAttributes.Overlapped, IntPtr.Zero);

        // 4. create an attributes struct and initialize the size
        HIDImports.HIDD_ATTRIBUTES attrib = new HIDImports.HIDD_ATTRIBUTES();
        attrib.Size = Marshal.SizeOf(attrib);

        // get the attributes of the current device
        if(HIDImports.HidD_GetAttributes(mHandle.DangerousGetHandle(), ref attrib))
        {
            // if the vendor and product IDs match up
            if(attrib.VendorID == VID && attrib.ProductID == PID)
            {
                // 5. create a nice .NET FileStream wrapping the handle above
                mStream = new FileStream(mHandle, FileAccess.ReadWrite, REPORT_LENGTH, true);
            }
            else
                mHandle.Close();
        }
    }

    // move to the next device
    index++;
}

// 6. clean up our list
HIDImports.SetupDiDestroyDeviceInfoList(hDevInfo);

CreateFile and SafeFileHandles

If you read through the above code, you may have noticed that the handle to the Wiimote is opened using Win32's CreateFile function instead of directly using a FileStream object or some other managed way.  This is required due to the way that the handle needs to be created.  The DevicePath member of the detail struct contains a non-file system path that Win32 can use to open a handle to the device.  The .NET methods for performing this action will only allow file system paths, so we must use the Win32 method instead.

You will also notice that we use the SafeFileHandle object to wrap the handle returned from the call to CreateFile.  The SafeFileHandle object wraps a native Win32 handle and allows one to safely manage the native type and cleanly close things up at the end of the application.  One could just as easily use an IntPtr, but I have found that this is a much cleaner method for dealing with the native type.

Wiimote I/O and HID Reports

In the world of HID, data is sent and received as "reports".  Simply, it is a data buffer of a pre-defined length with a header that determines the report contained in the buffer.  The Wiimote will send and can receive various reports, all of which are 22 bytes in length, and all of which are explained at the links above.  Given the number and complexity, I suggest you read through the wikis above if you wish to know more about the Wiimote's reports and the data they contain.

Now that we have a a FileStream on which to communicate with the Wiimote, we can start communication.  Because reports will be sent and received almost constantly, it is essential that asynchronous I/O operations are used.  In .NET, this is quite simple.  The process is to start an asynchronous read operation and provide a callback method to be run when the buffer is full.  When the callback is run, the data is handled and the process is repeated.

VB 

' sure, we could find this out the hard way using HID, but trust me, it's 22
Private Const REPORT_LENGTH As Integer = 22

' report buffer
Private mBuff As Byte() = New Byte(REPORT_LENGTH - 1){}

Private Sub BeginAsyncRead()
    ' if the stream is valid and ready
    If mStream.CanRead Then
        ' create a read buffer of the report size
        Dim buff As Byte() = New Byte(REPORT_LENGTH - 1){}

        ' setup the read and the callback
        mStream.BeginRead(buff, 0, REPORT_LENGTH, New AsyncCallback(AddressOf OnReadData), buff)
    End If
End Sub

Private Sub OnReadData(ByVal ar As IAsyncResult)
    ' grab the byte buffer
    Dim buff As Byte() = CType(ar.AsyncState, Byte())

    ' end the current read
    mStream.EndRead(ar)

    ' start reading again
    BeginAsyncRead()

    ' handle data....
End Sub

C#

// sure, we could find this out the hard way using HID, but trust me, it's 22
private const int REPORT_LENGTH = 22;

// report buffer
private byte[] mBuff = new byte[REPORT_LENGTH];

private void BeginAsyncRead()
{
    // if the stream is valid and ready
    if(mStream.CanRead)
    {
        // create a read buffer of the report size
        byte[] buff = new byte[REPORT_LENGTH];

        // setup the read and the callback
        mStream.BeginRead(buff, 0, REPORT_LENGTH, new AsyncCallback(OnReadData), buff);
    }
}

private void OnReadData(IAsyncResult ar)
{
    // grab the byte buffer
    byte[] buff = (byte[])ar.AsyncState;

    // end the current read
    mStream.EndRead(ar);

    // start reading again
    BeginAsyncRead();
    
    // handle data....
}

That's it!

You may not believe it, but that code is enough to open and start communicating with the Wiimote.  The rest of the code involves parsing the data sent from the Wiimote and writing properly formed data to the Wiimote.  As I said, I'm not going to go into the detail of the reports as the sites above will do a much better job than I, but one can write any command to the Wiimote simply by doing the following:

VB

mStream.Write(mBuff, 0, REPORT_LENGTH)

C#

mStream.Write(mBuff, 0, REPORT_LENGTH);

Reading is done with the async code you see above.  When a report of 22 bytes is received, the OnReadData method is called, and the data can be properly parsed and used.

Usage of the API

If you don't care about the implementation details, you've likely skipped down to this section to learn how to use the API in your own application.  The easiest way to learn how it all works is to look at the WiimoteTest application included with the source code.

First, you will need to add a reference to the WiimoteLib.dll included with the source code.  Next, you will need to pull the namespace into your application with the standard using/Imports statement.  With that in place, you can now create an instance of the Wiimote class and start using it.  Simply instantiate a new instance of the Wiimote class, setup events if you wish to use them, setup the report type of the data you want to be returned, and call the Connect method.

VB

Imports WiimoteLib

Private Sub Form1_Load(ByVal sender As Object, ByVal e As EventArgs)
    ' create a new instance of the Wiimote
    Dim wm As Wiimote = New Wiimote()

    ' setup the event to handle state changes
    AddHandler wm.WiimoteChanged, AddressOf wm_WiimoteChanged

    ' setup the event to handle insertion/removal of extensions
    AddHandler wm.WiimoteExtensionChanged, AddressOf wm_WiimoteExtensionChanged

    ' connect to the Wiimote
    wm.Connect()

    ' set the report type to return the IR sensor and accelerometer data (buttons always come back)
    wm.SetReportType(Wiimote.InputReport.IRAccel, True)
End Sub


Private Sub wm_WiimoteExtensionChanged(ByVal sender As Object, ByVal args As WiimoteExtensionChangedEventArgs)
    If args.Inserted Then
        wm.SetReportType(Wiimote.InputReport.IRExtensionAccel, True) ' return extension data
    Else
        wm.SetReportType(Wiimote.InputReport.IRAccel, True) ' back to original mode
    End If
End Sub

Private Sub wm_OnWiimoteChanged(ByVal sender As Object, ByVal args As WiimoteChangedEventArgs)
    ' current state information
    Dim ws As WiimoteState = args.WiimoteState

    ' write out the state of the A button    
    Debug.WriteLine(ws.ButtonState.A)
End Sub

C#

using WiimoteLib;

private void Form1_Load(object sender, EventArgs e)
{
    // create a new instance of the Wiimote
    Wiimote wm = new Wiimote();
    
    // setup the event to handle state changes
    wm.WiimoteChanged += wm_WiimoteChanged;
    
    // setup the event to handle insertion/removal of extensions
    wm.WiimoteExtensionChanged += wm_WiimoteExtensionChanged;
    
    // connect to the Wiimote
    wm.Connect();
    
    // set the report type to return the IR sensor and accelerometer data (buttons always come back)
    wm.SetReportType(Wiimote.InputReport.IRAccel, true);
}


void wm_WiimoteExtensionChanged(object sender, WiimoteExtensionChangedEventArgs args)
{
    if(args.Inserted)
        wm.SetReportType(Wiimote.InputReport.IRExtensionAccel, true);    // return extension data
    else
        wm.SetReportType(Wiimote.InputReport.IRAccel, true);            // back to original mode
}

void wm_WiimoteChanged(object sender, WiimoteChangedEventArgs args)
{
    // current state information
    WiimoteState ws = args.WiimoteState;

    // write out the state of the A button    
    Debug.WriteLine(ws.ButtonState.A);
}

If you wish to use multiple Wiimotes, instantiate a WiimoteCollection object, call the FindAllWiimotes method to initialize it, and then use each individual Wiimote object in the collection as a separate instance.

VB

Dim wc As New WiimoteCollection()
wc.FindAllWiimotes()

For Each wm As Wiimote In wc
    AddHandler wm.WiimoteChanged, AddressOf wm_WiimoteChanged
    AddHandler wm.WiimoteExtensionChanged, AddressOf wm_WiimoteExtensionChanged
    
    wm.Connect()
    wm.SetReportType(InputReport.IRAccel, True)
Next wm

C#

WiimoteCollection wc = new WiimoteCollection();
wc.FindAllWiimotes();

foreach(Wiimote wm in wc)
{
    wm.WiimoteChanged += wm_WiimoteChanged;
    wm.WiimoteExtensionChanged += wm_WiimoteExtensionChanged;
    
    wm.Connect();
    wm.SetReportType(InputReport.IRAccel, true);
}

Data can be retrieved from the API in two ways:  events and polling.  In event mode, one must subscribe to the WiimoteChanged event as shown above.  Then, when data is sent from the Wiimote to the PC, an event will be posted to your event handler for processing in your application.  If you elect to not use the event model, you may simply retrieve state information at any time from the WiimoteState property of the Wiimote class.

Report Types

The library currently supports only a handful of the many report types the Wiimote is capable of producing, however the ones that are implemented are enough to get all required data for the Wiimote and all current extensions.  Specifically, these reports are:

• Buttons - Button data only
• ButtonsAccel - Button and accelerometer data
• IRAccel - Button, accelerometer and IR data
• ButtonsExtension – Button and extension data
• ExtensionAccel - Button, accelerometer and extension data
• IRExtensionAccel - Button, accelerometer, extension and IR data

The report type can be set by calling the SetReportType method using the appropriate report type and whether or not you would like the data to be sent continuously or only when the state of the controller has changed.

Extensions

There are currently three Wii extensions supported by the library: the Nunchuk, the Classic Controller, and the Guitar Hero controller.  If you wish to use these, you must setup an event handler for the WiimoteExtensionChanged event.  When this event is called, you can check the event argument to determine if an extension was inserted or removed, and what type extension was inserted.  In this event handler, you will need to change the report type with the SetReportType method to one that supports extension data, otherwise the data will not be returned to you.

If you are using a strictly polled method, you may also check the Extension and ExtensionType parameters of the WiimoteState property to determine when an extension has been inserted or removed.

The Wii Fit Balance Board will show up as a Wiimote controller with an extension attached.  The report type is set internally and any attempt to set a new report type on this device will be ignored.  The power button maps to the Wiimote's A button, and the LED maps to the Wiimote's LED1.  The rest of the Wiimote's properties are ignored.  The remainder of the balance board information can be found in the BalanceBoard struct inside the WiimoteState object.

State Information

The heart of the entire library is contained in the WiimoteState object.  Check the download's included .chm help file for the full set of properties available. 

To Do...

At the moment, usage of the speaker on the Wiimote is not supported.  I will likely add this in a future update.  Additionally, there are several report types which are not implemented, however the reports which are implemented return all necessary information.  I would also like to add some "higher level" functionality that will return pitch/roll angles, mouse cursor position for the IR sensor, etc.  Check this article and my blog for information on updates to the library.

Conclusion

And there we have it.  A fully functional managed library for the Wiimote.  Give it a try and integrate it into your existing applications or build something new!  I'm looking forward to some creative uses of the library...

If you have any issues using the library, or have any feature requests (other than sound), please don't hesitate to contact me directly or post a message in my forum dedicated to the project.

Enjoy!

Links

• WiiLi Wiki
• WiiBrew Wiki
• Making USB C# Friendly
• P/Invoke Wiki
• Applications Using This Library

Bio

Brian is a Microsoft C# MVP who has been actively developing in .NET since its early betas in 2000, and who has been developing solutions using Microsoft technologies and platforms for even longer. Along with .NET, Brian is particularly skilled in the languages of C, C++ and assembly language for a variety of CPUs. He is also well-versed in a wide variety of technologies including web development, document imaging, GIS, graphics, game development, and hardware interfacing. Brian has a strong background in developing applications for the health-care industry, as well as developing solutions for portable devices, such as tablet PCs and PDAs. Additionally, Brian has co-authored the book "Debugging ASP.NET" published by New Riders, and is currently co-authoring a book titled "10 Coding4Fun Projects with .NET for Programmers, Hobbyists, and Game Developers" to be published by O'Reilly in late 2008. Brian also writes for MSDN's Coding4Fun website, contributing articles on a monthly basis.

Summary: In this installment of "Some Assembly Required" column, Scott Hanselman borrows Travis Illig's CueCat BarCode scanner and creates a plugin for Windows Live Writer than lets him blog more easily about books he's reading. We decode the bar code info, change UPCs into ISBNs, call Amazon's Web Service via REST and integrate with Windows Live Writer all in one article. Whew!

The CueCat BarCode Scanner

Here's what Wikipedia has to say about the CueCat BarCode scanner:

The :CueCat is a cat-themed handheld barcode reader developed in the late 1990s by the now-defunct DigitalConvergence Corporation, which connected to computers using the PS/2 keyboard port and later USB. The :CueCat enabled users to link to an Internet URL by scanning a barcode appearing in an article, catalog or on some other printed matter. In this way a user could be directed to a web page containing related information. The system that supported this functionality is no longer in operation.

Ah, nothing like the smell of obsolete hardware to get me thinking. You can get CueCats all over, there are millions of them, including on ebay. I asked around and my buddy Travis Illig loaned me his. It's ripe for hacking.

Decoding the CueCat Output

The CueCat speaks a funky encoded format that it "types" as if it were a keyboard. For all intents, it IS a keyboard as the PS2 version we're using is installed in series with your existing (if you have one) keyboard.

For example, my copy of Neil Gaiman's excellent "Stardust" produces this output:

.C3nZC3nZC3nZE3r0Chr3CNnY.cGf2.ENr7C3n1C3PWD3rYCxzYChnZ.

Which can be broken up into a series of tokens, some are one character and some are two. The periods are delimiters, so we'll toss them for now.

 C3 n Z C3 n Z C3 n Z E3 r 0 Ch r 3 CN n Y EN r 7 C3 n 1 C3 P W D3 r Y Cx z Y Ch n Z

The first part is the serial number of CueCat itself, which freaked a log of privacy folks out in the past. The scanned barcode is actually the last part after the third . delimiter:

EN r 7 C3 n 1 C3 P W D3 r Y Cx z Y Ch n Z

Each of these tokens is a key to a lookup table. The encoding is pretty lame, and very arbitrary, but it confounded folks for a few hours at least. There's dozens of implementations of this decoding to be found on the Web, although none in .NET, which is where we come in. Here's a resource page that includes several implementations in C, Perl and Python, http://www.beau.lib.la.us/~jmorris/linux/cuecat/ but it hasn't been updated in at least six years. There's also some very old but interesting discussion here http://cexx.org/cuecat.htm.

The lookup table looks like this (a C# example):

See how some tokens are two chars and some are one? We'll run through the tokens testing to see if they are in each dictionary and creating the result. For my Gaiman book, we end up with:

978006093471251300

Now what do we do with it?

UPCs and ISBN

Universal Product Codes or UPCs are different from ISBNs. This page at the Book-Scanning Project gave me the algorithm I needed to convert a UPC code into an ISBN that I could use to talk to Amazon.com. Most books have a bar code with 13 digits starting with "978." Turns out that the first 3 digits of a UPC are the country of origin, but "978" is "Bookland." Apparently the UPC number for a book is called the "Bookland EAN."

Here's the Javascript taken from http://isbn.nu.

if (indexisbn.indexOf("978") == 0) {
   isbn = isbn.substr(3,9);
   var xsum = 0;
   var add = 0;
   var i = 0;
   for (i = 0; i < 9; i++) {
        add = isbn.substr(i,1);
        xsum += (10 - i) * add;
   }
   xsum %= 11;
   xsum = 11 - xsum;
   if (xsum == 10) { xsum = "X"; }
   if (xsum == 11) { xsum = "0"; }
   isbn += xsum;
}

Now, converted into C#

So, my Gaiman UPC of 978006093471251300 turns into an ISBN of 0060934719. This ISBN can be used to talk to Amazon. Note that the source code to this article includes both C# and VB.

Calling Amazon's REST Web Service

Amazon.com has a rich Web Services platform that includes formal SOAP-based Web Services, but for our project, their HTTP GET-based system will work nicely. I visited the Amazon Web Services Home Page and applied for a free Amazon Web Service Developer Key. You'll need to do the same if you want to get the code to work. Amazon has some create documentation on their site that makes this really easy.

I'll make an HTTP GET call to Amazon like this (broken up for readability):

http://webservices.amazon.com/onca/xml?
Service=AWSECommerceService&
AWSAccessKeyId=YOURAWSKEYHERE&
Operation=ItemLookup&
ItemId=0060934719&
IdType=ASIN

This HTTP GET will result in an XML document as seen in this screenshot.

Ah! Just the kind of information I was looking for.

A CueCatConsole

Before we integrate with Windows Live Writer, let's make a little test console to make sure this all works together. I put the CueCatDecoder into a small reusable assembly and called it from the Console Application. I also pull the Amazon Web Services key from the configuration file. Note that VS.NET 2005 creates a strongly typed accessor for the "AWSAccessKeyId" automatically.

I've removed the error handling in this listing for clarity.

Notice that since the Amazon XML includes a default namespace, we have to let SelectSingleNode know about that namespace by registering a namespace prefix - in this case "def:" - that we use later in our XPath. There are many ways to query XML within the .NET Base Class Library that are faster, but for a small application like this the XmlDocument is a reasonable choice when performance doesn't matter.

Here's the output from my CueCatConsole:

Coding4Fun: CueCatDecoder by Scott Hanselman @ http://www.hanselman.com
Scan item with CueCat now or press Q then Enter to exit
.C3nZC3nZC3nZE3r0Chr3CNnY.cGen.ENr7C3T7D3f0C3vYDW.
Type: ISBN.
FOUND a Book called The Goal: A Process of Ongoing Improvement by Eliyahu M. Goldratt
Scan item with CueCat now or press Q then Enter to exit
.C3nZC3nZC3nZE3r0Chr3CNnY.cGf2.ENr7CNz6C3z6DxvXChzXD3P6.
Type: ISBN.
FOUND a Book called Programming Sudoku (Technology in Action) by Wei-Meng Lee
Scan item with CueCat now or press Q then Enter to exit
.C3nZC3nZC3nZE3r0Chr3CNnY.cGf2.ENr7C3n1C3PWD3rYCxzYChnZ.
Type: ISBN.
FOUND a Book called Stardust by Neil Gaiman
Scan item with CueCat now or press Q then Enter to exit

Fantastic. It works. Let's integrate it with Windows Live Writer, the new Offline Blogging Editor in BETA (at the time of this writing) from http://www.live.com.

Integrating with Windows Live Writer

Windows Live Writer (download) is a new desktop application that folks with blogs can use to write their blog posts offline. I use DasBlog for my blog at http://www.hanselman.com, and since DasBlog supports the standard MetaWebLog API also supported by Live Writer, I was in business right away.

Windows Live Writer also has an SDK that supports "Content Source Plugins" to extend the capabilities of the application and let folks insert new kinds of data. That's exactly what I want to do! I'd like to click "Blog about Book" and have the book title and author appear linked with an image of the book cover. I'd also like have the Amazon.com URL to include my Amazon Associates ID so I could make a few cents if someone clicks my link and buys the book.

Writing a plugin for Windows Live Writer couldn't be easier. The SDK includes step-by-step instructions. I created a class that derived from WindowsLive.Writer.Api.ContentSource and overrode the CreateContent method, returning my new snippet of HTML in the newContent string variable.

Here's the gist of the source, again with error handling removed for clarity. VB is included in the download.

Notice the crazy Amazon.com URL. Most folks with blogs become Amazon Associates to make a little money to pay for hosting (you won't get rich doing this, believe me). The Amazon Associates site includes link builders that create these complex snippets of HTML. I selected the large image, basic HTML option, then inserted String.Format tokens like {0} in the appropriate places.

Installation is simple, just copy Coding4Fun.AmazonLiveWriterPlugin.dll and Coding4Fun.CueCatDecoder.dll to C:\Program Files\Windows Live Writer\Plugins and they will be automatically detected.

Our plugin appears automatically in the Windows Live Writer sidebar. Notice the text and the icon was picked up from the Plugin class attributes. My Amazon Web Services ID and Associates ID are both stored internally by an Options bag managed by Windows Live Writer, making management of preferences very easy for the plugin developer. Once the CueCat data is scanned, we decode it and make the HTTP GET Web Services call using our external CueCat assembly created earlier.

Conclusion

There's things that could be extended, added, and improved on with this project. Here are some ideas to get you started:

• Include support for CDs and other products available on Amazon.
• Add support for more web-based databases with UPC codes.
• Support different Amazon link styles. 
• Create a system to print out product tags for a Garage Sale.

Have fun and have no fear when faced with the words - Some Assembly Required!

Scott Hanselman is the Chief Architect at the Corillian Corporation, an eFinance enabler. He has thirteen years experience developing software in C, C++, VB, COM, and most recently in VB.NET and C#. Scott is proud to be both a Microsoft RD and Architecture MVP. He is co-author of Professional ASP.NET 2.0 with Bill Evjen, available on BookPool.com and Amazon. His thoughts on the Zen of .NET, Programming and Web Services can be found on his blog at http://www.computerzen.com. He thanks his wife and son Zenzo for indulging him in these hobbies!

Scott Hanselman

Summary: In this installment of the "Some Assembly Required" column, Scott Hanselman creates a Family Fingerprint Manager using .NET 2.0 that interfaces with the Microsoft Fingerprint Reader and the GrFinger SDK from Griaule.

The Microsoft Fingerprint Reader

I love my Microsoft Fingerprint Reader and when I got it I knew I had to hack it write some software for it. Since there's already a lot of great security software out there like the Digital Persona software that Microsoft shipped with it as well as 3rd party software like GrFinger's Desktop Identity, I wanted to write something a little different.

My wife and I made a New Year's Resolution to be more prepared for emergencies and the like in the wake of Hurricane Katrina. I figured, since we've moved all our identification and important papers into the a safety deposit box, why not a copy of our fingerprints as well?

There are do-it-yourself-at-home fingerprinting kits one can buy, but I had this nice Fingerprint Reader lying around....

Interfacing with the Reader

Griaule Software out of Brazil has a nice, clean COM API SDK that abstracts away the fingerprint hardware supporting a number of readers including Microsoft's. You can download a trial of their SDK software and use it for 90 days. They also include an open-source driver for the Microsoft Fingerprint Reader so you don't have to use the Digital Persona standard stuff if you don't want to. They've got an excellent sample code section with details on how to interface with a read using VB.NET, C#, Java, VB6, even Excel.

I started by adding references to their COM APIs and generating an interop assembly as seen in the picture at left. The AxGrFinger control library includes a small control that you can drag on to your WinForms design surface that exposes a number of Fingerprint-related events like FingerDown and ImageRequired that make interfacing with the hardware incredibly easy.

I created a PictureBox on my WinForm that would display the fingerprint being acquired along with ten other PictureBoxes, one for each finger. The large picture box will appear only when an image is being acquired then disappear after we copy the large acquired image into one of the smaller "fingertip" images.

The large PictureBox is passed in to a utility class along with a reference to the AxGrFinger control. As the image is acquired a valid "Handle to a Device Context" or HDC (remember those?) is passed into the CapRawImageToHandle method on the AxGrFinger control. When that handle comes back, we just update the large picture box and it displays the newly received image. Once the image has move into the world of WinForms, it's considerably easier to manipulate using the System.Drawing APIs.

The images from the large picture box are of very high quality and are copied into the smaller fingers one at a time.

Loading and Saving the Fingerprints

There's ten fingerprints (assuming you have ten fingers) to save, but should we save them as one file or ten? I decided that folks would probably want one file per family member using File|Open and File|Save method, but they'd want to reserve the right to export all the fingers as individual (PNG) images.

I created a very simple in-memory representation of a Fingerprint Database like this. It's actually slightly more complex with custom constructors as such in the actual code, but this is the general idea. I'm going to be storing the high-quality image as a byte array rather than a System.Drawing.Bitmap or Image because a byte array is very easily serialized as XML.

Each per-person finger database, like scott.finger for example, will be saved as XML similar to the example below. The byte[] is automatically turned into a BASE64'ed string by the XmlSerializer. This might be considered inelegant by some, but disk space is cheap, and the code was fantastically simple. An easy enhancement would ZIP up the file, but the savings would only be about 30-40%.

<?xml version="1.0"?>
<Fingerprints xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <FingerImages>
    <FingerImage>
      <Image>iVBOR...big opaque BASE64'ed image</Image>
      <Finger>L1</Finger>
    </FingerImage>
</FingerPrints>

Loading the fingers back into the Form is the exact opposite of saving. We use a standard Windows File Dialog with the filter set to show only .finger files. The file is deserialized using the XmlSerializer and the images are loaded back into the appropriate PictureBox.

I used the Tag Property of the PictureBox to store the name of the finger. The name is also stored in the .finger file and used to correlate the fingers and PictureBoxes.

The export works similarly, except I save the contents of each PictureBox to the selected directory using the given name as a basefilename. For example, if the user said they wanted to export to Foo.png, I'd create 10 files with names like Foo.l1.png, Foo.r2.png, etc. You can see how easy it is to change image formats using System.Drawing.Image.

To get this sample to work, you'll need a Microsoft Fingerprint Reader (or other compatible reader) and the Griaule SDK installed. I recommend you uninstall the Digital Persona drivers that came with the Fingerprint Reader otherwise this application may compete for the same hardware if the Digital Persona software is running in the background.

Conclusion

There's a number of fun things that could be extended, added, and improved on with this project. Here are some ideas to get you started:

• Adding a PDF export feature using an open-source .NET PDF Library
• ...or export to Word using Office Automation and the Microsoft Office Interop Assemblies.
• Add security by using DPAPI or encrypt the Fingerprint files directly.
• Add compression by zipping the .finger files up programmatically.

Have fun and have no fear when faced with the words - Some Assembly Required!

If you do extend this application, be sure to release the source. Thanks again to Griaule for the use of their SDK.

Scott Hanselman is the Chief Architect at the Corillian Corporation, an eFinance enabler. He has thirteen years experience developing software in C, C++, VB, COM, and most recently in VB.NET and C#. Scott is proud to be both a Microsoft RD and Architecture MVP. He is co-author of Professional ASP.NET 2.0 with Bill Evjen, available on BookPool.com and Amazon. His thoughts on the Zen of .NET, Programming and Web Services can be found on his blog at http://www.computerzen.com.

Summary: In this Part Two of "Some Assembly Required", Scott Hanselman extends his original Microbric Viper Robot/Iguanaworks IR solution to script-enable control of the robot with Lee Holme's "PowerShell LOGO."

Background

The Microbric Viper is a robot construction kit based on a new solderless construction technique. The Viper uses a Basic Atom microcontroller that you program, shockingly, using BASIC. Modules like an Infrared Receiver, motors and switches can be fairly easily controlled as you just screw them directly into the mainboard and address them by number. Once I got my hands on a Microbric, I started trying to talk to the robot using my laptop's Infrared Port. It turns out after a number of failed attempts that Windows doesn't expose standard laptop IR ports as Serial Ports or in any way other than via the IRDA protocol. IRDA was not only overkill for this kind of communication but also a hassle to program to. What I really needed was an IR transmitter that I could talk to using an interface that was clear and clean like the Serial Port interface exposed by System.IO.Ports.

I approached the guys at Iguanaworks as they make an infrared transceiver that is addressable via a standard Serial Port. The Viper comes with a Sony Remote Control so I set off to figure out how to get the Iguanaworks IR transmitter to speak to the Viper using .NET.

There's Video of the whole thing working together posted in the Channel 9 Forums Screencast topic as seen in the screenshot above.

Remote Scripting the Microbric Viper Robot

I used the custom IR Serial Port and created a console application that let me control the Microbric Viper using the keyboard of my computer. Now I'll create an application that'll let me control the Viper with script. What better script to use than Windows PowerShell?

Rather than just writing some PowerShell script that sends IR commands, I thought it'd be more interesting to host PowerShell within a WinForms application and write script that controls both the Viper and an an on screen "turtle," just like the LOGO Programming Language you may have learned in school.

Hosting Windows PowerShell

Hosting PowerShell within your own WinForms application is amazingly easy. Almost embarrassingly easy. Since I wanted a LOGO-like experience, I started by calling Lee Holmes who has a deeply technical blog dedicated to PowerShell. He happily let me extend his existing application to include support for the Microbric Viper.

Lee's application is a simple WinForms project with an Image control and a textbox. There's a .NET object called turtle that you can think of as the "pen." The turtle has methods like Forward, Left, Right, etc.

The turtle can be controlled on the Actions tab via a series of links. Moving the turtle forward 10 pixels is as trivial as:

turtle.Forward(10);

Controlling the turtle via the UI is simple enough, but scripting the turtle is the interesting part.

It's very significant to note that PowerShell is much more than a shell. It's not a shell at all at its heart - It's a UI-non-specific "runspace" for executing commands. PowerShell doesn't need to be seen in order to do its work. You can host PowerShell in any of your applications and instantly script enable them.

The first step is adding a .NET Assembly reference to System.Management.Automation. Within this assembly we'll be focused on the System.Management.Automation.Runspaces namespace.

We create a Runspace where our script will run like this:

runspace = RunspaceFactory.CreateRunspace();
runspace.Open();

Runspaces host Pipelines of commands strung together. We can create a pipeline within our Runspace and pass in the contents of the scriptText textbox from the main WinForm.

Pipeline pipeline = runspace.CreatePipeline(scriptText.Text);

The pipeline can be executed with

pipeline.Invoke()

But we need to let PowerShell have access to an instance of our Turtle class so that it can call methods like turtle.Forward(). You can pass any object from your application into the PowerShell runspace like this:

   1:  runspace = RunspaceFactory.CreateRunspace();

   2:  runspace.Open();

   3:  runspace.SessionStateProxy.SetVariable("turtle", turtle);

   4:  Pipeline pipeline = runspace.CreatePipeline(scriptText.Text);

   5:  pipeline.Invoke();

   6:  runspace.Close();

In this snippet an instance of the turtle class is passed into the SetVariable method along with the string "turtle." The string is the name of the PowerShell variable you want. If we'd passed in "foo",turtle then there'd be an instance of a turtle in a PowerShell variable accessible as "$foo."

Now that I've got access to the turtle from PowerShell, I could draw a square in PowerShell script however I like:

   1:  $turtle.Reset() 

   2:   

   3:  $a = 30

   4:  $turtle.Forward($a)

   5:  $turtle.Right(90) 

   6:   

   7:  $turtle.Forward($a)

   8:  $turtle.Right(90) 

   9:   

  10:  $turtle.Forward($a)

  11:  $turtle.Right(90) 

  12:   

  13:  $turtle.Forward($a)

  14:  $turtle.Right(90)

At this point, I've got full control of the turtle, so I can write whatever script I like. If you're going to host PowerShell in your own application be sure to consider the security ramifications. Someone could certainly include "dir" or "format c:" in the script, so you might consider a whitelist of allowed commands.

Extending the Turtle

In the current version on Lee's website the turtle doesn't update as he draws. Instead the image is updated after the turtle has finished. Changing this won't make our robot integration work any better, but it would certainly be nice to see the robot move in the real world as it draws on screen.

To solve this we need to understand the relationship between our application and the PowerShell runspace we're hosting. When we pass the turtle instance in PowerShell, that instance is being accessed on another thread, in the "PowerShell world."

Since the PowerShell script is going to be accessing the turtle instance, and the turtle instance will be drawing to a Canvas object that we passed in, we'll need a way to refresh the WinForm's image. First we need to invoke the PowerShell pipeline asynchronously. The pipeline will fire off and immediately return control to our application. It will continue executing, accessing the turtle, while a Timer control refreshes the Image control on the WinForm every 1/4 second or so.

The turtle class is changed to a ViperTurtle class and we'll add the IR class from the last article. When the turtle moves, we'll not only draw to the canvas but also send Viper-specific IR codes to move the robot.

Problems with Timing

I had a few timing-related problem with this project.

First, since the IR transmitter is going as fast as I can send signals to it, I had to add a "WaitASec" method that sleeps for a a third of a second. The commands were coming so fast that the robot would get "stuck" as if a real person was pressing buttons too fast on a remote control. If a Forward command is followed by a Left command too quickly, the robot thinks that the Forward command is continuing...as if the button were being held down. If we wait a fraction of a second for the IR beam to shut off, it works much better.

The second issue was that the commands the Viper understands are relative - like "turn left." The longer we "hold down," or continue to send, a command, the longer the robot turns. However, the LOGO-like PowerShell script thinks in absolutes, like "left(90)" where 90 means ninety degrees exactly. If the robot skids or slides on the floor, it can also throw the turns off. I could, and can still, deal with this in a number of ways. First, I could create new IR commands for the Viper to respond to and program its firmware with these custom commands, like "TurnLeft5Degrees" or even at 1 or 2 degree increments. Then I'd need to calculate the circumference of the wheels on the robot and do the math backwards in order to find the correct number of milliseconds to pulse the Viper's onboard motors to turn those wheels. Whew!

Or, I could just build in a "fudge factor" and get it working by trial and error. This is a classic software engineering problem, isn't it? The amount of effort required to get 100% perfect turning weighed against how long it would take to get it working 85-90%.

I came up with these two constant values that worked pretty well for me. You might need to try different values based on the kind of surface you're using, or if you've built your own IR transmitter.

   1:  const double VIPERADJUST = 0.35;

   2:  const double VIPERADJUSTTURN = 0.06;

   3:   

   4:  public void Left(int degrees)

   5:  {

   6:      direction = (direction + degrees) % 360;

   7:      WaitASec();

   8:      ir.Send(LEFT, (int)(VIPERADJUSTTURN * degrees));

   9:  }

The IR's send method takes a Repeat value as its second parameter. This indicates how many times the IR command should repeat. I found that multiplying the number of degrees I wanted the robot to turn by a value of 0.06 would get me the correct turn. However, I can't send fractional packets, so I'm forced to lose precision as I cast the result of the double calculation down to an int. So a 90 degree turn becomes 5.4, which becomes the left command sent via IR 5 times. In practice it works pretty well, though. If I were to hook a pen up to the Viper in the future and create PenUp and PenDown commands in its firmware, I'd want to get tighter, more accurate turning.

Again, here's Video of the whole thing working together posted in the Channel 9 Forums Screencast topic as seen in the screenshot above.

Purchasing these Parts

The Microbric Viper can be ordered online in North America, check out www.microbric.com for North American distributors. It's only US$89 at Saelig and CAD$99 at RobotShop. They have a number of educational robots that can be assembled by kids of all ages and skill levels. They're great for the classroom, and include projects like Sumo Robots, and a line-following bot, as well as a Spiderbot that climbs rope - all from the same kit.

You can order the IR Transmitter/Receiver from IguanaWorks. The serial version works on Windows or Linux, and there's a Linux USB version. It's not just a Transmitter, but also a learning receiver that works with WinLIRC and turns your computer into a learning remote control and can be used for nearly any project that utilizes IR.

Conclusion

As with all my projects, there's things that could be extended, added, and improved on with this project. Here are some ideas to get you started.

• Extend the IR program running inside the Viper to support other commands. 
• Integrate the Viper with the Microsoft Robotics Toolkit.
• Use the IR class to control your home theater and television. 
• Change this program to control a Roomba, or any IR-controllable robot.

Have fun and have no fear when faced with the words - Some Assembly Required!

Scott Hanselman is the Chief Architect at the Corillian Corporation, an eFinance enabler. He has thirteen years experience developing software in C, C++, VB, COM, and most recently in VB.NET and C#. Scott is proud to be both a Microsoft RD and Architecture MVP. He is co-author of Professional ASP.NET 2.0 with Bill Evjen, available on BookPool.com and Amazon. His thoughts on the Zen of .NET, Programming and Web Services can be found on his blog at http://www.computerzen.com. He thanks his wife and Zenzo for indulging him in these hobbies!

Control a remote controlled car with your computer using Microsoft Robotics Studio. Then, add a wireless IP camera for stealthy remote operation.

Brian Peek ASPSOFT, Inc. Difficulty: Advanced Time Required: 3-6 hours Cost: $50-$100 Software: Visual Basic or Visual C# Express Editions, Microsoft Robotics Studio, Phidgets .NET and MSRS libraries Hardware: A remote-controlled car, PhidgetInterfaceKit 0/0/4 OR PhidgetInterfaceKit 0/16/16, Airlink101 AIC-250W wireless camera (optional), Xbox 360 Controller for Windows (optional), batteries, wire, and solder Download: Download   In 2005, we at ASPSOFT built the world's first .NET-powered battlebot known as The Finalizer. Since then, many robotics enthusiasts and hobbyists have become intrigued with the idea of a .NET-controlled robot, but don't have the cash or hardware required to build something as elaborate as The Finalizer. With that in mind, I set out to create a very simple and very cheap computer controlled vehicle that anyone with a couple bucks and a computer could build.

What You Will Need

• A remote-controlled vehicle with a digital controller. That is, a controller that uses switches for forward/backward/left/right, not potentiometers. The car used in this article is a vehicle from the “Dub City” collection by Jada Toys.
• PhidgetInterfaceKit 0/0/4 – a USB-controlled board with 4 relays, OR PhidgetInterfaceKit 0/16/16, - USB-controlled digital IO boards (if you need to control more than 4 controls on your remote controlled car)
• A soldering iron and some solder
• Some spare wire
• Microsoft Visual C# or Visual Basic 2005 Express Edition
• Microsoft Robotics Studio
• Optional: WiFi-enabled webcam. This article uses the Airlink101 AIC-250W wireless camera.
• Optional: 4 AA battery holder (6V total) to power the webcam. I used a battery holder from Radio Shack with part number 270-391A.
• Optional: DC Power Plug to connect to AA battery holder above. The size required is 5.5mm outer diameter and 2.5mm inner diameter. The Radio Shack “Size N” connector with part number 274-1573 is the one I chose.
• Optional: Xbox 360 Controller for Windows

The Car

First, we will modify the remote controlled car so that it can be interfaced with the Phidget controller. The easiest way to accomplish this is to modify the hand-held controlling unit to receive input from the Phidget controller instead of a human driver.

Start by unscrewing all screws on the back of the unit and then open up the controller.

If you have chosen the right type of controller, that is, one with digital inputs, you will see that the joysticks simply push down on contacts on the board, closing the circuit for forward/backward/left/right. We will be wiring these contacts to our Phidget board so when the relays/IO ports open and close, they will be closing and opening the circuits.

To do this, cut 8 equal lengths of wire of about 6 inches. Strip off a bit of insulation from each end.

Next, solder one end of each wire to each contact of the controller. You will need to connect one wire to both the ground and the active points of the contact as shown in the picture.

When all wires are soldered, insert the opposite ends into the screw terminals of the Phidget interface board.

Phidget 0/0/4 Interface Kit

You'll notice on the Phidget board that there are 3 terminals per relay, labeled NO, XC, and NC, where “X” is a number from 0 to 3. These stand for “Normally Open”, “Relay X Common”, and “Normally Closed”. Since our circuits are normally open and closed when you press the joystick, you will want to connect one wire from each contact to the NO and the XC terminals of each relay. Wire the relays as follows:

0 – Forward

1 – Backward

2 – Left

3 - Right

Phidget 0/16/16 Interface Kit

This board is divided into two sections and is labeled as such: Inputs and Outputs.  On the Output side, there are 16 outputs, numbered 0 through 15.  There are also 8 common ground terminals labeled "G".  As with the above board, you will want to connect one wire from each contact to a numbered output, and one to a common ground contact.  Wire them using the same numbering scheme as above.

The Camera

If you decide to add the network camera to the project continue reading. If you choose to skip the camera, move on to the next section.

The camera will need to be battery powered. The camera I used in this project is the Airlink101 AIC-250W. This camera requires 5V to operate. Getting 5V out of a series of standard batteries isn't easy without a voltage regulator, so I cheated and decided to try pumping 6V (4 AA batteries at 1.5V each) to the device. I have run the cam for over 2 hours on one set of batteries with no ill effects.

To easily connect the batteries to the camera, I used a 4 AA battery holder and a power connector that matched the size of the connector on the included AC power supply. As stated above, the size required is a 5.5mm outer diameter with a 2.5mm inner diameter.

On the back of the camera, above the power connector, there is a symbol showing the positive and negative connections for the power connector.

The symbol shows that the negative terminal is the outside ring and the positive terminal is the inside. With that in mind, we will solder the black wire of the holder to the outside connector, and the red to the ring connector. The wires from the battery holder can be easily soldered onto the power connector as shown.

When all is said and done, you will have a battery holder with a DC power end that can be plugged directly into the camera. Pop 4 AA batteries into the holder and you should see the Power light on the camera turn on. I chose to mount the battery pack to the top of the camera using some double-sided tape as shown.

Finally, mount the camera to the vehicle with some double-sided tape or velcro.

The Software

Now that the hardware is complete, we need to write the software to control the car. First, the required software must be installed.  Install the software in the following order:

1. Microsoft C# or Visual Basic 2005 Express Edition
2. Microsoft Robotics Studio
3. Phidgets libraries
4. Phidgets Services for Microsoft Robotics Studio

Once all of the software is installed, the files phidget21.dll and Phidget21.NET.dll must be copied from the \Program Files\Phidgets directory to the bin directory of your MSRS installation.  If this is not done, the Phidget services will all fail to start when requested, causing you as much pain and torment as it caused me.

The Code

Before we get started, it is essential that you have some familiarity with Microsoft Robotics Studio. Please read through the included help file as well as the base tutorials. I have also included several links at the bottom of this article with even more information on the internals of MSRS.  The documentation and tutorials will be your best guidance for understanding what we are about to build.

At its core, the MSRS runtime aids in developing applications where concurrency and performance are key issues. In a robot where there are multiple sensors receiving input that needs to be concurrently handled, MSRS excels.  While our remote controlled car may not fit into that mold exactly, it is a great way to delve into Robotics Studio and learn about what it has to offer.

To start, we must first create a service for our car. This can be done by opening the Microsoft Robotics Studio Command Prompt from the Microsoft Robotics Studio (1.0) program group on the Start menu.

To generate the template for our service, the command dssnewservice.exe is used. Change to the directory where you wish to create your project and type one of the following commands, depending on the language you wish to use, to create a new service named “RCCar”:

C#:

dssnewservice /service:RCCar

VB

dssnewservice /language:VB /service:RCCar

This will generate a folder with several items, including project and solution files.  Note that it will lower-case RCCar to Rccar in various places.  This is normal.

Open the generated RCCar.sln file in Microsoft C# 2005 Express Edition or Microsoft Visual Basic 2005 Express Edition. You will see that several files were generated.

RCCar.cs/.vb – Contains actual implementation of the RCCar service

RCCarTypes.cs/.vb – Contains list of messages and default state object

RCCar.manifest.xml – A description of the services started when the application is run

Take some time to review the base generated code. This will help as we move forward.

Open the RCCar.sln file to start working.  The first thing we will need to do is access the service which controls our Phidget Interface Kit. Robotics Studio includes a built-in service for this device. To access it, first, set a reference to the PhidgetBoards.Y2006.M08.proxy namespace. This can be done by right-clicking on the project name in the Solution Explorer and selecting Add Reference from the context menu. When the Add Reference dialog appears, select PhidgetBoards.Y2006.M08.proxy and click the OK button.

To use the contents of this library in our code, we need to import the library.  This can be done by adding the following line to the Rccar class.

C#

using phidgetinterfacekit = Phidgets.Robotics.Services.PhidgetInterfaceKitBoards.Proxy;
using phidgetcommon = Phidgets.Robotics.Services.Proxy;

VB

Imports phidgetinterfacekit = Phidgets.Robotics.Services.PhidgetInterfaceKitBoards.Proxy
Imports phidgetcommon = Phidgets.Robotics.Services.Proxy

Next, we need to setup a partner relationship with the PhidgetInterfaceKit service and declare a port on which to communicate with it. At the top of the RccarService class implementation, add the following code to the service:

C#

[Partner("PhidgetInterfaceKit", Contract = phidgetinterfacekit.Contract.Identifier,
CreationPolicy = PartnerCreationPolicy.UseExistingOrCreate)]
private phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations _ikPort = 
    new phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations();

VB

<Partner("PhidgetInterfaceKit", Contract:=phidgetinterfacekit.Contract.Identifier, _
CreationPolicy:=PartnerCreationPolicy.UseExistingOrCreate)> _
Private _ikPort As New phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations()

Next, we will setup a few items that will be required to talk to the Phidget controller.  First, we must subscribe to the PhidgetInterfaceKit service.  We will be subscribing to additional services later on, so create a SubscribeToServices method in the RccarService class and then call this method from the bottom of the Start method:

C#

private void SubscribeToServices()
{
    _ikPort.SelectiveSubscribe(null, 
        new phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations());
}

VB

Private Sub SubscribeToServices()
    _ikPort.SelectiveSubscribe(Nothing, New phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations())
End Sub

This will subscribe to the PhidgetInterfaceKit service without requesting any notifications on events, as they will not be needed for this project.

Next, create an enumeration above the definition for the RccarService class for the possible directions:

C#

public enum Direction
{
    Forward = 0,
    Backward,
    Left,
    Right,
    None
}

VB

Public Enum Direction
    Forward = 0
    Backward
    Left
    Right
    None
End Enum

The value of each item in the enumeration will correspond to the relay used for that direction as described above.  Next, we will write a method named SetOutput to communicate with the Phidget Interface Kit and enable or disable the output relay as appropriate:

C#

public void SetOutput(Direction dir, bool enabled)
{
    // if no direction sent, return
    if(dir == Direction.None)
        return;

    // create a new request to send to the Phidgets service
    phidgetcommon.SetOutputRequestType req = new phidgetcommon.SetOutputRequestType();

    // assign the index and state (true/false)
    req.Index = (int)dir;
    req.State = enabled;

    // send the request to the port
    _ikPort.SetOutput(req);
}

VB

Public Sub SetOutput(ByVal dir As Direction, ByVal enabled As Boolean)
    ' if no direction sent, return
    If dir = Direction.None Then Return

    ' create a new request to send to the Phidgets service
    Dim req As New phidgetcommon.SetOutputRequestType

    ' assign the index and state (true/false)
    req.Index = dir
    req.State = enabled

    ' send the request to the port
    _ikPort.SetOutput(req)
End Sub

The SetOutput method creates a new SetOutputRequestType request object and fills in the appropriate Index and State. The Index refers to the index number of the relay to change, and the State property is a boolean of whether the relay should be opened or closed.

Next, we will need a user interface to interact with to control the robot. Right-click on the RCCar project and select Add -> Windows Form from the context menu. Name this form RemoteControlForm.cs/.vb.

<NOTE FOR VB USERS>

Due to an oddity in the dssproxy application which runs during the compilation process, one addition must be made to the form created in the Visual Basic project.  To make this change, first click on the Show all files button on the Solution Explorer toolbar as shown:

Once that is done, you should see a "+" next to the RemoteControlForm.vb file.  First, right-click on the RemoteControlForm.vb file and select View Code.  Add the following namespace definition around the the class definition as shown:

Namespace Robotics.Rccarvb
    Public Class RemoteControlForm

    End Class
End Namespace

Next, open the RemoteControlForm.Designer.vb file using the same method above and add the same namespace definition around the class implementation.  Once this is done, verify that the project compile and continue on.

</NOTE FOR VB USERS>

For the moment, we will setup the form with four buttons (up, down, left and right) and control the vehicle with those. Drag four buttons onto the form and label them appropriately. You can easily get arrow glyphs on the buttons by using the Marlett font and setting the Text property of each button to one of the following:

3 = Left

4 = Right

5 = Up

6 = Down

Name the buttons btnForward, btnBackward, btnLeft, and btnRight.  In an effort to make things easy for determining which button is pressed, assign the values 0, 1, 2, and 3 to the Tag property of each button to match the enumeration created above.  So, the Tag property of the forward button would be 0, backward would be 1, and so on.  When you are done, you will have a form similar to the following:

This form will need to call the SetOutput method we created in our service above.  Therefore, it will need an instance of the RccarService available to it.  To easily accomplish this, the constructor for the form will be written to accept the current instance of the service and store it in a member variable for later use.  Add the following code to RemoteControlForm.cs/.vb to facilitate this:

C#

RccarService _service = null;

public RemoteControlForm(RccarService service)
{
    InitializeComponent();

    _service = service;
}

VB

Private _service As RccarService

Public Sub New(ByVal service As RccarService)

    ' This call is required by the Windows Form Designer.
    InitializeComponent()

    ' Add any initialization after the InitializeComponent() call.
    _service = service
End Sub

To properly drive the car, we need to listen for both the button being clicked (close the circuit) and the button being released (open the circuit). This can be done by hooking the MouseDown and MouseUp events on each button. Open RemoteControlForm.cs/.vb in the designer. Then, choose the events list in the properties window by clicking on the “lightning bolt” icon. This will toggle the event list into view.

Highlight all of the buttons on the form. Then, enter btn_MouseDown into the MouseDown event and btn_MouseUp in the MouseUp event. The IDE will automatically generate method stubs for each of those events.

Next, add the code for the MouseDown and MouseUp events.  If you are using VB, you will need to import the System.Windows.Forms namespace into the RemoteControlForm code.  The Tag property created above will be used from each button to determine which was clicked:

C#

private void btn_MouseDown(object sender, MouseEventArgs e)
{
    _service.SetOutput((Direction)int.Parse((sender as Button).Tag.ToString()), true);
}

private void btn_MouseUp(object sender, MouseEventArgs e)
{
    _service.SetOutput((Direction)int.Parse((sender as Button).Tag.ToString()), false);
}

VB

Imports System.Windows.Forms

...

Private Sub btn_MouseDown(ByVal sender As System.Object, ByVal e As System.Windows.Forms.MouseEventArgs) Handles btnRight.MouseDown, btnLeft.MouseDown, btnForward.MouseDown, btnBackward.MouseDown
    _service.SetOutput(CType(Integer.Parse(CType(sender, Button).Tag.ToString()), Direction), True)
End Sub

Private Sub btn_MouseUp(ByVal sender As System.Object, ByVal e As System.Windows.Forms.MouseEventArgs) Handles btnRight.MouseUp, btnLeft.MouseUp, btnForward.MouseUp, btnBackward.MouseUp
    _service.SetOutput(CType(Integer.Parse(CType(sender, Button).Tag.ToString()), Direction), False)
End Sub

The events simply look at the button being sent in as the sender parameter, convert the Tag to an entry in the Direction enumeration, and set the associated relay using the SetOutput method.

Now that our base user interface is setup, we need to launch the form from our service. To do this, we need to add reference to the Ccr.Adapters.WinForms namespace. This can be done in the same way as we added the reference to the Phidget library. Once that is done, back in Rccar.cs/.vb, bring the WinForms library into the class by adding the following using statement:

C#

using Microsoft.Ccr.Adapters.WinForms;

VB

Imports Microsoft.Ccr.Adapters.WinForms

To actually launch the form, add the following line to the bottom of the Start method, below the previous call to SubscribeToServices:

C#

WinFormsServicePort.Post(new RunForm(StartForm));

VB

WinFormsServicePort.Post(New RunForm(AddressOf StartForm))

This will launch the method StartForm which actually creates the form. Add a member variable to the class to store the instance of the created form and add the StartMethod form as follows:

C#

RemoteControlForm _remoteForm = null;

public System.Windows.Forms.Form StartForm()
{
    _remoteForm = new RemoteControlForm(this);
    return _remoteForm;
}

VB

Private _remoteForm As RemoteControlForm

Private Function StartForm() As System.Windows.Forms.Form
    _remoteForm = New RemoteControlForm(Me)
    Return _remoteForm
End Function

This will create a new instance of the form, assign it to the local member variable, and return the instance back to the calling method.

At this point, everything should be in place to actually run the project. Make sure the Phidget board is connected to the PC via the USB cable and press the F5 key to build and execute the project. If all went to plan, you should see the UI form start up and you should be able to click the arrow buttons and hear the relays click on the 0/0/4 board, or the lights flicker on the 0/16/16 board. If the car is turned on and the remote is turned on, the car will drive around based on the buttons you click.

The Keyboard

We can easily add support to drive the car with the keyboard's arrow keys. To use the arrows, we cannot use the native KeyDown event because the arrow keys are normally used to cycle through the active controls of a form. Therefore, we need to trap the key down events earlier in the message chain. To do this, we must override the ProcessCmdKey method of the Form class. Additionally, the form's KeyPreview property must be set to false.  Add the following code to the RemoteControlForm.cs/.vb file:

C#

const int WM_KEYDOWN = 0x100;

protected override bool ProcessCmdKey(ref Message msg, Keys keyData) 
{ 
    if(msg.Msg == WM_KEYDOWN) 
    {
        switch(keyData) 
        { 
            case Keys.Up: 
                _service.SetOutput(Direction.Forward, true); 
            break; 

            case Keys.Down: 
                _service.SetOutput(Direction.Backward, true); 
                break; 

            case Keys.Left: 
                _service.SetOutput(Direction.Left, true); 
                break; 

            case Keys.Right: 
                _service.SetOutput(Direction.Right, true); 
                break; 
        } 
        return true; 
    } 
    return false; 
} 

VB

Dim WM_KEYDOWN As Integer = 256

Protected Overrides Function ProcessCmdKey(ByRef msg As Message, ByVal keyData As Keys) As Boolean
    If (msg.Msg = WM_KEYDOWN) Then
        Select Case (keyData)
            Case Keys.Up
                _service.SetOutput(Direction.Forward, true)
            Case Keys.Down
                _service.SetOutput(Direction.Backward, true)
            Case Keys.Left
                _service.SetOutput(Direction.Left, true)
            Case Keys.Right
                _service.SetOutput(Direction.Right, true)
        End Select
        Return true
    End If
    Return false
End Function

This looks much like the button processors from above.

Just like the buttons, we also need a KeyUp event. Go back to the designer for the form and double-click on the KeyUp event in the properties pane, just like we did with the mouse button events. This will generate an event stub for the KeyUp event. Fill in the method with the following:

C#

private void RemoteControlForm_KeyUp(object sender, KeyEventArgs e)
{
    switch(e.KeyCode)
    { 
        case Keys.Up: 
            _service.SetOutput(Direction.Forward, false); 
        break; 

        case Keys.Down: 
            _service.SetOutput(Direction.Backward, false); 
            break; 

        case Keys.Left: 
            _service.SetOutput(Direction.Left, false); 
            break; 

        case Keys.Right: 
            _service.SetOutput(Direction.Right, false); 
            break; 
    } 
} 

VB

Private Sub RemoteControlForm_KeyUp(ByVal sender As System.Object, ByVal e As System.Windows.Forms.KeyEventArgs) Handles MyBase.KeyUp
    Select Case (e.KeyCode)
        Case Keys.Up
            _service.SetOutput(Direction.Forward, False)
        Case Keys.Down
            _service.SetOutput(Direction.Backward, False)
        Case Keys.Left
            _service.SetOutput(Direction.Left, False)
        Case Keys.Right
            _service.SetOutput(Direction.Right, False)
    End Select
End Sub

Now, if you execute the application again, you will be able to drive the car around just by pressing the arrow keys on the keyboard.

Gamepad Support