The Netduino is an open source electronics platform based on a 32-bit microcontroller running the .NET Micro Framework. The Netduino Plus is similar to the original Netduino, but adds a built-in Ethernet controller and MicroSD slot. Since the Netduino Plus can connect directly to a network, it can independently communicate with Twitter’s API without being connected to a computer.

Hardware Overview

The MakerShield is a simple prototyping shield that is compatible with the standard Arduino and Netduino boards.

In this configuration, the MQ-3 alcohol gas sensor will output an analog voltage between 0 and 3.3V to indicate the amount of alcohol detected. This output will be connected to one of the Netduino’s analog input pins and read by its ADC.

While it would possible to convert the sensor’s output to a numeric BAC level, this would require careful calibration and would be prone to error. For this project, I will use approximate value ranges to determine which of several messages should be posted to Twitter. An approximate reading will be displayed on an RGB LED.

RGB LED

The RGB LED is the primary status indicator. During normal operation, it shows the level of alcohol, represented by colors ranging from green to red.

Three transistors are used to provide power to the RGB LED. The microcontroller used on the Netduino has a relatively low current limit per IO pin (around 8 mA for most pins) so it is generally not advised to drive LEDs (which can require 20-30 mA) directly from these pins. Using a transistor (or another LED driver) helps ensure that enough power will be made available to each LED without damaging the Netduino.

This page shows some common transistor circuits, including a few "transistor as a switch" circuits. Since the RGB LED I am using has a common cathode (low side) lead, I am using PNP transistors to switch the anode (high side) of each color.

Alcohol Gas Sensor

The MQ-3 essentially has two components: a heater and an alcohol gas sensor. The heater literally applies heat to the sensor in order to provide more accurate measurements and requires a constant 5V power source. The alcohol sensor acts as a variable resistor, where the resistance decreases as the level of alcohol increases.

While the heating element requires 5V, the alcohol sensor can actually operate at a different voltage. Applying 3.3V to the sensor (V_C in the above diagram) will ensure that the output voltage (V_RL) doesn’t exceed the Netduino’s input limits.

The sensor and R_L act as a voltage divider. The datasheet for this sensor shows that the resistance across the sensor is between 2K-20K ohms, so an R_L value of 10K ohms will provide a wide voltage output range.

Other Components

The MakerShield has two LEDs that I am using to display some additional status information. The red LED indicates whether the device is ready to post to Twitter, and the Green LED indicates that a post is in progress.

I am also using the button on the MakerShield to trigger the Netduino to post a message on Twitter. You could also write code to automatically post based on level peak detection, but for simplicity I decided to just use the button as a trigger.

Code

Pin configuration, Twitter API keys, the NTP server address, and other settings are stored in a Config class (Config.cs in the project root). If you download the source code for this project from CodePlex, make sure you copy the Config.sample.cs to Config.cs and take a look through the file to confirm all the settings:

public static class Config
{
    // NTP Server
    public const string NTPServer = "time-a.nist.gov";
    public const int TimeOffset = -7;    // MakerShield LEDs
    public const Cpu.Pin RedLEDPin = Pins.GPIO_PIN_A5;
    public const Cpu.Pin GreenLEDPin = Pins.GPIO_PIN_A4;    // RGB LED
    public const Cpu.Pin RGBRedPin = Pins.GPIO_PIN_D10;
    public const Cpu.Pin RGBGreenPin = Pins.GPIO_PIN_D6;
    public const Cpu.Pin RGBBluePin = Pins.GPIO_PIN_D9;
    public const bool RGBInverted = true;    // Button
    public const Cpu.Pin ButtonPin = Pins.GPIO_PIN_A3;    // Alcohol Gas Sensor
    public const Cpu.Pin SensorPin = Pins.GPIO_PIN_A0;
    // The Netduino's 10-bit ADC has a maximum value of 1023, but the sensor may not reach the highest possible voltage.
    // Input values >= the SensorMaxValue will be treated as 100%.
    public const int SensorMaxValue = 900;    // Twitter
    public const string ConsumerKey = "YourConsumerKey";
    public const string ConsumerSecret = "YourConsumerSecret";
    public const string UserToken = "YourUserToken";
    public const string UserTokenSecret = "YourUserTokenSecret";
}

In order to post messages on Twitter, it is necessary to register an application through Twitter’s developer portal. Instructions for this can be found on the MicroTweet CodePlex Page.

Startup

When the Netduino first powers on it will wait for a few moments to let the alcohol gas sensor heat up and establish a baseline reading. The sensor’s datasheet recommends a preheat time of 24-48 hours, but the readings typically stabilize after a few minutes. During this time the RGB LED will slowly fade from blue to red to indicate that the sensor is heating up.

RGB LED

A simple RGB LED helper class is included to help make controlling the color and output of the LED easier. The class must be instantiated with three Cpu.Pin addresses, one for each color:

protected static RGBLED rgbLed = new RGBLED(Config.RGBRedPin, Config.RGBGreenPin, Config.RGBBluePin, Config.RGBInverted);

In the sample code, these pins are pulled from the Config class. The last parameter for the RGBLED constructor, invertOutput, is used to determine whether a logical high output value turns the LED on or off.

NTP

Next, the Netduino will attempt to update the system time from an NTP server. This is necessary for communication with Twitter’s OAuth API because all requests must contain an accurate timestamp. If the NTP server is unreachable, the Twitter functionality will be disabled, but it will still be possible to see the alcohol level on the RGB LED.

The code to communicate with an NTP server is included with the MicroTweet library. The UpdateTimeFromNTPServer method will contact the specified NTP server and update the Netduino’s time:

NTP.UpdateTimeFromNTPServer(Config.NTPServer);

If the time is updated successfully, a new instance of the TwitterClient class is created:

twitterClient = new TwitterClient(Config.ConsumerKey, Config.ConsumerSecret, Config.UserToken, Config.UserTokenSecret);

Measurements

Measurements from the alcohol sensor are repeatedly taken within the main program loop. The 10-bit ADC will return a value between 0 and 1023, but these values don’t necessarily correspond to the minimum/maximum readings from the alcohol sensor so some calculations are performed to convert the ADC value to a number between 0 and 100:

protected static AnalogInput SensorInput = new AnalogInput(Config.SensorPin);// ...public static int GetSensorValue()
{
    int rawValue;
    lock (SensorInput)
    {
        rawValue = SensorInput.Read();
    }
    int adjustedValue = (rawValue - SensorMinValue) * 100;
    int result = adjustedValue / (Config.SensorMaxValue - SensorMinValue);
    return result;
}

SensorMinValue is the baseline measurement established when the program was starting up. Config.SensorMaxValue is the configured "maximum" value that indicates the highest detectable alcohol level. Depending on your sensor, this maximum value may need to be adjusted.

It’s also worth noting that it can take about 30 seconds for the sensor to return to its baseline value after a measurement. This screenshot shows the sensor’s output before and after taking a reading:

In this test, the baseline measurement (with no alcohol applied to the sensor) was about 1V. When alcohol was applied to the sensor the output quickly peaked at just under 3V, and then slowly decreased once the alcohol was removed.

Posting to Twitter

Sending a tweet from a TwitterClient instance is simple:

twitterClient.SendTweet("Hello, world!");

In this project, the MakerShield’s button is used to trigger the Netduino to post a message on Twitter. This button is monitored by an instance of the InterruptPort class, which fires a C# event when the input changes on the selected pin:

protected static InterruptPort button = new InterruptPort(Config.ButtonPin, true, Port.ResistorMode.PullUp, Port.InterruptMode.InterruptEdgeLevelLow);// ...static void button_OnInterrupt(uint data1, uint data2, DateTime time)
{
    if (twitterClient == null)
        return;
    greenLED.Write(true);    int sensorValue = GetSensorValue();    string message;
    if (sensorValue < 20)
        message = "No alcohol detected";
    else if (sensorValue < 40)
        message = "Tipsy";
    else if (sensorValue < 60)
        message = "Drunk";
    else if (sensorValue < 80)
        message = "Whoa! #WINNING";
    else
        message = "x_x";    DateTime adjustedTime = DateTime.Now + new TimeSpan(Config.TimeOffset, 0, 0);    message += " at " + adjustedTime.ToString("h:mm:ss tt");    try
    {
        twitterClient.SendTweet(message);
    }
    catch (Exception e)
    {
        Debug.Print(e.ToString());
    }    greenLED.Write(false);
    button.ClearInterrupt();
}

The button event handler code in this project is relatively simple: the current level from the alcohol sensor is read, a message is generated, and it attempts to post the message to Twitter. The current time is appended to the end of the message to avoid issues with multiple identical tweets.

Conclusion

The code for this project is included in the Samples folder of the MicroTweet library. Be sure to check out the MicroTweet project page on CodePlex for the source code and more information.

MicroTweet is released under the Apache 2.0 license, so it can be freely used in your own projects with attribution.

About the Author

Matt Isenhower is a desktop and mobile application developer who primarily works with the .NET Framework. His company, Komodex Systems, is currently focused on building new and exciting apps for Windows Phone 7 and other platforms. He also has a blog and can be followed on Twitter at @mattisenhower.

Glenn shares his expertise gained at Newsgator with ISV Architect Evangelist Bruce Kyle. Glenn's book, SQL Server Hardware from Simple Talk Press provides the fundamental knowledge and resources you need to make intelligent decisions about choice, and optimal installation and configuration, of SQL Server hardware, operating system and SQL Server.

Glenn provides a few tips from his book in this video.

You find out more about the book on Amazon. See SQL Server Hardware.

Chapter Outline

1. Processors and Associated Hardware
2. The Storage Subsystem
3. Benchmarking Tools
4. Hardware Discovery
5. Operating System Selection and Configuration
6. SQL Server Version and Edition Selection
7. SQL Server Installation and Configuration

Glenn's Blog

Glenn Berry's SQL Server Performance is one of the best resources for SQL Server tips and tricks. He blogs daily exploring both the hardware and software sides of SQL.

Follow Glenn on Twitter at @GlennAlanBerry.

See his interview with John O'Donnell at Talking with NewsGator at PDC 2009.

About Most Valued Professionals (MVP)

Microsoft recognizes the best and brightest from technology communities around the world with the
Microsoft Most Valuable Professional (MVP) Award. These exceptional community leaders come from a wide range of backgrounds. They are teachers, artists, doctors, engineers, as well as technologists, who actively share their high-quality, real-world technical expertise with the community and with Microsoft.

About NewsGator

NewsGator was founded in 2004 to realize the potential of RSS (Really Simple Syndication) to transform the way people consume information. Today NewsGator's flagship product - Social Sites – is social computing software that’s built directly into Microsoft SharePoint.

Other ISV Videos

For other videos about independent software vendors (ISVs):

• Quark Extends SharePoint for Dynamic Publishing
• ProModel Adds Simulation, Visualization to Microsoft Project
• Thumb-Driven Workflow on Windows 7 Slates from Blue Dot Solutions
• Accumulus Makes Subscription Billing Easy for Windows Azure
• Azure Email-Enables Lists, Low-Cost Storage for SharePoint
• Crowd-Sourcing Public Sector App for Windows Phone, Azure
• Food Buster Game Achieves Scalability with Windows Azure

Up to Date News for ISVs and Software Developers

See US ISV Community blog.

It has gone through quite a few changes and I have expanded it to easily display MJPEG streams on a variety of platforms.  The developer just references the assembly appropriate to their platform, adds a few lines of code, and away it goes.

Usage

For those that are just interested in the usage, it's as simple as this:

1. Reference one of the following assemblies appropriate for your project:
   • MjpegProcessorSL.dll - Silverlight (Out of Browser Only!)
   • MjpegProcessorWP7.dll - Windows Phone 7 (XNA or Silverlight, performance maxes out around 320x240 @ 15fps, so set your camera settings accordingly)
   • MjpegProcessorXna4.dll - XNA 4.0 (Windows)
   • MjpegProcessor.dll - WinForms and WPF
2. Create a new MjpegDecoder object.
3. Hook up the FrameReady event.
4. In the event handler, take the Bitmap/BitmapImage and assign it to your image display control:
5. In the case of XNA, use the GetMjpegFrame method in the Update method, which will return a Texture2D you can use in your Draw method.
6. Call the ParseStream method with the Uri of the MJPEG "endpoint".

That's it!  The source code and binaries above both include projects demonstrating how to use the library on each of these platforms.  As long as you set the appropriate reference, you can just copy and paste the code in the sample to get your project running (changing the Uri, of course).

public partial class MainWindow : Window
{
    MjpegDecoder _mjpeg;

    public MainWindow()
    {
        InitializeComponent();
        _mjpeg = new MjpegDecoder();
        _mjpeg.FrameReady += mjpeg_FrameReady;
    }

    private void Start_Click(object sender, RoutedEventArgs e)
    {
        _mjpeg.ParseStream(new Uri("http://192.168.2.200/img/video.mjpeg"));
    }

    private void mjpeg_FrameReady(object sender, FrameReadyEventArgs e)
    {
        image.Source = e.BitmapImage;
    }
}

If that doesn't fit your needs, you can also access the Bitmap/BitmapImage properties directly from the MjpegDecoder object, or the CurrentFrame property, which will contain the raw JPEG data prior to being decoded.

A Word About Network/IP Cameras

I have tested this against several different cameras.  Each device has its own quirks, but all of them seem to work with this library with one exception:  several cameras will respond differently when an Internet Explorer user agent header is sent with the HTTP request.  Instead of sending down an MJPEG stream, it will send a single JPEG image as Internet Explorer does not properly support MJPEG streams.  Unfortunately, this causes the Silverlight processor to not work properly as the header cannot be changed from the Internet Explorer default.  When this happens, only a single frame will be sent, and the decoding will fail.  The only fix I have found is to use a different camera that doesn't work in this way.

What is MJPEG?

Pretty simply, it's a video format where each frame of video is sent as a separate, compressed JPEG image.  A standard HTTP request is made to a specific URL, and a multipart response is sent.  Parsing this multipart stream into separate images as they are sent results in a series of JPEG images.  The viewer displays those JPEG images as quickly as they are sent and that creates the video.  It's not a well documented format, nor is it perfectly standardized, but it does work.  For more information, see the MJPEG article on Wikipedia.

How Do I Find the MJPEG URL of My Camera?

Excellent question.  Not an excellent answer.  The user manual may mention the URL.  A quick internet search with the model number should get you a result.  Or, you can also try this company's lookup tool.

How Does It Work?

Glad you asked.  If you take a look at the project, you'll notice there isn't much code.  One single file is used with a variety of compiler directives to compile certain portions based on the platform assembly being generated.  The MjpegDecoder.cs/.vb contains the entire implementation.

First, an asynchronous request is made to the provided MJPEG URL inside the ParseStream method.  If we are in a Silverlight environment, the AllowReadStreamBuffering property must be set to false so the response returns immediately instead of being buffered.  Additionally, we need to register the http:// prefix to use the client http stack vs. the browser stack.  Finally, the request is made using the BeginGetResponse method, specifying the OnGetResponse method as the callback.  This will be called as soon as data is sent from the camera in response to our request.

public void ParseStream(Uri uri)
{
#if SILVERLIGHT
    HttpWebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);
#endif
    HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);

#if SILVERLIGHT
    // start the stream immediately
    request.AllowReadStreamBuffering = false;
#endif
    // asynchronously get a response
    request.BeginGetResponse(OnGetResponse, request);
}

OnGetResponse grabs the response headers and uses the Content-Type header to determine the boundary marker that will be sent between JPEG frames.

private void OnGetResponse(IAsyncResult asyncResult)
{
    HttpWebResponse resp;
    byte[] buff;
    byte[] imageBuffer = new byte[1024 * 1024];
    Stream s;

    // get the response
    HttpWebRequest req = (HttpWebRequest)asyncResult.AsyncState;
    resp = (HttpWebResponse)req.EndGetResponse(asyncResult);

    // find our magic boundary value
    string contentType = resp.Headers["Content-Type"];
    if(!string.IsNullOrEmpty(contentType) && !contentType.Contains("="))
        throw new Exception("Invalid content-type header.  The camera is likely not returning a proper MJPEG stream.");
    string boundary = resp.Headers["Content-Type"].Split('=')[1];
    byte[] boundaryBytes = Encoding.UTF8.GetBytes(boundary.StartsWith("--") ? boundary : "--" + boundary);
...

It then streams the response data, looks for a JPEG header marker, then reads until it finds the boundary marker, copies the data into a buffer, decodes it, passes it on to whoever wants it via an event, and then starts over.

...
    s = resp.GetResponseStream();
    BinaryReader br = new BinaryReader(s);

    _streamActive = true;

    buff = br.ReadBytes(ChunkSize);

    while (_streamActive)
    {
        int size;

        // find the JPEG header
        int imageStart = buff.Find(JpegHeader);

        if(imageStart != -1)
        {
            // copy the start of the JPEG image to the imageBuffer
            size = buff.Length - imageStart;
            Array.Copy(buff, imageStart, imageBuffer, 0, size);

            while(true)
            {
                buff = br.ReadBytes(ChunkSize);

                // find the boundary text
                int imageEnd = buff.Find(boundaryBytes);
                if(imageEnd != -1)
                {
                    // copy the remainder of the JPEG to the imageBuffer
                    Array.Copy(buff, 0, imageBuffer, size, imageEnd);
                    size += imageEnd;

                    // create a single JPEG frame
                    CurrentFrame = new byte[size];
                    Array.Copy(imageBuffer, 0, CurrentFrame, 0, size);
#if !XNA
                    ProcessFrame(CurrentFrame);
#endif
                    // copy the leftover data to the start
                    Array.Copy(buff, imageEnd, buff, 0, buff.Length - imageEnd);

                    // fill the remainder of the buffer with new data and start over
                    byte[] temp = br.ReadBytes(imageEnd);

                    Array.Copy(temp, 0, buff, buff.Length - imageEnd, temp.Length);
                    break;
                }

                // copy all of the data to the imageBuffer
                Array.Copy(buff, 0, imageBuffer, size, buff.Length);
                size += buff.Length;
            }
        }
    }
    resp.Close();
}

The ProcessFrame method seen above takes the raw byte buffer, which contains an undecoded JPEG image, and decodes it based on the environment.  However this isn't called in the case of XNA which we'll see in a moment:

private void ProcessFrame(byte[] frameBuffer)
{
#if SILVERLIGHT
    // need to get this back on the UI thread
    Deployment.Current.Dispatcher.BeginInvoke((Action)(() =>
    {
        // resets the BitmapImage to the new frame
        BitmapImage.SetSource(new MemoryStream(frameBuffer, 0, frameBuffer.Length));

        // tell whoever's listening that we have a frame to draw
        if(FrameReady != null)
            FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, BitmapImage = BitmapImage });
    }));
#endif

#if !SILVERLIGHT && !XNA
    // I assume if there's an Application.Current then we're in WPF, not WinForms
    if(Application.Current != null)
    {
        // get it on the UI thread
        Application.Current.Dispatcher.BeginInvoke((Action)(() =>
        {
            // create a new BitmapImage from the JPEG bytes
            BitmapImage = new BitmapImage();
            BitmapImage.BeginInit();
            BitmapImage.StreamSource = new MemoryStream(frameBuffer);
            BitmapImage.EndInit();

            // tell whoever's listening that we have a frame to draw
            if(FrameReady != null)
                FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, Bitmap = Bitmap, BitmapImage = BitmapImage });
        }));
    }
    else
    {
        // create a simple GDI+ happy Bitmap
        Bitmap = new Bitmap(new MemoryStream(frameBuffer));

        // tell whoever's listening that we have a frame to draw
        if(FrameReady != null)
            FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, Bitmap = Bitmap, BitmapImage = BitmapImage });
    }
#endif
}

In the case of Silverlight, the BitmapImage object has a SetSource method which takes a stream to be decoded and turned into the image.  In WPF, BitmapImage works differently.  In this case, BeginInit is called, then the StreamSource property is set to the stream of bytes, and finally EndInit is called.  In WinForms, the library will return a Bitmap object which can be initialized with the stream right in the constructor.

In the code above, I look at the Application.Current property to determine if the library is being used by a WPF project.  If that property is not null, it is assumed the library is being called from a WPF project.

When compiled as an XNA library, we have no use for a BitmapImage or a Bitmap…we need a Texture2D object.  The GetMjpegFrame method seen below is what is called by an XNA application during the Update method to pull the current frame:

public Texture2D GetMjpegFrame(GraphicsDevice graphicsDevice)
{
    // create a Texture2D from the current byte buffer
    if(CurrentFrame != null)
        return Texture2D.FromStream(graphicsDevice, new MemoryStream(CurrentFrame, 0, CurrentFrame.Length));
    return null;
}

Conclusion

And that's about it for the library.  Give it a try and please contact me with feedback or if you run into any issues.  Enjoy!

Follow Ben on Twitter or see more on the Windows Team Blog.

00:25 - Acer ICONIA - Dual screen multitouch laptop. We looked at one in the back of the booth because there was a constant crowd around the one out front. Place 5 fingers on the bottom screen to bring up a wheel launcer or 10 fingers to bring up the keyboard.
02:00 - Toshiba Satellite A665-3D - A fast i7-based gaming laptop with 3D glasses with a retail price of around $1300.
02:33 - Sony Vaio P - 8" screen that fits in a back pocket. It has available turn-by-turn GPS navigation and Verizon Broadband built in.
03:18 - HP Envy 14 - The brains inside the much-feared and arguably dangerous Coding4Fun T-Shirt cannon, the 17" version is my main PC, used to edit many of my Channel 9 videos. Why? Because it's a huge screened beauty with Beats Audio that gets a 7.1 WEI score.
04:05 - Samsung Note PC 9 Series 900X - You've never seen a PC like this before. It's one of the lightest, thinnest 13" laptops in the world. The body is Duralumin aircraft metal which feels both solid as a rock and light as a feather. Backlit keyboard, i5 processor. The screen is twice as bright as what you're probably used to, and for speed - world record holder at 12 seconds boot time. When you retire this machine you could use it to split wood. 
05:00 - ASUS Eee Slate EP121 - A slate PC that works as a multitouch device, as a Tablet PC, or with a BlueTooth keyboard. Forward facing camera for video chat and a home button that drives AeroFlip3d.
06:33 - Dell Inspiron duo - Very cool little convertible tablet that works as a laptop and with a push the screen flips around and it works as a touch tablet. It has an optional Audio Station, 2GB of RAM and will play HD movies.

Awesome job Don and thanks for telling us about it! 

Update:  Don has shared his sequence file with us!

To get started, we purchased and retrofitted a Spin Master Tron Identity Disc replica, which kept us from having to create a new physical disk, and so sidestepped the most difficult aspect of the project.

Operation Specifications

We decided that we wanted our version of the Tron Idenity Disc to have the following specifications:

Power

• Rechargeable lithium-ion battery
• LED to indicate the Tron Identity Disc is being charged
• An On/Off switch enabling the battery to disconnect from the Netduino to conserve power while the disc is being stored

Operation

• Scenario 1 - Power On
  • All LEDs turn on so we can make sure they are all working
• Scenario 2 – Animations
  • Pressing a momentary button will cycle through LED animations
• Scenario 3 – Charge Detection
  • Placing the Tron Idenity Disc on or removing the Tron Identity Disc from a charging station will raise an event handler. Placing the disc on the charger will start a pulsating animation. Removing the disc will resume the previous animation.

Hardware

• DeluxeTronIdentityDisc
• Netduino
• Dremel with Cutoff Wheel
• Soldering Iron
• Solder
• Hot Glue Gun
• Phillips Screw Driver
• Wire Strippers
• Heat Shrink
• Lithium-Ion Battery (2x)
• SPST On/Off Switch
• SPST Momentary Switch
• Prototype Board
• Blue LEDs (33x)
• MAX7219
• LM317T Adjustable Voltage Regulator
• Heat sink for TO-220 devices
• 0.1uF capacitor (2x)
• 2222 Transistor
• 1k Ohm potentiometer
• 470 Ohm resistor, ¼ watt (2x)
• 10k Ohm resistor, ¼ watt
• 2.2k Ohm resistor ¼ watt
• 1.5 Ohm resistor, 1 watt
• 1N4001 Rectifier Diode

Tron Disc Light Ring

The original DeluxeTronIdentityDisc contained six LED on the outer ring, which left much to be desired. Our modified disc includes 30 LEDs on the outer ring and two LEDs illuminating the inner arc.

To get started, we opened the disc by removing its four Phillips head screws. Next, we removed all electronics including the wires, tilt sensor, LEDs, switch, speaker, and microcontroller board, taking care to not lose the plastic momentary button, since we later used it to switch between our animations. Then, we used a Dremel with a cutoff wheel to remove any unnecessary plastic in order to make room our own electronics. See Figure 1.

Figure 1: Removed Plastic with Dremel

As shown in Figure 2, we used superglue to attach the narrow end of the momentary button to the plastic Tron ring. Then, using the Dremel, we cut a small L bracket from a scrap piece of acrylic and attached it with cyanoacrylate. We reinforced the L bracket using hot glue and applied a small amount of cyanoacrylate to the momentary switch and attached it to the L bracket. Consequently, the original plastic momentary button now activates our momentary switch.

While we had the hot glue gun out, we also attached the SPST On/Off switch and the two inner ring LEDs.

Figure 2: Mounting Momentary Switch

Next, we cut a circle from a piece of cardboard. Using a straight edge, we carefully marked the location of each LED to arrange an equally spaced circular pattern. Then, with a hobby razor knife, we cut notches in the cardboard at a slight angle to hold the LEDs. An additional benefit of the cardboard is it helps keep the anodes and cathodes from shorting out.

Figure 3: LED Nestled in Notched Cardboard

Next, we connected all of the components together per the following schematic diagrams. The MAX7219 IC is capable of controlling 64 LEDs. However, we chose to use only 30 in order to simplify of wiring. The circuit is divided into five segments, each containing six LEDs. In each segment, all the cathodes are tied together and then tied to a single segment pin on the MAX7219 and the anodes are routed back to its digit pins. There is one digit for each LED in the segment and the segments share digit pins. The Netduino controls the logic level of the digit and segment pins using “bit-banged” I2C. All that said, the schematic is the best reference as to how the LEDs were wired.

Figure 4: Netduino and MAX7219 Schematic

Charging System

The charging system is comprised of a charging circuit and a detection circuit. A lithium-ion battery inside the disc is charged from an external power source. The charging circuit, which was modified from a design by Scott Henion ofSHDesigns.org, provides a method of safe charging. The detection circuit allows the Netduino to know when it's being charged.

Charging Circuit

Figure 5: Charging Station Schematic

The LM317 is an adjustable voltage regulator that may also be used as an adjustable constant-current source. In this case, it's used as a voltage regulator. The 470 ohm resistor forms a voltage divider with the 1k ohm potentiometer and the 2.2k ohm resistor to set the output voltage at 8.4V. Since lithium-ion batteries should be charged using constant current-constant voltage, the transistor and resistor form the current limiting in the circuit. When the charge current reaches a certain threshold, the resistor's voltage drop exceeds the turn-on voltage of the transistor and the transistor starts to conduct. That takes current away from the voltage divider, dropping the voltage until the current is below the threshold. In this way, the current is limited until the voltage limit is reached; then, the current slowly drops off at a constant voltage.

Detection circuit

Figure 6: Charge Detection Schematic

The detection circuit works by applying a voltage to the disc whenever it is plugged in. When the disc is plugged in, 8.4V from the charger run through the 470 ohm resistor, then most of the current runs through the LED while a minimal amount runs through the 10k ohm resistor. The voltage to the Netduino is the voltage across the LED, equaling somewhere around 3V. When the charger is disconnected, however, the 10K ohm resistor pulls the voltage detection line to 0V. SW2 is a disconnect switch which feeds the main power to the Netduino. From there we use the Netduino's 5v power supply.

After all the connections were made we carefully placed all the wires and components inside the disc. We learned the hard way that you must pay extra attention since it's very easy to put a screw through one of your wires.

Creating a driver for the MAX7219:

To create a driver for the MAX7219, we took a little information from the data sheet. First, we created a basic class structure and initialized some static defines for the command reference as well as a few OutputPort's to hold our pin's to bit-bang. We bit-banged the protocol because the SPI implementation of the MAX7219 isn't 100% compliant with the Netduino's SPI Library:

namespace TronDisc
{
    public class max7219
    {
        // Command reference
        public byte max7219_reg_noop           = 0x00;
        public byte max7219_reg_digit0         = 0x01;
        public byte max7219_reg_digit1         = 0x02;
        public byte max7219_reg_digit2         = 0x03;
        public byte max7219_reg_digit3         = 0x04;
        public byte max7219_reg_digit4         = 0x05;
        public byte max7219_reg_digit5         = 0x06;
        public byte max7219_reg_digit6         = 0x07;
        public byte max7219_reg_digit7         = 0x08;
        public byte max7219_reg_decodeMode     = 0x09;
        public byte max7219_reg_intensity      = 0x0a;
        public byte max7219_reg_scanLimit      = 0x0b;
        public byte max7219_reg_shutdown       = 0x0c;
        public byte max7219_reg_displayTest    = 0x0f;

        // Pin ports for spi
        private OutputPort loadPin;
        private OutputPort dataPin;
        private OutputPort clkPin;

        // Constructor, pass pin definitions
        public max7219( OutputPort in_dataPin,
                        OutputPort in_clockPin,
   OutputPort in_loadPin )
        {
            // Assign local port pins to ports passed from constructor
            dataPin = in_dataPin;
            clkPin = in_clockPin;
            loadPin = in_loadPin;
     }
     }
}

Next we added a method to our class to transmit a single byte to the MAX7219:

// Transmits 1 byte over SPI, bitbang method
public void putByte(byte data)
{
    byte i = 8;
    int mask;

    while (i > 0)
    {
        mask = (1 << i - 1);
        clkPin.Write(false);
        if (((int)data & mask) == 0)
            dataPin.Write(false);
        else
            dataPin.Write(true);
        clkPin.Write(true);
        --i;
    }
}

And finally, we added one more method that pulled our load pin low, transmitted the register and column bytes, and pulled load high again to latch the data:

// Sends 1 Command / Data pair to a single driver chip
public void maxSingle(byte reg, byte col)
{
    // LOAD low
    loadPin.Write(false);

    // Transmit Register
    putByte(reg);
    // Transmit Column
    putByte(col);

    // LOAD high latches data sent
    loadPin.Write(true);
}

Complete with a function to send commands to the MAX via the maxSingle, we went back to our constructor and passed some initialization values after our pin assignments. maxSingle is so named because it only addresses one MAX7219, and since they are daisy-chainable, we can control multiple MAX7219s with little modification to our code:

// Initialize MAX7219
// set scan limit
maxSingle(max7219_reg_scanLimit, 0x07);
// using an led matrix mode (not digits)
maxSingle(max7219_reg_decodeMode, 0x00);
// not in shutdown mode
maxSingle(max7219_reg_shutdown, 0x01);
// no display test
maxSingle(max7219_reg_displayTest, 0x00);
// set max intensity  (range 00-0f)
maxSingle(max7219_reg_intensity, 0x0f);

Using the driver:

To use the driver, we created a variable to hold the instance of our driver, created the actual OutputPort references to pins, and passed to our instanced MAX7219 driver. The pins we are using are D10, D11, D13:

public static max7219 driver;

public static void Main()
{
    // Define spi pins
    OutputPort loadPin = new OutputPort(Pins.GPIO_PIN_D10, false);
    OutputPort dataPin = new OutputPort(Pins.GPIO_PIN_D11, false);
    OutputPort clkPin = new OutputPort(Pins.GPIO_PIN_D13, false);

    // Instance of driver, passing pin assignments
    driver = new max7219(dataPin, clkPin, loadPin);
}

At this point, we used our driver by calling driver.maxSingle(reg, col) to light either a single LED or a group of LEDs depending on specifications.

Animations

We completed two different types of animations, one based off of an algorithm and one using frame-by-frame animation for more complex patterns.

Our first animation is my personal favorite, the pulsating animation, which uses pulses to raise and lower the brightness:

// Main loop
while (true)
{
   // if direction is 1, we are going to fade down
   if (dir == 1)
      cnt--;
   else // otherwise fade up
      cnt++;

   // if direction is fading down and we are at 0 (the bottom)
   // switch directions to fading up
   if (dir == 1 && cnt == 0)
      dir = 0;
   // and if we are fading up (dir == 0) and we are at the max intensity
   // lets start fading down
   else if (dir == 0 && cnt == 15)
      dir = 1;

    // loop through driver.MAX7219_reg_digit0-7 and make sure all led's
    // are on by passing 0xFF which is all 8 bits on per digit
   for (j = 1; j < 8; j++)
      driver.maxSingle(j, 0xFF);

   // Pass the cnt value to the intensity register
   driver.maxSingle(driver.max7219_reg_intensity, (byte)(cnt & 0x0f));

   // Add a small delay between levels of intensity
   Thread.Sleep(15);
}

Wow, that looks great! Now we can create all kinds of animations using simple algorithms. To create more complex animations, however, a simpler method was needed. The disc is wired up around the ring using five digits of six LED segments. To simplify the math, we made a graphical ring of 30 LEDs, recorded each frame, and stored five characters per frame using six bits per character.

The creator is posted on the web at http://harfordhackerspace.org/ledwiz/tron/ and was created using HTML/Javascript and CSS. We won't get into the details of that code in this article, but feel free to use it to create as many animations as your Netduino will hold!

After creating an animation, click the ‘generate' button to get an output similar to the following:

byte[] animation = new byte[] {
  0x2,0x11,0x4,0x21,0x8,
  0x4,0x22,0x8,0x2,0x11,
  0x8,0x4,0x11,0x4,0x22
};

Here you can see three frames (one line per frame) with five characters using six bits per character. To display the animation, keep track of which digit (0-5) is active as the register and pass the character as the column:

int i;

for (i = 0; i < animation.Length; i++)
{
    driver.maxSingle((byte)((i % 5) + 1), animation[i]);
}

It's a simple animation. To slow it down, add a Thread.sleep(x) (x being the number of milliseconds between each command).

Dealing with multiple frame by frame animations

To create multi-frame animations, make a multi-dimensional array and modify the code to support multiple animations:

byte animation_num = 0;
int i;
byte[][] animations = new byte[][]
{
   new byte[]
   {
      0x2,0x11,0x4,0x21,0x8,
      0x4,0x22,0x8,0x2,0x11,
      0x8,0x4,0x11,0x4,0x22
   },
   new byte[]
   {
      0x1,0x10,0x2,0x8,0x2,
      0x3,0x18,0x6,0x2,0x8,
      0x7,0x3,0x18,0x1,0x8
   }
}
// Main loop
while (true)
{
   for (i = 0; i < animations[animation_num].Length; i++)
   {
      driver.maxSingle((byte)((i % 5) + 1), animations[animation_num][i]);
   }
}

Once it hits the main loop, it will keep playing the first animation (which is 0) until a change in the value of animation_num to 1 (then it will play the second animation).

A method is needed to read the button on the Disc and increment/loop the animation_num value. To do this, use the InterruptPort class.

First, add the InterruptPort code to the Main() function before the Main loop:

// Tron Button to Change Annimations, pulls high
InterruptPort changeBtn = 
   new InterruptPort(
      Pins.GPIO_PIN_D0, 
      false, 
      Port.ResistorMode.Disabled,
      Port.InterruptMode.InterruptEdgeHigh);
changeBtn.OnInterrupt += new NativeEventHandler(changeBtn_OnInterrupt);

Then, add a method called changeBtn_OnInterrupt to the program:

//Interrupt Handler for changeBtn to change annimations
static void changeBtn_OnInterrupt(uint data1, uint data2, DateTime time)
{
    if (++new_animation_num > animation_total)
    {
        new_animation_num = 0;
    }

     // reset intensity to max incase previous animation exits with 
     // intensity less than max
    driver.maxSingle(driver.max7219_reg_intensity, 0x0f);

    // Print debug string
    Debug.Print("Animation Num: ");
    Debug.Print(new_animation_num.ToString());
}

Finally, to get it all working, add a switcher at the top of the loop, change the animation_num values to be public throughout the class, as well as static since the program is static:

public class Program
{
    public static byte new_animation_num = 0;
    public static byte animation_num = 0;
    public static byte animation_total = 2;

    /* more goodness */

    // Main loop
    while (true)
    {
        animation_num = new_animation_num;

        // adding this to the top of our main loop
    }
    /* more goodness */
}

Finishing touches

To make it all work together, we mixed algorithmic animations by filling our animations array, hard-coded a few “if” statements to do algorithmic animations, as well as “else” statements for all the frame-based animations.  We also added another InterruptHandler to both act like a button when the Disc is placed on the charging dock and change the animation to the pulse animation.  Also, we tested the power by turning on all of the LEDs, waiting three seconds, and then proceeding.

About

Harford Hackerspace is a non-profit 501(C)(3) charitable organization set forth to create a place for people to collaborate on hardware and software projects. Members of the space have been meeting weekly since January 2009 to socialize, learn, and work on projects related to science and technology. In 2009 they built a Computer Numerical Controller (CNC) Router capable of precision cutting of wood, plastics, and aluminum. In August 2010 Harford Hackerspace hosted the first ever Netduino class with Secret Labs CTO Chris Walker. Most recently in November 2010 Harford Hackerspace won first place in the Baltimore Hackathon with their RotoFoto Project.

Here's another look at the phones

My first step was buying an Energy Monitor. After looking at a few different models, I spotted one with a USB cable that would allow me to “send the readings to a home PC.” Ohh! I could do something with the data instead of just looking at it on a handheld monitor. Interesting…

Getting Started

The first thing to do is install the monitor and the next thing is to install the drivers for the USB cable—these allow the USB connection to present itself as a COM port and make it easy to interact with.

Next, test the basic functionality. To do this (and to see the data transfer in action) you will need a Terminal client. Open your terminal software, set the COM port to whatever the USB Cable driver surfaced as (typically COM 3), set the baud rate to 57600bps, 8 data bits, “none” for parity, 1 stop bit, and no handshaking. When you “open” the COM port you should now see an Xml string being received every six seconds (for example):

HTML

<msg>
   <src>CC128-v0.11</src>
   <dsb>00089</dsb>
   <time>13:02:39</time>
   <tmpr>18.7</tmpr>
   <sensor>1</sensor>
   <id>01234</id>
   <type>1</type>
   <ch1>
      <watts>00345</watts>
   </ch1>
   <ch2>
      <watts>02151</watts>
   </ch2>
   <ch3>
      <watts>00000</watts>
   </ch3>
</msg>

The format of the Xml data can be found in this document, available on the CurrentCost website.

Project Outline / Design

Thinking about the project outline and design, I knew that I wanted to be able to make the received data readings available to multiple applications and websites. I also wanted to be able to tweet my energy usage every couple of hours, upload the data to some data tracking/recording web app, and be able to store the data myself so I could chart it at a later date. I also wanted the ability to see the real-time data when I was away from home, and I wanted to be able to use my phone as a kind of remote monitor.

I sketched out some ideas and came up with a design for a core Windows service that simply grabbed the data from the device, decoded it, and then passed it on to a number of modules/plugins, each of which used the data to complete a specific action. This kind of module/plugin/extension framework is a good candidate for the Managed Extension Framework (MEF) in .NET 4.0:

In fact, we'll be targeting the .NET 4.0 framework because we need some of the newer System.Composition namespace elements for the Managed Extension Framework to work.

Step 1 – The Windows Service

The Windows Service is the core of the application and needs a serial port connection to receive the data from the monitor, making it able to alert each of the plugins of the new reading.

Coding the Service

To begin coding the service, you must first create a new “Windows Service” project type. Then, change the class name to EnergyService and the ServiceName to EnergyService, add the serial port, and wire it up. To do this, you'll need a System.IO.Ports.SerialPort member variable in the class:

C#

SerialPort comm;

Create the object in the class constructor, and wire up the ‘DataReceived' event:

C#

comm = new SerialPort(portName, baudRate, parity, dataBits, stopBits);
comm.DataReceived += new SerialDataReceivedEventHandler(SerialDataReceived);

Then, we pull the bytes from the serial port in the ‘SerialDataReceived' event handler and add them to a buffer, immediately checking the buffer to see if the bytes we have stored in there so far make up a full reading. If so, we process it. If we don't have a full reading, however, we simply wait for more bytes.

void SerialDataReceived(object sender, SerialDataReceivedEventArgs e)
{
    rxBuffer += comm.ReadExisting();
    while (IsReading(rxBuffer))
    {
        // we have a reading (or partial reading), so let's process
        EnergyReading rdg = ExtractReading(ref rxBuffer);
        if (rdg.IsValid)
        {
            NotifyPlugins(rdg);
        }
    }
}

private EnergyReading ExtractReading(ref string rxBuffer)
{
    string rdgXml;
    int pos = rxBuffer.IndexOf(Environment.NewLine);
    rdgXml = rxBuffer.Substring(0, pos);
    rxBuffer = rxBuffer.Substring(pos + 2);
    return new EnergyReading(rdgXml);
}

Debug

I also added a debug feature allowing you to inject a reading every six seconds. To enable the debug mode, simply set “debug_enabled” to “true” in the appSettings section of the app.config file.

Parsing the Readings

The xml text is passed to the constructor of the EnergyReading class and this parses the xml into the required properties, which gives us an object to pass around as needed. The EnergyReading class only extracts the timestamp, temperature and energy values, though there is scope for parsing much more, especially from the “history” readings (see the Xml definition for more info). Note: we also override the ToString() method so that we have a simple method of debugging the app/values.

C#

class EnergyReading
{
    public DateTime TimeStamp { get; private set; }
    public float Energy { get; private set; }
    public float Temperature { get; private set; }
    public bool IsValid { get; private set; }

    public EnergyReading(string readingData)
    {
        TimeStamp = DateTime.MinValue;
        Energy = float.NaN;
        Temperature = float.NaN;
        IsValid = false;

        try
        {
            // parse the xml based reading
            XmlDocument xDoc = new XmlDocument();
            xDoc.LoadXml(readingData);

            XmlNode xNode = xDoc.SelectSingleNode("/msg/time");
            TimeStamp = DateTime.Parse(xNode.InnerText);
            xNode = xDoc.SelectSingleNode("/msg/tmpr");
            Temperature = float.Parse(xNode.InnerText);
            xNode = xDoc.SelectSingleNode("/msg/ch1/watts");
            Energy = int.Parse(xNode.InnerText);
            IsValid = true;
        }
        catch (Exception ex)
        {
            // invalid reading
            Debug.WriteLine(ex.Message);
        }
    }

    public override string ToString()
    {
        StringBuilder sb = new StringBuilder(TimeStamp.ToLongTimeString());
        sb.Append(" - ");
        sb.Append(Energy.ToString("N"));
        sb.Append(" Watts - ");
        sb.Append(Temperature.ToString("N"));
        sb.Append(" Degrees");

        return sb.ToString();
    }
}

Adding the Plugin Infrastructure

Next up is adding the plumbing for the plugins. As previously mentioned, we will be using MEF, which makes it all incredibly simple.

Basically, we define an interface supported by all of the plugins, create a private variable of type IList<PluginInterface> and tag it with the [ImportMany] attribute (this holds our plugin objects), create a catalog of plugins (loaded from a specific directory), and then add that catalog to a CompositionContainer that automatically wires everything up for us. Sounds complex, but it only takes about 10 lines of code:

C#

using System.ComponentModel.Composition;
using System.ComponentModel.Composition.Hosting;

// snip....

CompositionContainer container;

// snip....

[ImportMany]
private IList<IEnergyMonitorPlugin> plugins { get; set; }

// snip....

private void LoadPlugins()
{
    string baseDir = AppDomain.CurrentDomain.BaseDirectory;
    DirectoryCatalog currDirCatalog = new DirectoryCatalog(baseDir);

    container = new CompositionContainer(currDirCatalog);
    container.ComposeParts(this);
}

Now that we have all of the plugins correctly wired up, we need to notify them. For the time being, we'll also leave a few lines of debugging messages in there:

C#

private void NotifyPlugins(EnergyReading rdg)
{
    Debug.WriteLine("Notify : " + rdg.ToString());
    if (null != plugins)
    {
        foreach (IEnergyMonitorPlugin plugin in plugins)
        {
            try
            {
                Debug.WriteLine("Notify : " + plugin.Name);
                plugin.Notify(rdg);
            }
            catch (Exception ex)
            {
                Debug.WriteLine("Plugin '" + plugin.Name + "' : " + ex.Message);
            }
        }
    }
}

Adding an Installer to the Windows Service

Now that we have completed the Windows Service, we need to test it out. To do so, we need to install it, and for that we need to add an Installer Class, as well as adding both a ServiceInstaller and a ServiceProcessInstaller to the designer (set the ServiceProcessInstaller Account property to “Local System” and run the service under this account).

Installing a service is different than installing a normal application. Luckily, we can use .NET framework's InstallUtil.exe to simplify things. I like to set InstallUtil.exe up as an External Tool in Visual Studio when I'm working on Windows Service projects, which makes the whole process as simple as highlighting the Windows Service project and clicking on the Install (or Uninstall) option under the Tools menu:

The configuration to set this up as an “external tool” is as follows:

Title	Install Service (.NET 4)
Command	C:\Windows\Microsoft.NET\Framework64\v4.0.30319\InstallUtil.exe
Arguments	/i "$(ProjectDir)\bin\debug\$(TargetName)$(TargetExt)"
Title	Uninstall Service (.NET 4)
Command	C:\Windows\Microsoft.NET\Framework64\v4.0.30319\InstallUtil.exe
Arguments	/i "$(ProjectDir)\bin\debug\$(TargetName)$(TargetExt)"

For 32-bit operating systems, change the Framework64 folder name to Framework.

Testing the Windows Service

Once you start the service, if you open the Output windows in Visual Studio and view the output from Debug, you should start seeing the readings come in:

Step 2 – The Twitter Plugin

The first plugin we'll write is a simple Twitter poster. Every 60 minutes, this plugin will send a tweet with your machine name and the average energy use over the last hour.

Next, create a new Class Library project and name it “TwitterPlugin.” Then add a reference to the “EnergyService_CSharp” project and have the class implement the IEnergyMonitorPlugin interface. We will also need to add some class variables to hold the last tweet time, energy total, and reading count.
As Twitter no longer supports Basic Auth for API calls we will need to use oAuth. For this we will use the excellent oAuth class developed by Eran Sandler and extended by Shannon Witley.

C#

[Export(typeof(IEnergyMonitorPlugin))]
public class TwitterPlugin : IEnergyMonitorPlugin
{
    double twInterval;
    double twEnergy = 0;
    int twReadings = 0;
    DateTime twLastTweet = DateTime.MinValue;
    oAuthTwitter _oAuth = new oAuthTwitter();

Next, we give the plugin values for “Name” and “Description” and set the DeInit() function to return true (this is a simple plugin so no deinitialization is needed). For the Init() function, we want to read from the app.config the twitter oAuth parameters, and the interval between tweets—remember, this plugin is loaded into the AppDomain of the service, so we need to add the entries to the app.config file of the service, not the app.config of the twitterplugin project/class.

The code looks like this:

C#

bool IEnergyMonitorPlugin.Init()
{
    _oAuth.Token = GetSetting("twitterplugin_token", "");
    _oAuth.TokenSecret = GetSetting("twitterplugin_tokensecret", "");
    _oAuth.Pin = GetSetting("twitterplugin_pin", "");

    twInterval = double.Parse(GetSetting("twitterplugin_interval", "60"));
    return true;
}

private string GetSetting(string key, string defaultValue)
{
    NameValueCollection appSettings = ConfigurationManager.AppSettings;
    string val = appSettings.Get(key);
    if (null == val)
    {
        val = defaultValue;
    }

    return val;
}

The crux of a plugin is the “Notify” function, and for this one we simply add the energy value to the total energy count and increase the reading count. We then check if more than 60 minutes have passed since the last tweet, and if so we send the tweet:

C#

bool IEnergyMonitorPlugin.Notify(EnergyReading rdg)
{
    bool retVal = false;

    twEnergy += rdg.Energy;
    twReadings++;

    if (twLastTweet.AddMinutes(twInterval) < DateTime.Now)
    {
    // it's time to tweet
        retVal= SendTweet(
            string.Format(
                "In the last {0} mins {1} has used an average of {2} watts", 
                twInterval, 
                Environment.MachineName, 
                (twEnergy/twReadings)));
        twEnergy = 0;
        twReadings = 0;
        twLastTweet = DateTime.Now;
        retVal = true;
    }

    return retVal;
}

Sending the tweet requires that we UrlEncode the message and then call the oAuthWebRequest method of the oAuth class with the post data as a parameter.

private bool SendTweet(string tweet)
{
    bool retVal = false;
    try
    {
        string encTweet = System.Web.HttpUtility.UrlEncode(tweet);
        string xml = _oAuth.oAuthWebRequest(
            oAuthTwitter.Method.POST,
            "http://twitter.com/statuses/update.xml",
            "status=" + tweet);
        retVal = true;
    }
    catch (Exception ex)
    {
        System.Diagnostics.Debug.WriteLine("Error : " + ex.ToString());
    }

    return retVal;
}

Now we have the Notify functionality of the TwitterPlugin completed. To get it working, simply copy the plugin to the same folder as the running Windows Service, edit the setting for “twitterplugin_interval” in the service app.config files, and restart the service. The tweets will not yet be accepted by Twitter as we have not yet authorized the application – for this we need user interaction so it is not something we do in the Windows Service. Instead we build a ConfigTool that uses MEF to load in each of the plugins and calls the “Configure” method.
We add a Windows Form to the TwitterPlugin, add a button that takes the user to a Twitter application authorization page, add a textbox that the user enters the PIN displayed on the webpage into and save the resulting oAuth Token and TokenSecret we get back from calling the AccessTokenGet method of the oAuth class. When we have all the oAuth parameters back we simply save them into the app.config file:

If all is well, you should see tweets appearing every X minutes (depending on the interval you have set) along with your energy averages.

Now that we have completed our first plugin, it should be easy to see how simple it is to write further plugins. You might consider writing the energy values to a local database, sending Facebook updates, or posting to a dedicated service that uses API to store sensor or energy values, such as http://www.pachube.com or http://www.enio.co.uk.

Step 3 – The WebService and Plugin

The next stage we'll develop will give us a little more flexibility. Essentially, this stage consists of a RESTful Windows Communication Framework (WCF) Service and a corresponding WebSvcPlugin that we'll use to update the WCF Service with the latest readings. We'll also build a couple of client applications that will retrieve the reading from the WCF Service (one is a simple webpage using jQuery AJAX to get the reading from the WCF Service, and the other is a Windows Phone 7 application using a HttpWebRequest to get the reading from the WCF Service).

The plugin and clients will need the WCF Service to be in place before they can be written, so we'll tackle the WCF Service first by adding an “ASP.NET Empty Web Application” project to the solution and calling it “EnergyMonitorWs”:

Then we add a new “AJAX-enabled WCF Service” and name it EnergyMonitor.svc:

The WCF Service will expose OperationContracts that allow clients to set the current values and receive each of the values in either JSON or XML format. The current values will be stored in an HttpRuntime.Cache object, so we don't need to worry about persisting the reading data to MSSQL or the FileSystem.

The cache object is in the System.Web.Caching namespace, so we'll need a reference to that. We'll also need to make sure we are working in AspNetCompatibility mode. To do so, we'll add the following attribute to the class:

C#

[AspNetCompatibilityRequirements(
    RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]

Now let's code the “SetCurrentValues” function, which exposes a “/set” endpoint and takes date, time, energy, and temperature as parameters on the HTTP GET querystring and stores them in the cache for an hour (this function should be called every 6 seconds, so caching each reading for up to an hour gives us plenty of room for failed updates).

C#

[OperationContract]
[WebInvoke(
    Method="GET",
    UriTemplate="set?energy={energy}&temp={temp}&date={date}&time={time}")]
public void SetCurrentValues(string energy, string temp, string date, string time)
{
    Cache cache = HttpRuntime.Cache;

    try
    {
        cache.Remove("energy");
        cache.Remove("temp");
        cache.Remove("date");
        cache.Remove("time");
    }
    catch { }

    cache.Add(
        "energy", 
        energy, 
        null, 
        DateTime.Now.AddHours(1), 
        Cache.NoSlidingExpiration, 
        CacheItemPriority.Default, 
        null);
    cache.Add(
        "temp", 
        temp, 
        null, 
        DateTime.Now.AddHours(1), 
        Cache.NoSlidingExpiration, 
        CacheItemPriority.Default, 
        null);
    cache.Add(
        "date", 
        date, 
        null, 
        DateTime.Now.AddHours(1), 
        Cache.NoSlidingExpiration, 
        CacheItemPriority.Default, 
        null);
    cache.Add(
        "time", 
        time, 
        null, 
        DateTime.Now.AddHours(1), 
        Cache.NoSlidingExpiration, 
        CacheItemPriority.Default, 
        null);

    return;
}

We want to make request functions available via a simple HTTP GET and we want to provide the option of JSON or XML as the response. Therefore, we add a “/json/{x}” endpoint and a “/xml/{x}” endpoint where {x} is the reading data we want (energy, temp, date or time):

C#

[OperationContract]
[WebInvoke(
    Method = "GET", 
    UriTemplate = "xml/{value}")]
public string GetValueXml(string value)
{
    WebOperationContext.Current.OutgoingResponse.ContentType = "text/xml";
    return GetValue(value.ToLower());
}

[OperationContract]
[WebInvoke(
    Method = "GET", 
    UriTemplate = "json/{value}", 
    ResponseFormat = WebMessageFormat.Json)]
public string GetValueJson(string value)
{
    return GetValue(value.ToLower());
}

private string GetValue(string key)
{
    Cache cache = HttpRuntime.Cache;

    string retVal = (string)cache.Get(key);
    if (null == retVal)
    {
        retVal = "N/A";
    }

    return retVal;
}

Now that we have the WCF Service in place, we can build the plugin that updates it. This is another Class Library project that we add to the solution (this time naming it WebServicePlugin), again making sure it implements the IEnergyMonitor interface (you'll need to add a reference to the EnergyMonitor_CSharp for this).

Give the plugin a Name and Description and set the Init() and DeInit() functions to return true. Then, simply have the code in the “Notify” function create an HTTP GET request with the values in the querystring to the “/set” endpoint (the SetCurrentValues function). This can be accomplished with a System.Net.WebClient object:

C#

bool IEnergyMonitorPlugin.Notify(EnergyReading rdg)
{
    bool retVal = false;

    DateTime ts = rdg.TimeStamp;

    string getUrl = baseUrl + "/set?";
    getUrl += "energy=" + rdg.Energy.ToString() + "&";
    getUrl += "temp=" + rdg.Temperature.ToString() + "&";
    getUrl += "date=" + ts.ToString("dd/MM/yy") + "&";
    getUrl += "time=" + ts.ToShortTimeString();

    WebClient wc = new WebClient();

    try
    {
        wc.DownloadString(getUrl);
        retVal = true;
    }
    catch (Exception ex)
    {
        System.Diagnostics.Debug.WriteLine(
            "Error (Notify) : " + ex.Message);
    }


bool IEnergyMonitorPlugin.Init()
{
    // get the base url from the (Service) app.config
    baseUrl = GetSetting(
        "websvcplugin_baseurl", 
        "http://localhost/EnergyMonWs.svc");
    return true;
}

// snip....


private string GetSetting(string key, string defaultValue)
{
    NameValueCollection appSettings = ConfigurationManager.AppSettings;
    string val = appSettings.Get(key);
    if (null == val)
    {
        val = defaultValue;
    }

    return val;
}

Now that we're done with the WCF Service and WebSvcPlugin, we need to work on the two WCF Service client applications.

Step 4 – The WebService Html Page Client

We'll first tackle the html webpage with jQuery—a simple webpage that uses the jQuery library to make AJAX calls to our WCF Service to get the current values—by adding a new ‘HTML Page' to the EnergyMonitorWs project. To begin, name the page EnergyMonitor.htm.

Next, we'll add some html to the page. This html is linked to some CSS, giving us a simple remote version of the energy monitor:

Then we use jQuery to call the WCF Service and update the actual values with real reading data. We also add a JavaScript timer to make the WCF Service calls every 15 seconds. Obviously, we need to reference the jQuery code and the CSS in the head section:

HTML

<head>
    <title>Energy Monitor - Ken Hughes</title>
    <script 
        src="scripts/jquery-1.4.1.js" 
        type="text/javascript"></script>
    <link 
        rel="Stylesheet" href="css/style.css" 
        type="text/css" />
</head>

Then, after the main page display elements, we add the javascript code so the values are updated every 15 seconds:

Javascript

function getReading(fname, elementid) {
    ajaxUrl = "EnergyMonWs.svc/json/" + fname;
    $.ajax({
        type: "GET",
        url: ajaxUrl,
        contentType: "application/json; charset=utf-8",
        dataType: "json",
        success: function (msg) {
            // Replace the div's content with the value returned.
            $("#" + elementid).text(msg);
        }
    });
}


function updateReadings() {
    getReading("temp", "tempval");
    getReading("energy", "energyval");
    getReading("time", "timeval");
   
}

$(document).ready(function () {

    updateReadings();
    window.setInterval(function () {
        updateReadings();
    }, 15000);
});

Step 5 – The WebService Windows Phone 7 Client

The purpose of the Windows Phone 7 client is to act as a remote monitor. The client pulls its reading from the web service and updates a monitor-style display. To complete this part of the project, you will need to have the Windows Phone 7 Development tools installed.

Begin by adding a new “Windows Phone Application” project and calling it “WinPhoneEnergyMonitor.”

First, in the constructor, add a Timer object, set the interval to 15 seconds, and have it call a function to update the screen values. This function calls the webservice for each value, receiving the latest reading data and updating the relevant TextBlock.

In Windows Phone 7, uses an asynchronous manner to call to things such as the WebClient and HttpWebRequest. Accordingly, we create a new WebClient object, assign a delegate to the “DownloadStringCompleted” event, and then call “DownloadStringAsync” with the relevant uri along with the TextBlock we want to update as the ‘userState' object. All of this allows us to update the correct item in the event handler.

The code looks like this:

C#

private void UpdateScreen(object state)
{
    UpdateValueFromWebService("energy", this.energyValue);
    UpdateValueFromWebService("temp", this.tempValue);
    UpdateValueFromWebService("time", this.timeValue);
    UpdateValueFromWebService("date", this.dateValue);
}

private string UpdateValueFromWebService(string reading, TextBlock tb)
{
    string retVal = "N/A";
    Uri uri = new Uri(baseUri + "/json/" + reading);

    try
    {
        WebClient wc = new WebClient();
        wc.DownloadStringCompleted += 
            new DownloadStringCompletedEventHandler(DownloadStringCompleted);
        wc.DownloadStringAsync(uri, tb);
    }
    catch (Exception ex)
    {
        System.Diagnostics.Debug.WriteLine(
            "Error (UpdateValueFromWebService) : " + ex.Message);
    }

    return retVal;
}

void DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
{
    try
    {
        string val = e.Result.Replace("\"", "").Replace("\\", "");
        System.Diagnostics.Debug.WriteLine(val);
        ((TextBlock)e.UserState).Text = val;
    }
    catch (Exception ex)
    {
        System.Diagnostics.Debug.WriteLine(
            "Error (DownloadStringCompleted) : " + ex.Message);
    }
}

Conclusion

This was a really interesting project. Not only did it help me understand how much energy I was using (and help me reduce it), it also presented a software challenge covering a number of areas and technologies: Windows Service, WCF, Windows Phone 7, and javascript/jQuery.

I already have plans for a number of other plugins and new ways to visualize the usage data—keep an eye on the Codeplex site for updates.

If you want to try this out, the download link for the source code is at the top of the article!

About The Author

Ken Hughes is Chief Technical Officer at C2C Systems, a Microsoft Gold Partner ISV. He is passionate about technology, productivity, and automating everything. He stays up-to-date with and keeps his hand in new technologies by working on personal and open source projects such as dasBlog. Contact information may be found on both his blog and Twitter.

This upcoming special edition of the Wireless Mobile Mouse 4000 has been made in partnership with Susan G. Komen for the Cure, which you probably know is the world’s largest breast cancer charitable organization.

Microsoft will be donating 10% of the selling price ($4.00 USD) with a minimum donation of $200,000 to Susan G. Komen for the Cure when the mouse goes on sale mid-September (retail price: $39.95 USD). The money will be used for research grants and community outreach projects, reports the Microsoft Hardware blog.

The mouse offers pretty great hardware specs, too, including BlueTrack technology, a tiny nano transceiver and customizable buttons. Windows 7 integration is supported as well. For more details, check out the full feature list here on Microsoft.com while you wait for the pink one to arrive in stores and online.

• Supports all EM41xx family 125 kHz RFID tags
• USB, XBee, and TTL Serial communication
• Serial Communication Protocol and Shared Serial Devices Protocol (SSDP)
• .NET programming API
• Operates as a slave device to PC or microcontroller
• Onboard programmable microcontroller
• Xbee point to point
• Xbee Mesh networking
• 3 power input options (Barrel Connector, USB, or Vcc)
• 4 Configurable I/O pins
• Valid/Invalid Tag LED
• Xbee Comm LED

There is even a drywall with a gangbox cover even!  You can bet we'll be releasing a project with one of these shortly.

Below is the basic hardware setup of the Laser Graffiti System.

Building a Laser Tracker Engine

Giving Sight to the Blind

The very first thing that needed to be proven was the ability to track a laser point. The most obvious solution was to give our application some vision with a webcam. Since this was going to be a WPF application, I decided to use my open-source project, WPF MediaKit, which comes with a webcam control called the VideoCaptureElement.

At the time, VideoCaptureElement was not very robust, and it occurred to me that my control wasn't very useful beyond looking at your webcam in a WPF app! I needed a way to get high-performance access to every pixel of every frame the webcam spat out. This was a good time to add that ability and increase value in my project. Without getting into the gory details of DirectShow and p/invoke, I was able to add a hook to pass me the pixel buffer, where I would wrap it in a Bitmap class and raise an event for each frame.

The XAML required to get each video sample from the web camera.

<MediaKit:VideoCaptureElement LoadedBehavior="Play"
    RenderOptions.BitmapScalingMode="LowQuality"
    UnloadedBehavior="Manual"
    DesiredPixelWidth="{Binding DesiredPixelWidth}"
    DesiredPixelHeight="{Binding DesiredPixelHeight}"
    VideoCaptureDevice="{Binding SelectedItem, ElementName=videoCapDevices}"
    FPS="30"
    EnableSampleGrabbing="True"
    NewVideoSample="VideoCaptureElement_NewVideoSample"
    Margin="0"
    x:Name="videoElement"
    Stretch="Fill"
    MinHeight="380"
    MinWidth="500" />

Did I Mention Lasers?

Now that I had the ability to look at every pixel sent to me, I needed to make sense of it all. In order to complete the project, we have to be able to find a laser point shined on a wall. The difficulty lies in the fact that a laser can be of varying size and color. For example, if you take two consecutive video frames with the same green laser point, you will find the point may slightly differ between the two. This is because we are dealing with analog data and slightly changing lighting conditions. Whatever algorithm we choose to find the laser must take this into account.

There are many advanced ways to use video analytics to find the laser pointer, but this project did have a deadline, and I wanted to keep it fun, so I went with a simple method, which allows a user to filter the video based off ranges of hue, saturation, and luminance. To do this, we make use of the AForge image processing library. AForge is an open-source library that comes with tons of useful utilities for just what we want to do.

Filtering pixels and counting blobs…

As previously mentioned, we need to first filter the image based on hue, saturation, and luminance. Luckily, the AForge library comes with such a filter, so there is not much work to be done here:

/* This AForge class helps us filter out the pixels we do not want */
var hsl = new HSLFiltering
{
    Hue = new IntRange(HueMinimum, HueMaximum),
    Saturation = new DoubleRange(SaturationMinimum, SaturationMaximum),
    Luminance = new DoubleRange(LuminanceMinimum, LuminanceMaximum)
};

var bitmapData = bitmap.LockBits(
    new Rectangle((int)targetSearchArea.X, 
        (int)targetSearchArea.Y, 
        (int)targetSearchArea.Width, 
        (int)targetSearchArea.Height),
    System.Drawing.Imaging.ImageLockMode.ReadWrite, 
    m_lastBitmap.PixelFormat);


/* Apply the AForge filter.  Doing it "in place" is more efficient as
 * new bitmap does not have to be allocated and copied */
hsl.ApplyInPlace(bitmapData);

So, what needs to be done now that we have a filtered image showing only a laser dot? The answer: Blob counting! A blob, in the context of video analytics, is a set of pixels that are all touching, and AForge comes with a class ready to tackle that too!

The AForge blob counter takes in a gray-scale image for processing, and then returns all blobs found in the image. The information included on a blob includes the rectangular pixel area where the laser was found in the video frame. The center of that area should be the center of our laser. It's important to note that if the HSL filter is improperly calibrated, we will get an overabundance of blobs and there will accordingly be no way we can tell if it is the laser point or not.

/* Create an initialize our blob counter */
var blobsCounter = new BlobCounter
    {
       FilterBlobs = true,
       ObjectsOrder = ObjectsOrder.Size,
       MinHeight = BlobMinimumHeight,
       MinWidth = BlobMinimumWidth,
       MaxWidth = 25,
       MaxHeight = 25
    };

/* We first let the blob counter process our image */
blobsCounter.ProcessImage(grayImage);

/* Retrieve a list of blobs that were found */
Blob[] blobs = blobsCounter.GetObjects(grayImage);

The early laser tracking prototype. We can literally have it track anything, based off color.

Building the Laser Graffiti Application

Our Rocket Engine Needs a Rocket

At this point, I had a working proof of concept and a laser tracker. What I needed now was an application to use it. You know, the laser-graffiti application.

There are many ways to construct a WPF application and one pattern that's been getting a little bit of traction is MVVM. I chose to make a composite application using Prism and MVVM. Prism is a great guidance library pack for making composite WPF applications with tons of tools to help in common scenarios.

Composite What? Huh?

When developing applications, usually of the larger variety, things can become complex. Complexity turns into a mess. A mess turns into time investment. Time investment turns into money. The more money spent, the fewer bonuses you will receive. To make sure we get that end-of-year pay raise, we want to control complexity from the get-go by making our code and our internal systems loosely coupled. A composite application is just a collection of loosely coupled systems that is composited at runtime.

The advantages of a composite application are:

· Components can be developed independently

· Components can be swapped out more easily than if they were tightly coupled

· The application remains extendable, meaning you can more easily extend your application

· Maintenance is much clearer with less chance of breaking other components

· Unit testing is more feasible

Composite Application Entrails

There are many, many excellent resources about Prism available on the internet, so I'd rather not compete with them. I did, however, want to cover the basics, and at least how I've used it within the Laser Graffiti application.

The Shell Assembly – This contains the main entry point of the application and the initial or main UI container for the application. Before anything else, this assembly will initialize something called a bootstrapper. The bootstrapper simply does any pre-initialization that needs to be done before initializing the rest of the application.

Module Assemblies – Each module assembly can be considered a component. The bootstrapper initializes each component and then the module spins up any application services and adds any user interface to the shell. For instance, the MediaKit module handles the capture of the video and the AForgeModule handles the vision routines.

The Infrastructure Assembly – This contains any base classes, well known/shared interfaces or general infrastructure code. Simply put, most if not all modules will have reference to this.

Rendering the Graffiti

I needed a way to draw the graffiti to a projector. Overall, this will be accomplished by drawing the graffiti to a window. The window will be set to maximized with no border. I also needed to write the code to draw the graffiti. Initially, I just used the WPF InkCanvas control, but I found it wasn't well tuned for what I wanted to do with it. So, I decided XNA would give the most performance and flexibility with the level of effects desired for drawing graffiti.

I had never really done any XNA development, so this project is certainly not built using best practices. The important part to notice is that the project is more or less a regular XNA application with the exception that the executable is added as a library reference. This is because we instantiate the XNA from our WPF application's process. The following snippet of code accomplishes this:

private void ShowGraffitiWall(bool show)
{
    if (m_game != null)
        m_game.Exit();

    if (!show) 
        return;

    var t = new Thread((ThreadStart)delegate
    {
        m_game = new XnaGraffitiGame();
        m_game.SetScreen(m_lastScreen);
        m_game.Effect = m_lastGraffitiEffect;
        m_game.Run();
        m_game.Dispose();
    });

    t.SetApartmentState(ApartmentState.STA);
    t.IsBackground = true;
    t.Start();
}

We instantiate the XNA window this way because it allows our WPF code to send the XNA messages of where the laser pointer was detected.

Drawing with the GPU

A blanket statement about XNA is that it is a drawing API. XNA applications can get quite complex when you add in things like 3D or shaders, but for this article, I'm only covering the required simple stuff. That said, I do make the assumption that the reader has had at least a few hours of XNA experience.

Our primary goal is to render virtual graffiti using XNA. This means we need to be able to draw a line. I started by creating a PNG file that would act as a “brush.” In XNA, these 2D graphics are also known as sprites. The brush sprite looks like this:

Now we can use XNA's SpriteBatch class to draw our brush. The first problem I had, though, was that most XNA tutorials show drawing the sprite, continue, clear the screen, draw the sprite, continue, etc. I needed to retain previous brush strokes, which means drawing onto an intermediate drawing surface (RenderTarget2D), then drawing that to the screen.

The next issue I had was that the line I was trying to draw was very broken. See, the laser detection code runs at a maximum 30 FPS (the limit of my web camera). If I moved the laser fast enough, the screen would just look like a bunch of random points were being drawn. I had to move the laser very slowly to make it appear like a line. I remember reading an excellent article detailing a project by Rick Barraza, which led me to a link to some popular algorithms for creating a line where your input is two, 2D points.

So, drawing a line required only a simple, two-step process of A) getting a new point from laser detection, and B) drawing a line from last_laser_coordinate to current_laser_coordinate. After putting that together, we can now produce some useful looking graffiti!

Adding Crazy Graffiti Effects

The excitement of drawing simple lines in XNA wore off pretty fast. After all, the GPU is capable of some pretty amazing things. What about drawing with fire? Fire isn't boring:

The fire effect is done using a method slightly similar to the solid color sprite method described earlier. However, since it uses a particle system, the fire effect is much more complex. Each particle has its own shader to control position and color. With the time constraints of this project, I had modified code from this XNA tutorial. There weren't many changes from the tutorial's code, but a big hurdle was that the particles took 3D coordinates instead of the 2D coordinates I needed to draw the graffiti. I found a small snippet to handle that, though I was informed by other XNA gurus that there is an easier way to accomplish this. If you are curious, this is what it looks like (and it's not for the faint of heart):

protected Vector3 ScreenPointToVector3(Point coords)
{
    Matrix viewMatrix = 
        Matrix.CreateTranslation(0, -25, 0) *
        Matrix.CreateRotationY(
            MathHelper.ToRadians(CameraRotation)) *
        Matrix.CreateRotationX(
            MathHelper.ToRadians(CameraArc)) *
        Matrix.CreateLookAt(
            new Vector3(0, 0, -CameraDistance),
            new Vector3(0, 0, 0), 
            Vector3.Up);

    float aspectRatio = 
        (float)m_game.GraphicsDevice.Viewport.Width /
            (float)m_game.GraphicsDevice.Viewport.Height;

    Matrix projectionMatrix = 
        Matrix.CreatePerspectiveFieldOfView(
            MathHelper.PiOver4,
            aspectRatio,
            1, 10000);

    Vector3 nearScreenPoint = new Vector3(coords.X, coords.Y, 0);
    Vector3 farScreenPoint = new Vector3(coords.X, coords.Y, 1);
    Vector3 nearWorldPoint = 
        m_game.GraphicsDevice.Viewport.Unproject(
            nearScreenPoint, 
            projectionMatrix, 
            viewMatrix, 
            Matrix.Identity);
    Vector3 farWorldPoint = 
        m_game.GraphicsDevice.Viewport.Unproject(
            farScreenPoint, 
            projectionMatrix, 
            viewMatrix, 
            Matrix.Identity);

    Vector3 direction = farWorldPoint - nearWorldPoint;

    float zFactor = -nearWorldPoint.Z / direction.Z;
    Vector3 zeroWorldPoint = nearWorldPoint + direction * zFactor;

    return zeroWorldPoint;
}

The End Result

Web-camera device selection:

Fine tuning the laser detection:

About Jeremiah

Jeremiah Morrill is a software developer and 2010 MVP living in Las Vegas, Nevada. He owns a software company called HJT, http://www.hjtcentral.com/, with a couple partners. There he focuses on multimedia and rich user interfaces with WPF and Silverlight. Jeremiah spends a lot of his free time learning, listening and helping others in the online development communities. 

All the source code can be found at http://netmfbikecomputer.codeplex.com/ and just a reminder, the NETMF is 100% open source as well!

1. Introduction
2. Getting the project set up
3. Sensors and Integration
4. UI – Font and touch
5. UI custom controls
6. Complex sensor integration
7. Working with emulator

Brian Peek, author of WiimoteLib, saw the video and asked me to write a practical article on how to create something like this yourself. Let’s get started!

Hardware Setup

To get this working, you only need a digital projector, a Wiimote, a few cheap components, and, of course, a computer. The latter won’t need a lot of horsepower; I’m even able to run it on my $300 netbook with an acceptable framerate! Let’s take a look at how to prepare these items before we get working on the software.

Creating the tabletop display

First, let’s create the poor man’s version of the Microsoft Surface table. If you are like me and aren’t burdened by a fat bank account, just take some ugly photo-in-a-frame off the wall and lay its protective sheet of glass down on a table-without-tabletop. Trust me: as long as you don’t sit on it, it works like a charm! Now go to a nearby grocery store and buy some white fat-free paper. Tape it on top of the glass to create your DIY projection screen. You may also tape it to the bottom side, but in my experience it will start hanging down after some time has passed. As shown in the video, use an inclined mirror and a projector connected to your computer. Now you have your own backlit tabletop display!

Making the display interactive

Now comes the trick: making the display interactive. Enter the brilliant Nintendo Wiimote. Simply put it in a stable position facing the mirror on top of the projector, preferably right above the lens. The Wiimote contains a high-performance camera that can simultaneously track up to 4 infrared light sources. Now, if you fashion a simple infrared pen (or do it the easy way and buy one online), the Wiimote can track its position all across the display!

Fashioning the point-of-view device

We’re not finished yet, because for this project we’re also going to need a point-of-view device. This device is nothing more than a small glass turned upside down, with two infrared LEDs in its bottom as shown in the photo. Building this is similar to building 2 IR pens.  Make sure the small glass is large enough to fit an AAA battery in there! It’s best to also incorporate a pushbutton switch right between the LEDs, so that they only emit light when the point-of-view is slightly pressed onto the tabletop display. It may be a bit tricky to get the button position exactly right, but I’m sure you’ll manage with some perseverance.

Connecting to the Wiimote

Now it’s time to connect to the Wiimote. For your convenience, I’ve copy-pasted Brian’s short manual here:

1. Start up your Bluetooth software and have it search for a device.

2. Hold down the 1 and 2 buttons on the Wiimote. You should see the LEDs at the bottom start flashing. Do not let go of these buttons until this procedure is complete.

3. Wiimotes should show up in the list of devices found as Nintendo RVL-CNT-01. If it's not there, start over and try again.

4. Click Next to move your way through the wizard. If at any point you are asked to enter a security code or PIN, leave the number blank or click Skip. Do not enter a number.

5. You may be asked which service to use from the Wiimote. Select the keyboard/mouse/HID service if prompted (you should only see one service available).

6. Finish the wizard.

If you can’t get it to connect, you might try using a Wiimote compatible Bluetooth adapter and stack.

Inserting Some Third-Party Code

With the hardware all set up, we’ll continue to the software. Make sure you have installed a (free) version of Visual Studio with C#, and Microsoft XNA Game Studio version 3.0 or higher. XNA integrates neatly into Visual Studio and we’ll use it as an easy way to harness your graphic card’s 3D power. Our home designer relies heavily on work done by the open-source community, so let’s start by inserting these components first.

XNA Surface As A WinForms Control

We’re going to use a hack that allows us to use XNA-controlled surfaces on as many forms as we like. There’s no need to know exactly how the hack works—we’ll just use it! Therefore, instead of starting a normal XNA project within Visual Studio, we’ll make use of a sample project that has been posted online at the XNA Creators club. Download and open it. If you run the solution you just downloaded, you’ll see two separate XNA-powered surfaces (the left one contains some text and the right one is a spinning triangle).

Both are standard Controls, but why is that important? Because now we can easily create multiple WinForms, each containing its own XNA surface—one for the blueprint and the other for the 3D perspective view! Later on, it will look like this:

A Little Housekeeping

Since we don’t want to do any more work than necessary, we’ll start our project from the this XNA control example project. Let’s do a little housekeeping to keep everything neat and tidy. First, make sure you delete all controls present on the MainForm. In the solution explorer, rename “SpinningTriangleControl.cs” and “SpriteFontControl.cs” to “BluePrintControl.cs” and “_3Dcontrol.cs” respectively, and move all files except “MainForm.cs” to a new directory called “Controls”. By the way, this would also be an appropriate moment to rename both Control classes. We’ll change their functionality later on. Your solution explorer should now look like this:

It may also be smart to change the name of our project from “WinFormsGraphicsDevice” to “HomeDesigner” or something (I actually call it “DualDesign”, but choose whatever you like).

Enabling whiteboard functionality

Time to honor the Wiimote community’s all-time great: Johnny Lee Chung! He’s the guy that used Brian’s managed Wiimote library to build some amazing applications. We’ll tweak his Whiteboard application to make our blueprint interactive. Check out his video:

I downloaded the source code for Johnny’s Whiteboard application and then updated to support version 1.7 of WiimoteLib, which is more compatible than the version Johnny used.

Now, by adding the following code to the Click event handler, let’s add a button on our MainForm that starts Chung’s app inside our own:

C#

try

{

    wiimoteForm = new WiimoteWhiteboard.WiimoteWhiteboardForm();

    wiimoteForm.Visible = true;

}

catch (Exception x)

{

    System.Console.Out.WriteLine("Exception: " + x.Message);

}

Make sure you have established a Bluetooth connection to the Wiimote before running the code. What Lee’s app basically does is emulate a standard mouse. The operating system (and thus our blueprint control) won’t see the light pen but will receive standard MouseDown, MouseMove, and MouseUp events. This is very convenient, because we don’t need the Wiimote to be connected while debugging: we can simply draw on the form with our mouse since our blueprint won’t be able to tell the difference!

Pretty Lines

There’s one last component to insert before we can really start. In the demo, you saw the big, fat white lines I drew on the blueprint. These pretty RoundLines have been coded by Microsoft’s Mike Manders and released into the public domain. Download them and insert the code into our project. Make sure to manually copy all .xnb files to our project’s “bin\ … \debug\Content” directory using the Windows Explorer (not the Solution Explorer), otherwise your code might not run.

Coding the BluePrintControl

We’ve got all the third-party code we need, so now it’s time to get our hands dirty. However, the entire solution is pretty comprehensive and much of the code is trivial. Therefore, we won’t be getting into every nitty-gritty detail; instead, I will single out some of the interesting parts. A lot of the code is XNA-related. If you want to read more about it, “Learning XNA 3.0” (O’Reilly) is an excellent starting point, as are Riemer’s online tutorials.

XNA Basics

In order to understand the code, you’ll need a grasp of some of the very basics of XNA Game Studio. XNA Game Studio is a comprehensive set of tools that allows the development of entire games, but we’ll only use it for basic visualization. XNA games run in a continuous loop, basically drawing frames to the viewport at as fast a rate as possible. A lot of things happen behind the scenes in each frame, but the most important part is the Draw() method in which all the drawing is done. Here, we draw all elements to the frame buffer, at the end of which the buffer “empties” to the viewport and becomes visible to the user.

Drawing to the GraphicsDevice

Because we were lazy and started with a pre-coded demonstration control, we’ll have to remove the unnecessary code in “BluePrintControl.cs” (the drawing of the spinning triangle). After cleaning up, make sure our Draw() method looks like this:

C#

/// <summary>

/// Draws the control.

/// </summary>

protected override void Draw()

{

    // clears the graphicsdevice

    GraphicsDevice.Clear(defaultBackgroundColor);

    // draws all elements to the viewport

    this.DrawWalls();

    this.DrawAllSprites();

    this.DrawObjectHandles();

    this.DrawButtons();

}

The GraphicsDevice is an abstraction of the graphics card and is what we draw to. The Clear() statement clears the entire frame buffer to our default background color on top of which we can draw. I used “Color.DarkBlue” but you can use anything you want here as long as it’s blue (otherwise you should have differently named the BluePrintControl class ). Next, we call four methods that draw all elements to the blueprint.

Drawing the Walls

Let’s take a look at the DrawWalls() method:

C#

private void DrawWalls()

        {

            // Draw lines

            roundLineManager.Draw(

                ConvertWallListToRoundLineList(wallList), // converts Walls to RoundLines

                defaultWallThickness,

                Color.White,

                camera.View * camera.Projection, // 4x4 matrix to that allows XNA to map 3D coordinates to screen coordinates

                1f,

                roundLineManager.TechniqueNames[defaultTechnique]

                );

        }

This is where we make use of the RoundLine component. The RoundLineManager.Draw() method takes a list of RoundLine objects and draws them to the graphicsdevice in one go. Internally, we store the walls in our own format, so the ConvertWallListToRoundLineList method does exactly what its name implies.

The fourth argument, camera.View * camera.Projection, tells XNA how to map 3D coordinates to the viewport. The View matrix contains information regarding the position and orientation of the camera, and the Projection matrix can be seen as a matrix describing the lens. Since we want the blueprint to be displayed without any distortion, we previously defined the projection matrix to be orthographic:

C#

camera.Projection = Matrix.CreateOrthographic(this.Width, this.Height, 0, 100);

To get an idea of what an orthographic camera is, imagine a real camera that is hanging 500 feet above our blueprint. In order to see the blueprint, it has to really zoom in. In this case, everything becomes very flat and undistorted—basically, it has become an orthogonal camera.

Drawing the Objects Using Sprites

The walls we just drew to the screen are flat lines floating in a true 3D space. However, with XNA it is also possible to just draw to the viewport coordinates; no 3D to 2D mapping has to be done in this case.

XNA handles this in a very convenient way using Sprites. A sprite is a bitmap drawn into a rectangle that you have specified in advance. It offers a very high performance since your graphics card does all the hard work! The following code draws the furniture to the blueprint using sprites:

C#

private void DrawAllSprites()

        {

            spriteBatch.Begin(SpriteBlendMode.AlphaBlend, SpriteSortMode.BackToFront, SaveStateMode.SaveState);

            foreach (ImageTextureObject tempObject in objectList)

            {

                Rectangle tempBoundingRectangle = this.GetBoundingScreenRectangleFromObject(tempObject);

                spriteBatch.Draw(

                    tempObject.Texture, // the bitmap

                    tempBoundingRectangle, // the bounding rectangle (not yet rotated)

                    null,

                    Color.White,

                    -tempObject.Rotation.Y, // determines the rotation of the sprite

                    new Vector2((float)tempObject.Texture.Width / 2, (float)tempObject.Texture.Height / 2),

                    SpriteEffects.None,

                    1

                );

            }

            spriteBatch.End();

        }

Note that we created an ImageTextureObject such that it contains all information about a piece of furniture: position, orientation, and size in addition to the image of that particular piece of furniture. Now, we loop through the entire list of furniture and use the SpriteBatch.Draw() method draw to the graphicsdevice. Note that any SpriteBatch.Draw() call has to occur within the SpriteBatch.Begin() and .End() statements. This prepares the graphics device to receive the data and allows some optimization to occur after a group of sprites has been received. We draw the handles that appear (see image below) when clicking an object in exactly the same manner:

Receiving User Input

Now, how do we let the user draw lines and drag furniture around? First, we’ll have to subscribe to the MouseDown, MouseMove, and MouseUp events:

C#

this.MouseDown += new System.Windows.Forms.MouseEventHandler(this.bluePrintControl_MouseDown);

this.MouseMove += new System.Windows.Forms.MouseEventHandler(this.bluePrintControl_MouseMove);

this.MouseUp += new System.Windows.Forms.MouseEventHandler(this.bluePrintControl_MouseUp);

Recognizing Button Clicks

When receiving a MouseDown event, we’ll first check if there is a button pressed. We do this using the ButtonPressed() method:

C#

private int ButtonPressed(System.Drawing.Point point)

{

    foreach (ButtonSprite button in buttonList)

    {

        if (button.Clicked(new Point(point.X, point.Y))) return buttonList.IndexOf(button);

    }

    return -1;

}

Each Rectangle object comes with a nice Intersects() method that allows us to check if two Rectangles well…intersect J. We exploit this by constructing a new 1x1 rectangle at the cursor location:

C#

public bool Clicked(Point location)

{

    return this.TargetRectangle.Intersects(new Rectangle(location.X, location.Y, 1, 1));

}

If true, the index of the button is returned by the ButtonPressed() method so that we know what button was pressed. In a similar way, we check if an object was clicked.

Drawing Walls with the Light Pen

If it turns out that there is no button, object, or handle at the MouseDown location, we assume that the user wants to draw a wall. The location sent along with the MouseDown event is in screen coordinates (in pixels); however, in order to draw a wall we need it in “world coordinates” (in meters). For that we employ the following method:

C#

private Vector3 ConvertLocalMouseToWorldCoors(System.Drawing.Point value)

{

    Vector3 worldCoors = new Vector3(

        this.camera.Target.X - this.camera.Width / 2 

            + (this.camera.Width / (float)this.ClientSize.Width) * value.X,

        0,

        -this.camera.Target.Y - this.camera.Height / 2 

            + (this.camera.Height / (float)this.ClientSize.Height) * value.Y

        );

    return worldCoors;

}

This method is called a lot throughout the code, and there is also a method that converts world coordinates to screen coordinates: ConvertWorldCoorsToLocalCoors().

Reading the Point-of-View Device

Now comes some good old hackin’! The whiteboard component – the one that uses the light pen to emulate a mouse – only “sees” one infrared dot and ignores any others it may detect. We are going to tweak his code so that it does not discard these dots but instead passes these on to the BluePrintControl in the form of events, so that we can also track the point-of-view.

Whenever the Wiimote state changes (this may be up to 100 times per second), the mouse emulator will receive an event fired by Brian Peeks WiimoteLib. Instead of immediately passing the first IR dot to the emulator, we’ll tap into the process. No matter how many dots are visible, we’re gonna make sure that the emulator gets only the one dot it needs to “impersonate” a mouse.

Here’s how to tap into the emulator code. Open WiimoteWhiteboardForm.cs from the solution explorer and change the first part of the event listener as follows:

C#

void wm_OnWiimoteChanged(object sender, WiimoteChangedEventArgs args)

{

    mut.WaitOne();

    // wiimote state with the real data

    WiimoteState wsReal = args.WiimoteState;

    // gets which IR point is mouse pointer and which are POV

    int mouseIR = new int();

    int[] POVIR = new int[2];

    int[,] rawIRCoors = new int[5, 2];

    // will contain the raw IR coors in array form for easy access

    this.IdentifyIRpoints(wsReal, ref rawIRCoors, ref mouseIR, ref POVIR);

 // REST OF CODE OMITTED

}

This means that the “hard” work of recognizing point-of-view and light pen is done in the IdentifyIRpoints() method. Now imagine that there are three dots visible. We know that one belongs to the light pen and the others to the point of view. In the case of three visible points we’ll assume that:

1. the two points that are closest to each other belong to the point of view and

2. the remaining dot belongs to the light pen

We are able to make these assumptions since there is only one light pen and one point of view. Because the Wiimote can track up to four points simultaneously, it’s also possible to add another light pen. In this case, the algorithm becomes a little bit more complicated (and probably also less robust) but with some commonsense reasoning it should also be possible. I’ll leave that up to you!

C#

/// <summary>

/// Recognizes which IR points are mouse point and POV points and returns

/// </summary>

/// <param name="ws">WiimoteState object</param>

/// <param name="rawIRCoors">Returns the IR-points in a raw form. Needs an initialized int[5,2]</param>

/// <param name="mouseIR">Returns the index of the IR mouse dot or 0 if not found</param>

/// <param name="POVIR">Returns the indices of the IR POV dots or null if not found</param>

void IdentifyIRpoints(WiimoteState ws, ref int[,] rawIRCoors, ref int mouseIR, ref int[] POVIR)

{

    // Holds all IR points. Note: @index 0 the default values are stored if no point is found.

    // This is needed for faking Chung's implementation.

    int[] IRState = new int[5];

    IRState[0] = 0;

    IRState[1]=Convert.ToInt16(ws.IRState.IRSensors[0].Found);

    // Contains the raw coors in a bidimensional array

    rawIRCoors[0, 0] = 1023; // default value when not found

    rawIRCoors[0, 1] = 1023; // default value when not found

    rawIRCoors[1, 0] = ws.IRState.IRSensors[0].RawPosition.X;

    rawIRCoors[1, 1] = ws.IRState.IRSensors[0].RawPosition.Y;

    // Calculates the total number of points seen

    int noOfPointsSeen = 0;

    for (int i = 1; i <= 4; i++) {

        noOfPointsSeen += IRState[i];

    }

    // Executes recognition of which dot is what based on

    // the amount of dots seen

    switch (noOfPointsSeen)

    {

        // No points seen

        case 0:

            mouseIR = 0;

            POVIR = null;

            break;

        // Only mouse seen

        case 1:

            // passes the dotnumber containing the mouse

            for(int i=1; i<=4; i++) {

                if(IRState[i]==1)

                {

                    mouseIR=i;

                    break; //exit for loop

                }

            }

            POVIR = null;

            break;

        // Only POV seen

        case 2:

            mouseIR = 0;

            POVIR = new int[2];

            // passes the dotnumbers containing the IR dots

            int tempIndex = 0;

            for (int i = 1; i <= 4; i++)

            {

                if ((IRState[i] == 1) && (tempIndex<=1))

                {

                    POVIR[tempIndex] = i;

                    tempIndex++;

                }

            }

            break;

        // determine which dots are closest. There are the POV dots

        // (so I do not remember points but analyze each frame individually

        // Three dots seen

        case 3:

            double smallestDistanceSoFar = 10000000;

            double currentDistance;

            int[] closestTwoPoints = new int[2];

            POVIR = new int[2];

            // Algorithm to calculate the points the smallest distance apart.

            // Only checks the first 3 points (which are all visible).

            for (int i = 1; i <= 3; i++)

            {

                for (int j = i + 1; j <= 3; j++)

                {

                    // Calculates the distance squared between points (using Pythagoras)

                    currentDistance =

                        (rawIRCoors[i, 0] - rawIRCoors[j, 0]) * (rawIRCoors[i, 0] - rawIRCoors[j, 0]) +

                        (rawIRCoors[i, 1] - rawIRCoors[j, 1]) * (rawIRCoors[i, 1] - rawIRCoors[j, 1]);

                    // If this distance is smaller than the previously smallest distance,

                    // the smallest distance so far is updated, as well as the array

                    // containing the indices of this pair

                    if (currentDistance < smallestDistanceSoFar)

                    {

                        smallestDistanceSoFar = currentDistance;

                        POVIR[0] = i; POVIR[1] = j;

                    }

                }

            }

            // The mouse index is returned (can be calculated easily

            // based on the fact that the the sum of all three indexes must be 0)

            mouseIR = 6 - (POVIR[0] + POVIR[1]);

            break;

        // Four dots seen: not supported so no output

        case 4:

            mouseIR = 0;

            POVIR = null;

            break;

    }

}

Consequently, we create a new WiimoteState object only if a light pen is seen. This stripped-down state is then passed on to the emulator code, effectively filtering the information received by the emulator. An event with the point-of-view data is received by the BluePrintControl, which subsequently updates the camera position. I’ll leave checking for that in the source files to you.

Inserting Images

At the bottom of the BluePrintControl, there is a strip with images that can be inserted as objects. It is a so-called FlowLayoutPanel (found in the System.Windows.Forms namespace) and displays all .png images (“contained” in PictureBoxes) in a certain directory. Make sure you set the right directory by modifying the defaultImageDirectory private variable in the blueprint form (not the control).

I won’t run you through the code to add images to the FlowLayoutPanel because it is quite obvious. However, it’s worth noting that we’ll use a System.IO.FileSystemWatcher to update the list:

C#

private bool SetUpFileSystemWatcher()

{

    try

    {

        fileSystemWatcher = new System.IO.FileSystemWatcher(defaultImageDirectory, defaultSupportedImageMasks);

    }

    catch (Exception ex)

    {

        System.Console.Out.WriteLine("Error: {0}", ex.Message);

        return false;

    }

    this.fileSystemWatcher.Changed += new FileSystemEventHandler(fileSystemWatcher_Changed);

    this.fileSystemWatcher.EnableRaisingEvents = true;

    return true;

}

Drag ‘n’ drop

The drag procedure is initiated when the user clicks an image:

C#

// initiates the drag procedure

private void pictureBox_MouseDown(object sender, MouseEventArgs e)

{

    PictureBox pictureBox = (PictureBox)sender;

    pictureBox.DoDragDrop(pictureBox.Tag, DragDropEffects.Copy);

} 

Now, we’ll have to subscribe to the DragEnter (when the cursor enters the control while dragging is in progress) in the BluePrintControl and DragDrop (when the user drops an object by releasing the button) events:

C#

// Subscribes to (drag and) drop user actions

this.DragEnter += new DragEventHandler(BluePrintControl_DragEnter);

this.DragDrop += new DragEventHandler(BluePrintControl_DragDrop);

In the method handling the DragDrop event, we can receive the object containing information regarding the image using the GetData method:

C#

// called on a drop (as in drag & drop)

void BluePrintControl_DragDrop(object sender, DragEventArgs e)

{

    ImageObject tempImageObject = 

        (ImageObject)e.Data.GetData("DualDesign.Data_containers.ImageObject");

    // code below omitted

}

Coding the 3D Control

Thread-safety

We’ve come a long way by programming the blueprint. Since the 3D view doesn’t have to accept direct user input – all that is handled by the blueprint – all it has to do is display the home as-is. There are three types of data that the 3Dcontrol has to receive:

1. The walls

2. The objects

3. The location and orientation of the point of view

This introduces some problems related thread-safety: the 3D control and blueprint run asynchronously on separate threads. Imagine that the 3dview is reading an object that is just in the process of being deleted; in this case data corruption may occur and the application may crash with some seemingly unrelated error code.

We’ll use two different methods to avoid these issues in order to assess performance. The objects are stored in a shared static class that uses locking to prevent data corruption. On the other hand, the walls and the point of view location object are sent in the form of events that have a copy of the data attached. Now I hear you ask: why use two different methods? That’s because I wanted to assess the performance of both methods, which, by the way, turned out to be similar.

Here, I’ll focus on only the objects because the locking procedure they employ is cleaner and easier to implement. If I’d have to do the project again, I wouldn’t even bother using the data-exchange-through-events method.

Locking the object list

First, we’ll have to go back to the BluePrintControl for a moment. Internally, this control keeps track of the list containing all objects that have been placed on the blueprint. Also, we’ll have to make sure we can access this data from the 3D Control by implementing a static property in the BluePrintControl. In order to make it thread-safe, we’ll lock access to the list: simultaneous reads are possible, though simultaneous reads/writes are not.

ReaderWriterLockSlim, a class written by a programmer who was unhappy by the low-performing ReaderWriterLock class included in .NET, is what we are going to use for locking. This slimmed down version performed so well that Microsoft eventually bought it and included it in subsequent versions of .NET, and so we are now reaping the fruits J :

C#

static ReaderWriterLockSlim texturedObjectListLock = new ReaderWriterLockSlim(LockRecursionPolicy.SupportsRecursion);

static List<ImageTextureObject> objectList = new List<ImageTextureObject>();

/// <summary>

/// Contains all ImageTextureObjects

/// </summary>

public static List<ImageTextureObject> ObjectList

{

    get 

    {

        texturedObjectListLock.EnterReadLock();

        try { return BluePrintControl.objectList; }

        finally { texturedObjectListLock.ExitReadLock(); }

    }

    private set

    {

        texturedObjectListLock.EnterWriteLock();

        try { BluePrintControl.objectList = value; }

        finally { texturedObjectListLock.ExitWriteLock(); }

    }

}

Unfortunately, however, we haven’t achieved complete thread-safety: access to the list is locked, but not to the objects themselves! For this reason the access to properties of the ImageTextureObjects need locking too:

public Vector3 Position

{

    get 

    {

        objectLock.EnterReadLock();

        try { return position; }

        finally { objectLock.ExitReadLock(); }

    }

    set 

    {

        objectLock.EnterWriteLock();

        try

        {

            dataChanged = true;

            position = value; 

        }

        finally { objectLock.ExitWriteLock(); }

    }

}

Always use a try/finally combination to ensure that the writelock is released in all cases, no matter what.

Drawing to the GraphicsDevice

Now that we have reliable access to all the data we need, let’s actually draw our house in 3D! Of course, we rely heavily on XNA to get this done. This is what the Draw() method looks like:

C#

protected override void Draw()

{

    // sets up device for new rendering round

    this.GraphicsDevice.Clear(Color.White);

    this.GraphicsDevice.RenderState.CullMode = CullMode.CullClockwiseFace; // hides clockwise faces

    basicEffect.World = Matrix.Identity;

    basicEffect.View = camera.View;

    basicEffect.Projection = camera.Projection;

    // draws elements to screen

    this.drawWalls();

    this.drawSkySphere();

    this.drawImageObjects(); // these should be drawn last due to transparency

}

A graphics cards works by rendering large amounts of triangle surfaces. Often, you only need to draw one side of the triangle, as is the case for us: since our walls have thickness, one side of each triangle is facing “into” the wall itself and will never be seen from that side. Therefore, we use the GraphicsDevice.RenderState.CullMode to tell the graphics card to not even bother drawing the triangle when the camera is on the “wrong” side of the triangle. This enhances performance, though not so much in our case since our scenes are very simple, but it is still a small effort to tell the graphics card this.

Three essential matrices are also set: the Projection matrix determines the camera’s lens, the View matrix the camera’s position, and the World matrix the position of the drawn objects.

Looking at the drawWalls() method, the walls are drawn using the userfriendly DrawUserIndexedPrimitives method. We basically give two lists. The first contains a set of coordinates in 3D space (the corners of the walls) and the second contains a list of integers that refer to coordinates in the other list. In this way, we can easily construct a series of triangles while only specifying the coordinates of the corners once:

C#

GraphicsDevice.DrawUserIndexedPrimitives<VertexPositionNormalColor>

(

    PrimitiveType.TriangleList,

    this.All3DWallVertices, 0,

    this.All3DWallVertices.Length,

    this.All3DWallIndices, 0,

    this.All3DWallIndices.Length / 3

);

The images are displayed by drawing textured triangles. In the DrawImageObjectTexturedFaces() method, we use the VertexPositionTexture custom vertex format to this end. This not only defines the texture itself, but also defines texture coordinates: these specify which part of the texture should be drawn onto a triangle:

C#

this.GraphicsDevice.DrawUserIndexedPrimitives<VertexPositionTexture>

(

    PrimitiveType.TriangleList,

    imageTextureObject.TexturedFaceVertices,

    0, 4,

    imageTextureObject.TexturedFaceIndices,

    0, 2

);

Conclusion

The code is quite comprehensive, so I’ve only been able to explain a few parts and ideas. If you want to try this out, the download link for the source code is at the top of the article! And be creative! Why not create a smart Google Earth mash-up in which you can use point-of-view to navigate intuitively though a landscape? Surely there are also some innovative games you can think of. Be inspired by the Nintendo DS games that make use of a similar dual display layout!

About The Author

Thijs Brilleman has a Master in Mechanical Engineering but has always been interested in fun, tangible machines that are somewhere in-between computers and mechanical machines. Currently employed at the University of Twente, he designs new interaction methods that can be used in collaborative design. You can reach him at: thijsbrilleman@gmail.com

Introduction

Back in July of 2009 I learned of Neurosky's MindSet device via Engadget.com.  The MindSet is a a brain-wave sensing headset costing only $200.  I contacted the company and was placed on a short list to receive hardware when available to build a demo for PDC 2009.  This is the result of that project.

MindBlaster is a pretty simple game written using XNA which runs on a Windows PC in conjunction with the MindSet headset and a Nintendo Wiimote.  The object of the game is to move your head, which moves a targeting reticule on the screen, to aim at enemy alien ships, then focus your concentration on the selected ship to heat it up and eventually make it explode.  The more you focus, the faster the alien ship explodes.  As more enemies are destroyed, they faster they fire back at you, causing a pretty frenetic gameplay experience until the onslaught becomes too much and you are destroyed.

So let's take a look at some of the components that went into creating MindBlaster and how the integrate into the final game.

MindSet Headset

As stated, the Mindset is a headset from Neurosky that is capable of reading alpha, beta, theta and gamma brain waves via 4 dry sensors, 3 located on the left earpiece, and one on an arm that touches the forehead.  In addition, the device contains a microphone and ear speakers, making it a full-on audio headset as well.

This device connects to a PC via Bluetooth and shows up as a Bluetooth serial port.  This means we can communicate with the device extremely easily.  However, Neurosky also includes their own API with a simple C# wrapper that makes .NET development even easier.  On top of that, I wrote a simple API which somewhat mimics my Managed Wiimote Library, which is available as a separate download from this application to make .NET development with the Mindset a bit easier.

One of the interesting things about the MindSet is that you don't actually need to know what brain-waves are or how they work.  I don't know the difference between an alpha wave and a theta wave, but I was still able to produce a decent game using brain waves.  Neurosky has developed a proprietary algorithm which takes your brain-wave values and turns them into a single score from 1 to 100 of “attention” and “meditation” allowing you can very easily determine how hard a person is concentrating or relaxing.

Using the provided API is pretty simple.  Here is a chunk of code that connects to the headset and pulls brain-wave values in a loop.  You wouldn't want to use this directly in your application, but it covers the important points:

// get a connection ID
int connectionId = ThinkGear.TG_GetNewConnectionId();
if(connectionId < 0)
    return false;

// connect to a COM port
int connect = ThinkGear.TG_Connect(_connectionId, @"\\.\COM1", baud, ThinkGear.STREAM_PACKETS);
if(connect < 0)
    return false;

while(true)
{
    // read data packets
    int packetsRead = ThinkGear.TG_ReadPackets(_connectionId, -1);
    if(packetsRead > 0)
    {
        // get the attention value
        if(ThinkGear.TG_GetValueStatus(_connectionId, ThinkGear.DATA_ATTENTION) != 0)
            int attention = ThinkGear.TG_GetValue(connectionId, ThinkGear.DATA_ATTENTION);

        // get the meditation (relaxation) value
        if(ThinkGear.TG_GetValueStatus(_connectionId, ThinkGear.DATA_MEDITATION) != 0)
            int attention = ThinkGear.TG_GetValue(connectionId, ThinkGear.DATA_MEDITATION);
    }
}

This opens a connection ID with the API, opens a connection to the headset on a provided COM port, then, in a loop, reads all available packets from the headset and retrieves the current state of the values requested.

I have created a very simple library which interfaces with the headset and put it up as a separate download.  If you're just interested in using this headset with .NET, please have a look at my ThinkGearNET library.

Wiimote

A Nintendo Wiimote is strapped upside-down to the top of the headset and its IR camera is used for 2D head tracking.  For PDC, I removed the Wiimote's guts from its case and mounted them directly to the headset, adding a battery holder box from Radio Shack.  This isn't really necessary, and you can just mount your own cased Wiimote on top.

The Wimote contains a 1024x768 camera which can detect the positions of up to 4 IR sources.  For this application, I only needed a single IR source since rotation is unimportant.  I used the Nyko Wireless Sensor Bar for this source with one side covered in electrical tape.

Using my Managed Wiimote Library, I am able to poll the current state of the IR camera to pull the XY position of the IR source and translate that to the position of the targeting reticule on the screen.

WiimoteState wmState = MindBlasterGame.Wiimote.WiimoteState;
if(wmState.IRState.IRSensors[0].Found)
{
    _reticule.Position.X = (wmState.IRState.IRSensors[0].Position.X) * MindBlasterGame.ScreenSize.X;
    _reticule.Position.Y = MindBlasterGame.ScreenSize.Y - (wmState.IRState.IRSensors[0].Position.Y) * MindBlasterGame.ScreenSize.Y;
}

Game Architecture

The game uses parts of the XNA Creators Club Game State Management sample for the overall game and screen architecture.  Each screen is broken out into its own object with its own Update, Draw, LoadContent, etc. methods.  This allows the title screen, settings screen, gameplay screen, and final scoring screen to remain separate “entities” for easier development.

Let's talk a little bit about each screen and important points for each.

Settings Screen

While developing at home and demonstrating this at PDC, I needed a very quick and dirty settings screen to easily enable and disable various features of the game.  For example, I really didn't want to wear the headset the entire time I was developing, so I wrote some code which allowed me to use the mouse in place of the actual hardware.  The settings screen allowed me to easily toggle those types of things.  This was accomplished by creating an object that inherits from the MenuScreen class from the Game State Management sample.  Menu entries can be then created as follows:

private readonly MenuEntry _menuEntry;
_menuEntry = new MenuEntry("My menu entry");
_menuEntry.Selected += _menuEntry_Selected;
MenuEntries.Add(_menuEntry);

This creates the entry, sets its event handler, and adds it to the list.  In the event handler, you can do whatever might be required; toggle an option on or off, increase or decrease a value, etc.

Title Screen

The title screen doesn't do much other than display the title graphic and the “Press Start” text, or any additional text noting if the headset is detached, the Wiimote is not connected, or other error conditions. 

Gameplay Screen

This is where the magic happens.  Perhaps surprisingly, there isn't a ton of code in the GamePlayScreen.cs file itself.  The logic for the game is quite simple:  move the targetting reticule based on the Wiimote IR position, detect if it's over an enemy, if it is, remove hitpoints from the enemy, and if the hitpoints drop below 0, blow up the enemy.  In addition, the enemies fly around the screen and fire missiles at the player.  Let's break down some of these elements.

Enemies

Every drawable object in the game derives from the Drawable2D or Drawable3D base classes.  These classes define some simple properties like position, origin, scale, etc. and provide some methods which must be implemented by the derived classes, like LoadContent, Update and Draw, matching the architecture of a standard XNA application.  Here is what the Drawable2D class looks like:

public abstract class Drawable2D
{
    public Vector2 Position;
    public Vector2 Origin;
    public float Rotation;
    public float Scale;

    public Drawable2D()
    {
        Scale = 1.0f;
        Visible = true;
    }

    public Drawable2D(Vector2 position)
    {
        Position = position;
    }

    // Update/Draw/Load which needs to be implemented by all Drawable types
    public virtual void LoadContent(ContentManager contentManager) {}
    public virtual void Update(GameTime gameTime) {}
    public virtual void Draw(GameTime gameTime) {}

    public bool Visible { get; set; }
}

The Drawable3D class looks almost identical to this, but moves everything to 3D space using Vector3 to denote positions.  It also includes a Model property to hold the geometry of the object, and a property to hold the current World transformation matrix. 

public class Drawable3D
{
    public Vector3 Position;
    public Vector3 Rotation;
    public float Scale;

    protected Model Model;
    protected Matrix World;

    private Vector2 _position2D;

    protected Drawable3D()
    {
        Scale = 1.0f;
        Visible = true;
    }

    public Drawable3D(Vector3 position)
    {
        Position = position;
    }

    public Vector2 GetPosition2D(Camera camera)
    {
        Vector3 position = MindBlasterGame.ScreenManager.GraphicsDevice.Viewport.Project(this.Position, camera.Projection, camera.View, Matrix.Identity);
        _position2D.X = position.X;
        _position2D.Y = position.Y;
        return _position2D;
    }

    // Update/Draw/Load which needs to be implemented by all Drawable types
    public virtual void LoadContent(ContentManager contentManager) {}
    public virtual void Update(GameTime gameTime, Camera camera) {}
    public virtual void Draw(GameTime gameTime, Camera camera) {}
    public bool Visible { get; set; }

The enemy ships and missiles were modeled in 3D Studio Max and exported to the .X file format to be used with the XNA Content Pipeline by kW X-port, a free plugin for 3D Studio Max that can export models and animations to the .X format with a variety of user-controllable options.

Rotate and fly

If you've played the game or watched the video above, you've seen how the enemies rotate toward a position, fly to that position, and then rotate back toward the camera.  To make the ships behave in this way, a random position in 3D space is calculated.  Then the following code is run to rotate the ship to that position:

protected Matrix RotateToTarget(Vector3 position, Vector3 target)
{
    // get direction from position to target
    Vector3 vecToTarget = Vector3.Normalize(target - position);
 
   // get angle to rotate
   double angle = Math.Acos(Vector3.Dot(vecToTarget, Vector3.Backward));

   // get axis to rotate on
   Vector3 axisToRotate = Vector3.Cross(Vector3.Backward, vecToTarget);

   axisToRotate.Normalize();

   // create a rotation matrix from the calculated axis and angle
   return Matrix.CreateFromAxisAngle(axisToRotate, (float)angle);
}

This code uses some vector math to get the direction the random point is in relation to the current ship orientation.  The rotation angle is then calculated, the axis to actually rotate on depending on the location, and finally a rotation matrix is calculated and returned.

This returned rotation matrix is used to rotate the ship over a short period of time using the Vector3.SmoothStep method, which interpolates between two values using a cubic equation.  Once it has reached the target rotation, the ship then “smooth steps” to its destination position, then “smooth steps” back to being rotated toward the camera, using that same RotateToTarget method above, this time with the target position being the camera position.

Using XNA Samples

I used a few samples from the XNA Creators' Club in MindBlaster.  I think there are some fabulous samples available on the site that make life easier for all XNA developers.  I can't speak for everyone, but I'd rather pull out a known working, tested particle engine than spend the time to create one from scratch, especially on a deadline like I had for this project.  Earlier in the article I explained how I used the Game State Management sample to provide the base screen architecture for my application.  I also used its MenuScreen class to build my settings screen as described above.

Particle Engine

I also used the 2D particle engine from the NetRumble sample.  While my game is really in 3D, I was able to use some trickery to throw a particle effect up on the screen in 2D space on top of the ship's location in 3D space which provide a reasonable looking effect.

The particle engine provides sample effects for various explosions and trails.  I used these effects for the enemy ship exploding, the enemy ship firing missiles, and the enemy ship spawning into view.

To get the 2D screen-space position of a 3D object in world-space, I used the following method, which is part of the Drawable3D class:

public Vector2 GetPosition2D(Camera camera)

{
    Vector3 position = MindBlasterGame.ScreenManager.GraphicsDevice.Viewport.Project(this.Position, camera.Projection, camera.View, Matrix.Identity);
    _position2D.X = position.X;
    _position2D.Y = position.Y;
    return _position2D;
}

This uses XNA's Viewport.Project method to return the 2D position of this 3D object using the current 3D position, the current camera projection matrix, and the camera's view matrix and turns it into a 2D point with no additional effort.

So, when an enemy ship explodes or a missile is fired, the 3D object's 2D position is calculated, and the 2D particle effect is displayed at that position.

The background Starfield was also taken from NetRumble.  This is a single class file that can be was easily pulled right out of the sample code and plopped into any XNA application to give a very nice moveable background.

Bloom effect

The Bloom Postprocessor sample on the XNA Creators Club site provides a very easy to use bloom DrawableGameComponent.  A DrawableGameComponent is a class which implements a few methods such as Update and Draw, and then added to the Game object's Component collection.  During each frame, the appropriate methods are called and the object updates and draws.

Blooming is a neat effect that produces some artifacts caused by bright lights.  Think of how your vision reacts when you walk from a dark room out into a bright sunny day.  I use this effect throughout the game to give the ships a neat glowing effect as they heat up.  Also, when an enemy missile hits the player's screen, I flash the screen white for a few milliseconds, and the bloom effect in combination with this produces a nice effect similar to staring at a bright light and then looking quickly looking at something dark as your eyes adjust to the light difference.

Using the BloomComponent is amazingly simple.  In the game's Initialize method:

Bloom = new BloomComponent(this);
Components.Add(Bloom);
Bloom.Settings = BloomSettings.PresetSettings[3];

With this in place, the object does the rest by hooking into the drawing surfaces and applying the effect every frame.  The sample includes several different bloom effects with varying parameters, so its easy to experiment with these values and see what works best for your game.

How to Install and Play

Ensure you have either XNA Game Studio 3.1 installed, or the XNA 3.1 redistributable package.

To play, start the executable and configure the settings that are appropriate to your setup.  For example, you can turn off the Wiimote or the headset if you don't have the hardware to support it.

Up/Down/Left/Right - navigate setup menus
1 - Turn on debug info in game
F1 - Back
F2 - Start
Page Up - Reset the game

If you have turned off Wiimote support in the setup screen, the mouse can be
used to move the targetting reticule.

If you have turned off headset support in the setup screen, the attention
level can be increased/decreased with the left/right mouse buttons.

Move the reticule over an enemy and focus your attention on the enemy to make
your attention score increase.  The higher your attention score, the faster
the enemy heats up and explodes.  As you play, enemies will start to fire back
at you.  Move the reticule over the missiles and focus your attention to blow
them up before they hit you.

Conclusion

There is obviously more to the game than what is seen here, however it would be impossible to cover every single bit of the code in an article here.  I have tried to cover the important and more difficult parts here and show how it's very easy to leverage existing samples to get an XNA game up and running quickly.

If you have any questions or would like further explanation on how a certain portion of the game works, please contact us and I will be happy to answer the questions and perhaps expand the article further.

Thanks

• Joey Buczek for all of the 3d models and artwork in the game
• David Wallimann for all of the sound effects and music in the game
• Rick Barraza for the idea of moving the game into a space environment.  Better than the road I was heading down…
• Greg Hyver and Johnny Liu from Neurosky for getting me set up with headsets and providing loaners for PDC
• Michelle Leavitt for an infinite amount of playtesting
• The XNA folks who created the various tutorials and starter kits whose code saved me a ton of time in creating the game.  Without these, I'd probably still be writing particle engines, bloom shaders and various other pieces from scratch….these are a huge time saver!

Bio

Brian is a Microsoft C# MVP who has been actively developing in .NET since its early betas in 2000, and who has been developing solutions using Microsoft technologies and platforms for even longer. Along with .NET, Brian is particularly skilled in the languages of C, C++ and assembly language for a variety of CPUs. He is also well-versed in a wide variety of technologies including web development, document imaging, GIS, graphics, game development, and hardware interfacing. Brian has a strong background in developing applications for the health-care industry, as well as developing solutions for portable devices, such as tablet PCs and PDAs. Additionally, Brian has co-authored the book "Coding4Fun: 10 .NET Programming Projects for Wiimote, YouTube, World of Warcraft, and More" published O'Reilly. He previously co-authored the book "Debugging ASP.NET" published by New Riders.  Brian is also an author for MSDN's Coding4Fun website.