Legacy software written for Windows 7 and prior targeted the traditional Windows API, often referred to as Win32. They resulting apps would be a combination of an executable, assets and libraries. The Windows 8 operating system introduces a new type of application, called a Windows Store app. The new architecture is called Windows Runtime, or WinRT for short. Windows Store apps are written exclusively against this new WinRT architecture.

Windows 7 and prior only had one mode of interacting with software, launching the application's executable from the desktop or file path. Windows 8 now defaults to a Windows Runtime start screen, where Windows Store apps are presented and can be accessed through their respective Live Tiles. An application's Live Tile is registered automatically when that app is installed from the Windows Store. If a user needs to access legacy software that hasn't been migrated to the new Windows Store format, they can still get to the desktop from the Start Screen.

Windows desktops and laptops have traditionally run on Intel processors, referred to frequently as x86 or x64 machines. However, in parallel with the release of Windows 8, an optimized version of the operating system designed for mobile ARM based devices was released called Windows RT. Windows RT devices, however, will not run legacy Win32 software, nor can you develop Win32 software for content.

So unlike a Windows 8 computer, which is optimized for the Windows Store apps written against WinRT but still provides access to Win32 software from the desktop, Windows RT devices do not.

What about the Microsoft Surface?

There are two versions of the Microsoft Surface. The first version runs the Windows RT operating system and uses an ARM CPU. The Surface Pro, however, is an x64 device and runs Windows Pro 8. Both devices are optimized for Windows Store apps, but the Surface Pro is a fully featured computer as well, allowing you to install Visual Studio, Photoshop, or whatever else you wish. You can think of the Microsoft Surface as a light, long lasting consumer consumption device, whereas the Surface Pro is a powerful, mobile content creation device and netbook equivalent.

What's up with Cinder and DirectX?

Cinder is a cross platform framework for C++ design engineering. In order to run and Apple, Android and Windows devices prior to Windows 8, the rendering libraries leveraged OpenGL as the open source, cross platform solution.

Even though DirectX has been Microsoft's preferred rendering solution for advanced graphics programming, Windows traditionally provided both DirectX and OpenGL support from the desktop. However, Windows 8 Store applications are now exclusively DirectX based, to maximize reusability across the entire domain of Microsoft experiences including Windows Phone, Windows RT, Windows 8, and gaming. This means that Cinder apps that leverage OpenGL for their rendering will not work as Windows Store apps.

To get a Cinder project running as a Windows Store app, the Cinder project needs to leverage a DirectX render instead of the default OpenGL renderer.

Both OpenGL and DirectX have undergone significant changes over the years, with DirectX migrating away from the simpler fixed function pipeline of OpenGL early on. As hardware and software continue to advance, both libraries continue to progress in their own distinct ways. Windows 8 Store apps leverage the most recent version of DirectX 11.1, which itself has some breaking changes from DirectX 11, most notably is the loss of D3DX, a very popular helper library.

As this document targets using Cinder for Windows Store apps, we will be using DirectX 11.1. Whereas you could have a DirectX renderer for desktop based Cinder experiences, because of the history and community support behind the existing OpenGL render, it would be a difficult choice to argue. However, to create a Windows Store app, DirectX 11.1 must be used as the underlying rendering technology in the Cinder framework.

What about Visual Studio?

Previous versions of Cinder were written to make it easy to use for Visual Studio 2010, and targeted Win32 applications. Since Windows Store apps require targeting the WinRT architecture, this has changed.

To develop software for Window Store 8, you need to use Visual Studio 2012. We will be walking through the setup assuming you have downloaded the free version of the software, Visual Studio Express 2012.

You should know, however ,that the professional versions of Visual Studio 2012 comes with several new features for professional graphics engineering, including a robust graphics debugger, DirectX support, visual shader editors, and tools for working with virtual models and objects.

Enough theory, lets get started

To create Windows Store apps, you need to have Windows 8 installed and running at least Visual Studio Express 2012 for Windows 8. Do not download Visual Studio Express 2012 for Web, Phone or Desktop. We are targeting Windows 8 here.

Setting up Visual Studio Express 2012 for Windows 8

Once you have a copy of Windows 8 running natively or on a virtual machine from another Operating System, you can get the bytes for Visual Express 2012 for Windows here:

http://www.microsoft.com/visualstudio/eng/products/visual-studio-express-products

When the installer has finished and you launch the application, you will be taken to a registration screen. If you choose not to register, Visual Studio Express 2012 will expire after 30 days. Go ahead and register the product.

Once registration is complete, Visual Studio will launch for the first time. In order to create a Windows Store app, you need to setup a Developer license. A developer license allows you to develop, install and test apps on your computer before submitting them to the Microsoft store for publishing and deployment to the community.

At this time, it would also be a good idea to go to the menu item Tools > Extensions and Updates and check for any updates to Visual Studio as well:

Grabbing the Git branch

Now that we have Windows 8 and Visual Studio Express 2012 setup, our next step is getting the correct Cinder bits on our machine. Since this is still under development, we need to pull from the correct developer branch of Cinder from Cinder's GitHub account.

The professional version of Visual Studio 2012 allows for rich Git integration, but the free Express version doesn't support this, so you will need to have a standalone Git client if you choose to use the Express version of Visual Studio. If so, we recommend downloading the GitHub client, which installs both itself and helpful Git utilities.

You can download the GitHub client here:

https://help.github.com/articles/set-up-git

After the GitHub client is installed, you may find it helpful to enter or setup your account. However, you should be able to pull the project if you just want to get up and running quickly using the Git Shell as explained further below.

As a community developed, open source project, Cinder separates the stable, official version of its Cinder framework from newer branches that developers setup as they program future functionality. WinRT support for Cinder is currently being developed off of one of these feature branches, with the goal of integrating in to a future iteration of the master project.

So we will need to clone the Cinder project locally, which defaults to the master branch, and then explicitly move to the dx_rt branch to get to the version under development that also includes support for Windows Store 8 projects.

To clone Cinder, either open your Git enabled Power Shell in the directory you want to use and type:

git clone https://github.com/cinder/Cinder.git

Or open a browser and go to the Cinder project page on GitHub.com. There you will see a Clone In Windows button that should launch the GitHub client automatically and start the local cloning process:

https://github.com/cinder/Cinder/tree/dx_rt

This should take several minutes to download the Cinder project and open it in your GitHub client. Once it appears as a line item in your GitHub client, you can right click the repository icon and select open a shell here. We will do the rest of the commands from the Power Shell directly.

If you look at your command prompt and you are still on the master branch, you can switch to the dx_rt branch by typing the following:

git checkout dx_rt

Or if you prefer to use the GitHub client you installed, you can switch from the master branch to the dx_rt branch from there as well.

If you are using the GitHub client, though, it would probably be a good idea to switch to a command shell for the next steps. You can do that from the client directly by right clicking on project's list item like so:

The master, dev and dx_rt branches all require you to also install certain Boost libraries to the Cinder project directory before you can compile the Cinder framework. However, on the dx_rt branch we will be using, this process has been greatly automated for you. So once you've switched to the dx_rt branch, type in the following two commands to download the required Boost libraries that are compatible with this particular branch of Cinder for Windows Store:

git submodule init

git submodule update

This may take a few minutes, but once it is done you should be ready to open the Cinder project and compile it for use.

Compiling Cinder for Windows Store apps

Once you have pulled the Cinder project from its GitHub repository, switched to the dx_rt branch that contains the bits for working on Windows Store apps, and run the submodules that download the additional Boost dependencies, you are ready to open up the Cinder solution to compile the library.

Go to where GitHub cloned the Cinder project, the default is Libraries\Documents\GitHub\Cinder. There you will find multiple version of the Cinder framework separated by development environment. You should see the xcode and vc10 solutions, but you will also three additional folders. Vc11 provides support for compiling Cinder for Visual Studio 2012, but still using OpenGL and thus limited to Desktop mode. VC11-DirectX is provides support for compiling Cinder for Visual Studio 2012 using DirectX 11 but also for Desktop mode. We will be focusing on the third option, the winrt folder. This is the solution that supports Windows Store apps and development.

Launch Cinder/winrt/cinder.sln and the Cinder project should open up in Visual Studio 2012 automatically. If you are using the professional version of Visual Studio 2012, you can Batch Build all the versions you need. Since we are using the Express edition in this walkthrough, though, we will Build individually the debug and release versions of the Cinder library we need.

We will build the debug version of the framework by making sure that our Solution Configuration in the toolbar is set to Debug and the Solution Platform is set to Win32.

To build the solution, simply go to the Build menu option and select Build Solution.

Some warnings may pop up, but there should be no errors. Once that is done, you will find a new file in the Cinder project directory under:

..\Cinder\lib\winrt\Win32\Debug\cinder.lib

This is the Cinder library your Windows Store projects will link to when you are programming and debugging your application. Now we need to rebuild this framework to provide a release version of this same framework. Going back to the Cinder project, and on the toolbar change the Solution Configuration option from Debug to Release and Build the solution again.

You now have a Debug and Release version of the Cinder framework and are ready to start creating Windows Store programs against it for any Windows 8 device, including the Surface Pro. However, any WindowsRT device, such as the non-Pro version of Surface, will be running on ARM and so you would need to follow the above steps targeting the ARM platform instead of just the Win32 platform.

Testing out the Library

At this point, you are ready to explore the collection of DirectX samples created specifically for Windows Store apps. You can find various projects for Windows Store in the DirectX folder inside the samples folder. For our initial test, we will open the following sample:

..\Cinder\samples\DirectX\_winrtBasic\winrt\winrtBasic.sln

When you open the solution in Visual Studio Express 2012, you will find basicApp.cpp in the Solution Explorer. This contains all the code we need to start using the Cinder framework.

The code in this sample illustrates a few basic tasks, including loading an image from the assets folder and using it as a texture, loading fonts and displaying them on the screen, handling mouse events, drawing various geometries and implementing a basic follower easing algorithm in the update() call.

Once you're looked over the code, you can press F5 or the Play button on the main toolbar to run the program on the machine. If you compiled both the Debug and Release versions of the framework, you can select either Solution Configuration. You will find a significant increase in speed in Release mode over Debug mode.

You will also notice that the program has automatically installed its own Live Tile on your Start Screen, which you can use to launch the program again whenever you wish.

 Congratulations! You should now be ready to start exploring Cinder for Windows Store apps.

@rickbarraza

You can learn more about this branch by going to the Cinder forums found here:

https://forum.libcinder.org/#Topic/23286000001540037

The system has two core components:

• A server application, which is a Win32 console application that handles incoming queries and returns data to the connected clients. This application runs on the desktop machine with the connected sensor.
• The Windows Phone application that sends requests to the target server and can trigger a variety of pre-programmed commands.

The Basics

Hardware and Communication Infrastructure

One of the most important parts of the project is the signal capture and replication hardware. For the purposes of this project, I decided to use a dual-mode Phidgets IR sensor. It supports both IR code capture and subsequent replication. From a user’s perspective, this device also eliminates a substantial code-learning overhead as well as the potential error rate. Instead of searching for a device-specific hexadecimal sequence that later has to be transformed in a working IR code, the user simply has to point his remote control at the sensor and press the button that he wants accessible from a mobile device. Given that the capturing software is running on the target machine, once the sensor detects that a code can be repeated within an acceptable precision range, it will be automatically captured and stored, with all required transformations worked out in the backend using the free Phidgets SDK.

Even though I can, I don’t have to handle the binary code content received through the sensor—the Phidgets .NET libraries carry built-in types that contain all the processed metadata that I will discuss later in this article.

This sensor is connected through a USB port to a machine that acts as a communication gateway. This server should have port 6169 open for inbound connections.

NOTE: The port number can be changed, but you have to keep it consistent between your server and client applications.

The communication between the phone and the computer running the client is performed via a TCP channel—sockets are used to perform the initial connections and serialized data transfer. You can see the generalized data flow between the devices that are involved in the procedure in the graphic below:

The server (desktop client) handles the local storage and release of all incoming IR codes. The mobile client has to know the location of the server—once specified and confirmed, it can send one of the pre-defined commands to it and either query the server for existing command groups (sets) or invoke one of the stored IR codes. When I pass data between devices, I use JSON for the serializable components. The data is also processed before being sent in order to speed-up the process—for example, on the server side the sets are serialized together with the associated codes. Like this:

[
{
    "Name":"batman",
    "IsList":false,
    "Commands":[
    {
        "Name":"test command",
        "Code":{
            "Mask":{
                "BitSize":12,
                "CodeData":"AAA="
            },
            "BitSize":12,
            "Encoding":2,
            "CarrierFrequency":38000,
            "DutyCycle":50,
            "Gap":44761,
            "Header":[
            2374,
            606
            ],
            "CodeData":"DJA=",
            "MinRepeat":5,
            "One":[
            1189,
            606
            ],
            "Repeat":null,
            "Trail":0,
            "Zero":[
            582,
            606
            ]
        }
    },
    {
        "Name":"turn off",
        "Code":{
            "Mask":{
                "BitSize":12,
                "CodeData":"AAA="
            },
            "BitSize":12,
            "Encoding":2,
            "CarrierFrequency":38000,
            "DutyCycle":50,
            "Gap":44770,
            "Header":[
            2360,
            613
            ],
            "CodeData":"DJA=",
            "MinRepeat":5,
            "One":[
            1169,
            613
            ],
            "Repeat":null,
            "Trail":0,
            "Zero":[
            585,
            613
            ]
        }
    }
    ]
}
]

The inherent problem with the JSON data above is the fact that the phone client does not need the information related to the code binary sequence and all the metadata that goes with it. So it is effectively stripped down and reduced to the names of the sets (when a list of sets is requested) and commands (when a list of commands is requested).

The Data Model

As you saw from the description above, the server organizes individual infrared codes in sets. A single set is a bundle of codes that may or may not be related to each other—ultimately, this is the user’s decision. A good example of using sets is organizing IR commands by rooms, devices or code types. Each set has a unique name on the server, therefore eliminating the possibility of a request conflict.

Each set stores individual commands built around the Command model:

namespace Coding4Fun.MissionControl.API.Models 
{ 
    public class Command 
    { 
        public Command() 
        { 
        } 
    
    public string Name { get; set; }     
    public SerializableIRCode Code { get; set; } 
    } 
}

Despite the obvious Name property, you can see that I am using a SerializableIRCode instance that is specific to each model. Before going any further, I need to mention that the Phidgets SDK offers the IRLearnedCode model to store code contents. I could have used it instead, but there is an issue that prevents me from doing that—there is no public constructor defined for IRLearnedCode, therefore there is no way to serialize it, either with the built-in .NET serialization capabilities or JSON.NET, which I am using in the context of the project.

Instead, I have this:

using Phidgets; 
namespace Coding4Fun.MissionControl.API.Models 
{     
    public class SerializableIRCode     
    {         
        public SerializableIRCode()         
        { 
            
        } 
        
    IRLearnedCode code;         
    public ToggleMask Mask { get; set; } 
    public int BitSize { get; set; } 
    public Phidgets.IRCodeInfo.IREncoding Encoding { get; set; } 
    public int CarrierFrequency { get; set; } 
    public int DutyCycle { get; set; } 
    public int Gap { get; set; } 
    public int[] Header { get; set; } 
    public byte[] CodeData { get; set; } 
    public int MinRepeat { get; set; } 
    public int[] One { get; set; } 
    public int[] Repeat { get; set; } 
    public int Trail { get; set; } 
    public int[] Zero { get; set; } 
    }     
}

It is an almost identical 1:1 copy of the original class, storing both the layout of the IR code and additional information related to its replication mechanism. You can learn more about each property listed in the model above by reading the official document on the topic.

ToggleMask, the identity bit carrier that helps marking the code as repeated or not, is also implemented through a built-in Phidgets SDK model, and it has the same problem as IRLearnedCode. I implemented this model to replace it in the serializable code:

namespace Coding4Fun.MissionControl.API.Models 
{ 
    public class ToggleMask 
    { 
        public ToggleMask() 
        { 
            
        } 

    public int BitSize { get; set; }         
    public byte[] CodeData { get; set; } 
    } 
}

I also needed an easy way to store all sets at once and carry all associated codes in a single instance retrieved from the storage. Here is the Set class:

namespace Coding4Fun.MissionControl.API.Models 
{     
    public class Set 
    { 
        public Set() 
        { 
            Commands = new List<Command>(); 
        } 
        
    public string Name { get; set; } 
    public bool IsList { get; set; } 
    public List<Command> Commands { get; set; } 
    } 
}

Notice that there is an IsList flag that allows me to specify how to display this specific list on the connecting device. This adds some level of flexibility for situations where the user wants to build a virtual remote for closely-related keys, such as digits. With that in mind, displaying those as a list might be inconvenient, wasting visual space on the client. But if the flag is set to false, the list can be displayed as a pad.

Also, when the server performs the data exchange, it provides a single “envelope” that allows the connecting device to easily understand what the server is trying to do:

namespace Coding4Fun.MissionControl.API.Models 
{     
    public class ServerResponse     
    {     
    public string Identifier { get; set; }     
    public string Marker { get; set; }     
    public string Content { get; set; }     
    } 
}

The Identifier property carries the server IP address. That way, when a device receives a response, it is able to either accept it, because it knows that a response is requested from a target location, or discard it because the user is no longer using the specific server.

Marker carries the command type of the sent command, therefore giving the Windows Phone application a hint as to what to do with the data. The server can send the following commands:

• SET_LIST – returns the list of sets that are currently available on the server.
• SET_COMMANDS:SET_NAME:IS_LIST – returns the list of commands that are associated with a given set that is currently stored on the server.
• NOTIFICATION – send a simple notification to the client; no further action is required.

Last but not least, Content is used to push the necessary data that is associated with the given Marker. It can be either a JSON-based string that lists the sets or commands, or a plain-text message that is used as an alert for the end-user.

Server Architecture

The server is the only component of this entire system that does all the heavy lifting. It learns commands, stores them and then generates new IR signal requests, as controlled from any of the connected clients. Let’s take a closer look at what happens behind the scenes—to start, I am going to document the network infrastructure.

The Network Layer

In order to be a reliable system, the server needs to be always ready to accept an incoming connection. For that purpose, it is possible to use the TcpListener class—an “always on” receiver that can handle incoming TCP connections. I integrated it in my CoreStarter class that is used to start the listener when the application is launched:

namespace Coding4Fun.MissionControl.API 
{ 
    public class CoreStarter 
    { 
        static TcpListener listener; 
        
        public static void LaunchSocket()     
        {     
            Console.WriteLine("Starting socket server on port {0}...", Constants.DEFAULT_PORT); 
            listener = new TcpListener(NetworkHelper.GetLocalIPAddress(), Constants.DEFAULT_PORT); 
            listener.Start(); 
            
            for (int i = 0; i < Constants.MAX_CONCURRENT_CLIENTS; i++) 
            {     
                Thread socketThread = new Thread(new ThreadStart(ListenForData));     
                socketThread.Start(); 
            } 
        } 
        
        private static void ListenForData() 
        { 
            Console.WriteLine("Listener thread started."); 

            while (true) 
            { 
                Socket acceptedSocket = listener.AcceptSocket(); 
                using (MemoryStream coreStream = new MemoryStream()) 
                {     
                    try 
                    { 
                        Console.WriteLine("Incoming connection: {0}", acceptedSocket.RemoteEndPoint); 
                        
                        using (Stream sourceStream = new NetworkStream(acceptedSocket)) 
                        {     
                            sourceStream.ReadTimeout = Constants.SOCKET_READ_TIMEOUT; 
                            
                            byte[] buffer = new byte[Constants.DEFAULT_BUFFER_SIZE]; 
                            int i; 
                
                            while ((i = sourceStream.Read(buffer, 0, buffer.Length)) != 0) 
                            { 
                                coreStream.Write(buffer, 0, i); 
                            } 
                        } 
                    } 
                    catch 
                    { 
                        string data = Encoding.ASCII.GetString(coreStream.ToArray()); 
        
                        CommandHelper.InterpretCommand(data, acceptedSocket.RemoteEndPoint.ToString()); 
                    } 
                } 
            } 
        } 
    } 
}

When LaunchSocket is called, the listener is activated on the current machine. As I mentioned above, the port number can be arbitrarily assigned, but has to be consistent between connecting apps in order for the TCP links to be established. Because I expect that more than one device will be connecting to the service at a time, the listener is set as active across a constant number of threads.

NOTE: By default, a there is a maximum limit of 5 simultaneous clients. Although this number can be adjusted, be aware of the requirements of each environment in which a limited number of potential devices can connect. Even though the performance footprint of each thread is minimal, it can have a negative effect if used in unnecessarily large instances.

ListenForData is used to read the incoming stream. When an inbound connection is accepted, the data is read with the help of a fixed content buffer. Then a read timeout is specified to prevent situations where the stream was completely read but the application still waits to pull non-existent data. Once the timeout milestone is hit, an exception is thrown, which marks the end of the stream—at this point, the plain text data that was received (remember that both the server and client exchange text data only) is passed to the command interpreter—CommandHelper, with a reference to the source of the command.

The commands from the device are passed as serialized key-value pairs (KeyValuePair<T, T>), the key being the command with any possible suffixes, and the value being the contents of the command itself that helps the server identify the specific item in the local storage.

InterpretCommand,in this case, does three things sequentially:

1. Deserialize the incoming string and create a KeyValuePair<string,string> instance.
2. Process the command and check whether it is recognizable.
3. Send a response to the client, if deemed necessary by the command type.

The serialization and deserialization is done via JSON.NET. You can install this package in your console managed Win32 project and the Windows Phone application project via NuGet:

The deserialization step is as simple as one line of C# code:

KeyValuePair<string, string> result = JsonConvert.DeserializeObject<KeyValuePair<string, string>>(rawCommand.Remove(0, rawCommand.IndexOf('{')));

The string is sanitized to ensure that only JSON content is being passed to the serializer.

Because of a relatively limited command set, I can put together the entire interpretation stack like this:

// Get the initial list of sets on the target server
if (result.Key == Constants.COMMAND_INIT)
{
    SendSets(sourceLocation);
}
// Create a new set on the target server
else if (result.Key.Contains(Constants.COMMAND_CREATE_SET))
{
    CreateSet(result, sourceLocation);
    SendSets(sourceLocation);
}
// Get the commands that are associated with a given set.
else if (result.Key == Constants.COMMAND_GET_COMMANDS)
{
    SendCommands(result.Value, sourceLocation);
}
// The client requested the server to learn a new command.
else if (result.Key.Contains(Constants.COMMAND_LEARN_COMMAND))
{
    LearnCommand(result, sourceLocation);
}
// The client requested one of the commands to be executed on the 
// target server.
else if (result.Key.Contains(Constants.COMMAND_EXECUTE))
{
    ExecuteCommand(result);
}
// The client has requested a set to be deleted from the target server.
else if (result.Key == Constants.COMMAND_DELETE_SET)
{
    DeleteSet(result.Value);
    SendSets(sourceLocation);
}
// The client has requested a set to be deleted from the target server.
else if (result.Key.Contains(Constants.COMMAND_DELETE_COMMAND))
{
    DeleteCommand(result);
    SendCommands(result.Key.Split(new char[] { ':' })[1], sourceLocation);
}

All commands are constants, declared in the local helper class:

public const string COMMAND_INIT = "INIT"; 

public const string COMMAND_CREATE_SET = "CREATE_SET"; 

public const string COMMAND_GET_COMMANDS = "GET_COMMANDS"; 

public const string COMMAND_LEARN_COMMAND = "LEARN_COMMAND"; 

public const string COMMAND_EXECUTE = "EXECUTE"; 

public const string COMMAND_DELETE_SET = "DELETE_SET";
public const string COMMAND_DELETE_COMMAND = "DELETE_COMMAND";

Notice that these are not the commands that the server sends back, but rather the commands it receives from connecting Windows Phone devices.

Let’s now take a look at the breakdown for each command.

SendSets:

/// <summary> 
/// Send the list of sets to the client that requested those. 
/// </summary> 
/// <param name="sourceLocation">The location of the requesting client.</param> 
private static void SendSets(string sourceLocation) 
{ 
    Console.WriteLine("Received an initial set query from {0}", sourceLocation);     
    ServerResponse response = new ServerResponse(); 
    response.Marker = "SET_LIST"; 
    response.Content = JsonConvert.SerializeObject(StorageHelper.GetRawSetNames()); 
    response.Identifier = NetworkHelper.GetLocalIPAddress().ToString(); 
    NetworkHelper.SendData(sourceLocation, JsonConvert.SerializeObject(response)); 
    Console.WriteLine("Sent the set list to {0}", sourceLocation);     
}

When this command is received, the server does not have to do much processing. It is only invoked when the client establishes the initiating link and needs to know what possible sets it can get from the target machine. The request is logged in the console and a server response is prepared that contains a serialized list of set names, which is later serialized as well and sent back to the source machine location.

StorageHelper and NetworkHelper will be documented later in this article.

CreateSet:

/// <summary> 
/// Create a new set and store it on the local server. 
/// </summary> 
/// <param name="result">The original deserialized command.</param> 
/// <param name="sourceLocation">The location of the requesting client.</param> 
private static void CreateSet(KeyValuePair<string,string> result, string sourceLocation) 
{ 
    bool isSuccessful = false; 
    string[] data = result.Key.Split(new char[] { ':' }); 
    
    Console.WriteLine("There is an attempt to create the {0} set from {1}.", result.Value, sourceLocation); 
    
    if (data[1].ToLower() == "list") 
        isSuccessful = StorageHelper.AddSet(result.Value); 
    else 
        isSuccessful = StorageHelper.AddSet(result.Value, false); 
    
    if (isSuccessful) 
        Console.WriteLine("The {0} set was successfully created.", result.Value); 
    else 
        Console.WriteLine("Something happened and the {0} set was not created.", result.Value);     
}

When a mobile device attempts to create a new set on the server, it sends a command in the following format:

CREATE_SET:list/pad, SET_NAME

CreateSet will get the type of the set that was created, will check whether a set with the same name already exists and will either create it or ignore the command altogether. No notification is sent to the connecting device, but either the failure or the success of the command is registered in the local console.

SendCommands:

/// <summary>
/// Send a list of commands that are associated with the pushed set.
/// </summary>
/// <param name="setName">The original deserialized command.</param>
/// <param name="sourceLocation">The location of the requesting client.</param>
private static void SendCommands(string setName, string sourceLocation)
{
    Console.WriteLine("There was a request to get the commands for the {0} set from {1}.", setName, sourceLocation);
    
    bool isList = StorageHelper.IsSetAList(setName);
    
    ServerResponse response = new ServerResponse();
    response.Marker = string.Format("SET_COMMANDS:{0}:{1}", setName, isList);
    response.Identifier = NetworkHelper.GetLocalIPAddress().ToString();
    response.Content = JsonConvert.SerializeObject(StorageHelper.GetRawCommandNames(setName));
    
    NetworkHelper.SendData(sourceLocation, JsonConvert.SerializeObject(response));
    
    Console.WriteLine("Command list for the {0} set were sent to {1}.", setName, sourceLocation);
}

Commands are sent in the same manner as sets—once the set is recognized, the names of the associated commands are retrieved and serialized inside a ServerResponse instance and then pushed back to the requesting device.

LearnCommand:

/// <summary> 
/// Learn a new command and store it on the target server. 
/// </summary> 
/// <param name="result">The original deserialized command.</param> 
/// <param name="sourceLocation">The location of the requesting client.</param> 
private static void LearnCommand(KeyValuePair<string,string> result, string sourceLocation) 
{     
    Console.WriteLine("[!] Server in COMMAND LEARNING MODE! Point the remote towards the sensor and send a command."); 
    
    string[] data = result.Key.Split(new char[] { ':' });     
    var set = StorageHelper.GetSingleSet(StorageHelper.GetSets(), data[1]); 
    
    if (set != null)     
    {     
        if ((from c in set.Commands where c.Name == result.Value select c).FirstOrDefault() != null) 
        {     
            Console.WriteLine("Cannot learn command {0} for the following set: {1}. Command already exists.", data[1], result.Value); 
            
            ServerResponse response = new ServerResponse();     
            response.Marker = "NOTIFICATION";     
            response.Identifier = NetworkHelper.GetLocalIPAddress().ToString(); 
            response.Content = "We could not save the following command - " + result.Value + ". It already exists in the set."; 
            
            NetworkHelper.SendData(sourceLocation, JsonConvert.SerializeObject(response));     
        } 
        else 
        { 
            if (sensor == null) 
                sensor = new IR(); 
            
            sensor.open(-1); 

            sensor.waitForAttachment(); 
    
            sensor.Learn += (sender, args) => 
            { 
                Console.WriteLine("[!] Server learned the command and is no longer in COMMAND LEARNING MODE."); 
                IRLearnedCode code = args.LearnedCode; 
                code.CodeInfo.MinRepeat = 5; 
            
                Command command = new Command(); 
                command.Name = result.Value; 
                command.Code = IRCodeWorker.GetSerializableIRCode(code); 
                
                StorageHelper.AddCommand(command, set.Name); 
                
                ServerResponse response = new ServerResponse(); 
                response.Marker = "NOTIFICATION"; 
                response.Identifier = NetworkHelper.GetLocalIPAddress().ToString(); 
                response.Content = "The following command has been stored: " + result.Value; 
                
                NetworkHelper.SendData(sourceLocation, JsonConvert.SerializeObject(response)); 
            }; 
        } 
    } 
}

Once a request was received that the server needs to learn a new command, an initial verification is done to make sure that the requested command name and set are not already taken. If neither the command nor the set exist, both will be created.

After the basic setup is complete, the IR sensor is activated and will be waiting for the command to be learned. The way it works is quite simple – the sensor will remain in learning mode until the point where it recognizes a command without error, being 100% sure that it can be reproduced internally. You will need to point your remote towards the sensor and hold the button you want captured for one or two seconds in order for the command to be learned.

NOTE: To ensure that a proper transmission is done, I manually set the minimal repeat value to 5. This is the number of times the sensor will fire the same code towards the target. That is the optimal value for a target device to receive the code if the remote is pointed directly at it without necessarily triggering the same command twice or more.

After the command is learned, the code is processed and transformed into a serializable instance. The connecting client is then notified about whether the command was learned.

ExecuteCommand:

/// <summary> 
/// Execute one of the commands currently stored on the local server. 
/// </summary> 
/// <param name="result">The original deserialized command.</param> 
private static void ExecuteCommand(KeyValuePair<string,string> result) 
{ 
    string[] data = result.Key.Split(new char[] { ':' }); 

    var set = StorageHelper.GetSingleSet(StorageHelper.GetSets(), data[1]); 
    
    if (set != null) 
    {     
        var command = StorageHelper.GetSingleCommand(StorageHelper.GetCommands(set.Name), result.Value); 

        IRLearnedCode code = IRCodeWorker.GetLearnedCode(command.Code); 
        
        if (sensor == null) 
            sensor = new IR(); 

        sensor.open(-1); 
        sensor.waitForAttachment(); 
        sensor.transmit(code.Code, code.CodeInfo); 
        sensor.close(); 
    } 
}

Command execution relies on the hardware sensor. The phone sends a command execution request in the following format:

EXECUTE:SET_NAME, COMMAND_NAME

Once the command is parsed out and found in the local storage, the IR code is transformed back to a model that is recognizable by the Phidgets SDK and transmitted towards the location where the sensor is pointed at the time of the execution.

DeleteSet:

/// <summary> 
/// Delete a single set and all the associated commands 
/// </summary> 
/// <param name="target">The name of the set.</param> 
private static void DeleteSet(string target) 
{ 
    var sets = StorageHelper.GetSets(); 
    var targetSet = StorageHelper.GetSingleSet(sets, target); 
    
    if (targetSet != null) 
    { 
        StorageHelper.RemoveSet(sets, targetSet); 
    } 
}

When deleting a set, only the name of the set should be specified. The user will get a warning on the client side that requires a confirmation of the deletion. The server will blindly execute the command.

DeleteCommand:

private static void DeleteCommand(KeyValuePair<string, string> result)
{
    var sets = StorageHelper.GetSets();
    string setName = result.Key.Split(new char[] {':'})[1];
    var targetSet = StorageHelper.GetSingleSet(sets, setName);
    var command = (from c in targetSet.Commands where c.Name == result.Value select c).FirstOrDefault();
    
    if (command != null)
    {
        targetSet.Commands.Remove(command);
        StorageHelper.SerializeSets(sets);
    }
}

Not only can the user remove entire sets, but he can also target specific commands from a given set. Once a DELETE_COMMAND directive is recognized, the set name is parsed out from the original string, that follows the DELETE_COMMAND:SET_NAME, COMMAND_NAME format, and a simple LINQ query extracts the command instance, removes it and stores the set content on the local hard drive.

Notice that for some commands, particularly for set creation, deletion and command deletion, the server will return a list of the remaining items. The contents will be automatically updated on the devices, which will be waiting for that response. This measure was deliberately introduced to minimize the chances of a user triggering a command that was already deleted or trying to query a previously removed set.

Transforming Codes

You might have noticed that I am using IRCodeWorker.GetSerializableCodeType to transform a Phidgets SDK native IR code model into a serializable one. This is a helper function that performs a field copy of the existing object. Because of the differences in the model structure, it has to be done manually:

public static SerializableIRCode GetSerializableIRCode(IRLearnedCode code) 
{ 
    SerializableIRCode sCode = new SerializableIRCode();     
    sCode.BitSize = code.Code.BitCount;     
    sCode.Encoding = code.CodeInfo.Encoding;     
    sCode.CarrierFrequency = code.CodeInfo.CarrierFrequency;     
    sCode.CodeData = code.Code.Data;     
    sCode.DutyCycle = code.CodeInfo.DutyCycle;     
    sCode.Gap = code.CodeInfo.Gap;     
    sCode.Header = code.CodeInfo.Header;     
    sCode.MinRepeat = 5;     
    sCode.One = code.CodeInfo.One;     
    sCode.Repeat = code.CodeInfo.Repeat;     
    sCode.Trail = code.CodeInfo.Trail;     
    sCode.Zero = code.CodeInfo.Zero;     
    sCode.Mask = new ToggleMask()     
    {         
        BitSize = code.CodeInfo.ToggleMask.BitCount,         
        CodeData = code.CodeInfo.ToggleMask.Data         
    }; 
    
    return sCode;     
}

The reverse process is easier because I can pass each of the existing properties to the IRCodeInfo constructor. The only difference is the fact that I need to use Reflection to create an instance of IRLearnedCode because there is no public constructor defined and a dynamic object has to be created:

internal static IRLearnedCode GetLearnedCode(SerializableIRCode serializableIRCode) 
{ 
    IRCode code = new IRCode(serializableIRCode.CodeData, serializableIRCode.BitSize);     
    IRCodeInfo info = new IRCodeInfo(serializableIRCode.Encoding, serializableIRCode.BitSize, serializableIRCode.Header,     
    serializableIRCode.Zero, serializableIRCode.One, serializableIRCode.Trail, serializableIRCode.Gap, serializableIRCode.Repeat, 
serializableIRCode.MinRepeat, serializableIRCode.Mask.CodeData, IRCodeInfo.IRCodeLength.Constant,
serializableIRCode.CarrierFrequency, serializableIRCode.DutyCycle); 
    
    object[] parameters = new object[] { code, info }; 
    
    BindingFlags flags = BindingFlags.NonPublic | BindingFlags.Instance; 
    object instantType = Activator.CreateInstance(typeof(IRLearnedCode), flags, null, parameters, null); 

    return (IRLearnedCode)instantType; 
}

Command and Set Management

Looking back at the code that I put together for the command interpreter, there is one class that does all local content manipulation—StorageHelper. This is a simple class that performs LINQ queries on set as well as command collections, and makes sure that all the changes are preserved in the sets.xml file in the application folder that is used as the only storage place for all the content that is being manipulated by the server.

namespace Coding4Fun.MissionControl.API.Helpers 
{ 
    public class StorageHelper 
    { 
        
        /// <summary> 
        /// Lists all available sets that are currently stored on the server.         
        /// </summary> 
        /// <returns>List of sets on the machine.</returns>         
        internal static List<Set> GetSets()         
        {             
            List<Set> sets = null; 
            
            string rawContent = GetRawSets();             
            sets = JsonConvert.DeserializeObject<List<Set>>(rawContent); 
            
            return sets;             
        } 
        
        /// <summary>         
        /// Returns the list of commands that are associated with the given set.         
        /// </summary>         
        /// <param name="setName">The name of the target set.</param>         
        /// <returns>List of commands associated with the given set.</returns>         
        internal static List<Command> GetCommands(string setName)         
        {             
            List<Command> commandList = null; 
            
            var sets = GetSets(); 
            
            Set singleSet = null;             
            if (sets != null)             
                singleSet = (from c in sets where c.Name == setName select c).FirstOrDefault(); 
            
            if (singleSet != null)         
            {         
                commandList = singleSet.Commands;     
            } 
            
            return commandList; 
        } 
        
        /// <summary>     
        /// Gets the list of names for the commands in the requested set. 
        /// </summary> 
        /// <param name="setName">The name of the target set.</param> 
        /// <returns>List of commands associated with the given set.</returns> 
        internal static List<string> GetRawCommandNames(string setName) 
        { 
            List<Command> commandList = GetCommands(setName); 
            
            List<string> stringSet = null; 
            
            if (commandList != null) 
            {     
                stringSet = commandList.Select(x => x.Name).ToList();     
            } 
            
            return stringSet;     
        } 
        
        /// <summary> 
        /// Get the list of names for all sets on the local server. 
        /// </summary> 
        /// <returns>List of sets on the machine.</returns> 
        internal static List<string> GetRawSetNames() 
        {     
            List<Set> sets = GetSets(); 
            
            List<string> stringSet = null; 
    
            if (sets != null) 
            {     
                stringSet = sets.Select(x => x.Name).ToList();     
            } 
            
            return stringSet; 
        } 
        
        /// <summary> 
        /// Get the raw string contents of sets.xml. Should only be used in the 
        /// context of this class. 
        /// </summary> 
        /// <returns>JSON string representing stored sets and commands.</returns> 
        internal static string GetRawSets() 
        { 
            string sets = string.Empty; 
            
            if (File.Exists("sets")) 
            { 
                using (StreamReader reader = new StreamReader(File.OpenRead("sets"))) 
                { 
                    sets = reader.ReadToEnd(); 
                } 
            } 
            else 
            { 
                FileStream stream = File.Create("sets.xml"); 
                
                stream.Close();     
            } 
            
            return sets; 
        } 
        
        /// <summary> 
        /// Check whether a set is marked with a IsList flag. 
        /// </summary> 
        /// <param name="setName">The name of the target set.</param> 
        /// <returns>TRUE - set is a list. FALSE - set is not a list.</returns> 
        internal static bool IsSetAList(string setName) 
        { 
            bool isList = true; 
            var sets = GetSets(); 
            Set set = null; 

            if (sets != null) 
                set = (from c in sets where c.Name == setName select c).FirstOrDefault(); 
            
            if (set != null) 
                isList = set.IsList; 
            
            return isList;     
        } 
        
        /// <summary> 
        /// Serialize the set collection to sets.xml 
        /// </summary> 
        /// <param name="sets">Collection to be serialized.</param> 
        /// <returns>true if sets are serialized.</returns> 
        private static bool SerializeSets(List<Set> sets) 
        { 
            try 
            { 
                using (StreamWriter writer = new StreamWriter("sets.xml", false)) 
                {     
                    string data = JsonConvert.SerializeObject(sets); 
            
                    writer.Write(data); 
                } 
                
                return true; 
            } 
            catch 
            {     
                return false;     
            } 
        } 
        
        /// <summary> 
        /// Add a new set to the existing global set collection. 
        /// </summary> 
        /// <param name="name">Set name.</param> 
        /// <returns>true if successfully added set.</returns> 
        internal static bool AddSet(string name, bool isList = true) 
        { 
            var sets = GetSets(); 

            if (sets == null) 
                sets = new List<Set>(); 
            
            var singleSet = GetSingleSet(sets, name); 

            if (singleSet == null) 
                sets.Add(new Set { Name = name, IsList = isList }); 
            
            if (SerializeSets(sets)) 
                return true; 
            else 
                return false; 
        } 
        
        /// <summary> 
        /// Retrieves a single set from a collection that has a specific name. 
        /// </summary> 
        /// <param name="sets">The source collection from which to extract the set.</param> 
        /// <param name="name">The name of the set to get.</param> 
        /// <returns>An instance of the found set, if any.</returns> 
        internal static Set GetSingleSet(List<Set> sets, string name) 
        { 
            if (sets != null) 
                return (from c in sets where c.Name == name select c).FirstOrDefault(); 
            else 
                return null; 
        } 
        
        /// <summary> 
        /// Add a IR command to an existing set. If the set is not found, it will be created. 
        /// </summary> 
        /// <param name="command">The command instance to be added.</param> 
        /// <param name="targetSet">The name of the target set.</param> 
        /// <returns>true if the command was successfully added.</returns> 
        internal static bool AddCommand(Command command, string targetSet) 
        { 
            var sets = GetSets(); 
            
            if (sets == null) 
                sets = new List<Set>(); 
            
            var singleSet = GetSingleSet(sets, targetSet); 
            
            if (singleSet == null) 
                singleSet = new Set { Name = targetSet }; 
            
            var singleCommand = (from c in singleSet.Commands where c.Name == command.Name select c).FirstOrDefault(); 
            
            if (singleCommand == null) 
            { 
                singleSet.Commands.Add(command); 
                
                if (SerializeSets(sets)) 
                    return true; 
                else 
                    return false; 
            } 
            else 
                return false; 
        } 
        
        /// <summary> 
        /// Retrieve a single command instance from one of the sets on the local server. 
        /// </summary> 
        /// <param name="commands">Original list of commands.</param> 
        /// <param name="name">Name of the command to be retrieved.</param> 
        /// <returns>An instance of the command, if found. NULL if not.</returns> 
        internal static Command GetSingleCommand(List<Command> commands, string name) 
        { 
            if (commands != null) 
                return (from c in commands where c.Name == name select c).FirstOrDefault(); 
            else 
                return null; 
        } 
        
        /// <summary> 
        /// Remove a set from a local machine. 
        /// </summary> 
        /// <param name="sets">Original list of sets.</param> 
        /// <param name="targetSet">Name of the set to remove.</param> 
        internal static void RemoveSet(List<Set> sets, Set targetSet) 
        { 
            sets.Remove(targetSet); 
            
            SerializeSets(sets); 
        } 
    } 
}

Sending Data Back to the Client

SendData in the NetworkHelper class handles all outbound connections. Here is its structure:

/// <summary> 
/// Send data to the target network machine. 
/// </summary> 
/// <param name="destination">The target machine IP.</param> 
/// <param name="data">Data to be sent, in string format.</param> 
/// <param name="sanitizeIp">Determines whether to remove the port from the given IP string.</param> 
public static void SendData(string destination, string data, bool sanitizeIp = true) 
{ 
    using (Socket client = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)) 
    {     
        string completeIp = string.Empty; 
        
        if (sanitizeIp)     
            completeIp = destination.Remove(destination.IndexOf(":"), destination.Length - destination.IndexOf(":")); 
        
        client.Connect(completeIp, 6169); 
        client.Send(Encoding.UTF8.GetBytes(data));     
    }     
}

A new stream socket is created in order to connect to the target machine over the TCP pipe. If IP sanitization is enabled, the port is stripped from the address in order to pass a valid IP. A Socket instance cannot directly handle IPs of the format:

255.255.255.0:PORT_NUMBER

Later, in a synchronous manner, a connection is established and the data is sent.

At this point, you can see that the barebones service offers a flexible way to manage content. It can be accessed by any application type as long as the server can be accessed and the application can send commands in the pre-defined format and the content requested is actually located on the target server. This allows for high levels of extensibility and interoperability, as the server usage is not limited to a single platform. If I decide to create a Windows Store application that would allow me to control my TV, I simply need to add socket connection layer that will send plain strings to the machine where the IR sensor is connected.

Similarly, if some functionality needs to be added, it is possible to do so without ever touching the client applications. A modification in the endpoint will be reflected with no direct effect on all connection applications as long as all handled returned and requested values are preserved. The only additional requirement is that if the client applications want to take advantage of newly introduced capabilities, they need to have an updated command transmission layer for the new command types.

In Program.cs, I simply need to start the server through the CoreStarter class:

namespace Coding4Fun.MissionControl.API 
{ 
    class Program     
    {         
        static void Main(string[] args)         
        {             
            Console.WriteLine("Coding4Fun MissionControl Server");             
            CoreStarter.LaunchSocket();             
        }         
    }     
}

Mobile client overview

The mobile client does not have the capability to send commands directly to the IR sensor. Instead, it connects to a remote machine that has the IR sensor plugged in and attempts to invoke a command from the list returned by the service. A single mobile client can support control over multiple servers.

NOTE: Make sure that at the time of working with the Windows Phone client, the server is actually running on your local machine. To make it easier to test, also open port 6169 for incoming connections in Windows Firewall.

When building a Windows Phone application, make sure you have the proper version of the SDK installed, as well as a SLAT-compatible machine if you plan on testing the application in the emulator.

Networking Infrastructure

The Windows Phone application also relies on a network infrastructure somewhat similar to that of the server. There is a TCP listener that is created when the application is started:

// Code to execute when the application is launching (eg, from Start) 
// This code will not execute when the application is reactivated 
private void Application_Launching(object sender, LaunchingEventArgs e) 
{ 
    ServiceSerializer.DeserializeServices(); 
    
    listener.OnClientConnected += listener_OnClientConnected; 
    listener.Start(6169); 
}

Here, listener is an instance of TcpSocketListener—a custom class designed to handle incoming network connections:

namespace Coding4Fun.MissionControl.WP.Network 
{ 
    public class TcpSocketListener : SocketConnectorBase     
    {     
        StreamSocketListener coreSocket; 
        
        public async void Start(int port)     
        {     
            coreSocket = new StreamSocketListener();         
            coreSocket.ConnectionReceived += coreSocket_ConnectionReceived; 
            
            try     
            {         
                await coreSocket.BindServiceNameAsync(port.ToString());         
            }     
            catch (Exception ex) 
            {     
                Debug.WriteLine(ex.Message); 
                
                coreSocket.Dispose();     
                coreSocket = null; 
                OnConnectionCompleted(new ConnectionEventArgs { IsSuccessful = false, DeviceID = string.Empty }); 
            } 
        } 
        
        async void coreSocket_ConnectionReceived(StreamSocketListener sender, StreamSocketListenerConnectionReceivedEventArgs args) 
        { 
            Debug.WriteLine("Connection received!"); 
            
            DataReader reader = new DataReader(args.Socket.InputStream); 
            
            try 
            { 
                while (true) 
                { 
                    StringBuilder builder = new StringBuilder(); 
        
                    uint actualLength = 1; 

                    while (actualLength > 0) 
                    {     
                        actualLength = await reader.LoadAsync(256); 
                        builder.Append(reader.ReadString(actualLength)); 
                    } 
                    
                    OnConnectionCompleted(new ConnectionEventArgs 
                    { 
                        Socket = args.Socket, 
                        IsSuccessful = true, 
                        DeviceID = args.Socket.Information.RemoteHostName.DisplayName, 
                        Token = builder.ToString() 
                    }); 
                    break; 
                } 
            } 
            catch (Exception exception) 
            { 
                Debug.WriteLine(exception.Message); 
                
                OnConnectionCompleted(new ConnectionEventArgs { IsSuccessful = false }); 
            } 
        } 
    } 
}

A StreamSocketListener is used for the connection core. When a connection is received, a continuous loop reads the entire contents of the incoming stream. OnConnectionCompleted is declared in the base class—SocketConnectorBase.

namespace Coding4Fun.MissionControl.WP.Network 
{ 
    public class SocketConnectorBase 
    { 
        public event EventHandler<ConnectionEventArgs> OnClientConnected; 
        public virtual void OnConnectionCompleted(ConnectionEventArgs connectionArgs) 
        { 
            if (OnClientConnected != null) 
            { 
                OnClientConnected(this, connectionArgs); 
            } 
        } 
        
        public event EventHandler<bool> OnSendCompletedEvent; 
        public virtual void OnSendCompleted(bool succeeded) 
        { 
            if (OnSendCompletedEvent != null) 
            { 
                OnSendCompletedEvent(this, succeeded); 
            } 
        } 
    } 
    
    public class ConnectionEventArgs : EventArgs 
    { 
        public StreamSocket Socket { get; set; }     
        public string DeviceID { get; set; }     
        public string Token { get; set; }     
        public bool IsSuccessful { get; set; }     
    } 
}

ConnectionEventArgs here is used to identify the content that is passed to the client. DeviceID gives access to the source IP, IsSuccessful tells the developer whether the established connection is active and the Token carries the raw string if any was received.

Sending data is simplified to the maximum with the help of the SocketClient class, which relies on a StreamSocket instance that handles outbound connections and writing to the output stream:

namespace Coding4Fun.MissionControl.WP.Network 
{ 
    public class SocketClient : SocketConnectorBase 
    {     
        StreamSocket _socket; 
        
        public SocketClient() 
        {     
            _socket = new StreamSocket();     
        } 
        
        public SocketClient(StreamSocket socket) 
        {     
            _socket = socket;     
        } 
        
        public async void Connect(string hostName, int portNumber) 
        { 
            try 
            { 
                await _socket.ConnectAsync(new HostName(hostName), portNumber.ToString(), SocketProtectionLevel.PlainSocket); 
            
                OnConnectionCompleted(new ConnectionEventArgs { IsSuccessful = true }); 
            } 
            catch (Exception ex) 
            {     
                Debug.WriteLine(ex.Message); 
                
                OnConnectionCompleted(new ConnectionEventArgs { IsSuccessful = false });     
            } 
        } 
        
        public async void Send(string dataToSend) 
        { 
            try 
            {     
                using (DataWriter writer = new DataWriter(_socket.OutputStream)) 
                { 
                    // Write the length of the binary data that is being 
                    // sent to the client. 
                    writer.WriteUInt32((UInt32)dataToSend.Length); 
                    
                    writer.WriteString(dataToSend); 
        
                    // Send the actual data. 
                    await writer.StoreAsync(); 
                    
                    writer.DetachStream(); 
                    
                    OnSendCompleted(true); 
                } 
            } 
            catch 
            { 
                _socket.Dispose();     
                _socket = null;     
                OnSendCompleted(false);     
            }     
        }     
    }     
}

As with the listener class, SocketClient supports OnConnectionCompleted to notify the application that the connection attempt completed.

Back in App.xaml.cs, the data from the incoming connection captured by the TcpSocketListener instance is passed to the ResponseHelper class:

void listener_OnClientConnected(object sender, ConnectionEventArgs e) 
{ 
    ResponseHelper.HandleIncomingResponse(e.Token); 
}

This class reads the possible three commands sent by the server and interprets them, creating internal collections from the raw data if the current server IP matches the one obtained in the ServerResponse (the same model in the desktop application):

using Coding4Fun.MissionControl.WP.Models;
using Coding4Fun.MissionControl.WP.ViewModels;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Windows;

namespace Coding4Fun.MissionControl.WP.Misc
{
        public class ResponseHelper
        {
                public static void HandleIncomingResponse(string rawResponse)
                {
                        if (rawResponse != null)
                        {
                                ServerResponse response = JsonConvert.DeserializeObject<ServerResponse>(rawResponse);
                
                                if (response.Marker == Constants.COMMAND_SERVER_NOTIFICATION)
                                {
                                        Deployment.Current.Dispatcher.BeginInvoke(() =>
                                        {
                                                MessageBox.Show(response.Content, "Server Response", MessageBoxButton.OK);
                                        });
                                }
                                else
                                {
                                        if (CommonViewModel.Instance.IsWaiting)
                                        {
                                                if (response.Identifier == CommonViewModel.Instance.CurrentServer.Location)
                                                {
                                                        // returns the list of sets that are associated with the current server.
                                                        if (response.Marker == Constants.COMMAND_SERVER_SET_LIST)
                                                        {
                                
                                                                List<string> items = JsonConvert.DeserializeObject<List<string>>(response.Content);
                                
                                                                if (items != null)
                                                                {
                                                                        List<Group<string>> groupedItems = Group<string>.CreateGroups(items,
                                                                        CultureInfo.CurrentCulture, (string s) => { return s[0].ToString(); }, true);
                                                                        SetsPageViewModel.Instance.Sets = groupedItems;
                                                                }
                                                                else
                                                                {
                                                                        SetsPageViewModel.Instance.Sets = new List<Group<string>>();
                                                                }
                                
                                                                Deployment.Current.Dispatcher.BeginInvoke(() =>
                                                                    {
                                                                            CommonViewModel.Instance.IsWaiting = false;
                                    
                                                                            if (!App.RootFrame.CurrentSource.ToString().Contains("SetsPage"))
                                                                            {
                                                                                    App.RootFrame.Navigate(new Uri("/Views/SetsPage.xaml", UriKind.Relative));
                                                                            }
                                                                    });
                                
                                                        }
                                                        // returns the list of commands associated with a given set.
                                                        else if (response.Marker.Contains(Constants.COMMAND_SERVER_SET_COMMANDS))
                                                        {
                                                            string[] data = response.Marker.Split(new char[] { ':' });
                                                                if (data[1] == CommonViewModel.Instance.CurrentSet)
                                                                {
                                                                        bool isList = false;
                                                                        bool.TryParse(data[2].ToLower(), out isList);
                                    
                                                                        if (isList)
                                                                        {
                                                                                CommonViewModel.Instance.CurrentSetType = "list";
                                                                        }
                                                                        else
                                                                        {
                                                                                CommonViewModel.Instance.CurrentSetType = "pad";
                                                                        }
                                    
                                                                        CommandsPageViewModel.Instance.Commands = new System.Collections.ObjectModel.ObservableCollection<string>(JsonConvert.DeserializeObject<List<string>>(response.Content));
                                    
                                                                        Deployment.Current.Dispatcher.BeginInvoke(() =>
                                                                            {
                                                                                    CommonViewModel.Instance.IsWaiting = false;
                                                                                    App.RootFrame.Navigate(new Uri("/Views/CommandsPage.xaml", UriKind.Relative));
                                                                            });
                                                                }
                                                        }
                                                }
                                        }
                                }
                        }
                }
        }
}

If the response comes from a server that is different than the one that is currently active, the data is discarded as the user no longer needs it. Also, for specific commands, the mobile application will be on standby, waiting for a response (unless the user decides to cancel the request) – the IsWaiting flag is an application-wide indicator that a pending server action is in the queue.

Same as with the server, the commands in the Windows Phone application are represented through pre-defined constants:

public const string COMMAND_SERVER_SET_LIST = "SET_LIST"; 

public const string COMMAND_SERVER_SET_COMMANDS = "SET_COMMANDS"; 

public const string COMMAND_SERVER_NOTIFICATION = "NOTIFICATION";

Let’s now take a closer look at how it is handled internally to build the visual layer.

Handling the Data

The first thing users will see when the application is launched is the list of registered servers:

This is ServiceListPage.xaml. The list of servers that were added is retrieved from the isolated storage on application startup, with the help of the standard serialization routine implemented in the Coding4Fun Toolkit—specifically, its storage subset (you can get it via NuGet):

The one-liner that initializes the internal server collection is as follows:

MainPageViewModel.Instance.Servers = Serialize.Open<ObservableCollection<Server>>(Constants.SERVERS_FILE);

Here, the SERVERS_FILE constant is equal to servers.xml. It is a good idea to use constants for file names in order to be able to later modify the location through a single change instead of digging through the many source files in a solution to find references to the old location.

The user can define an unlimited number of servers, as long as he can access those. There is no restriction on the location of the server itself—it can work with the desktop in your room just as well as with a PC on the other end of the world (yes, this was tested).

When adding a new server, the user is redirected to AddServicePage.xaml, where he can fill in connection details, as well as the location of an image that would help him identify that specific item in the general list:

Once data entry is complete, it is validated internally to make sure that the server is not already registered with the same name and location. If the validation step passes, the server is added to the list of local access points and the user is returned back to the server selection page:

private void AttemptAddService()
{
        if (!string.IsNullOrWhiteSpace(txtName.Text) && !string.IsNullOrWhiteSpace(txtLocation.Text))
        {
                Server server = new Server
                    {
                            Name = txtName.Text,
                            Location = txtLocation.Text,
                            ImageURL = !string.IsNullOrWhiteSpace(txtAvatar.Text) ? txtAvatar.Text : string.Empty
                    };
        
                if (!CollectionHelper.CheckServerExists(server))
                {
                        MainPageViewModel.Instance.Servers.Add(server);
            
                        Serialize.Save(Constants.SERVERS_FILE, MainPageViewModel.Instance.Servers);
                        NavigationService.GoBack();
                }
                else
                {
                        Alert.Send("The service with this name or location is already registered.");
                }
        }
        else
        {
                Alert.Send("The service needs a name and a location.");
        }
}

When a server selection is made by the user, it is necessary to show SetsPage.xaml. However, it is necessary to also check whether the server is active or not prior to the actual navigation. With the help of internal bindings, I am doing it through a RelayCommand:

public RelayCommand SelectServerCommand { get; private set; }
private async void SelectServer(object server)
{
        CommonViewModel.Instance.IsWaiting = true;
        CommonViewModel.Instance.CurrentServer = (Server)server;
    
        bool result = await CommonViewModel.Instance.CommandClient.SendCommand(CommonViewModel.Instance.CurrentServer.Location, 
        Constants.COMMAND_SERVER_HELLO, string.Empty);
        if (!result)
        {
                Alert.Send(Constants.MESSAGE_SERVER_CONNECT_FAIL);
                CommonViewModel.Instance.CurrentServer = null;
        
                CommonViewModel.Instance.IsWaiting = false;
        }
}

COMMAND_SERVER_HELLO represents the initial handshake command that I mentioned earlier—it requests the list of sets on the target server. To streamline command processing, CommandClient is used and wraps around the SocketClient class, giving me the possibility to call SendCommand with the command metadata without having to explicitly handle socket interactions in my views:

namespace Coding4Fun.MissionControl.WP.Network 
{     
    public class CommandClient     
    {     
        private SocketClient client; 
        
        public Task<bool> SendCommand(string key, string value, Action<bool> onCompleted = null) 
        {     
            var taskCompletionSource = new TaskCompletionSource<bool>(); 
            
            client = new SocketClient();     
            client.OnClientConnected += (s, args) =>     
            {     
                if (args.IsSuccessful)     
                {     
                    string data = JsonConvert.SerializeObject(new KeyValuePair<string, string>(key, value));     
                    client.Send(data);     
                } 
            
                taskCompletionSource.SetResult(args.IsSuccessful); 
                client = null; 
            }; 
            
            client.Connect(Binder.Instance.CurrentService.Location, 6169); 
            return taskCompletionSource.Task;     
        } 
    } 
}

From here on, ResponseHelper is once again involved, grouping all the data alphabetically—remember this call:

// returns the list of sets that are associated with the current server. 
if (response.Marker == Constants.COMMAND_SERVER_SET_LIST) 
{     
    List<string> items = JsonConvert.DeserializeObject<List<string>>(response.Content);     
    if (items != null) 
    {     
        List<Group<string>> groupedItems = Group<string>.CreateGroups(items,     
    CultureInfo.CurrentCulture, (string s) => { return s[0].ToString(); }, true);     
        Binder.Instance.Sets = groupedItems;     
    } 
    else 
    {     
        Binder.Instance.Sets = new List<Group<string>>();     
    } 
}

The grouped collection is later bound to a LongListSelector instance:

For each handshake call to the server, the set collection will be re-initialized, in case the server was updated by another device while the user was not taking any actions.

Adding a set takes the user to AddSetPage.xaml, where the user input is once again validated and the appropriate command sent to the currently selected server:

private async void AttemptAddSet()
{
        if (!string.IsNullOrWhiteSpace(txtName.Text))
        {
                this.Focus();
        
                bool commandSent = await CommonViewModel.Instance.CommandClient.SendCommand(CommonViewModel.Instance.CurrentServer.Location, 
                    string.Format(Constants.COMMAND_CREATE_SET, ((ListPickerItem)lstType.SelectedItem).Content.ToString()), txtName.Text);
                if (!commandSent)
                {
                        Alert.Send(Constants.MESSAGE_SERVER_CONNECT_FAIL);
                }
                else
                {
                        CommonViewModel.Instance.IsWaiting = true;
                        NavigationService.GoBack();
                }
        }
        else
        {
                Alert.Send(Constants.MESSAGE_NO_NAME_FAIL);
        }
}

The end-user is also able to specify whether the new set is a list or a pad. Since the server does not explicitly define the type of a set beyond marking whether it’s a list, it is possible to have an arbitrary type here.

To give you an idea of what it looks like in the current release of MissionControl, here is the pad representation of a set of commands:

It is a convenient way to display buttons for typical actions, such as channel switching through digits. Since we can safely assume many of those will be tapped sequentially, a list would be inconvenient to scroll through.

On the other hand, some remote control commands work well with a list because no sequences are invoked most of the time:

If the pad is not desired, it can easily be swapped with another design and internal template - the appearance is swapped dynamically and is not hard-bound to a string value.

Once a set is selected, a connection attempt is made to the current server in order to check whether there is still a communication channel available with the resource that fetched the initial list of commands. If a connection is established, the server will also return a set of commands that are available in the set at the time of the request.

private async void AttemptLoadCommands()
{
        bool commandSent = await CommonViewModel.Instance.CommandClient.SendCommand(CommonViewModel.Instance.CurrentServer.Location,
            Constants.COMMAND_GET_COMMANDS, CommonViewModel.Instance.CurrentSet);
        if (!commandSent)
        {
                Alert.Send(Constants.MESSAGE_SERVER_CONNECT_FAIL);
                CommonViewModel.Instance.IsWaiting = false;
        }
}

You’ve probably already noticed that both for commands and sets, the initial routine verifies the connection to the server. The server might go dark after the set list is loaded, therefore rendering any attempt to process other commands impossible. To avoid scenarios in which the user is waiting for a response from a server that doesn’t run, the user is notified before being redirected to the subsequent view, if the connection fails. That way unnecessary navigation passes are out of the picture.

If the user selects a command from one of the lists demonstrated above, an EXECUTE directive is issued via the CommandClient class:

private async void lstCommands_SelectionChanged(object sender, SelectionChangedEventArgs e) 
{ 
    if (lstCommands.SelectedItem != null)
    { 
        string selectedItem = lstCommands.SelectedItem.ToString(); 
        
        CommandClient commandClient = new CommandClient(); 
        bool commandSent = await commandClient.SendCommand(string.Format(Constants.COMMAND_EXECUTE, 
        
        Binder.Instance.CurrentSet), selectedItem); 
    
        if (!commandSent) 
        {     
            Alert.Send(Constants.MESSAGE_SERVER_CONNECT_FAIL);     
        } 
        
        lstCommands.SelectedItem = null; 
    } 
}

Once the server receives the command, it will send it to the target without additional notifications being released to the connecting client.

When it comes to learning a new remote control code in LearnCodePage.xaml, the procedure is exactly the same as with any other part of the server communication process — a LEARN_CODE command is sent to the server with the associated set and new command name, and the server will wait for incoming IR input, leaving the connecting device free (no waiting lock is issued):

private async void AttemptLearnCode()
{
        if (!string.IsNullOrWhiteSpace(txtName.Text))
        {
                CommonViewModel.Instance.IsWaiting = true;
                this.Focus();
        
                bool commandSent = await CommonViewModel.Instance.CommandClient.SendCommand(CommonViewModel.Instance.CurrentServer.Location,
                    string.Format(Constants.COMMAND_LEARN_NEW, CommonViewModel.Instance.CurrentSet), txtName.Text);
                if (!commandSent)
                {
                        Alert.Send(Constants.MESSAGE_SERVER_CONNECT_FAIL);
                }
                else
                {
                        Alert.Send(Constants.MESSAGE_COMMAND_LEARN_WAIT);
                        NavigationService.GoBack();
                }
        
                CommonViewModel.Instance.IsWaiting = false;
        }
        else
        {
                Alert.Send(Constants.MESSAGE_NO_NAME_FAIL);
        }
}

Once the server learns a new command — if, and only if, the user still works in the context of the same server — an alert will be displayed, telling the user whether the command was successfully learned.

For convenience purposes, I also implemented a quick launch panel, where frequently-used commands can be placed. Whenever a user wants to add something here, he will tap-and-hold on an existing command in any of the sets that are available for any given server, and select the "add to quick launch" option. Once completed, the stored commands will be available on the main page, even when the user is not directly connected to the server that carries the command:

Because this interaction layer is placed outside the boundaries of a single server or set, I needed to create a special data model to store the quick commands and the related connection information, that would let me call the server even when it is not the currently selected one:

namespace Coding4Fun.MissionControl.WP.Models
{
        public class Favorite
        {
            public string CommandName { get; set; }
            public string ParentSet { get; set; }
            public string ServerLocation { get; set; }
        }
}

Same as with the list of servers, the list of favorites is deserialized on application startup:

MainPageViewModel.Instance.Favorites = Serialize.Open<ObservableCollection<Favorite>>(Constants.FAVORITES_FILE);

Logically, we would also need to have a way to eliminate trailing commands for servers or sets that have been removed, since those can no longer be invoked or might have a different meaning on servers that were added and have the same IP as the previous owner. This is easily done with a simple LINQ expression that is passed to RemoveTrailingFavorites in the CollectionHelper class:

internal static void RemoveTrailingFavorites(Func<Favorite,bool> predicate)
{
        var favorites = MainPageViewModel.Instance.Favorites.Where(predicate).ToList();
    
        if (favorites.Count() > 0)
        {
                foreach (var favorite in favorites)
                {
                        Deployment.Current.Dispatcher.BeginInvoke(() =>
                            {
                                    MainPageViewModel.Instance.Favorites.Remove(favorite);
                            });
                }
        
                Serialize.Save(Constants.FAVORITES_FILE, MainPageViewModel.Instance.Favorites);
        }
}

A typical usage scenario is reflected in the server removal snippet:

public static bool RemoveServer(Server server)
{
        try
        {
                RemoveTrailingFavorites(x=> x.ServerLocation == server.Location);
        
                MainPageViewModel.Instance.Servers.Remove(server);
        
                Serialize.Save(Constants.SERVERS_FILE, MainPageViewModel.Instance.Servers);
        
                return true;
        }
        catch
        {
                return false;
        }
}

Because an ObservableCollection<T> is used for both the list of servers and quick launch commands, the view will be instantly updated to reflect the changes.

Improvements to the project

This specific project relies on a hybrid IR transmitter and receiver, which is not exactly cheap. As a step forward for this project, it can be adapted to use a central microcontroller that acts as a server (e.g. Netduino) and a series of IR emitters (instead of using a composite receiver/emitter) connected to it. Reduced cost for the IR infrastructure is key, as not every single component needs the capability to learn IR commands. You can have a single command capturing endpoint and multiple transmitters. This will also eliminate the need for a desktop client, since the server on the microcontroller can be built to be accessible via a web-browser.

Another important aspect not covered in this article is security. With the current workflow, anyone who has direct access to the server IP is able to do anything he wants with the data handled by the server. I am basing my writing on the assumption that you are testing the application on a secure local network and that the the odds of something like this happening are close to zero. However, for other environments where tampering with a server might be unacceptable, consider implementing a layer of security between the server and the client.

Conclusion

With affordable microcontrollers and sensors, home and office automation can be a nice bonus resulting from little investment. This article covers the implementation of a proof-of-concept server and application that can be easily extended and adapted to a variety of environments and devices.

So Philips recently introduced their Hue Connected Bulbs: an easy-to-use set of LED light bulbs and Wi-Fi connected bridge which allows you to dynamically change the color of your home lighting using their iOS or Android app. What’s particularly cool is that the bridge has a web API which you can access to set the colors of each bulb with your own app.

We at untitled network developed our own Philips Hue app called Oni: light Control, which is currently available on the Windows Phone Store. In addition to allowing you to set the color of your home lighting with defined “Moods”, the app also allows you to do the same using your phone’s built in voice commands or inexpensive NFC stickers. Here’s a demo of the Oni: Light Control in action:

I’m going to show you how you can develop your own app using the color picker from the Coding4Fun Toolkit for Windows Phone and the Json.Net library from Newtonsoft. These libraries can be found on NuGet, but you’ll obviously need the Philips Hue Connected Bulbs kit to test things out.

Getting Started:

The Philips Hue API uses a restful JSON interface you can access using any http client. Documentation on all of the supported methods by the Philips Hue bridge can be found at http://blog.ef.net/2012/11/02/philips-hue-api.html.

To get started, you’ll need authorized access to your bridge’s API. Once the bridge has successfully established a network connection with your router, discover its internal IP address via the URL: http://www.meethue.com/api/nupnp

You should get a response similar to:

[{"id":"ffss00fffe123456","internalipaddress":"192.168.1.100","macaddress":"0aa:bb:cc:dd:00:11"}]

Note the internalipaddress value and use the IP to access the bridge’s API directly the with the URL http://192.168.1.100/api

Now, since we haven’t registered a user for authorization, attempting to access the hub will return an error from the bridge:

[{"error":{"type":1,"address":"/","description":"unauthorized user"}}]

To register a new user, we’ll first POST the username we wish to use.

var client = new WebClient();
 
//our uri to perform registration
var uri = new Uri(string.Format("http://{0}/api", HostnameTextBox.Text));
 
//create our registration object, along with username and description
var reg = new
{
    username = UsernameTextBox.Text,
    devicetype = "Coding4Fun Hue Light Project"
};
 
var jsonObj = JsonConvert.SerializeObject(reg);
 
 
//decide what to do with the response we get back from the bridge
client.UploadStringCompleted += (o, args) =] Dispatcher.BeginInvoke(() =]
{
    try
    {
        ResponseTextBox.Text = args.Result;
    }
    catch (Exception ex)
    {
        ResponseTextBox.Text = ex.Message;
    }
});
 
//Invoke a POST to the bridge
client.UploadStringAsync(uri, jsonObj);

Note the response we get back from our hub will be

[{"error":{"type":101,"address":"","description":"link button not pressed"}}]

This is because bridge requires you to first push the link button before new registrations can be made. After pushing the button and invoking the registration function again, you should receive the following result from the bridge:

[{"success":{"username":"coding4fun"}}]

Setting the Bulb Color:

We should now be able to access all methods on the bridge. You can get all of the configuration details, including all of the bulbs and their statuses with the same base url: http://192.168.1.100/api/coding4fun

Now comes the fun part. There are three color modes in which you can use to set the color of your bulbs:

• hue & sat: ‘hue’ is a color range between 0-65535 which represent 182.04*degrees, ‘sat’ is saturation with a range of 0-254
• xy: are coordinates in the CIE 1931 space
• ct: is a color temperature expressed in mireds from 154 to 500, coolest to warmest respectfully

Source: http://rsmck.co.uk/hue

We’ll be setting the colors of our bulbs using hue & saturation parameters. Luckily, the Coding4Fun Toolkit for Windows Phone has three awesome color picker controls and some useful color extensions which makes setting the bulb colors a breeze. We start by building our state object – a list of parameters we want our bulb to be set to. Then we use the PUT verb to set the light with the following URL: http://{BRIGE-IPADDRESS}/api/coding4fun/lights/1/state

The “1” in the url is the 1-based index of the bulb you want to set.

//Get the HSV Value from the currently selected color
var hsv = LightColorSlider.Color.GetHSV();
 
//build our State object
var state = new
{
    on = true,
    hue = (int)(hsv.Hue * 182.04), //we convert the hue value into degrees by multiplying the value by 182.04
    sat = (int)(hsv.Saturation * 254)
};
 
//convert it to json:
var jsonObj = JsonConvert.SerializeObject(state);
 
//set the api url to set the state
var uri = new Uri(string.Format("http://{0}/api/{1}/lights/{2}/state", HostnameTextBox.Text, UsernameTextBox.Text, LightIndexTextBox.Text));
 
var client = new WebClient();
 
//decide what to do with the response we get back from the bridge
client.UploadStringCompleted += (o, args) => Dispatcher.BeginInvoke(() =>
{
    try
    {
        ResponseTextBox.Text = args.Result;
    }
    catch (Exception ex)
    {
        ResponseTextBox.Text = ex.Message;
    }
     
});
 
//Invoke the PUT method to set the state of the bulb
client.UploadStringAsync(uri, "PUT", jsonObj);

That’s it! Be sure to check out all of the other functions the (other functions the Hue API supports) at http://developers.meethue.com/index.html

Bio:
Jarem Archer is a self-taught Software Developer and UX Designer. He’s part of a small team at untitled network who have a passion for video games, digital motion, entertainment and gadgets. Follow him on Twitter at http://twitter.com/unt1tled.

What do you need

You will need to download and install ASP.NET MVC3 to work on the web frontend and Windows Phone 8 SDK to work on the mobile applications. An Azure Mobile Services account will be necessary, and of course don’t forget to download and install the Azure Mobile Services client libraries. All three components are available at no additional charge.

NOTE: Without the Azure Mobile Services SDK installed on the development machine, the compilation process will fail for the Windows Phone application.

Setting up The Data Store

First we need to establish the general design of the application and organize the workflow. The application will provide two ways to assign the dynamic lockscreen:

• With the help of custom image sets that are provided by the service;
• With the help of self-created image sets, aggregated from images provided by the service but ultimately managed by the end-user.

Let’s talk about the general data model. Every image belongs to a certain category and to keep track of each we need a table with two columns—category ID and category name. We also need another core table containing the image references themselves, with the following columns: image URL, descriptive name, and the category ID to which it belongs. The overall structure looks like this:

Now to the Windows Azure Management Portal and creating a new Mobile Service.

Once created, you need to specify database information, just like you would with a standard SQL Server database:

As the database is being created, you can easily integrate it with SQL Server Management Studio. You will need the server address, which may be obtained in the Azure Management Portal. To login, use the credentials that you set when creating the core database.

Create the two tables mentioned above, with the following column configuration:

Categories

• ID - int
• Name – varchar(100)

Images

• ID – int
• URL – varchar(500)
• Name – varchar(100)
• CategoryID – int

You can create these tables either in the SQL Server Management Studio or through the Azure Management Portal. However, you will need the Management Studio to create the column structure, as the Azure Management Portal does not offer this functionality right now.

By default, the id column will be created automatically. To add the Name column to the Categories table, run this query:

ALTER TABLE c4flockscreen.Categories
ADD Name VARCHAR(100)

To add the missing columns to the Images table, simply execute this query:

ALTER TABLE c4flockscreen.Images
ADD URL VARCHAR(500),
Name VARCHAR(100),
CategoryID INT

Now that the database is ready, we’ll proceed to working on the web layer, which will effectively be the administrative portal for the service.

Creating the Web Portal

There should be a way to easily manage images and constantly expand the collection of possible lockscreen wallpapers. One way to do this is create a basic management portal that can carry basic CRUD operations.

Start by creating an empty project:

If you are not yet aware of the Model-View-Controller (MVC) development pattern, here is a good read explaining the fundamentals.

Create a new controller in the Controllers folder, named HomeController. This will be the only controller created in this project. For now, add an ActionResult-based function that will return the main view:

using System.Web.Mvc;

namespace Coding4Fun.Lockscreen.Web.Controllers
{
    public class HomeController : Controller    
    {    
        public ActionResult MainView()    
        {    
            return View();    
        }
    }
}

Having the controller without the proper views is pointless, so create a new view in Views/Home and name it MainView. For now, do not focus on the visual layout of the page, but rather on the functional aspect of the web frontend. If you run the application now, you will most likely get a 404 response. That is because the associated home view is by default not found. Open App_Start/RouteConfig.cs and make sure that the default view is set to MainView instead of Index.

routes.MapRoute(
name: "Default",
url: "{controller}/{action}/{id}",
defaults: new { controller = "Home", action = "MainView", id = UrlParameter.Optional }
);

The core is created and now if running the web application you will see a basic HTML page:

We now need to handle data from the Azure Mobile Services database. Out-of-the-box, there is no ASP.NET SDK available, but the database can be easily accessed through a REST API. But before that, we need to define the data models for the Categories and Images table. Begin by creating two classes in the Models folder:

Category.cs:

public class Category
{
public int? id { get; set; }
public string Name { get; set; }
}

Image.cs:

public class Image
{
public int? id { get; set; }    
public string URL { get; set; }
public string Name { get; set; }
public int CategoryID { get; set; }
}

Each of the properties is tied to the associated column in the database we created earlier. Notice that the ID values are nullable. This is introduced because the index will by default be automatically assigned. When new instances of Category or Image are created, I will not explicitly set the id property, so keeping it null instead of at a potential default value of 0 will ensure that it is properly set on the backend.

Let’s now create the connectivity engine that will allow us to query the content of the data store. For this purpose, I created a DataStore folder and a DataEngine class inside it. We will need a unique API key for each of our requests, so open the Azure Management Portal and obtain it from there:

In order to keep consistency between projects, and to be able to re-use the same Azure Mobile Services API key and core URL, I created an AuthConstants class in the context of the Coding4Fun.Lockscreen.Core project. It carries three static fields:

public static class AuthConstants
{
    public static string AmsApiKey = "YOUR_KEY_HERE";    
    public const string AmsUrl = "https://c4flockscreen.azure-mobile.net/";
    public const string AmsTableUrl = AmsUrl + "tables/";
}

Back in the ASP.NET project, the query operations are carried with the help of HttpClient initialized in the class constructor, which also includes the key used to authenticate the requests via the X-ZUMO-APPLICATION header:

private HttpClient client;

public DataEngine()
{    
    client = new HttpClient();    
    client.DefaultRequestHeaders.Add("X-ZUMO-APPLICATION", KEY);
    client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));    
}

This is the basic data harness. I also implemented two core methods in order to get all existing categories:

public IEnumerable<Category> GetAllCategories()
{
    var result = client.GetStringAsync(string.Concat(CORE_URL,"Categories")).Result;
    IEnumerable<Category> categories = JsonConvert.DeserializeObject<IEnumerable<Category>>(result);
    return categories;    
}

And images:

public IEnumerable<Image> GetAllImages()
{
    var result = client.GetStringAsync(string.Concat(CORE_URL, "Images")).Result;
    IEnumerable<Image> images = JsonConvert.DeserializeObject<IEnumerable<Image>>(result);
    return images;    
}

For each of these, a basic request is made with the table name appended to the base URL (represented by the CORE_URL constant). Since JSON.NET is now bundled with ASP.NET, I am able to easily deserialize the returned JSON data array to an IEnumerable<Type>. There is one problem, however, with the GetAllImages approach. It implies that even if I want to use LINQ to query the existing image collection, I have to first download the entire set locally.

Fortunately, the Azure Mobile Services REST API provides an endpoint with filtering, and that’s what I am using in GetCategoryById and GetImagesByCategoryId:

public Category GetCategoryById(int id)
{    
    string composite = string.Concat(CORE_URL, "Categories?$filter=(id%20eq%20", id.ToString(), ")");    
    var result = client.GetStringAsync(composite).Result;
    IEnumerable<Category> categories = JsonConvert.DeserializeObject<IEnumerable<Category>>(result);
    return categories.FirstOrDefault();
}

public IEnumerable<Image> GetImagesByCategoryId(int id)
{    
    string composite = string.Concat(CORE_URL, "Images?$filter=(CategoryID%20eq%20", id.ToString(), ")");    
    var result = client.GetStringAsync(composite).Result;
    IEnumerable<Image> images = JsonConvert.DeserializeObject<IEnumerable<Image>>(result);
    return images();
}

Notice the ?$filter= parameter, in which the conditional is URL encoded and is wrapped in parentheses. For the category query, I am checking the id value, and for the image I’m checking CategoryID.

In the Views/Home folder, create a new view and name it Images. It will be used to list existing images that are associated with one of the selected categories. You also need to adjust the controller code to handle the incoming data:

using Coding4Fun.Lockscreen.Web.DataStore;
using System.Web.Mvc;

namespace Coding4Fun.Lockscreen.Web.Controllers
{    
    public class HomeController : Controller    
    {    
        DataEngine engine;    
        public HomeController()
        {    
            engine = new DataEngine();    
        }
        
        public ActionResult MainView()
        {    
            var categories = engine.GetAllCategories();    
            return View(categories);
        }
        
        public ActionResult Images(int categoryId)
        {    
            var images = engine.GetImagesByCategoryId(categoryId);    
            if (images != null)
            {    
                return View(images);    
            }
            
            return View("MainView");
        }
    }
}

For the main view, I am getting the list of categories and passing them as the bound model. For the Images view, the category ID is passed as an argument that will later enable the engine to return a list of all images that have CategoryID set to that value. In case the returned list is not null, the view is shown. Otherwise, the main view is the terminal point.

In its current state, I’ll be able to use the frontend to list existing categories and images, but not to add, remove, or update items. Adding a category and an image is a matter of modifying an HttpClient request, with the help of HttpRequestMessage. For example, here is how I can add a category through my DataEngine class:

public HttpStatusCode AddCategory(Category category)
{    
    var serializedObject = JsonConvert.SerializeObject(category, new JsonSerializerSettings() { NullValueHandling = NullValueHandling.Ignore });
    var request = new HttpRequestMessage(HttpMethod.Post, string.Concat(CORE_URL, "Categories"));
    request.Content = new StringContent(serializedObject, Encoding.UTF8, "application/json");
    var response = client.SendAsync(request).Result;
    return response.StatusCode;
}

JSON.NET capabilities are used to serialize the object that needs to be inserted. The POST request is executed against the standard table URL, with the UTF8 encoded JSON string. Since the client is already carrying the basic authentication header, all that needs to be done is calling the SendAsync function.

Updating a category follows the same approach, though a PATCH method is used for the request and the URL contains the index of the category that needs to be updated:

public HttpStatusCode UpdateCategory(Category category)
{    
    var request = new HttpRequestMessage(new HttpMethod("PATCH"), string.Concat(CORE_URL, "Categories", "/", category.id));    
    var serializedObject = JsonConvert.SerializeObject(category);
    request.Content = new StringContent(serializedObject, Encoding.UTF8, "application/json");
    var response = client.SendAsync(request).Result;
    return response.StatusCode;
}

To delete a category from the data store, one simply needs to pass a parameter to it that identifies the index of the category that needs to be removed:

public HttpStatusCode DeleteCategoryFromId(int categoryId)
{    
    var request = new HttpRequestMessage(HttpMethod.Delete, string.Concat(CORE_URL, "Categories", "/", categoryId));    
    var response = client.SendAsync(request).Result;
    return response.StatusCode;
}

For images, the same methods can be used, with the Images table passed as the name for the target in the composite URL. Let’s now get back to working on some of the views. A static category list is not fun, so let’s create a way to add new categories. Right click on the Views/Home folder and select Add View:

A great thing about the view creation process in Visual Studio is the fact that you are able to use a basic scaffold template for a strongly-typed view. In this case, I am associating it with a Category class and using the Create template. I now need to modify the controller code to process requests to AddCategory. I need to handle two types of requests, GET and POST, because the view will be displayed to both add an item and submit an item:

public ActionResult AddCategory()
{    
    return View();    
}

[HttpPost]
public ActionResult AddCategory(Category category)
{    
    if (ModelState.IsValid)    
    {    
        engine.AddCategory(category);    
        return RedirectToAction("MainView");    
    }
    
    return View();
}

For a GET request, I am simply returning the view. For a POST view, I am adding the category that was defined by the bound model through the local DataEngine instance, after which the user is redirected to the main view. But we also need to add an ActionResult for the MainView to obtain the list of items that are currently in the Categories table:

public ActionResult MainView()
{    
    var categories = engine.GetAllCategories();    
    return View(categories);
}

The DataEngine instance will return all categories in an IEnumerable<Category> form that are passed as the model for the main view. The layout of MainView.cshtml can be as simple as a table:

@{    
    ViewBag.Title = "Coding4Fun Dynamic Lockscreen";    
}

<h2>Coding4Fun Dynamic Lockscreen - Categories</h2>

<table>
<tr>
<td>
<b>ID</b>
</td>
<td>
<b>Category Name</b>
</td>
</tr>

@foreach (var p in Model)
{    
    <tr>    
    <td>
    @p.id
    </td>
    <td>
    @p.Name
    </td>
    <td>
@Html.ActionLink("Images", "Images", new { categoryId = p.id })
    </td>
    <td>
@Html.ActionLink("Edit", "EditCategory", new { categoryId = p.id })
    </td>
    <td>
@Html.ActionLink("Delete", "DeleteCategory", new { categoryId = p.id })
    </td>
    </tr>
}

</table>

@Html.ActionLink("Add Category", "AddCategory")

The ActionLink helper allows me to invoke a view and, if necessary, pass specific parameters to it (e.g., when I need to identify the category that needs to be deleted or edited). Some of the views listed here are not yet created, but I can easily use placeholder names in any case.

The ultimate result for the main page will look like this:

Notice that you are also able to add new categories now by clicking on the Add Category link on the bottom. This will redirect you to the AddCategory view that we created:

Let’s see how to implement the category editing in the web frontend. First of all, create a new view in Views/Home and name it EditCategory. Use the Edit scaffold template. Like AddCategory, EditCategory needs to be handled in two separate ways for GET and POST requests in the controller:

public ActionResult EditCategory(int categoryId)
{    
    Category category;    
    category = engine.GetCategoryById(categoryId);
    if (category != null)
       return View(category);
    return View("MainView");
}

[HttpPost]
public ActionResult EditCategory(Category category)
{    
    if (ModelState.IsValid)    
    {    
        engine.UpdateCategory(category);    
        return RedirectToAction("MainView");
    }
    
    return View();
}

For a GET request, we need to identify the category that needs to be added by its index, so we are using a categoryId argument passed to the view, which is later used by the DataEngine instance to retrieve the category from the data store. For a POST action, the implementation for UpdateCategory from above is used, where a PATCH request is run with the serialized object bound to the view.

For the Delete action, no additional view is necessary but the controller still needs a handler, so we can use a snippet like this:

public ActionResult DeleteCategory(int categoryId)
{    
    engine.DeleteCategoryFromId(categoryId);    
    return RedirectToAction("MainView");
}

You can use the same approach to add, delete, and edit items in the list of images. For adding images, however, you might want to pass the category identifier. When images are listed after the category has been selected, it is necessary to provide a way to identify the category to which new entities should be added. To do this, we can. in the main controller. pass the category index to the view when the Images action is being triggered:

public ActionResult Images(int categoryId)
{    
    var images = engine.GetImagesByCategoryId(categoryId);    
    if (images != null)
    {    
        ViewData["CID"] = categoryId;    
        return View(images);
    }
    
    return View("MainView");
}

Afterwards, the categoryId value can be obtained by using the CID key for ViewData inside the view itself.

Let’s now take a look at how images are represented for each category. I created a custom view to list all the images associated with the Images category. If you look above at the controller code, you will notice that I am passing the category ID, through which the image set query is executed, and the returned collection is set as the bound model:

public ActionResult Images(int categoryId)
{    
    var images = engine.GetImagesByCategoryId(categoryId);    
    if (images != null)
    {    
        ViewData["CID"] = categoryId;    
        return View(images);
    }
    
    return View("MainView");
}

When an image needs to be added, call the AddImage view. In HomeController.cs, it carries implementations for both GET and POST requests:

public ActionResult AddImage(int categoryId)
{    
    Image image = new Image();    
    image.CategoryID = categoryId;
    return View(image);    
}

[HttpPost]
public ActionResult AddImage(HttpPostedFileBase file, Image image)
{    
    if (file != null && file.ContentLength > 0)    
    {    
        var fileName = Path.GetFileName(file.FileName);    
        var path = Path.Combine(Server.MapPath("~/Uploads"), image.CategoryID.ToString(), fileName);
        string dirPath = Path.GetDirectoryName(path);
        
        if (!Directory.Exists(dirPath))
            Directory.CreateDirectory(dirPath);
        
        file.SaveAs(path);

        string applicationUrl = string.Format("{0}://{1}{2}",        
            HttpContext.Request.Url.Scheme,
            HttpContext.Request.ServerVariables["HTTP_HOST"],
            (HttpContext.Request.ApplicationPath.Equals("/")) ? string.Empty : HttpContext.Request.ApplicationPath
        );
        
        image.URL = Path.Combine(applicationUrl, "Uploads", image.CategoryID.ToString(), fileName);
    }
    
    if (ModelState.IsValid && image.URL != null)
    {    
        engine.AddImage(image);    
        return RedirectToAction("Images", new { categoryID = image.CategoryID });
    }
    
    return View();
}

When a GET request is executed against the AddImage endpoint, I pass the category ID as the flag, signaling which category the image should be included in. When a POST request is executed, it can go two ways—either the user is passing an existing link to a hosted image or the user is uploading his own image to the local server. When an upload is inbound, HttpPostedFileBase carries the content that needs to be pushed to the server.

The upload component on the view itself is done by creating a form with a file input:

<h2>Or you could upload your own file: </h2>

@if (Model != null)
{    
    using (Html.BeginForm("AddImage", "Home", FormMethod.Post, new { enctype = "multipart/form-data", image = Model }))
    {    
        @Html.HiddenFor(model => model.CategoryID);    
        <input type="file" name="file" />
        <input type="submit" value="OK" />
    }
}

If there is no file selected, the system assumes that the user just decided to add an existing URL.

It’s important to mention that the upload workflow relies on the availability of the Upload folder. It is created by default when the project is deployed to the server, but you also need to make sure that the ASP.NET user on the machine where IIS is located has the appropriate write permission for the folder.

The Windows Phone 8 Application Foundation

Create a new Windows Phone 8 application and add a reference to Windows Azure Mobile Services Managed Client. It should be available in the Extensions section if you installed the Windows Azure Mobile Services SDK as I mentioned at the beginning of the article:

In App.xaml.cs you need to create an instance of MobileServiceClient that will be used as the central connection point to the database. Notice that I am using the predefined AMS and API KEY string constants:

public static MobileServiceClient MobileService =
new MobileServiceClient(AuthConstants.AmsUrl, AuthConstants.AmsApiKey);

The mobile application should also carry the data models for both the categories and images. That said, we can reorganize those a bit for a more convenient data binding layout. To ensure that we can reuse the classes from different application components, I am once again using the Coding4Fun.Lockscreen.Core project.

Create a new folder called Models and add a new class called Category:

using System.Collections.ObjectModel;

namespace Coding4Fun.Lockscreen.Core.Models
{    
    public class Category    
    {    
        public Category()    
        {    
            Images = new ObservableCollection<Image>();    
        }
    
        public int? id { get; set; }    
        public string Name { get; set; }

        public ObservableCollection<Image> Images { get; set; }
        
        public override string ToString()        
        {    
            return Name;    
        }
    }
}

We are still relying on a nullable index value, but now there is an ObservableCollection for images. The reason for using this specific collection type is because with an ObservableCollection, binding updates are performed automatically when new items are added or removed, therefore cutting the need to implement the notification mechanism.

The ToString function is overridden to simplify data extraction on binding. When a collection with categories will be hooked to a list, for example, I don’t have to create a converter or a property link.

For the Image model, create a new class called Image in the same Models folder:

namespace Coding4Fun.Lockscreen.Core.Models
{    
    public class Image    
    {    
        public int? id { get; set; }
        public string URL { get; set; }
        public string Name { get; set; }
        public int CategoryID { get; set; }    
    }
}

Application Workflow & Storage

Let’s talk about how image categories will be handled in the application. On application startup, the database is queried for the available categories and each of them is listed on the home screen. If the user taps on one of the categories, the database is queried for the images that are associated with the category index.

However, the user should also be able to create his own custom categories that will only be available in-app. Those categories can carry images from multiple other categories, if necessary, with the default reference set to the internal storage.

Since we are working with local storage, let’s create a helper class called LocalStorageHelper in the Coding4Fun.Lockscreen.Core project in the Storage folder. This class will carry basic read and write functions, allowing us to store data internally:

public static class LocalStorageHelper
{    
    public async static void WriteData(string folderName, string fileName, byte[] content)    
    {    
        IStorageFolder rootFolder = ApplicationData.Current.LocalFolder;
        
        if (folderName != string.Empty)    
        {    
            rootFolder = await rootFolder.CreateFolderAsync(folderName,    
            CreationCollisionOption.OpenIfExists);    
        }
        
        IStorageFile file = await rootFolder.CreateFileAsync(fileName, CreationCollisionOption.ReplaceExisting);
        using (var s = await file.OpenStreamForWriteAsync())
        {    
            s.Write(content, 0, content.Length);    
        }
    }
    
    public static async void ClearFolder(string folderName)
    {    
        var folder = await ApplicationData.Current.LocalFolder.GetFolderAsync(folderName);    
        if (folder != null)
        {    
            foreach (IStorageFile file in await folder.GetFilesAsync())    
            {    
                await file.DeleteAsync();    
            }
        }
    }
    
    public static async Task<string> ReadData(string fileName)
    {    
        byte[] data;    
        StorageFolder folder = ApplicationData.Current.LocalFolder;
        StorageFile file = await folder.GetFileAsync(fileName);
        using (Stream s = await file.OpenStreamForReadAsync())
        {        
            data = new byte[s.Length];    
            await s.ReadAsync(data, 0, (int)s.Length);
        }
        
        return Encoding.UTF8.GetString(data, 0, data.Length);
    }    
}

Notice that I am using the newly-introduced StorageFolder/StorageFile capabilities. If you worked with Windows Store application development, you are probably already familiar with them. Application.Current.LocalFolder gives me direct access to the local directory. which can be modified from within the application itself. It works in a manner similar to IsolatedStorageFile in Windows Phone 7, but with more flexibility when it comes to creating new folders and files and well doing file sweeps.

As I mentioned above, there will be internal data stored as XML. For this purpose, I need a class that carries serialization and deserialization routines, and I can simplify this task by using the Coding4Fun Toolkit Serialize.Save<T> and Serialize.Open<T> capabilities. Calls to these functions allow flexible serialization, where by default the static class is not aware of the serialization type, but is instead able to dynamically infer it from the incoming data. Once the byte layout is obtained for the content, I use the LocalStorageHelper class to write it to a file.

As there are multiple UI items that need to be bound to collections and object instances, I have a CentralBindingPoint class in my main project that is my main view model (it implements INotifyPropertyChanged). It implements the singleton pattern, so that the main instance is created on initialization and is subsequently re-used as necessary:

using Coding4Fun.Lockscreen.Core.Models;
using System;
using System.Collections.ObjectModel;
using System.ComponentModel;

namespace Coding4Fun.Lockscreen.Mobile
{    
    public class CentralBindingPoint : INotifyPropertyChanged    
    {    
        static CentralBindingPoint instance = null;    
        static readonly object padlock = new object();
        
        public CentralBindingPoint()
        {    
            Categories = new ObservableCollection<Category>();    
            CustomCategories = new ObservableCollection<Category>();    
        }
        
        public static CentralBindingPoint Instance
        {    
            get    
            {    
                lock (padlock)    
                {    
                    if (instance == null)    
                    {    
                        instance = new CentralBindingPoint();    
                    }
                
                    return instance;        
                }
            }
        }
        
        private ObservableCollection<Category> _categories;
        public ObservableCollection<Category> Categories
        {    
            get    
            {    
                return _categories;    
            }
            set
            {    
                if (_categories != value)        
                {    
                    _categories = value;    
                    NotifyPropertyChanged("Categories");
                }
            }
        }
        
        private ObservableCollection<Category> _customCategories;
        public ObservableCollection<Category> CustomCategories
        {    
            get    
            {    
                return _customCategories;    
            }
            set
            {    
                if (_customCategories != value)    
                {    
                    _customCategories = value;    
                    NotifyPropertyChanged("CustomCategories");
                }        
            }
        }
    
        private Category _currentCategory;    
        public Category CurrentCategory
        {    
            get    
            {        
                return _currentCategory;    
            }
            set
            {    
                if (_currentCategory != value)    
                {    
                    _currentCategory = value;    
                    NotifyPropertyChanged("CurrentCategory");
                }
            }
        }
        
        public event PropertyChangedEventHandler PropertyChanged;
        private void NotifyPropertyChanged(String info)
        {    
            if (PropertyChanged != null)    
            {    
                System.Windows.Deployment.Current.Dispatcher.BeginInvoke(    
                () =>
                {    
                    PropertyChanged(this, new PropertyChangedEventArgs(info));    
                });
            }
        }
    }
}

On the main page, I create a Pivot-based layout to have an easy way to transition between the web collections (categories) and the local ones:

For each of the collection types, there is a ListBox with a custom DataTemplate assigned for each item. The items are obtained from the Categories collection for web sets and the CustomCategories collection for local sets, both in the CentralBindingPoint view model.

The categories are loaded with the help of the DataEngine class that I added in the Data folder in the main application project. It is a wrapper for the Azure Mobile Services data operations, allowing me to aggregate the list of categories and images, given that I know the category index:

public class DataEngine
{    
    async public Task<List<Category>> GetCategoryList()    
    {    
        IMobileServiceTable<Category> table = App.MobileService.GetTable<Category>();    
        List<Category> data = await table.ToListAsync();
        
        return data;
    }
    
    async public Task<List<Image>> GetImagesByCategoryId(int categoryId)
    {    
        IMobileServiceTable<Image> table = App.MobileService.GetTable<Image>();    
        List<Image> data = await table.Where(x => x.CategoryID == categoryId).ToListAsync();
        
        return data;
    }
}

When the main page loads, I use the local DataEngine instance to call GetCategoryList and obtain a List<Category> collection that is subsequently transformed into an ObservableCollection through one of the default constructors:

async void MainPage_Loaded(object sender, RoutedEventArgs e)
{    
    CentralBindingPoint.Instance.Categories = new ObservableCollection<Category>(await dataEngine.GetCategoryList());    
}

When a category is selected in the web sets list, I assign the selected item as the current category and navigate to the ImageSetPage.xaml page that will display the associated images:

async void ListBox_SelectionChanged_1(object sender, System.Windows.Controls.SelectionChangedEventArgs e)
{    
    var box = (ListBox)sender;
    
    if (box.SelectedItem != null)    
    {    
        Category selectedCategory = (Category)box.SelectedItem;    
        selectedCategory.Images = new ObservableCollection<Coding4Fun.Lockscreen.Core.Models.Image>
        (await dataEngine.GetImagesByCategoryId((int)selectedCategory.id));
        
        CentralBindingPoint.Instance.CurrentCategory = selectedCategory;
        NavigationService.Navigate(new Uri("/ImageSetPage.xaml", UriKind.Relative));
    }
}

Notice that the images are not loaded at the same time as the categories; rather, they’re loaded only when a category has been selected, hence the GetImagesByCategoryId call on selection.

For a custom set, the procedure is pretty much the same, the only difference being the fact that image references are already present since those were deserialized from the local storage:

private void lstCustomSets_SelectionChanged_1(object sender, SelectionChangedEventArgs e)
{    
    var box = (ListBox)sender;
    
    if (box.SelectedItem != null)    
    {    
        Category selectedCategory = (Category)box.SelectedItem;    
        CentralBindingPoint.Instance.CurrentCategory = selectedCategory;
        NavigationService.Navigate(new Uri("/ImageSetPage.xaml", UriKind.Relative));
    }
}

In ImageSetPage.xaml I use a ListBox with a WrapPanel in the ItemsPanelTemplate, which ensures that I can have only two images in a row and any additions will be wrapped, with a fixed row length. You can get that control from the WPToolkit (formerly known as Silverlight Toolkit for Windows Phone, available on NuGet).

Here is the basic XAML layout:

<ListBox 
    SelectionMode="Single"
    Margin="24"
    x:Name="lstImages" SelectionChanged="lstImages_SelectionChanged_1"
    ItemsSource="{Binding Path=Instance.CurrentCategory.Images,
    Source={StaticResource CentralBindingPoint}}"
    ItemTemplate="{StaticResource ListItemTemplate}">
    <ListBox.ItemsPanel>
        <ItemsPanelTemplate>
            <toolkit:WrapPanel ItemWidth="216" ItemHeight="260"/>
        </ItemsPanelTemplate>
    </ListBox.ItemsPanel>
</ListBox>

Now that we have a basic skeleton for the incoming data, let’s see how it can be transformed into a live lockscreen, on which wallpapers can be cycled. In the ImageSetPage.xaml page I have a button in the application bar that allows me to set the current category as the source for the switching wallpapers.

Currently, each Image instance carries an image URL and the images can be located anywhere outside the application. This can cause problems with the wallpaper setting process, however, since the API only allows local images to be set as background. This means that I need to download each image to the local application folder:

private async void btnSetStack_Click_1(object sender, EventArgs e)
{    
    var isProvider = Windows.Phone.System.UserProfile.LockScreenManager.IsProvidedByCurrentApplication;
    
    if (!isProvider)    
    {    
        var op = await Windows.Phone.System.UserProfile.LockScreenManager.RequestAccessAsync();    
        isProvider = op == Windows.Phone.System.UserProfile.LockScreenRequestResult.Granted;
    }
    
    if (isProvider)
    {    
        downloadableItems = new List<string>();    
        fileItems = new List<string>();

        foreach (var image in CentralBindingPoint.Instance.CurrentCategory.Images)        
        {    
            downloadableItems.Add(image.URL);    
            fileItems.Add(Path.GetFileName(image.URL));    
        }
        
        SerializationHelper.SerializeToFile(fileItems, "imagestack.xml");
        LocalStorageHelper.ClearFolder("CurrentSet");
        DownloadImages();
        grdDownloading.Visibility = System.Windows.Visibility.Visible;
    }
}

First of all, I need to make sure that the application can set a lockscreen background and is registered in the OS as a provider. The application needs to state its intent to be able to access the wallpaper by adding this snippet to the WMAppManifest.xml, right after the Tokens node:

<Extensions>
      <Extension ExtensionName="LockScreen_Background" ConsumerID="{111DFF24-AA15-4A96-8006-2BFF8122084F}" TaskID="_default" />
</Extensions>

downloadableItems is a collection that represents the download queue. fileItems contains the local file names for each image that is about to be downloaded and will be serialized and used in the background agent to iterate through the category files. Whenever the download process is started, an overlay becomes visible to notify the user that the image acquisition process is in progress.

Also, notice the fact that I am calling LocalStorageHelper.ClearFolder, passing the name of the folder as the first argument. I do not want to keep images for sets that are not active, therefore when a new set is selected, the currently stored images are deleted from the CurrentSet folder and replaced by the ones that are about to be downloaded. The implementation of the ClearFolder function looks like this:

public static void ClearFolder(string folderName
{    
    if (store.DirectoryExists(folderName))    
    {    
        foreach (string file in store.GetFileNames(folderName + "\\*.*"))    
        {    
            store.DeleteFile(folderName + "\\" + file);    
        }
    }
}

Once the file names are stored in imagestack.xml, the image contents are downloaded via DownloadImages:

void DownloadImages()
{    
    WebClient client = new WebClient();    
    string fileName = Path.GetFileName(downloadableItems.First());
    client.OpenReadAsync(new Uri(downloadableItems.First()));
    client.OpenReadCompleted += (sender, args) =>
    {    
        Debug.WriteLine("Downloaded " + fileName);    
        LocalStorageHelper.WriteData("CurrentSet", fileName, StreamToByteArray(args.Result));
        downloadableItems.Remove(downloadableItems.First());
        if (downloadableItems.Count != 0)
            DownloadImages();
        else
        {    
            grdDownloading.Visibility = System.Windows.Visibility.Collapsed;    
            LocalStorageHelper.CycleThroughImages();
            
            //ScheduledActionService.LaunchForTest("LockscreenChanger", TimeSpan.FromSeconds(5));
        }
    };
}

Here you can see that I am making a call to LocalStorageHelper.CycleThroughImages—a function that reads the file that contains the current set and picks the first image, assigning it to be the current wallpaper and then pushing it to the back of the list, making the succeeding image the next in line for the wallpaper:

public static void CycleThroughImages()
{    
    List<string> images = Coding4Fun.Phone.Storage.Serialize.Open<List<string>>("imagestack.xml");    
    if (images != null)
    {    
        string tempImage = images.First();    
        Uri currentImageUri = new Uri("ms-appdata:///Local/CurrentSet/" + tempImage, UriKind.Absolute);
        Windows.Phone.System.UserProfile.LockScreen.SetImageUri(currentImageUri);
        images.Remove(tempImage);
        images.Add(tempImage);
        Coding4Fun.Phone.Storage.Serialize.Save<List<string>>("imagestack.xml", images);
    }
}

You might be wondering why I’m not using Queue<T> for this. After all, Enqueue and Dequeue would make things a bit easier. The problem is that a Queue instance cannot be directly serialized without being transformed to a flat list. Therefore, I am sticking to minimal resource processing by manipulating a List<T> instance instead.

The recursive image download method runs until the download queue is emptied, after which the overlay is hidden.

Background Agent

At this point, we have the images locally stored and listed in an XML file. If the user accepted the system prompt, the application has also been registered as a lockscreen background provider, but there is not yet a single piece of code that would actually set the wallpaper cycle. For that, create a new Background Agent project in your solution. I named mine Coding4Fun.Lockscreen.Agent.

The OnInvoke function in ScheduledAgent.cs is executed at 30-minute intervals. This is a time limit defined by the PeriodicTask background agent type that we’ll be using here. You need to add the following snippet to it:

protected override void OnInvoke(ScheduledTask task)
{    
    var isProvider = Windows.Phone.System.UserProfile.LockScreenManager.IsProvidedByCurrentApplication;    
    if (isProvider)
    {    
        LocalStorageHelper.CycleThroughImages();    
    }
    NotifyComplete();
}

As with the download snippet, I am ensuring that before I attempt to change the wallpaper the application is a registered provider. Otherwise, an exception will be thrown and the background agent will crash. The bad thing about periodic tasks crashing is the fact that once two consecutive crashes occur, the task is removed from the task queue and the backgrounds will not be changed.

If the application is a provider, call CycleThroughImages to set the new background and push the old one to the end of the list. To make sure that a different image is selected each time, the original deserialized list is modified, where the first image now becomes last, switching the stack up, after which it is serialized back into imagestack.xml.

The background agent needs to be registered in the WMAppManifest.xml. Inside the Tasks node, add an ExtendedTask:

<ExtendedTask Name="LockscreenChangerTask">
    <BackgroundServiceAgent Specifier="ScheduledTaskAgent"
        Name="LockscreenChanger"
        Source="Coding4Fun.Lockscreen.Agent"
        Type="Coding4Fun.Lockscreen.Agent.ScheduledAgent" />
</ExtendedTask>

Also, when the application starts, you need to ensure that the task is registered, and register it if it isn’t yet. Use the Application_Launching event handler for this task:

private void Application_Launching(object sender, LaunchingEventArgs e)
{    
    string taskName = "LockscreenChanger";    
    var oldTask = ScheduledActionService.Find(taskName) as PeriodicTask;

    if (oldTask != null)
    {    
        ScheduledActionService.Remove(taskName);    
    }
    
    PeriodicTask task = new PeriodicTask(taskName);
    task.Description = "Change lockscreen wallpaper.";
    
    ScheduledActionService.Add(task);    
    LoadCustomCategories();    
}

Here, LoadCustomCategories will deserialize the existing custom categories, so that those can be shown in the main page after the application starts:

private async void LoadCustomCategories()
{    
    try    
    {    
        CentralBindingPoint.Instance.CustomCategories =    
        (ObservableCollection<Category>)await SerializationHelper.DeserializeFromFile(
        typeof(ObservableCollection<Category>), "customcat.xml");
    }
    catch
    {    
        Debug.WriteLine("No customcat.xml - no registered custom categories.");    
    }
}

Now the backgrounds will automatically change based on the web sets that you will activate every 30 minutes.

Working with Custom Categories

Let’s create some custom sets. To manage user input, I leverage the CustomMessageBox control available in the Windows Phone Toolkit. It has enough flexibility to let me choose between adding a TextBox control, to have the user create the new category or use a ListPicker to show the available custom categories in a consistent UI layout.

When the user decides to create a new category, he taps the plus button in the application bar on the main page:

The implementation for the call is simple:

private void btnSetStack_Click_1(object sender, EventArgs e)
{    
    TextBox textBox = new TextBox();    
    CustomMessageBox box = new CustomMessageBox()
    {    
        Caption = "Add Custom Category",    
        Message = "Enter a unique name for the new category.",
        LeftButtonContent = "ok",
        RightButtonContent = "cancel",
        Content = textBox
    };
    
    box.Dismissed += (s, boxEventArgs) =>
    {    
        if (boxEventArgs.Result == CustomMessageBoxResult.LeftButton)    
        {    
                    if (!string.IsNullOrWhiteSpace(textBox.Text))    
                    {    
                        var categoryCheck = (from c in CentralBindingPoint.Instance.CustomCategories    
                        where
                        c.Name == textBox.Text
                        select c).FirstOrDefault();
                        
                        if (categoryCheck == null)
                        {    
                            Category category = new Category() { Name = textBox.Text };
                            CentralBindingPoint.Instance.CustomCategories.Add(category);
                            Coding4Fun.Toolkit.Storage.Serialize.Save<ObservableCollection<Category>>(
                            "customcat.xml", CentralBindingPoint.Instance.CustomCategories);
                        }
                        else
                        {    
                            MessageBox.Show("Add Custom Category",    
                            "This category name was already taken!",
                            MessageBoxButton.OK);
                        }
                    }
        }        
    };
    
    box.Show();    
}

When the message box is dismissed, I check which button is pressed to take the appropriate course of action. Let’s assume that the user decided to add the new category—we need to check and make sure that there isn’t already a category with the same name in the existing collection. If there isn’t one, a new Category instance is created, added to the collection in the main view model, and serialized to customcat.xml.

The user also needs to be able to add images from any category to another custom category. To do this, I decided to give the user the option to carry across the image name and URL when he taps on an image in the ImageSetPage.xaml.

Remember, if there are no current custom categories registered, the user should be informed that he should create some first, so the alternative route for the dialog with custom category name selection should be a message box alert:

Here is the snippet that does this:

private void lstImages_SelectionChanged_1(object sender, SelectionChangedEventArgs e)
{    
    if (CentralBindingPoint.Instance.CustomCategories.Count > 0)    
    {    
        if (lstImages.SelectedItem != null)    
        {    
            ListPicker picker = new ListPicker()    
            {    
                Header = "Custom category name:",    
                ItemsSource = CentralBindingPoint.Instance.CustomCategories,
                Margin = new Thickness(12, 42, 24, 18)
            };
            
            CustomMessageBox messageBox = new CustomMessageBox()
            {    
                Caption = "Add To Custom Category",    
                Message = "Select a registered custom category to add this image to.",
                Content = picker,
                LeftButtonContent = "ok",
                RightButtonContent = "cancel"
            };
            
            messageBox.Dismissing += (s, boxEventArgs) =>
            {    
                if (picker.ListPickerMode == ListPickerMode.Expanded)    
                {    
                    boxEventArgs.Cancel = true;    
                }
            };
            
            messageBox.Dismissed += (s2, e2) =>
            {    
                switch (e2.Result)    
                {    
                    case CustomMessageBoxResult.LeftButton:    
                    {    
                        if (picker.SelectedItem != null)    
                        {    
                            Category category = (from c in CentralBindingPoint.Instance.CustomCategories    
                            where c.Name == picker.SelectedItem.ToString()
                            select c).FirstOrDefault();
                            
                            if (category != null)
                            {    
                                category.Images.Add((Coding4Fun.Lockscreen.Core.Models.Image)lstImages.SelectedItem);    
                                Coding4Fun.Toolkit.Storage.Serialize.Save<ObservableCollection<Category>>(
                                "customcat.xml", CentralBindingPoint.Instance.CustomCategories);
                            }
                            
                            lstImages.SelectedItem = null;
                            lstImages.IsEnabled = true;
                        }
                        break;
                    }
            
                    case CustomMessageBoxResult.RightButton:        
                    case CustomMessageBoxResult.None:
                    {    
                        lstImages.SelectedItem = null;                    
                        break;
                    }
                }
            };
            
            messageBox.Show();
        }
    }
    else
    {    
        MessageBox.Show("Add To Custom Category",    
        "Tapping on an image will prompt you to add it to a custom category" + Environment.NewLine +
        "Seems like you don't have any custom categories yet.", MessageBoxButton.OK);    
    }
}

Once the category is selected from the list, the image is added to the Images collection in the Category instance, and the category list is serialized to preserve the changes. There are no restrictions as to which categories can fetch images to other categories—we can even select images from custom categories and include them in other categories. The image can be added multiple times to the same category as well.

Conclusion

With Azure Mobile Services and a managed SDK available for Windows Phone, as well as an open REST API, it is fairly easy to build connected applications on multiple platforms at once without major logic and code base modifications.

I’ve always wanted to take panoramic pictures with my camera from a tripod, so I decided to make a remote controlled panoramic tripod head that would screw on to any tripod. Using a motor and a shutter control, a series of pictures can be taken. With the use of Photosynth, those pictures can be stitched up to make an interactive 360-degree view. You can also use your Windows Phone 8 as a remote shutter release over a Bluetooth connection. Windows Phone 8 has a Bluetooth API that allows us to program connectivity using TCP/IP style sockets.

This is a popular project in the Arduino world and I wanted to complete it using Gadgeteer hardware and the .NET Micro Framework (NETMF). This project uses the FEZ Cereberus mainboard module, but you should be able to use other Gadgeteer mainboards without any problems.

What is the .NET Gadgeteer platform?

The .NET Gadgeteer platform is an open source toolkit for building small electronic projects using Gadgeteer hardware modules and programmed using the .NET Micro Framework with Visual Studio 2010 or Visual C# Express (2010 edition). The Gadgeteer modules connect together using special cables and very little soldering is required. You typically use a mainboard module, which has the CPU and memory, and then connect various modules (camera, sensors, networking, display, controllers, etc) to accomplish your chosen task.

This part is going to require Visual Studio 2010 C#. Whether the full version or the Express version, it has to be 2010. This is because the Gadgeteer libraries have as of this writing not yet been ported to Visual Studio 2012.

The programming module is easy to work with. Designers are installed into Visual Studio so that you get a visual display of the modules and how they are connected to each other.

You also get full Intellisense for the modules as you code and the best part is that you get to interactively debug through the Visual Studio IDE. In the reference assemblies, you can see a set of assemblies that start with “Microsoft.Spot”. The NETMF came out of the SPOT initiative. SPOT was the underlying technology used in a line of “smartwatches” and selected GPS receivers. The “Spot” assemblies make up the core of the NETMF and the Gadgeteer assemblies provide the “glue” between NETMF and Visual Studio. Also, the GHIElectronics assemblies provide the support for the GHI Gadgeteer modules.

Determining your camera’s remote shutter circuits

You will need a camera that has the ability to use a wired shutter release cable. Most DSLRs and a few point ‘n shoot cameras have a connector for a remote. The camera that I used for this project, a Panasonic Lumix FZ-30, falls in that category. You’ll also need to determine how your camera’s remote is wired. There is a great resource at http://www.doc-diy.net/photo/remote_pinout/ that lists the circuits and connectors for many popular camera makers.

Most of them use two wires, one to control the focus, the other to trip the shutter. My Lumix uses a single wire and controls the focus and shutter by changing the resistance over the wire. The hardest part here is getting the connector. I bought a generic Lumix wired remote online for a few dollars and then disassembled it to use the wiring and plug for this project.

Hardware list

Getting started with the FEZ Cerebus

GHI has a detailed tutorial for getting up and running on the FEZ Cerebus Mainboard. If you are new to the Gadgeteer world, start with the FEZ Cerebus Basic Kit Guide on the Cerebus download page. After installing the compiler and SDK bits, you’ll want to make sure that the board has the correct firmware on it. Detailed instructions on loading the firmware are on the TinyCLR wiki. I recommend using the tutorial to make sure that everything works before moving on.

Once we have the compiler and SDK bits installed along with the tutorial compiles and runs, it’s time to start building this thing.

Building the camera control board

The first part that I assembled was the mini-circuit board for the camera shutter control. I used a single wire to control the focus and the shutter—changing the resistance triggers the camera. So we need to short the circuit between the resistors to get the camera to focus and take a picture. We also want to isolate the circuit from the power coming from the Gadgeteer board. The camera supplies its own power to the circuit and the last thing we want to do is to fry our camera. Accordingly, we’ll use a pair of 2N222 NPN bipolar junction transistors—they are cheap; I bought a bag of 15 from RadioShack for about $3.50—and we’ll use general-purpose input/outputs (GPIO) pins from the Gadgeteer board to toggle the transistors.

This is the circuit that I used for my camera:

The jack is part of the cable that I took from the generic wired remote. Having a wired remote that you can take apart will make this project much easier. You can buy just the jack, but there really isn’t much of a difference in price and it’s easy to match up the wires and resistors after you disassemble the remote.

When the transistors are turned off, the camera shutter circuit flows through all three resistors and the combined resistance is below the threshold that triggers the camera. When we turn on the focus transistor, the combined resistance drops from 41.1kΩ to 5.1 kΩ. That gets the camera’s attention and puts it in focus mode. Then, turning on the shutter transistor, the resistance drops to 2.2kΩ and the camera takes the picture. You can test this circuit out with a breadboard and use jumpers instead of the transistors. If you can’t find exact matches for the resistors, try to get as close as possible. I was able to use 2k instead of 2.2k and the circuit still worked.

For Canon and Nikon DSLR cameras, the circuitry will be much simpler. Instead of using three resistors on one circuit, you will have separate circuits for focus and shutter, though the two will share a common ground. A Canon circuit might look like this:

I soldered the components to a small circuit board and used short jumper wires to connect the 3 lines from the header block to the Gadgeteer extension block. If I ever decide to use this with a different brand of camera, I can make a new board to match that circuit and connect it to Gadgeteer.

The red and black wires coming off to the left of the circuit board are connected to the positive and negative wires from the remote cable. The two red wires on the right side of the circuit board are connected to the GPIO pins (pins 3 & 4) on the extender module, and the black wire goes to ground (pin 10) on the extender. Also shown are the lines for the servo motor. Red going to the 5v line on pin 2 of the extender, green to the PWM line (pin 7), and ground is shared with the circuit board on the ground pin.

Building the extension block

Use a GHI extender module to provide access to power, pulse, GPIO, and ground pins from the mainboard. As shipped by GHI, it looks like this:

For ease of assembly and testing, I soldered a 10 pin header block to the board. In the Gadgeteer library, the pins are numbered from 1 to 10. For the camera control, we’ll be using pins 3, 4, and 10. On the mainboard, I connected this module to socket 5. Each socket on a mainboard has a unique number to identify it, and one or more letters to list the socket capabilities. Each socket can have different electrical and communications capabilities for its pins. Some types support Ethernet signals, others provide the control lines for an SD card controller.

When you pick out a Gagdeteer module, it will list the socket type requirements. For example, the Bluetooth module requires a socket with type “U” capabilities. Type U provides UART serial port support with pins 4 and 5 used for sending and receiving data. On the FEZ Cerebus mainboard, sockets 2 and 6 provide type U support.

Socket 5 on a FEZ Cerebus has the following types: P, C, and S. The socket type definitions can be found on Gadgeteer CodePlex wiki. The P type includes pulse-width modulated (PWM) outputs on pins 7, 8, and 9, and we need a PWM output to control the server motor. The S type has pins 3, 4, 5 set as general-purpose input/outputs (GPIO) and we need 2 of them to control the camera. The camera control and the servo share pin 10, which is the ground line. Pin 2 provides 5v of power and will be used to power the servo motor.

Connect each module to the mainboard to the following sockets:

Usually when you work with a Gadgeteer module, you drag the mainboard and modules from the toolbox onto the designer. Then you use the mouse to connect the mainboard to the module by dragging from one socket to another. The designer does some code-behind magic and brings in the appropriate reference libraries for each module and then you’re able to start working with the modules. We will do this with the Joystick, Extender, and LED modules. For the Bluetooth module, we will use an open source driver written by Eduardo Velloso. I downloaded the file Bluetooth.cs from Eduardo’s Codeplex site and added it to the project.

Connect the focus line from the camera control to pin 3 of the extender module. The shutter line goes to pin 4 of the extender and ground to pin 10. I made up a set of wires with jumper pins on each end to make it easy to connect the modules and soldered the crimp pins to sections of wire to make my jumper cables. Also, I used liquid electrical tape over the solder connections, which made the connections more secure and reduced the chances that wires in adjacent sockets would touch each other. Completed, I would wire up the camera board and servo motor to the extender module with the following arrangement:

Servo motor

For the servo motor, we are using a Parallax (Futaba) Continuous Rotation Servo. This is a servo motor that has been modified to allow a full and continuous 360 degree rotation. We need to connect three lines to the servo: power, ground, and pulse-width modulation (PWM). The PWM signal will determine the direction and rate the Servo turns. When the servo receives a 1.5 ms high pulse, the server will be at center position. Speed and direction are controlled by the length of the pulse, which can range from 1.3 ms to 1.7 ms. For correct operation, the servo needs a 20 ms pause between pulses.

The Gadgeteer SDK provides the PWMOutput class that allows us to define the PWM parameters. The SetPulse() method takes two parameters, period and hightime, which correspond to the pause and high pulse. These values are specified in nanoseconds (ns).

To connect the Servo wiring, connect the power line to pin 2 of the extender module and the ground line to pin 10. I ended up splicing a ground cable as a “Y” cable so that the ground from the extender module could be shared between the camera board and the servo. The white wire coming from the servo is the control line. That gets connected to extender pin 7:

The white wire from the servo goes to the green wire jumper wire and connects to pin 7, which is mapped as PWM pin. The red wire goes to pin 2, which provides 5v. The black wire goes to pin 10, which is ground. The camera circuit is not connected here, which makes it easier to see how the servo was wired up.

Gadgeteer App

With the Gadgeteer hardware assembled, it’s time to get the code working.

You’ll want to get the source code from the CodePlex site–everything is built and we can review the highlights here.

In ShutterControl.cs, we have the code that will tell the camera to take a picture. We have a class called ShutterControl, which descends from Gadgeteer.Modules.Module. The constructor looks like this:

public ShutterControl(int socketNumber)
{
    // Get the Gadgeteet Socket for the specified socket number
    _socket = Socket.GetSocket(socketNumber, true, this, null);

    // Define two GPIO pins and set their initial state to false
    _focus = new DigitalOutput(_socket, Socket.Pin.Three, false, this);
    _shutter = new DigitalOutput(_socket, Socket.Pin.Four, false, this);
}

This is where we wire up the pins to the code. We can turn on or off a GPIO pin by calling the Write method of the pin and passing true or false to set the state, and we can implement a method for taking a picture with the following code:

public void TakePicture(int shutterTime)
{
    // Tell the camera to focus
    SetFocus(true);

    // wait 1/10 of a second
    Thread.Sleep(100);

    // Tell the camera to open the shutter
    SetShutter(true);

    // Hold the shutter open for the specified time
    if (shutterTime > 0)
    {
        Thread.Sleep(shutterTime);
    }

    // Let go of the shutter
    SetShutter(false);

    // let go of the focus
    SetFocus(false);
}

The main execution block is in the Program partial class, Program.cs. Create an instance of ShutterControl with the following code:

ShutterControl shutterControl = new ShutterControl(5);

That gives a ShutterControl object bound to socket 5. Then, taking a picture is as simple as calling:

shutterControl.TakePicture(1000);

Bluetooth (on the Gadgeteer side)

So why Bluetooth when all the cool kids are using Wi-Fi? There are a couple of reasons:

• It’s cheaper. The Bluetooth Gadgeteer module is half the cost of the cheapest Wi-Fi module.
• It uses less power. There’s no point in draining your phone’s battery any faster than you have to.
• It’s much, much easier to configure. At the time of this writing, you cannot make the Gadgeteer the host in an Adhoc Wi-Fi network. That means it has to connect to an existing network. This opens up a box filled with ugly things. For example, there is neither a display nor a keyboard for this device, and so the user has no way of selecting a Wi-Fi network to join or entering a WEP or WPA key.

The Gadgeteer Bluetooth module supports the Serial Port Profile (SPP). Basically, it’s treated like a modem. When the app starts up, we call an InitBluetooth() method:

void InitBluetooth()
{
    // Keep a quick reference to the Bluetooth client, to make it
    // easier to send commands back over the module
    client = bluetooth.ClientMode;

    // Set the Bluetooth device name to make it easier to identify when
    // you pair this module with your phone
    bluetooth.SetDeviceName("Coding4Fun Pano Head");

    // We want to track when we are connected and disconnected
    bluetooth.BluetoothStateChanged += new Bluetooth.BluetoothStateChangedHandler(bluetooth_BluetoothStateChanged);

    // We want to collect all the data coming in
    bluetooth.DataReceived += new Bluetooth.DataReceivedHandler(bluetooth_DataReceived);
}

Command Parser

A simple command parser is implemented in CommandParser.cs. All of the data that comes in from the Bluetooth module gets appended to a custom Queue() class defined in CommandQueue.cs. A background thread calls the parser, which looks for command strings. Every command string starts with the “@” character and ends with a “#” character. The CommandParser.parse() goes through the buffer and queues up commands in the order they came in. The commands recognized by the app are defined as string constants in commands.cs.

The CheckForCommands() method is called after parsing the data and takes action for each command that comes in:

private void CheckForCommands()
{
    if (_commandParser.Commands.Count > 0)
    {
        ControlCommand controlCommand = _commandParser.Commands.Dequeue();
        switch (controlCommand.Action)
        {
            case Commands.CMD_LEDS:
                SpinLights();
                _client.Send("OK");
                break;

            case Commands.CMD_VERS:
                SendVersion();
                break;

            case Commands.CMD_STOP:
                _haltRequested = true;
                _servoControl.Stop();
                break;

            case Commands.CMD_RIGHT:
                _servoControl.ClockWise(_servoControl.Speed);
                break;

            case Commands.CMD_LEFT:
                _servoControl.ClockWise(-_servoControl.Speed);
                break;

            case Commands.CMD_SHUT:
                _shutterControl.TakePicture();
                break;

            case Commands.CMD_PANO:
                var p = new PanoSettings();
                p.Load(controlCommand.Params);
                PanoramaOperation(p);
                break;
        }
    }
}

To take a picture, the sender sends the start of command character, “@”, following by the CMD_SHUT string, and finally the end of command character, “#”. As a single string, it‘s “@SHUT#”.

Joystick processing

The joystick that comes with the kit has a button click event and you can poll it for the current position. I used the command sequence of a double click, followed by a direction. Double-click and up will put the Bluetooth module into pairing mode so that your phone can be paired to the device. A second double-click and up will reset the Bluetooth module to cancel pairing mode. Double-click and either left or right will rotate in that direction until the joystick is moved to another position. Double-click and down will put the servo into calibrate mode until the direction changes. The sequence is to double-click the joystick, then pick a direction. I coded the following functions:

Joystick Up: Puts the Bluetooth module into pairing mode. You need to do this once with the phone, to pair the phone with the device. Double-click, then Up will cancel pairing mode.

Joystick Left: Tells the servo to rotate left at the default speed. Continues until joystick changes direction.

Joystick Right: Tells the servo to rotate right at the default speed. Continues until joystick changes direction.

Joystick Down: Sends the 0 rotation signal to servo to allow you to set the set point of the servo. This is how you will calibrate the servo. Continues until joystick changes direction.

private void CheckJoyStick()
{
    if (_joyStickCommand.CheckState() == JoyStickCommandState.DoubleClicked)
    {
        var direction = _joyStickCommand.GetDirection(joystick.GetJoystickPosition());
        switch (direction)
        {
            case JoyStickDirection.Right:
                _servoControl.ClockWise(_servoControl.Speed);
                // turn to the right until the joystick is moved to a different direction
                while (direction == JoyStickDirection.Right)
                {
                    Thread.Sleep(300);
                    direction = _joyStickCommand.GetDirection(joystick.GetJoystickPosition());
                }
                _servoControl.Stop();
                break;

            case JoyStickDirection.Left:
                _servoControl.ClockWise(-_servoControl.Speed);
                // turn to the right until the joystick is moved to a different direction
                while (direction == JoyStickDirection.Left)
                {
                    Thread.Sleep(300);
                    direction = _joyStickCommand.GetDirection(joystick.GetJoystickPosition());
                }
                _servoControl.Stop();
                break;

            case JoyStickDirection.Up:
                if (!_inPairingMode)
                {
                    _client.EnterPairingMode();
                }
                else
                {
                    _bluetooth.Reset();
                }
                _inPairingMode = !_inPairingMode;
                break;

            case JoyStickDirection.Down:
                _servoControl.Calibrate();
                while (direction == JoyStickDirection.Down)
                {
                    Thread.Sleep(300);
                    direction = _joyStickCommand.GetDirection(joystick.GetJoystickPosition());
                }
                _servoControl.Stop();
                break;
        }

        if (direction != JoyStickDirection.None)
        {
            _joyStickCommand.ClearState();
        }
    }
}

Calibrating the servo

You can use the screwdriver included in the Parallax kit to turn the adjustment potentiometer on the servo. Put the device into calibrate mode via the joystick. Then, slowly turn the potentiometer in one direction. If the servo slows down, keep turning in that direction until the servo stops. If the servo starts going faster, then turn the potentiometer the opposite direction until the servo stops turning. If you turn the potentiometer past the zero point, the servo will start to spin in the opposite direction. If this happens, just turn the potentiometer very slowly in the other direction; it may take several tries to get it just right.

While writing this code, I managed to brick the board. I wrote some code to calibrate the servo and it used a while forever loop that was called when the device restarted. This eventually threw an error and prevented me from updating the app. I was forced to wipe the firmware and start from scratch. For the FEZ Cerberus board, the instructions are here: http://wiki.tinyclr.com/index.php?title=Firmware_Update_FEZ_Cerberus

Building the pano head

Cut two pieces of wood. The bottom piece is sized to match the Lazy Susan swivel, which is 6” square. Measure and mark the center of this board on both the top and bottom sides. Attach the bottom of the Lazy Susan to the top side of the bottom piece. Then, mount the larger reduction gear to the center of the top side of the bottom board:

Mount a tripod socket to the center of the bottom side of the board:

Mount the smaller reduction gear to the servo motor. This will replace the horn that comes with the servo:

Cut the top piece. This will be a little larger—I made a 10” by 7” rectangle. Drill the holes to mount it to the swivel and then mount the swivel just long enough to mark its center. Make sure that the two boards spin freely with the swivel unit. The first swivel that I used would bind up after 270 degrees of rotation. A replacement unit spun freely.

Next, cut a rectangular opening of about 1” by 3” in the top of the board, with one end over the center of swivel. This will allow the servo to be positioned in the hole so that the gears match up. Secure the top board to the swivel unit:

Position the servo so that its gear is flush against the reduction gear. Getting the correct fit is important in order for the pano head to reliably turn. I used metal mounting strips from the hardware store and was able to secure the servo with two sets on either end sandwiching the servo mounts:

Now we have to mount the Gadgeteer parts and provide a camera mount. I took the low road here and picked up a pair of PVC wall boxes for electrical outlets from the hardware store. I placed them on opposite sides of the servo, secured the various Gadgeteer modules to one box, and left the other one open to hold the battery back. For ease of assembly, the extender module was not yet connected:

Connect the jumper wires to the camera shutter board and extender module and connect the jumpers to the servo connectors. The camera board will connect the following lines to the extender module: Pin 1 (shutter) Pin3 (GPIO) Pin 2 (focus) Pin 4 (GPIO) Pin 4 (ground) Pin 10 (GRD) The servo will connect the following lines to the extender module: Positive (red) Pin 2 (5v) Control (white) Pin 7 (PWM) Ground (black) Pin 10 (GRND) The ground line of the extender module is shared between the camera control and the servo. Connect the extender module to socket 5 of the Cereberus mainboard.

For the camera mount, I took the low road again and cut a section from a yard stick to bridge the tops of the electrical boxes. In the center, I drilled ¼” hole and put a ½” thumbscrew to fit the tripod mount. At this point, you can use the joystick to test various functions of the hardware. Make sure that everything works before you tackle the Bluetooth remote control:

Windows Phone 8 App

Now that the Pano Head has been built, we need the remote control—a Windows Phone 8 device, I used my HTC 8X, but any WP8 device should work. As regards the compiler, we need to switch gears. While the Gadgeteer library required Visual Studio 2010, you need Visual Studio 2012 and Windows 8 for Windows Phone 8 development. Windows 8 Pro is required for the Windows Phone emulator, but that is moot in this case. Since you can’t emulate Bluetooth connectivity in the emulator, you’ll need to have an actual WP8 phone.

I used the Windows Phone Pivot App template, but ended up ditching the pivot control as the app really only has a single page. To use Bluetooth, I need to add the ID_CAP_PROXIMITY and ID_CAP_NETWORKING capabilities to the app manifest. The completed app has two pivot pages and they look like this:

About the icons in the apps

The app bar icons were generated with Syncfusion’s Metro Studio 2. Metro Studio 2 contains a collection of 1700 “Metro” style icon templates that be used to create customized icons for Windows Store Apps and Windows Phone Apps. Metro Studio 2 is a free gift from Syncfusion and the icons provided are royalty free without any usage restrictions.

The app icon that appears on the phone was derived from a camera image from the Open Clipart Library. The image that I used was a basic camera on a tripod image that works well with Windows Store and Windows Phone Store designs:

Using the app

The panoramic setup takes 5 controls:

The times are measured in milliseconds.

The app bar has two modes: connected and disconnected. When disconnected, the only button available is the connect button. All other controls are disabled until a connection is made. When there is a connection, 4 buttons and a menu option appear on the app bar:

The IntegerTextBox custom control

The panoramic setting controls use sliders and text boxes to change the settings. We are following a MVVM pattern and the sliders and text boxes are bound to view mode properties. When you change a slider, the associated text box is updated and vice versa.

For the text boxes, I made them accept only integer inputs. I also made them right-aligned, as that is what most people expect for numeric input controls. To make the process easier, I created a new IntegerTextBox class based on the standard TextBox.

I created a Windows Phone Class Library with just the IntegerTextBox class. Basically, I made three changes to the behavior of the TextBox. The popup keyboard is set to the Digits keyboard, TextAlignment is set to TextAlignment.Right, and OnKeyDown is overridden in order to only allow the digit keys and the backspace key. It all comes out like this:

public class IntegerTextBox: TextBox
{
    private readonly Key[] _num = new Key[] {Key.Back, Key.D0, Key.D1, Key.D2, Key.D3, Key.D4,
    Key.D5, Key.D6, Key.D7, Key.D8, Key.D9 };

    public IntegerTextBox()
    {
        InputScope = new InputScope();
        InputScope.Names.Add(new InputScopeName { NameValue = InputScopeNameValue.Digits });
        TextAlignment = System.Windows.TextAlignment.Right;
    }

    protected override void OnKeyDown(KeyEventArgs e)
    {
        if (Array.IndexOf(_num, e.Key) == -1)
        {
            e.Handled = true;
        }
        base.OnKeyDown(e);
    }
}

Once the custom control project compiles, add that project as a reference to the phone project. Then add the namespace to the MainPage.xaml so that you can use the new control:

<phone:PhoneApplicationPage xmlns:PhoneControls="clr-namespace:Coding4Fun.PhoneControls;assembly=Coding4Fun.PhoneControls"
x:Class="Coding4Fun.PanoHead.MainPage"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation">

Settings

Settings are persisted to IsolatedStorageSettings.ApplicationSettings. I used the sample code posted on MSDN, “How to create a settings page for Windows Phone”, to create a class with accessor functions in order for all of the properties to persist.

Hurray for Bluetooth

Microsoft greatly enhanced the API access to the Bluetooth hardware with Windows Phone 8, extending the Windows 8 StreamSocket and Peerfinder APIs to include Bluetooth. With this, you get TCP-style socket level programming over Bluetooth.

At first glance, it would appear that the lack of a SPP API would make it difficult to have a WP8 device talk to the Pano Head. It turns out that’s not the case—we can use a socket connection from the phone to send and receive data from the Pano Head.

The PeerFinder class was designed to allow Windows Store apps to discover other instances of the app on other devices. But it can also be used to locate other devices once they have been paired to your phone. The way to do this is to add “Bluetooth:PAIRED” to the PeerFinder.AlternateIdentities list. That tells PeerFinder to include the list of already paired devices when searching for matching apps. We can then find the Pano Head in the list of paired devices:

private async Task<HostName> QueryForHost()
{
    // Tell PeerFinder to only enumerate paired devices
    // See http://msdn.microsoft.com/en-us/library/windowsphone/develop/jj207007(v=vs.105).aspx#BKMK_Codeexamples
    PeerFinder.AlternateIdentities["Bluetooth:PAIRED"] = "";
    IReadOnlyList<PeerInformation> devices = null;

    try
    {
        // Get a list of matching devices
        devices = await PeerFinder.FindAllPeersAsync();
    }
    catch{}

    // If we can't find any devices, goto to the phone's Bluetooth settings
    if (devices == null || devices.Count == 0)
    {
        MessageBox.Show(AppResources.NoBlueToothDevices);
        await Windows.System.Launcher.LaunchUriAsync(new Uri("ms-settings-bluetooth:"));
        return null;
    }

    // Find the first device that identifies as the Pano Head, otherwise go to the Bluetooth settings
    PeerInformation peerInfo = devices.FirstOrDefault(c => c.DisplayName.Contains("Pano Head"));
    if (peerInfo == null)
    {
        MessageBox.Show(AppResources.NoPairedPanos);
        await Windows.System.Launcher.LaunchUriAsync(new Uri("ms-settings-bluetooth:"));
        return null;
    }

    // Store the device name so the next connection attempt will use that first
    saveString("lastdevice", peerInfo.HostName.RawName);

    // Return the hostname
    return peerInfo.HostName;
}

Once we find the Pano Head, we open a StreamSocket connection to the device. The Pano Head sees it as an SPP connection on its side, while on the phone-side it’s TCP -style socket coding. Once we have established the connection, it’s a straight forward task to send commands from the phone to the Pano Head.

The socket code is wrapped up inside a PanoControl class. We create an instance of a DataWriter on the socket’s OutputStream, and to send a command to the Pano Head we just have to write a string using the DataWriter’s WriteString method. For example, to tell the Pano Head to take a picture, implement a TakePicture() method like so:

public async Task TakePicture()
{
    await SendCommand(Commands.CmdShut);
}

public DataWriterStoreOperation SendCommand(string command)
{
    CheckForDevice();
    _dataWriter.WriteString(String.Format("@{0}#", command));
    return _dataWriter.StoreAsync();
}

Though we use the same string constant file from the Gadgeteer project, that’s pretty much the only code shared between the two projects.

Using the Pano Head

Taking the pictures is just a matter of finding the right location—one with the sun overhead and a stable location for the tripod. Since you are shooting a full 360 degrees, if the sun is lower in the sky, the camera will at some point be aimed at it and it will blow out the image.

Once you have the camera in the right spot, it’s time to try a few panoramic shots. You want to have overlapping shots, and I found that 16 pictures taken at 20-degree intervals worked for my camera. You’ll have to experiment a bit with the turn time setting to get that 20 degree turn. Remember to allow a couple of seconds for the settle and pause times, giving the Pano Head enough time to settle after turning as well as enough time for your camera to focus and take the picture.

There are various ways of stitching multiple images together to make up a panoramic picture. Microsoft has excellent tool with their Photosynth app and site. You can create a panoramic picture with their desktop app and upload to their site for free hosting.

You will need two desktop apps, the Image Composite Editor (ICE) and Photosynth. ICE is the application that will stitch the Images together into a panorama and Photosynth will publish the panorama to the Photosynth site. Register an account on the Photosynth site and download the two applications here.

Creating a panoramic picture using Photosynth

1. Take the pictures. You want some overlap, which will make it easier to stitch the images together.
2. Copy the pictures to your computer.
3. Launch Photosynth. If launched for the first time, the app will ask for Windows Live ID. Use the same account that you registered with on the Photosynth site. Once it comes up, you can ignore it.
4. Launch ICE.
5. Load your images into ICE. ICE will immediately start compositing the images. If you do not have sufficient overlap with the images, ICE will not be able to use all of the images.
6. Click the “Publish to Web…” button. ICE will generate the panorama and use Photosynth to upload the image.
7. Fill out the details when the “Upload Panorama” dialog appears and click the “Upload” button.
8. Photosynth will upload the panorama. When it’s complete, click the “view” on the Photosynth window.

My Sample

I tested the Pano Head at a local nature preserve. After some trial and error, I took a series of 16 pictures that yielded a full 360-degree view. I copied the pictures from my camera to my PC and loaded them into ICE. Here are the results:

Conclusion

This project was a lot of fun to build. Since I had never soldered anything before, it helped me develop a new skill set. I went to RadioShack and bought some components and a small circuit board to practice on. While it is possible to learn how to solder via YouTube and Twitter, it was tough going at first:

I had a lot of trouble getting the solder to stick to the board on my first attempt. It was a facepalm moment.

Needless to say, I did get a bit better at soldering.

This was also my first Windows Phone 8 project. It was very easy to write for and a breeze to debug. After doing some work on other mobile platforms, I felt much more productive with Windows Phone 8.

If you want to try this out, the download link for the source code is at the top of the article!

About The Author

Chris Miller is a senior R&D engineer for Tyler Technologies, working on the next generation of school bus routing software. He is also the leader of the Tech Valley .NET Users Group (TVUG). You can follow Chris at @anotherlab and check out his blog at anotherlab.rajapet.net. Chris started out with a VIC-20 and been slowly moving up the CPU food chain ever since.

The Parrot AR.Drone 2.0 is an awesome device packed with cool features. The drone contains two cameras, one pointed forward that streams live video, and one pointed downwards (for all your surveillance needs). Its got four powerful engines that make it fast and maneuverable. The drone’s firmware keeps it stable and level while stationary or performing maneuvers, handling a huge burden for the user. It’s a fun device to write code for, and even more fun to pilot when you’re done!

There’s a few projects that have created AR.Drone APIs that can be used for Windows 8 Desktop Apps. However, I wanted to fly my drone from a Windows Store App, so I decided to build my own API compatible with WinRT. The result is covered in this series of posts. Thanks to Nisha Singh (http://blogs.msdn.com/b/nishasingh/) and everyone else who helped me out with this project!

In the first post, I’ll go over how the API handles communicating with and controlling the drone, and then demonstrate how to use the API to make a simple control App.

API Information and Terms

The API consists of 5 files:

• DroneConnection.cs: functionality to connect to, and disconnect from, the drone.
• DroneControl.cs: contains the control loop.
• DroneMovement.cs: contains methods that construct and return fully-formatted commands.
• InputProcessing.cs: determines the next command to issue to the drone.
• InputState.cs: used to pass information between the UI and back-end.

• The terms the API code uses to refer to axes of movement are as follows:

• DroneStrafeX: left-right.
• DroneStrafeY: forward-backward.
• DroneRotateX: yaw, or rotation.
• DroneAscendY: up-down.

•  

  Communicating with the Drone

  The AR.Drone 2.0 communicates with other devices through its own wireless network. Sending the drone instructions is as simple as connecting your computer to the drone’s network, opening a UDP connection to the drone (I.P. 192.168.1.1, Port 5556), and then sending commands over this connection. The code to connect to the drone is contained inside DroneConnection.cs:

  // Initialize the connection to the drone
  public static async Task ConnectToDrone()
  {
        
           ...
        
           // Set up the UDP connection
           string remotePort = "5556";
           HostName droneIP = new HostName("192.168.1.1");
        
           udpSocket = new DatagramSocket();
           await udpSocket.BindServiceNameAsync(remotePort);
           await udpSocket.ConnectAsync(droneIP, remotePort);
           udpWriter = new DataWriter(udpSocket.OutputStream);
        
           ...
        
  }
  

   

  Once a connection has been established, we can begin to send the drone messages:

  // Send a command to the drone
  public static async Task SendDroneCommand(string command)
  {
           udpWriter.WriteString(command);
           await udpWriter.StoreAsync();
  }
  

  This networking code is discussed more in-depth here: Try, Catch, Finally... - UDP and Windows 8 Apps

  The drone needs to continuously receive commands for it to remain responsive. If the drone doesn’t receive any commands for two seconds, it assumes the connection has been lost, and it will ignore any further commands. If this happens, you need to close and re-establish the connection before you can continue to pilot the drone. DroneConnection.cs contains methods that allow you to do this.

   

  Commanding the Drone

  You can command the drone by sending it strings called AT commands. There are seven different categories of AT commands, but only two are required to fly the drone, and are the only ones included in the API:

• AT*REF – Used for basic behavior such as takeoff/landing, and emergency stop/reset.
• AT*PCMD – Used to fly the drone once airborne. This controls pitch, yaw, roll, and altitude.

• AT commands have very specific formatting, and consist of four parts. An improperly formatted command will be ignored by the drone. The different sections of an AT*PCMD command are highlighted below:

   

  AT*PCMD=23,1,0,0,0,0\r\n

   

  Header: The type of the AT command

  Sequence Number: The sequence number is used by the drone to keep track of which commands it needs to execute. The drone will not execute any commands with a sequence number lower than the highest sequence number it has received so far. The first command sent should always have a sequence number of 1, and each command sent subsequently should increment this number by 1. The sequence number can be manually reset by sending a command with the number 1.

  Payload: This carries the command arguments. Its format and content vary from command to command. In the AT*PCMD command, the first value is a flag indicating whether the drone should do nothing (low) or move according to the other arguments (high). The next four values denote roll, pitch, vertical speed, and angular speed respectively. These values are floats ranging from [-1, 1], where (0, 1] represents movement speed in one direction, and [-1, 0) represents the other. A 0 causes the drone to remain stationary along that axis, a 1 or –1 is maximum movement speed in their respective directions, and each value between is a fraction of the maximum possible speed. For example, in the pitch argument, a .75 would cause the drone to move backward at 3/4 maximum speed, while a –.5 would make it move forward at 1/2 the possible speed. However, the drone doesn’t respond to values in the [-1, 1] range. The actual value that needs to be inserted into the AT command is the signed integer value represented by the bytes the float is actually stored in. This conversion is done for you in the FloatConversion method in DroneMovement.cs.

  Carriage Character: A newline character. Use the one specific to your environment.

  DroneMovement.cs has a set of methods to generate AT*REF and AT*PCMD commands, all of which take the current sequence number as an argument, and return a ready-to-send AT command. Some methods take additional arguments to control the movement speed. An example of one of these methods is:

  // Strafe drone forward or backward at velocity in range [-1,1]
  public static string GetDroneStrafeForwardBackward(uint sequenceNumber, double velocity)
  {
           // Convert the ratio into a value the drone understands
           int value = FloatConversion(velocity);
           return CreateATPCMDCommand(sequenceNumber, "0," + value + ",0,0"); 
  }
    
  // Return a full ATPCMD command
  private static string CreateATPCMDCommand(uint sequenceNumber, string command)
  {
           return "AT*PCMD=" + sequenceNumber + ",1," + command + Environment.NewLine;
  }
  

  Control Infrastructure

  The drone is controlled with an continuous loop that sends a command to the drone every 30 ms. Each cycle, the loop determines the next command, sends the command to the drone, and then increments the sequence number. The method is asynchronous and awaits the call to System.Threading.Tasks.Task.Delay() so that it doesn’t block the UI between cycles.

  public static async void ControlLoop()
  {
           while (true)
           {
            
                   ...
            
                   // Get and send the next command
                   string commandToSend = InputProcessing.NextCommand(sequenceNumber);
                   await DroneConnection.SendDroneCommand(commandToSend);
            
                   sequenceNumber++;
            
                   await System.Threading.Tasks.Task.Delay(30);
           }
  }
  

  You need to implement the InputProcessing.NextCommand() method to return an AT command based on criteria of your own choosing. The API includes the class InputState.cs that you can use to pass information between your UI and the control infrastructure. The example App contains an example of this.

  Example App

  I’ve included a simple example App to demonstrate how to use the API. The App allows you to connect to the drone and make it take off, land, and rotate. It also illustrates how to pass input information between the UI and the control infrastructure using InputState instances. In the App, FlyingPage contains the method GetState() that returns an instance of InputState. GetState() looks at the input the App is receiving, then creates and returns an instance of InputState containing that information.

  // returns the current state of the input controls
  public static InputState GetState()
  {
           // slider value is in the range [-1,1]
           return new InputState(0, 0, 0, sliderValue, isFlying);
  }
  

  InputProcessing.NextCommand() calls GetState, and then interprets the state to decide on the next command to send:

  // returns the next command that should be executed
  public static string NextCommand(uint sequenceNumber)
  {
           InputState state = MainPage.GetState();
        
           // Determine if the drone needs to take off or land
           if (state.startFlying && !isFlying)
           {
                   isFlying = !isFlying;
                   return DroneMovement.GetDroneTakeoff(sequenceNumber);
           }
           else if (!state.startFlying && isFlying)
           {
                   isFlying = !isFlying;
                   return DroneMovement.GetDroneLand(sequenceNumber);
           }
        
           
           // Check if the drone needs to rotate
           if (state.rotateX != 0)
           {
                   return DroneMovement.GetDroneRotate(sequenceNumber, state.rotateX);
           }
        
           // Otherwise have the drone do nothing.
           return DroneMovement.NullCommand(sequenceNumber);
  }
  

  The command decision is made in order of importance. Takeoff and landing commands are the most important, followed by movement commands. If there is no other command to send, NextCommand() returns a null command so that the drone will continue hovering and listening to the connection.

  Once NextCommand() has been implemented, all the App needs to do is connect to the drone and initiate the control loop by calling DroneConnection.ConnectToDrone(), followed by calling DroneControl.ControlLoop(). In the example App, this is done in the handler for the connect button. Once the control loop is running, the drone is ready to fly!

  Get Flying

  The API should give you everything you need to get started writing drone Apps for Windows Store. In part 2, I’ll discuss a more complete App I’ve written to pilot the drone. There’s still so much drone functionality to be exposed, so get coding and get flying!

Developing FallFury was a fun and educational activity. Besides the fact that I got to work with an amazing team of people who were willing to assist me with the testing and development, I learned some important points about the development process as a whole and have outlined them in this article.  Check out the video for this article at http://channel9.msdn.com/Series/FallFury/Part-12-Project-Conclusion.  For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles.

Go Outside Your Comfort Zone

This past summer, I decided to tackle a new challenge—game development. Never before had I built a game from the ground up. Moreover, as I could already find my way around C# code, I decided that I wanted to make C++ the language of choice, and so worked with DirectX instead of XNA. Additionally, the project needed to be constructed with Windows 8 as the target platform and I’d have to rely on the WinRT stack.

The concepts were new to me, but I figured that learning about a new platform and a new language would be more interesting than sticking to my comfort zone. Through FallFury I extended my knowledge about pointers, trigonometry, how DirectX handles a lot of the graphics legwork, and how to write HLSL shaders.

It was a challenging but very rewarding experience.

Communicate

The most important part of any project is communication. There are always people who know more than you in one field or another, so don't hesitate to reach out to other developers and ask questions. Your code is not perfect, so talk about best practices used in production and how you can improve on what you already have—communication should not be limited to your team, floor, or even building.

One bonus of my experience at Microsoft was discovering that a lot of developers, when not in a meeting or in the middle of an important assignment, are more than happy to help a fellow employee with a piece of advice or some code review.

Plan the Small Details

Before you even touch the keyboard to write your first line of code, outline the components of your application and have a general idea of how you’ll implement those components as well as how they’ll interact with each other. It is tempting to start coding and then add or modify features, but doing so will create more problems than it will solve.

Build Small Prototypes

FallFury is a complex project composed out of blocks such as:

• XML level/metadata reader and writer
• The Share and Settings charm components
• SpriteBatch sub-system
• Level renderer
• Screen sub-system
• Menu interaction sub-system

Each block is responsible for its own set of tasks and each can be individually tested with either mocked data or a loose connection to the existing codebase. The necessity for small prototypes comes from individual requirements tied to the platform and possible integration scenarios. For example, when I began development, I created a non-hybrid DirectX project, which meant that if I needed to introduce XAML integration, I had to rewrite small parts of my swap chain preparation stack as well as work with a lot of XAML content in the code-behind. The Settings charm integration heavily relied on XAML, and I would have had to write the popup and control-related code in C#.

Obviously, that was not an option, so I created a hybrid project instead and ported most of my code into the new solution. Lesson learned here: prototype—and prototype early—in order to make sure that what you plan to implement will be implemented in an efficient manner.

Test Often, Test with Other People

When working on an application, it is important to understand the needs and expectations of your target audience, and a big part of this is making sure that the application provides a way for the user to easily learn about an action without a helper booklet. The menu system is a good example of this in FallFury. As I was designing the buttons on my screens, I tried to implement the approach used in games such as Dance Central—the user simply slides the button-panel and it takes him to the screen that was linked to that specific button. However, as I was testing the application with the local development crowd, I noticed a pattern—people were attempting to simply tap the button instead of sliding it. To avoid this confusion, I added a pulsating arrow on the right side of the button in order to highlight the fact that the button should be displaced in order to be activated.

The XML level loading system is another example of how secondary testing improved FallFury. Initial element positioning relied on a given level length, as well as on a ration-based placement. For example, if I wanted to put a button in the middle of the screen, I could position it with a value, such as .5. The level loader would then translate this value relative to the screen size. Rick Barraza was creating some test levels when he notified me about a big flaw—as the level extended, some elements were mis-positioned by a large margin compared to the original intended location. This was because when a level length was reset, the ratio was applied on a larger/smaller value, and some elements would therefore overlap. Ultimately, I rewrote the level structure to rely on pixel values, and designed the level-length to calculate automatically based on the included elements and their size.

Test on Different Hardware

FallFury started with portable hardware in mind. Its original design targeted Windows 8 tablet devices, though it was later decided that the project should support desktop machines as well. That automatically added thousands of possible hardware combinations on which the game might potentially run. Here are just some of the test machines we got to use:

• Samsung Series 7 Slate
• ARM Developer Tablet (Microsoft Surface)
• ASUS EP121 Slate

Each of these machines came with its own specific conditions that had to be accommodated. While Samsung Series 7 Slate featured an accelerometer, at the time of development the ASUS EP121 tablet did not have Windows 8 drivers for its sensor. When I sideloaded the game, I noticed that I could no longer control the character the way I wanted to. Also, though FallFury has keyboard controls, I had to develop controls for machines without keyboards. This is why I came up with the concept of touch-based, on-screen controls.

Moving on to the ARM device, first and foremost I noticed that my shader configurations were not compatible with the platform capabilities. All my testing was done on a machine that supported DirectX Feature Level 10, which was not the case for a low-power ARM machine that supports DirectX Feature Level 9.1. I had to work with some MSDN samples to find a way to properly modify the required shaders so they would work on Windows RT. ARM devices are built with battery life in mind. Therefore, their capabilities are reduced compared to those of a full-sized Windows tablet. This had an impact on the general game performance—some parts of the game, such as particle rendering, were working fine on a desktop machine, but were quite laggy on the ARM tablet. Knowing the possible source of the problem, I had to optimize the particle storage and rendering stack to ensure they didn’t affect the user experience on Windows RT hardware.

There Is Never Enough Time - Learn To Cut

When I started planning FallFury, I worked up a huge list of features. As the deadline approached, however, I noticed that I somewhat underestimated the allocated project time—a lot of time was spent on testing and eliminating bugs in existing features. That was the point at which I had to cut my feature set.

When designing the feature list for your app, make sure that you have a clear vision of which features get the priority and which can be cut if necessary. For example, FallFury was originally planned to be a multiplayer game. Although it was tempting to start working on it as such from the very beginning, I knew that it wasn’t a core feature and that it wouldn’t necessarily affect the general play experience on release day. Ultimately, multiplayer capability was dropped from initial development along with a number of other not-immediately-integral features.

Expect the Unexpected

When allocating time for the project, anticipate situations that might derail the process and cause a delay in the planned delivery date. It is easy to estimate that creating an accelerometer-based motion system will take you a day or two, but you also need to allow for the potential possibility of your code not working as expected, hardware accidents, natural disasters, etc. Doing so will help you focus on the right features at the right time while diminishing the risk of missing the deadline.

Platform Samples Are the Best Documentation - Learn to Read Others’ Code

While building the core of FallFury, I leveraged multiple features that hadn’t yet become mainstream among Windows developers. Therefore, the documentation was scarce and in some instances the only help I could get was internal. Luckily, Microsoft bundled hundreds of samples that demonstrate what the new platform can do. A lot of the development approaches gained during my internship were learned from looking at the official, publicly released sample code. The lesson here: if it's not on MSDN or blogs, check the official developer samples. For example, I used the DirectX sprite drawing sample to see how to build a proper sprite rendering mechanism with behavior similar to the one available in XNA.

Atomic Commits

When I started, source control wasn’t new to me, but there were proper usage practices that I didn’t apply before the internship. One of them is atomic commits, which I learned from Clint Rutkas.

With every new addition to the application, chances are something might go wrong. It could be a faulty third-party assembly, a misplaced class, or a broken reference that brings down the entire solution. If the commits include longer time intervals, for example, you will ultimately have to roll back to a build without a lot of the new functionality. Using atomic commits, however, you’ll be able to avoid unnecessary, redundant work in the case that something goes wrong during development.

A Non-standard Approach Is Not Necessarily a Bad Approach

For loading high scores, I needed the UI layout to display the registered items. To do so, I had could either use data binding or write a simple routine that would generate XAML items on the fly in the code-behind. Data binding is a pretty common practice among XAML developers, so my initial thought was to use that, but I would have had to create a C++ binding harness in addition to the existing data retrieval code. The second approach was much cleaner because it didn’t require me to rewrite or add much code. The lesson here: pick the best approach for the job and don’t blindly follow programming trends.

Focus on the User Experience

At the end of the day, the user doesn’t care whether you wrote the application in native DirectX or XNA, whether you applied the MVVM pattern, or whether you have easily readable code. The user cares about having a great time while playing the game, regardless of its backbone. It’s your goal as a developer to deliver that user experience and ensure that the user feels comfortable with the product. This means that the development cycle goes beyond simply writing code. In FallFury, levels are dynamic, which means the source code doesn’t necessarily need to be modified in order to create a different playable experience. But that experience matters in any case, and level design is a huge part of FallFury’s final product. Users need to be immersed in the game world, so levels must be planned in such as way that the user returns to the game and doesn’t abandon it after a couple of lost rounds.

Conclusion

FallFury was my first major project that I wrote almost entirely in C++, targeting the newly released Windows 8 operating system. Although the development process was challenging, it was proven to be a great educational experience that taught me about the approaches and practices that are used internally at Microsoft.

As previously mentioned, FallFury runs on multiple types of hardware as long as that hardware supports Windows 8. This article describes the project’s general testing and debugging process, including setting the debug configurations and remote debugging.

Check out the video for this article at http://channel9.msdn.com/Series/FallFury/Part-11-Hardware-Testing.  For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles.

Remote Debug

When working on an application that targets different machines, it’s probably out of the question to install Visual Studio on each instance and move the solution from one source to another in order to run it and diagnose potential problems. Here is where Remote Tools for Visual Studio 2012 come into play. Microsoft offers you three separate builds—tools for x86 systems, x64 systems, and ARM systems—also known as Surface RT.

Once the tools are running on the machine you want to debug, you have two choices. You can either install the remote debugger as a service, allowing it to constantly run in the background, or you can use the debugger on a per-launch basis. From a developer perspective, the choice doesn’t affect how your application is executed on the remote machine. From a security perspective, however, you need to be sure that you properly configure it so that no unwanted apps are remotely deployed.

Now you can start the Remote Debugger Configuration Wizard:

This works on both the initial start-up and also any subsequent launch as a way to easily and quickly set the necessary remote debugger settings. Specifically, it is useful to configure the machine’s network settings, allowing cross-domain communications for debugging purposes.

Once the wizard is completed, launch the Remote Debugger Monitor. Notice that it lists your machine name and the port on which it’s running. This is necessary for configuring the project to send the package to a remote machine instead of the local one:

The configuration depends on the network settings on both the local machine and the subnet as a whole. For example, in some cases, and especially on domain-joined machines, remote debugging is better done with the authentication disabled. Windows Authentication is used by default.

Since FallFury is a C++ project, the configuration for a remote session is different than, say, that of a C# Windows Store application. To configure the session, right click on the project in Solution Explorer and open the Debugging section. Make sure that Remote Machine is selected as the type of debugger to launch:

Next, specify the machine name as well as whether the current session will require authentication:

If you cannot specify the machine name, use the direct IPv4 address of that computer, minus the port (unless you’ve explicitly set it to a port other than 4016, which is the assumed default). Once the source machine connects to the remote one, you will see Visual Studio performing the deployment:

Configurations

As a matter of convenience, it is always good to have different debug configurations that will define how your projects are built, especially if the project targets multiple platforms (such as ARM and x86). Visual Studio provides a Configuration Manager that can be accessed from the debug target dropdown:

FallFury includes two separate projects—the game core, and the C#-based XML reader. Both need to be explicitly associated with separate target platforms in order to be correctly debugged. For that, profiles were created for both local and remote sessions:

As with standard Debug/Release configurations, the ones shown above determine whether the project will carry debug symbols and support debugging commands. It is important to mention that as you switch configurations, you must be careful how you configure the graphic shaders. For each shader in the container folder, explicitly set the HLSL shader type and model. Otherwise the application deployment will fail:

Remote Profiling

Last but not least, when working with different hardware configurations it might be useful to perform application profiling or performance review. Fortunately, Visual Studio also provides this capability through the vsperf tool, which is already integrated in the IDE if you are using Visual Studio 2012 Professional or above.

To initiate a profiling session on a remote device, the Remote Debugger Monitor must be active. Make sure that the Remote Machine debug target is selected, and go to Analyze > Start Performance Analysis:

On the remote machine, allow the vsperf process to run with administrative privileges. Once the profiling session completes and the data is analyzed, you can review the same performance indicators as you would when having a standard application run the profiler on the local machine:

Conclusion

Testing hardware outside the boundaries of a desktop machine is often a necessity, especially if the application relies on specific sensors, such as NFC, touch, or accelerometer. The remote debugging process is fairly streamlined and intuitive, with developing a proper network configuration allowing communication between machines requiring the most significant amount of effort. If you have problems getting the debugger to work, consult this article on MSDN.

With the release of the Windows 8 operating system, applications are now able to easily integrate with each other and have a unified way to control their workflow through unique system-wide contracts called Charms. Out of the multitude of available options, FallFury leverages two charms—Share contract and Settings. This article describes how the integration is implemented.

Check out the video for this article at http://channel9.msdn.com/Series/FallFury/Part-10-Charms .  For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles.

The Share Contract

When a user achieves a specific score in the game, he might decide to share it with his social circle. In Charms, that’s the purpose of the Share contract, which integrates directly with the OS.

Windows Store applications have the capability to expose their sharing capabilities and register as a share target. For example, if there is a Twitter client out, it can register itself as an app through which content can be shared. FallFury, on the other hand, acts as a consumer that aggregates existing share targets and lets the user pick the one through which he wants to let the message out:

Let’s take a look at how this process is built in the code-behind. The core is located in DirectXPage.xaml.cpp—the class responsible for XAML content manipulation in FallFury. First and foremost, you need to get the current instance of the DataTransferManager class:

auto dataTransferManager = Windows::ApplicationModel::DataTransfer::DataTransferManager::GetForCurrentView();

Consider this a proxy that allows you to pass content between your app and other Windows Store apps that are executed in the context of the same sandbox. As the instance is obtained, hook it to the DataRequested event handler that will handle the scenario where the user invoked the sharing capability:

dataTransferManager->DataRequested += ref new 
    TypedEventHandler<Windows::ApplicationModel::DataTransfer::DataTransferManager^,
    Windows::ApplicationModel::DataTransfer::DataRequestedEventArgs^>(this, &DirectXPage::OnShareDataRequested);

In the OnShareDataRequested, specify the information that goes into the sharing popup:

void DirectXPage::OnShareDataRequested(Windows::ApplicationModel::DataTransfer::DataTransferManager^ manager,Windows::ApplicationModel::DataTransfer::DataRequestedEventArgs^ params)
{
    auto request = params->Request;
    request->Data->Properties->Title = "Share Score";
    request->Data->Properties->Description = "Tell your friends how much you scored in [DEN'S PROJECT]!";
    request->Data->SetText("I just scored " + StaticDataHelper::Score.ToString() + " in [DEN'S PROJECT]! Beat me! http://dennisdel.com");
}

From that point, the control is in the user’s hand. The application cannot force the share, so unless you implement a direct API hook to a social service, the Share charm will only expose the endpoints available for sharing and will let you set the shareable content. You also don’t have to worry about the way the content will be shared—that will be handled by the target application:

You can show the popup triggered by the Share charm from your application without having the user open the Charms Bar. To do this, call the ShowShareUI method:

void DirectXPage::ShareTopScore_Selected(MenuItem^ sender, Platform::String^ params)
{
    Windows::ApplicationModel::DataTransfer::DataTransferManager::ShowShareUI();
}

This is exactly what the Share button does in the screenshot above. You should make this behavior predictable.

The Settings Charm

As you just saw, integrating basic sharing capabilities in FallFury is not too complicated. Working with settings is also a fairly easy task, though it involves some XAML work. While with sharing capabilities the work focused mostly on OS-based endpoints and application-specific popups, settings allow for full control over how they’re displayed.

For all Windows Store applications, settings should be handled via the Settings charm and not through dedicated application screens. Consider which settings that directly affect the user experience might be changed in FallFury:

• Music and SFX – the user can enable or disable music and sound effects, as well as control their volume.
• Accelerometer – depending on personal preferences, the user might decide to disable the accelerometer (it is enabled by default). The accelerometer can also be inverted—if the device is tilted to the right, the character will move to the left and vice-versa. Last but not least, even with dynamic screen rotation enabled on the device, the user can disable that rotation on the application level and lock the screen to one orientation, such as portrait or landscape.
• Character Movement – the character can be easily controlled via touch (swipe) or mouse. This behavior is enabled by default, but if the user decides to only use the mouse to direct shells, he can easily disable this feature here.

As seen in the image above, the operating system provides the basic shell used to list the possible settings. Once one option is selected, however, further UI displays are delegated to the developer.

As with the share UI, the settings UI can be shown to the user from the application and not from the Charms Bar. Here is how to do this:

void DirectXPage::Settings_Selected(MenuItem^ sender, Platform::String^ params)
{
    SettingsPane::GetForCurrentView()->Show();
}

SettingsPane is the core class that handles the settings display. It does not control how settings are stored or activated. When the main page loads, you need to make sure that you hook the current SettingsPane to a CommandRequested event handler. It will be triggered when the Settings capability is invoked:

SettingsPane::GetForCurrentView()->CommandsRequested += ref new TypedEventHandler<SettingsPane^, SettingsPaneCommandsRequestedEventArgs^>(this, &DirectXPage::OnSettingsRequested);

OnSettingsRequested is the function where the core setting selections are defined and hooked to their own event handlers:

void DirectXPage::OnSettingsRequested(Windows::UI::ApplicationSettings::SettingsPane^ settingsPane, Windows::UI::ApplicationSettings::SettingsPaneCommandsRequestedEventArgs^ eventArgs)
{
    if (m_renderer->CurrentGameState == GameState::GS_PLAYING)
            StaticDataHelper::IsPaused = true;
    
    UICommandInvokedHandler^ handler = ref new UICommandInvokedHandler(this, &DirectXPage::OnSettingsSelected);
    
        SettingsCommand^ generalCommand = ref new SettingsCommand("musicSfx", "Music & SFX", handler);
        eventArgs->Request->ApplicationCommands->Append(generalCommand);
    
    SettingsCommand^ accelerometerCommand = ref new SettingsCommand("accelerometer", "Accelerometer", handler);
    eventArgs->Request->ApplicationCommands->Append(accelerometerCommand);
    
    SettingsCommand^ charMovementCommand = ref new SettingsCommand("charMovement", "Character Movement", handler);
    eventArgs->Request->ApplicationCommands->Append(charMovementCommand);
}

Each SettingsCommand is an item in the list displayed in the settings pane. When one is selected, OnSettingsSelected is called:

void DirectXPage::OnSettingsSelected(Windows::UI::Popups::IUICommand^ command)
{
    if (command->Id->ToString() == "musicSfx")
    {
        stkMusicSfx->Width = 346.0f;
        grdSubMusicSfx->Height = m_renderer->m_renderTargetSize.Height;
        stkMusicSfx->IsOpen = true;
    }
    else if (command->Id->ToString() == "accelerometer")
    {
        stkAccelerometerSettings->Width = 346.0f;
        grdAccelerometerSettings->Height = m_renderer->m_renderTargetSize.Height;
        stkAccelerometerSettings->IsOpen = true;
    }
    else if (command->Id->ToString() == "charMovement")
    {
        stkCharacterMovement->Width = 346.0f;
        grdCharacterMovement->Height = m_renderer->m_renderTargetSize.Height;
        stkCharacterMovement->IsOpen = true;
    }

    WindowActivationToken = Window::Current->Activated += ref new WindowActivatedEventHandler(this, &DirectXPage::OnWindowActivated);
}

Looking back at OnSettingsRequested, each command has a string identifier. When a command is invoked, that string identifier is returned through the IUICommand instance in the Id property. Based on that, I decided which popups to open. Since each has a similar structure, I am going to cover the implementation of just one—Music & SFX.

Here is what the end result looks like:

The panel on the left is a Popup, with two ToggleButton controls used to enable or disable generic music and sound effects. There are also two Slider controls that are used to adjust the volume. The XAML for the above layout looks like this:

<Popup HorizontalAlignment="Right" IsLightDismissEnabled="True" x:Name="stkMusicSfx" >
    <Grid Background="Black" x:Name="grdSubMusicSfx"  Width="346">
        <Grid.Transitions>
            <TransitionCollection>
                <RepositionThemeTransition />
            </TransitionCollection>
        </Grid.Transitions>

        <Grid.RowDefinitions>
            <RowDefinition Height="120"></RowDefinition>
            <RowDefinition Height="*"></RowDefinition>
        </Grid.RowDefinitions>

        <StackPanel Grid.Row="0" Orientation="Horizontal" Margin="24,12,0,0" >
            <Button Margin="0" VerticalAlignment="Center" x:Name="dismissAudioSettings" Click="dismissAudioSettings_Click" Style="{StaticResource BackButtonStyle}"></Button>
            <TextBlock Margin="12,0,0,12" Height="Auto" VerticalAlignment="Center" Text="Music &amp; SFX" Style="{StaticResource SubheaderTextStyle}"></TextBlock>
        </StackPanel>

        <StackPanel Grid.Row="1" Margin="24,24,0,0">
            <StackPanel>
                <TextBlock Text="Music" Style="{StaticResource BodyTextStyle}"></TextBlock>
                <TextBlock  Width="346" Text="This includes the theme track and level background music." Style="{StaticResource CaptionTextStyle}" TextWrapping="Wrap" Margin="0,12,12,12" ></TextBlock>
                <ToggleSwitch x:Name="tglMusic" Toggled="tglMusic_Toggled" IsOn="{Binding ElementName=XAMLPage,Path=MusicEnabled}" Margin="-6,0,0,0"></ToggleSwitch>
            </StackPanel>

            <StackPanel Margin="0,24,0,0">
                <TextBlock Text="Music Volume" Style="{StaticResource BodyTextStyle}"></TextBlock>
                <Slider x:Name="sldMusicVolume" ValueChanged="sldMusicVolume_ValueChanged" Value="{Binding ElementName=XAMLPage,Path=MusicVolume, Mode=TwoWay}" Minimum="0" Maximum="100" Margin="0,0,12,0"></Slider>
            </StackPanel>

            <StackPanel Margin="0,24,0,0">
                <TextBlock Text="Sound Effects" Style="{StaticResource BodyTextStyle}"></TextBlock>
                <TextBlock Width="346" Text="Includes sounds played during the game (e.g. explosions)." Style="{StaticResource CaptionTextStyle}" TextWrapping="Wrap" Margin="0,12,12,12" ></TextBlock>
                <ToggleSwitch x:Name="tglSFX" Toggled="tglSFX_Toggled" IsOn="{Binding ElementName=XAMLPage,Path=SFXEnabled}" Margin="-6,0,0,0"></ToggleSwitch>
            </StackPanel>

            <StackPanel Margin="0,24,0,0">
                <TextBlock Text="SFX Volume" Style="{StaticResource BodyTextStyle}"></TextBlock>
                <Slider x:Name="sldSFXVolume" ValueChanged="sldSFXVolume_ValueChanged" Value="{Binding ElementName=XAMLPage,Path=SFXVolume, Mode=TwoWay}" Minimum="0" Maximum="100" Margin="0,0,12,0"></Slider>
            </StackPanel>

        </StackPanel>

    </Grid>
</Popup>

For every Popup instance used for settings, make sure that IsLightDismissEnabled is set to true. This allows the user to dismiss the panel with a touch outside its boundaries, just like the stock system panels. Other than that, you are working with the standard XAML control set and can include virtually anything in your settings.

Notice, that the switches and sliders are bound to internal properties, such as SFXEnabled and MusicEnabled, that perform the binding via dependency property references:

DependencyProperty^ DirectXPage::_musicEnabled = DependencyProperty::Register("MusicEnabled", 
bool::typeid, DirectXPage::typeid, nullptr);
DependencyProperty^ DirectXPage::_sfxEnabled = DependencyProperty::Register("SFXEnabled", 
bool::typeid, DirectXPage::typeid, nullptr);

The properties themselves are declared in the DirectXPage header file:

static property DependencyProperty^ SFXVolumeProperty
{
    DependencyProperty^ get() { return _sfxVolume; }
}
property int SFXVolume
{
    int get() { return (int)GetValue(SFXVolumeProperty); }
    void set(int value) 
    { 
        SetValue(SFXVolumeProperty, value);
    }
}

static property DependencyProperty^ MusicVolumeProperty
{
    DependencyProperty^ get() { return _musicVolume; }
}
property int MusicVolume
{
    int get() { return (int)GetValue(MusicVolumeProperty); }
    void set(int value) 
    { 
        SetValue(MusicVolumeProperty, value);
    }
}

Let’s take a quick look at how settings are stored. I have a class called SettingsHelper that allows me to save, read, and check if specific settings exist. Here is the implementation:

#include "pch.h"
#include "SettingsHelper.h"

using namespace Windows::Storage;
using namespace Coding4Fun::FallFury::Helpers;

SettingsHelper::SettingsHelper(void)
{
}

void SettingsHelper::Save(Platform::String^ key, Platform::Object^ value)
{
    ApplicationDataContainer^ localSettings = ApplicationData::Current->LocalSettings;
    auto values = localSettings->Values;

    values->Insert(key, value);
}

Platform::Object^ SettingsHelper::Read(Platform::String^ key)
{
    ApplicationDataContainer^ localSettings = ApplicationData::Current->LocalSettings;
    auto values = localSettings->Values;

    return values->Lookup(key);
}

bool SettingsHelper::Exists(Platform::String^ key)
{
    ApplicationDataContainer^ localSettings = ApplicationData::Current->LocalSettings;
    auto values = localSettings->Values;

    return values->HasKey(key);
}

It is clear that storage and retrieval heavily relies on the ApplicationDataContainer class, the container class for local settings that eliminates the need for the developer to create his own setting files, instead delegating this task to the OS and utilizing a centralized storage for all Windows Store applications.

A typical scenario that utilizes the class above is executed when the toggle that manages the sound effects is switched:

void DirectXPage::tglSFX_Toggled(Platform::Object^ sender, Windows::UI::Xaml::RoutedEventArgs^ e)
{
    SettingsHelper::Save("sfxEnabled", tglSFX->IsOn);
    SFXEnabled = tglSFX->IsOn;
    AudioManager::IsSFXStarted = SFXEnabled;

    if (SFXEnabled)
    {
        AudioManager::AudioEngineInstance.StartSFX();
    }
    else
    {
        AudioManager::AudioEngineInstance.SuspendSFX();
    }
}

The Boolean value above will be automatically serialized and stored. The files will be located at C:\Users\YOUR_USER_NAME\AppData\Local\Packages\PACKAGE_ID\Settings\settings.dat:

Conclusion

Implementing sharing through the OS channel in Windows 8 is extremely easy seeing as the developer does not necessarily have to worry about connecting the app to third-party API endpoints. Instead, the user controls the sharing, allowing flexibility of choice without requiring a major addition to the existing code base. It’s hard to predict which services might appear, and modifying the app to support each one of them would be next to impossible otherwise.

You can read more about settings in Windows Store applications here.

During gameplay there are scenarios during which users need to be visually notified that something has happened, such as a collision with an obstacle or an enemy shell. One way to do this is by having explosion or item breaking simulation, which brings us to the next large component in FallFury—the sprite-based particle system. It doesn’t offer as much power as a full-fledged particle system would, but it allows for effects that fit well within the overall theme and layout of the game.

Check out the video for this article at http://channel9.msdn.com/Series/FallFury/Part-9-Particle-System .  For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles.

The Core

ParticleSystem is the dedicated folder in the project that contains everything needed to render multiple textures at once and displace them to create the desired effect:

A single particle carries information regarding its size, position, velocity, color shading, rotation, circular velocity, and scale. As with any other rendered entity, it has a bounding box that can be used to detect its intersection with other elements on the screen. In FallFury, this functionality is not used.

Its structure is as follows:

#pragma once
#include "pch.h"
struct Particle
{
    Particle(float2 size);
    Particle(float2 size, float4 shading);
    float2 Size;
    float2 Position;
    float2 Velocity;
    float4 Shading;
    float Rotation;
    float RotationVelocity;
    float Scale;
    bool IsWithinScreenBoundaries(float x, float y, Windows::Foundation::Rect screenBounds);
    Windows::Foundation::Rect GetBoundingBox();
};

The Particle class also happens to have two constructors—one that sets the particle to have the default shading, effectively removing the effect, and one where shading is dynamic. Note that a particle on its own doesn’t do much—it neither carries the associated texture nor has an internal loop that can be used in any given application part to display it.

The next core class is ParticleSet. It is used as a container for all the particles associated with a specific effect. For example, if I want to create flying stuffing when the bear hits an obstacle, I create a new ParticleSet instance and define the necessary particle properties:

• Lifespan – a particle set does not constantly animate. It displays the particles for a limited amount of time, and displaces them by the given velocity values, and then self-destructs.
• Texture – all particles in a ParticleSet have the same texture. Going back to the stuffing example, there is a single PNG file used to render multiple variable-sized particles on collision.
• IsAlive – this is the flag that shows whether the particle set should be rendered in the first place. If it is set to false then, regardless of the conditions, this ParticleSet instance is ignored.
• ShouldScale – this flag determines whether particles will automatically increase their scale as they are being displaced, creating the effect of a particle approaching the screen. This effect is applied to each particle in the set.

The container class is used to update the particles through the Update function:

void ParticleSet::Update(float timeDelta)
{
    float quat = _lifespan / 0.016f;
    float decrement = 1.0f / quat;
    
    if (_totalTime <= _lifespan && _isAlive)
    {
        _totalTime += timeDelta;
        
        for (auto particle = _particles.begin(); particle != _particles.end(); particle++)
        {
            if (_shouldScale)
                particle->Scale += 0.2f;
            
            particle->Shading.a -= decrement;
            particle->Position = float2(particle->Position.x + particle->Velocity.x,
                            particle->Position.y + particle->Velocity.y);
            
            if (!_shouldScale)
                particle->Rotation += particle->RotationVelocity;
        }
    }
    else
    {
        _totalTime = 0.0f;
        _isAlive = false;
    }
}

As it relies on the _isAlive flag, the Update loop is only used when the particle displacement is activated via the Activate function:

void ParticleSet::Activate(float2 position, float2 velocity, bool randomize, bool scale)
{
    for (auto particle = _particles.begin(); particle != _particles.end(); particle++)
    {
        particle->Position = position;

        if (randomize)
            particle->Velocity = float2(RandFloat(-5.0f,5.0f), RandFloat(-5.0f, 5.0f));
        else
            particle->Velocity = float2(velocity.x + RandFloat(-0.6f, 0.6f), velocity.y + RandFloat(-0.6f, 0.6f));;
    }

    _shouldScale = scale;
    _isAlive = true;
}

When a set is activated, several user-defined parameters come into play. The position is set no matter what and is used to create the source point from which the particles start appearing. The velocity, on the other hand, can be randomized between the values of -5 and 5 pixels per update loop, on both the X-axis and the Y-axis. If randomized, large amount of particles will create an explosion that starts from the center point and expands towards all quadrants. When the velocity is not randomized, it creates a triangular expansion grid on which particles deviate from the center point in one of the given directions:

When it’s time to render the particles, the sprite batch associated with the current game screen is used to pass a texture for each particle registered in the set:

void ParticleSet::Render(SpriteBatch ^spriteBatch)
{
    for (auto particle = _particles.begin(); particle != _particles.end(); particle++)
        {
        if (GamePlayScreen::Manager->IsWithinScreenBoundaries(particle->Position))
                    spriteBatch->Draw(_texture.Get(), particle->Position, PositionUnits::DIPs,
                    particle->Size * particle->Scale, SizeUnits::Pixels, particle->Shading, particle->Rotation);
        }
}

Although there is no flag check inside the Render method that would make sure that the set is alive, this can be done outside of it by calling IsAlive, which will return the flag value:

bool ParticleSet::IsAlive()
{
    return _isAlive;
}

Let’s now take a look at the class that managed the particle flow—ParticleCore.

The Particle Manager

ParticleCore is the class that manages internal particle sets, and is also the proxy for set activation, update, and rendering. Here is its structure:

#pragma once
#include "pch.h"
#include "ParticleSet.h"
#include <list>
#include <map>

class ParticleCore
{
    public:
    ParticleCore();
    ParticleCore(Coding4Fun::FallFury::Screens::GameScreenBase^);
    virtual ~ParticleCore();
    
    void CreatePreCachedParticleSets();
    void ActivateSet(Platform::String^, float2);
    void ActivateSet(Platform::String^, float2, float2);
    void ActivateSet(Platform::String^, float2, bool);
    void ActivateSet(Platform::String^, float2, float2, bool);
    void ActivateSet(Platform::String^, float2, float2, bool, bool scale);
    void Update(float);
    void Render();
    
    private:
        std::list<ParticleSet>                                                    _renderParticleSets;
        std::map<Platform::String^, ParticleSet*>                                _particleSetCache;
        std::map<Platform::String^, Microsoft::WRL::ComPtr<ID3D11Texture2D>>    _textureCache;
        Coding4Fun::FallFury::Screens::GameScreenBase^                            _screenBase;
};

As previously mentioned, set activation can be done with some omitted parameters inferred by the system, such as randomization of velocity or particle scaling, which is the reason why you see multiple overloads for ActivateSet.

ParticleCore is also the container for pre-defined sets that have specific textures and properties that are dumped in the particle set cache (internal _particleSetCache). The cache is reusable and even though sets can be activated and destroyed, the cache remains intact for the duration of the game unless explicitly reset or modified. The texture cache is an addition to the particle set cache and is used as a helper container to temporarily store the textures used for individual particles.

Taking a look under the hood at the CreatePreCachedParticleSets function, you can see multiple sets that can be used in the game, each with different properties. Here is a snippet that shows how the bear stuffing explosion set is created:

auto smallExplosionSet = new ParticleSet(_textureCache["Stuffing"], LIFESPAN);
for (int i = 0; i < 20; i++)
{
    float size = RandFloat(50.0f, 100.0f);
    Particle particle(float2(size, size));
    smallExplosionSet->AddParticle(particle);
}

Once all the particles are in place for that specific set, it is added to the global particle set cache:

_particleSetCache["SmallExplosion"] = smallExplosionSet;

The particle set cache is not explicitly used to render anything on the screen. Rather, that task is delegated to the rendering cache. When a set is activated, the cache is inspected for the given key, its Activate function is called, marking it as alive, and the set itself is pushed on the rendering stack:

void ParticleCore::ActivateSet(Platform::String^ name, float2 position, float2 velocity, bool randomize, bool scale)
{
    ParticleSet set = ParticleSet(*_particleSetCache[name]);
    set.Activate(position, velocity, randomize, scale);
    _renderParticleSets.push_back(set);
}

The Render loop takes care of invoking the proper SpriteBatch drawing functions for each set that is in that stack:

void ParticleCore::Render()
{
    for (auto set = _renderParticleSets.begin(); set != _renderParticleSets.end(); ++set)
    {
        if ((*set).IsAlive())
        {
            (*set).Render(_screenBase->CurrentSpriteBatch);
        }
    }
}

Notice that, as previously mentioned, the set does not perform the life check on itself when rendering. Instead, the manager class performs this action. The same applies to the Update loop:

void ParticleCore::Update(float timeDelta)
{
    for (auto set = _renderParticleSets.begin(); set != _renderParticleSets.end();)
    {
        if ((*set).IsAlive())
        {
            (*set).Update(timeDelta);
            ++set;
        }
        else
        {
            set = _renderParticleSets.erase(set);
        }
    }
}

The textures for each set are individually loaded in the CreatePreCachedParticleSets. Each instance is internally pushed into the cache and also added to the sprite batch associated with the current game screen, where ParticleCore is used:

Loader->LoadTexture("DDS\\stuffing.dds", &_textureCache["Stuffing"], nullptr);
_screenBase->CurrentSpriteBatch->AddTexture(_textureCache["Stuffing"].Get());

Once everything is loaded, the ParticleCore is ready to go and you can use as many particles as necessary in any part of the application. In GamePlayScreen, particle sets are activated in many cases. For example, if the bear dies:

void GamePlayScreen::CheckBearHealth()
{
    if (GameBear->CurrentHealth <= 0)
    {
        m_particleSystem.ActivateSet("Buttons",GameBear->Position, true);
        GameBear->Kill();
        StopBackground();
    }
}

Conclusion

Implementing a sprite-based particle system is not complicated, but it requires depending on a number of assumptions. For example, when a particle set is created, you might consider the fact that some hardware can handle drawing only a given number of sprites at the same time. If a particle set is rendered on a desktop machine, there is no guarantee that the same set will successfully render on an ARM device. Therefore, plan accordingly. For each particle set, create a lifespan that fits the scenario without wasting resources on rendering unnecessary particles. As an additional failsafe, you might want to disable specific particle set types when a Direct3D feature level below 10.0 is detected.

During the gameplay multiple entities interact with each other to make the gaming experience what it is. The bear collides with obstacles and collects buttons, monsters shoot shells that can fly off-screen or hit the bear—all this is possible with the help of the basic collision detection techniques that are implemented in FallFury.

Check out the video for this article at http://channel9.msdn.com/Series/FallFury/Part-8-Element-Interaction . For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles.

Buttons

Buttons are bonus-boosters that can be placed by the level designer anywhere on the screen in game mode. These are relatively small entities, which are displaced vertically with each cycle of the Update loop and move in the opposite direction, but with the same velocity, as the main character.

Looking at the Update function in the GamePlayScreen class, you will notice this call:

UpdateButtons();

UpdateButtons can be considered the button manager function responsible for removing the collected buttons from the rendering stack, counting them, and checking for a button collision when the bear is in close proximity. The implementation looks like this:

void GamePlayScreen::UpdateButtons()
{
    Windows::Foundation::Rect livingEntityBoundingBox = GameBear->GetBoundingBox();
    for (auto button = m_buttons.begin(); button != m_buttons.end();)
    {
        (*button)->Position.x = (*button)->PixelDiff + LoBoundX;
        (*button)->Position.y -= GameBear->Velocity.y;
        if (Geometry::IsInProximity(GameBear->Position,(*button)->Position, 100))
        {
            Windows::Foundation::Rect obstacleRect = (*button)->GetBoundingBox();
            if (livingEntityBoundingBox.IntersectsWith(obstacleRect))
            {
                AudioManager::AudioEngineInstance.StopSoundEffect(Coin);
                AudioManager::AudioEngineInstance.PlaySoundEffect(Coin);
                m_particleSystem.ActivateSet("Sparkle", (*button)->Position,float2(RandFloat(-6.0f,6.0f),RandFloat(-10.0f, -5.0f)));
                StaticDataHelper::ButtonsCollected++;
                button = m_buttons.erase(button);
            }
            else
            ++button;
        }
        else
        {
            ++button;
        }
    }
}

For performance reasons, FallFury supports composite bounding box creation as well as simple box creation. As mentioned earlier in the series, the main character is not composed of a single texture, but rather multiple sprites that are cross-positioned to create a single visual entity. To give you a better idea of what composite vs. simple boxing looks like, take a look at the images below:

The image on the left shows how each part of the bear has its own bounding box, and each will be used for collision checking. In the image on the right, the bear has a single bounding box, creating minor potential gaps, but gaining performance.

Going back to UpdateButtons, once the bounding box is obtained, I iterate through the button collection and make sure that each item is located in the proper space:

(*button)->Position.x = (*button)->PixelDiff + LoBoundX;
(*button)->Position.y -= GameBear->Velocity.y;

The constant position checks are necessary because FallFury supports dynamic orientation changes. When the user switches from portrait to landscape mode and vice-versa, rendered elements on the screen are not automatically repositioned. Setting the X position is easy as long as there is a fixed button margin (from the left side of the screen: LoBoundX) and adding it to the current LoBoundX value results in a proper X location. There is no need to do the same check on the Y-axis because the level length remains the same regardless of the current screen orientation. The adjustment made relative the Y position is bound to the bear velocity. If the bear moves slower, buttons will also scroll slower.

Given that all buttons are properly positioned, a proximity check is performed on each button passed through the loop. If the bear position is at least 100 pixels away from the current button, the corresponding bounding box is obtained and an intersect check is performed. In simple boxing mode, this is done via Windows::Foundation::Rect::IntersectsWith:

If a collision occurs, the appropriate sound effect is played and a particle set is activated to create a visual notification of the action. After the button counter is incremented, the button is removed from the local collection, effectively being removed from the rendering stack.

Power-ups

As the bear flies towards the end of the level, it might encounter bonuses to improve its ability to fight incoming enemies or protect from damage caused by enemy ammo or obstacles. The process behind displaying power-ups on the screen and determining whether there was a collision with the main character is similar to UpdateButtons.

The core function for this task is UpdatePowerups:

void GamePlayScreen::UpdatePowerups(float timeDelta)
{
    if (m_powerups.size() > 0)
    {
        for (auto powerup = m_powerups.begin(); powerup != m_powerups.end(); powerup++)
        {
            (*powerup)->Update(timeDelta);
            (*powerup)->Position.x = (*powerup)->PixelDiff + LoBoundX;
            (*powerup)->Position.y -= GameBear->Velocity.y;
        }
    }
    CheckForCollisionWithPowerups();
}

One difference you probably noticed in the snippet above is the fact that the collision check is now done through a separate function—CheckForCollisionWithPowerups:

void GamePlayScreen::CheckForCollisionWithPowerups()
{
    Powerup^ currentPowerup;
    Windows::Foundation::Rect livingEntityBoundingBox = GameBear->GetBoundingBox();
    for (auto powerup = m_powerups.begin(); powerup != m_powerups.end();)
    {
        currentPowerup = (*powerup);
        if (Geometry::IsInProximity(GameBear->Position,currentPowerup->Position, 100))
        {
            Windows::Foundation::Rect obstacleRect = currentPowerup->GetBoundingBox();
            if (livingEntityBoundingBox.IntersectsWith(obstacleRect))
            {
                AudioManager::AudioEngineInstance.PlaySoundEffect(GenericPowerup);
                GameBear->PickupPowerup(currentPowerup, &m_particleSystem);
                powerup = m_powerups.erase(powerup);
            }
            else
            ++powerup;
        }
        else
        {
            ++powerup;
        }
    }
}

If an intersection is detected between the boxed bear and the power-up texture, the current power-up is passed to the Bear instance and the appropriate type of action is selected:

void Bear::PickupPowerup(Powerup^ powerup, ParticleCore* ParticleSystem)
{
    switch (powerup->Type)
    {
        case PowerupType::PARACHUTE:
        {
            m_previousVelocity = Velocity.y;
            SetParachute(powerup->Lifespan);
            Velocity.y -= powerup->Effect;
            ParticleSystem->ActivateSet("ScalableParachute", Position, 0.0f, false, true);
            break;
        }
        case PowerupType::HEALTH:
        {
            CurrentHealth = MaxHealth;
            ParticleSystem->ActivateSet("ScalableHeart", Position, 0.0f, false, true);
            break;
        }
        case PowerupType::BUBBLE:
        {
            if (IsHelmetEnabled)
            {
                IsHelmetEnabled = false;
                DamageDivider = 1.0f;
            }
            else
            {
                DamageDivider = powerup->Effect;
            }
            IsBubbleEnabled = true;
            m_maxBubbleCounter = powerup->Lifespan;
            ParticleSystem->ActivateSet("ScalableBubble", Position, 0.0f, false, true);
            break;
        }
        // [...]
        case PowerupType::BOOMERANG:
        {
            m_weaponType = powerup->Type;
            m_weaponTexture = m_boomerangTexture;
            CurrentDamage = powerup->Effect;
            m_weaponSize = float2(225.0f, 205.0f) * 0.5f;
            ParticleSystem->ActivateSet("ScalableBoomerang", Position, 0.0f, false, true);
            break;
        }
    }
}

Depending on the power-up, textures are added to the bear model and later passed to the rendering stack (if the power-up type is PARACHUTE), some textures and capabilities are replaced (BOOMERANG), or the bear capabilities are temporarily modified (BUBBLE):

If one of the temporary power-ups is enabled n the Update loop, dedicated timers ensure that ability enhancement does not last longer than necessary. As an example, here is the snippet that controls the bubble:

if (IsBubbleEnabled)
{
    m_currentBubbleCounter += timeDelta;
    if (m_currentBubbleCounter > m_maxBubbleCounter)
    {
        IsBubbleEnabled = false;
        DamageDivider = 1.0f;
        m_currentBubbleCounter = 0.0f;
    }
}

Ammo Collisions

There are also objects that direct ammo at either the enemy or the bear. After release, each shell follows a linear path towards the target, and while the user directs the shells that originate from the bear, those released by enemy entities automatically target the bear. In this case, the ammo needs to collide with an object to either damage or kill it. When a shell is released, it is added to the general ammo collection that is later updated internally:

void GamePlayScreen::UpdateAmmo(float timeDelta)
{
    for (auto shell = m_ammoCollection.begin(); shell != m_ammoCollection.end(); shell++)
    {
        (*shell)->Update(timeDelta, &m_particleSystem);
    }
}

At this point, the rendering system does not differentiate between friendly and enemy ammo—all it knows is that each item in the collection must have a new position when a new frame is rendered. The CheckForCollisionsWithAmmo function checks for ammo collisions:

void GamePlayScreen::CheckForCollisionWithAmmo(LivingEntity^ entity)
{
    if (entity != nullptr)
    {
        Windows::Foundation::Rect livingEntityBoundingBox = entity->GetBoundingBox();
        if (entity->IsFriendly)
        {
            for (auto ammo = m_ammoCollection.begin(); ammo != m_ammoCollection.end();)
            {
                if (!(*ammo)->IsFriendly)
                {
                    Windows::Foundation::Rect ammoBoundingBox = (*ammo)->GetBoundingBox();
                    if (livingEntityBoundingBox.IntersectsWith(ammoBoundingBox))
                    {
                        m_particleSystem.ActivateSet("SmallExplosion",entity->Position, true);
                        entity->InflictDamage((*ammo)->HealthDamage);
                        GameBear->RedShade();
                        AudioManager::AudioEngineInstance.PlaySoundEffect(OuchA);
                        AudioManager::AudioEngineInstance.PlaySoundEffect(HardSoftCollision);
                        CheckBearHealth();
                        ammo = m_ammoCollection.erase(ammo);
                    }
                    else
                    {
                        ++ammo;
                    }
                }
                else
                {
                    ++ammo;
                }
            }
        }
        else
        {
            for (auto ammo = m_ammoCollection.begin(); ammo != m_ammoCollection.end();)
            {
                if ((*ammo)->IsFriendly)
                {
                    Windows::Foundation::Rect ammoBoundingBox = (*ammo)->GetBoundingBox();
                    if (livingEntityBoundingBox.IntersectsWith(ammoBoundingBox))
                    { 
                        Monster^ monster = ((Monster^)entity);
                        if (!monster->IsDead && monster->IsActive)
                        {
                            m_particleSystem.ActivateSet("SmallExplosion", entity->Position, true);
                            entity->InflictDamage((*ammo)->HealthDamage);
                            monster->RedShade();
                            monster->CheckIfAlive();
                            AudioManager::AudioEngineInstance.PlaySoundEffect(SharpSoftCollision);
                        }
                        ammo = m_ammoCollection.erase(ammo);
                    }
                    else
                    {
                        ++ammo;
                    }
                }
                else
                {
                    ++ammo;
                }
            }
        }
    }
}

When the function is called, it is usually run against an entity that is present on the screen, such as the main character. Regardless of whether the enemy or the friendly character fired the shot, the shot cannot inflict damage to its source, and that’s why the function implements the ammo-to-entity crosscheck. If the ammo collides with any shells intersecting the entity bounding box, a collision is counted and health verification is performed to ensure that the character is still alive and that the game should continue. The bear’s health is checked via CheckBarHealth:

void GamePlayScreen::CheckBearHealth()
{
    if (GameBear->CurrentHealth <= 0)
    {
        m_particleSystem.ActivateSet("Buttons",GameBear->Position, true);
        GameBear->Kill();
        StopBackground();
    }
}

That said, not all ammo will collide with entities on the screen. Some of it will go out-of-bounds, and without an explicit cleanup process in place out-of-bounds ammo is constantly re-rendered even though the end user has no way of seeing it. To avoid this, there is a helper function—CheckForOutOfBoundsAmmo:

void GamePlayScreen::CheckForOutOfBoundsAmmo()
{
    for (auto l_Iter = m_ammoCollection.begin(); l_Iter != m_ammoCollection.end(); /* nothing here */ )
    {
        if (!GamePlayScreen::Manager->IsWithinScreenBoundaries((*l_Iter)->Position))
        {
            l_Iter = m_ammoCollection.erase(l_Iter);
        }
        else
        {
            ++l_Iter;
        }
    }
}

If any shell flies outside the screen bounding box, its instance is erased from the collection and the renderer no longer worries about allocating memory for an irrelevant item. To give you an idea of how that happens, here is a snippet that shows how the RenderScreen function handles the current ammo set:

if (!m_ammoCollection.empty())
{
    for (auto shell = m_ammoCollection.begin(); shell != m_ammoCollection.end(); shell++)
    {
        if (!(*shell)->IsFriendly)
        {
            (*shell)->Render();
        }
        else
        {
            GameBear->RenderShell((*shell)->Position, (*shell)->Rotation);
        }
    }
}

Conclusion

Element interaction is a core part of the FallFury experience. Separate handlers are implemented for each of them to ensure maximum flexibility when it comes to adding or removing components without breaking major parts of the code-base. Handling is mainly accomplished in the Update loop by iterating through registered entity sets, such as ammo, and verifying whether an action should be taken. Be cautious when implementing this kind of scenario with large entities and data sets—having multiple loops running simultaneously might tax machine performance, especially on low-power configurations such as ARM.

FallFury relies on a lot of dynamic content. As you already aware of how SpriteBatch is invoked inside the FallFury rendering stack, this article focuses on how dynamic activities are handled on existing textures and entities. If you need a quick look at what was already covered, refer to Part 3 of the series.

Check out the video for this article at http://channel9.msdn.com/Series/FallFury/Part-7-Animations . For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles.

Menu Screen

The first animated element shown in the menu screen is the bear, which is positioned in the top half of the viewport. Notice that the bear moves its paws as well as moving across the screen:

The bear is implemented in the GameBear class and is the normal playable entity that is invincible when animated outside the scope of the gameplay screen. Its full body image is composed of four textures:

Microsoft::WRL::ComPtr<ID3D11Texture2D>        m_head;
Microsoft::WRL::ComPtr<ID3D11Texture2D>        m_leftPaw;
Microsoft::WRL::ComPtr<ID3D11Texture2D>        m_rightPaw;

Microsoft::WRL::ComPtr<ID3D11Texture2D>        m_body // Exposed through LivingEntity;

Each of these textures is internally dependent on the coupled float rotation and position values passed to the sprite batch in the host container (the parent screen):

HostContainer->CurrentSpriteBatch->Draw(
    m_rightPaw.Get(),
    m_rightPawPosition,
    PositionUnits::DIPs,
    float2(522.0f, 141.0f) * Scale,
    SizeUnits::Pixels,
    m_shading,
    m_rightPawRotation - 0.5f);

HostContainer->CurrentSpriteBatch->Draw(
    m_body.Get(),
    Position,
    PositionUnits::DIPs,
    float2(400.0f, 400.0f) * Scale,
    SizeUnits::Pixels,
    m_shading,
    Rotation);

HostContainer->CurrentSpriteBatch->Draw(
    m_leftPaw.Get(),
    m_leftPawPosition,
    PositionUnits::DIPs,
    float2(522.0f, 141.0f) * Scale,
    SizeUnits::Pixels,
    m_shading,
    m_leftPawRotation + 0.5f);

Let’s take a look at how the position and rotation values are modified. All item animations are performed with the help of timers set to brief intervals in which the values are cycled through two value exchange operations. Here is a snippet taken from inside the Update method in the GameBear class that is responsible for bear arm movement:

if (m_armRotationTimer < 0.6f)
{
    m_rightPawRotation -= 0.005f;
    m_leftPawRotation += 0.005f;
}
else if (m_armRotationTimer > 0.6f && m_armRotationTimer < 1.2f)
{
    m_rightPawRotation += 0.005f;
    m_leftPawRotation -= 0.005f;
}
else
{
    m_rightPawRotation = 0.0f;
    m_leftPawRotation = 0.0f;
    m_armRotationTimer = 0.0f;
}

m_armRotationTimer is a float value that is initially set to 0.0f. In the Update loop and increments each iteration by the value of timeDelta, which represents the time difference between two loop cycles. As this value is less than 600 milliseconds (0.6 seconds), the right paw rotation is incremented and the left paw rotation decremented. The difference in the value adjustment direction is caused by the fact that one paw is rotated clockwise and the other is rotated counterclockwise. Once the timer tracks a value above 600 milliseconds but below than 1200 milliseconds, the process reverses and the paws rotate in the opposite direction. After 1.2 seconds, the timer resets, as does the rotation.

Since this rotation is automated, GameBear->Update is called in the menu screen:

m_showBear->Update(timeTotal, timeDelta, float2(Manager->m_windowBounds.Width / 2, Manager->m_windowBounds.Height / 2));

Outside the context of the bear update cycle, there is a similar timed mechanism used to randomly move the bear across the screen:

if (m_positionYAdj == 0.0f)
{
    m_positionYAdj = RandFloat(0.1f, 1.2f);
    m_positionXAdj = RandFloat(-1.0f, 1.0f);
}

if (m_positionTimer < 4.0f)
{
    m_showBear->Position.x += m_positionXAdj;
    m_showBear->Position.y += m_positionYAdj;
    m_showBear->Rotation += 0.001f;
    m_showBear->Scale += 0.0008f;
    
    m_showMonster->Scale += 0.002f;
}
else if (m_positionTimer > 4.0f && m_positionTimer < 8.0f)
{
    m_showBear->Position.x -= m_positionXAdj;
    m_showBear->Position.y -= m_positionYAdj;
    m_showBear->Rotation -= 0.001f;
    m_showBear->Scale -= 0.0008f;
}
else
{
    m_positionYAdj = 0.0f;
    m_positionTimer = 0.0f;
}

The X and Y adjustments are randomly generated and for the duration of a single timer cycle (in this case, 4 seconds) the bear’s horizontal and vertical positions are adjusted using those values. The rotation and scale are minimally adjusted to create a 3D motion effect. Because the displacement is minimal, the bear is always visible and does not move much outside the viewport.

There is also a flying monster animation in the main menu screen. Monster creation happens on top of the same instance that is replaced when an initial monster flies out of bounds:

m_showMonster->Position.y -= m_showMonster->Velocity.y;
if (m_showMonster->Position.y < -m_showMonster->Size.y)
{
    CreateNewMonster();
}

When a new monster is created, a random monster type is selected and its position on the X axis is randomized. The Y position is set to be right under the bottom screen boundary:

void MenuScreen::CreateNewMonster()
{
    m_showMonster = ref new Monster(this, (MonsterType)(rand() % (int)MonsterType::MT_CANDYLAND_E), true);
    m_showMonster->Scale = 0.3f;
    m_showMonster->Velocity.y = 1.0f;
    m_showMonster->Load();
    m_showMonster->Position = float2(RandFloat(LoBoundX, HiBoundX), Manager->m_windowBounds.Height + m_showMonster->Size.y * m_showMonster->Scale);
}

Monster scaling is done on the same timed loop as the bear. The results look like this:

Gameplay Screen

Bear

The gameplay screen carries the most animations in FallFury, including animations related to ammo displacement, monster movement paths, power-up animations, and secondary game character animations that are triggered in special cases such as character death.

Let’s start with character entrance. When the game begins, the teddy bear falls into the screen and stops at a specific point close to the top boundary of the gameplay screen. Once that transition is complete, the bear is no longer vertically displaced:

if ((GameBear->Position.y / HiBoundY) < 0.19f)
{
    GameBear->Position.y += GameBear->Velocity.y * (3.2f);
}

If the limit is not hit, the original bear velocity is multiplied by a fixed value, after which the condition is ignored for the rest of the gameplay.

Let’s now take a look at what happens when an enemy shell kills the bear. Generally, when the main character is killed, there is not much sense in maintaining other in-game activities. In FallFury, a killed bear results in a time freeze—the fall stops and the monsters are no longer shooting or moving. Not only that, but the bear is also flipped on its back:

In the Update loop, the position for the paws and the head are adjusted for the new body texture layout:

if (IsDead)
{
    if (m_armRotationTimer < 1.0f)
    {
        m_rightPawRotation += 0.001f;
        m_leftPawRotation -= 0.001f;
    }
    else if (m_armRotationTimer > 1.0f && m_armRotationTimer < 2.0f)
    {
        m_rightPawRotation -= 0.001f;
        m_leftPawRotation += 0.001f;
    }
    else
    {
        m_rightPawRotation = 0.0f;
        m_leftPawRotation = 0.0f;
        m_armRotationTimer = 0.0f;
    }

    m_rightPawPosition = Position + float2(40.0f, -50.0f) * Scale;
    m_leftPawPosition = Position + float2(10.0f, 150.0f) * Scale;
    m_headPosition = (Position - float2(-160.0f, 0.0f) * Scale);
}

The proper textures are assigned via the Kill method:

void Bear::Kill()
{
    IsDead = true;
    Rotation = -1.0f;
    m_head = m_deadHead;
    m_body = m_deadBody;
    m_leftPaw = m_deadLeftArm;
    m_rightPaw = m_deadRightArm;
}

The bear will continue falling as long as StopBackground in GamePlayScreen is called:

void GamePlayScreen::StopBackground()
{
    m_isBackgroundMoving = false;
}

This will cause the internal state check to fail in the Update loop, sending the bear off screen limits. Once the bear completes the fall, the state is set to GS_GAME_OVER:

if (!m_isBackgroundMoving)
{
    if (GameBear->Position.y > m_screenSize.y)
    {
        Manager->CurrentGameState = GameState::GS_GAME_OVER;
    }
    else
    {
        GameBear->Position.y += GameBear->Velocity.y * 1.5f;
    }
}

Monsters

Monsters, on the other hand, are being constantly moved during the duration of the game. Their basic displacement is performed by MoveMonsters:

void GamePlayScreen::MoveMonsters(float timeTotal ,float timeDelta)
{
    for (auto monster = m_monsters.begin(); monster != m_monsters.end();)
    {
        Monster^ currentMonster = (*monster);
        if ((currentMonster->Position.y - GameBear->Position.y) < -Manager->m_renderTargetSize.Height / 2)
        {
            monster = m_monsters.erase(monster);
        }
        else
        {
            currentMonster->Velocity.y = GameBear->Velocity.y;
            currentMonster->Update(timeTotal, timeDelta, GameBear->Position, GetScreenBounds());
            CheckForCollisionWithAmmo(currentMonster);
            ++monster;
        }
    }
}

This method adjusts the monster position relative to the bear:

The zig-zag like motion is defined in the Monster class in the Update method:

float adjustment = 0.0f;
adjustment = m_goingRight ? Position.x + Velocity.x : Position.x - Velocity.x;
if (adjustment >= (HostContainer->LoBoundX + (Size.x * Scale) / 2.0f) 
&& adjustment <= (HostContainer->HiBoundX - (Size.x * Scale) / 2.0f))
{
    Position.x = adjustment;
}
else
{
    m_goingRight = !m_goingRight;
}

The snippet above determines the direction in which the enemy has to move depending on which screen boundary is hit first. While the monster is active and is visible, its vertical adjustment is performed in a timed loop, like this:

m_jumpingTimer += timeDelta;
if (m_jumpingTimer > 0.0f && m_jumpingTimer < 1.0f)
{
    Position.y -= 1.0f;
}
else if (m_jumpingTimer >= 1.0f && m_jumpingTimer < 2.0f)
{
    Position.y += 1.0f;
}
else
{
    m_jumpingTimer = 0.0f;
}

Here you can either use the Y velocity or introduce a static value. The only condition has to be a number low enough that the monster does not leave the screen, which would prevent the bear from being able to kill it. When the monster is killed, it flies out by following an arch-like path:

The “death arc” is implemented via another timer:

Position.x += Velocity.y * 1.3f;
if (m_deathArcTimer > 0.4f)
{
    Position.y += Velocity.y;
    if (Scale > 0.1f)
    Scale -= 0.01f;
}
else
{
    Position.y -= Velocity.y;
    Scale += 0.01f;
    m_deathArcTimer += timeDelta;
}

Regardless of the monster position, the horizontal displacement is positive and the entity moves to the right. For 400 milliseconds its vertical position is decreased, moving the texture up, and the scale is also adjusted to create the proximity effect. After this time interval, the monster drops out of the screen boundaries.

As previously mentioned, each monster is composed of three cycled textures, which are loaded and stored in three ID3D11Texture2D containers:

Microsoft::WRL::ComPtr<ID3D11Texture2D> m_spriteA;
Microsoft::WRL::ComPtr<ID3D11Texture2D> m_spriteB;
Microsoft::WRL::ComPtr<ID3D11Texture2D> m_spriteC;

Each of these is assigned to the main body texture container every 300 milliseconds, replacing the previously assigned asset:

m_stateChangeTimer += timeDelta;
if (m_stateChangeTimer < 0.3f)
{
    m_body = m_spriteA;
}
else if (m_stateChangeTimer > 0.3f && m_stateChangeTimer < 0.6f)
{
    m_body = m_spriteB;
}
else if (m_stateChangeTimer > 0.6f && m_stateChangeTimer < 0.9f)
{
    m_body = m_spriteC;
}
else if (m_stateChangeTimer > 0.9f)
{
    m_stateChangeTimer = 0.0f;
}

This cycle can be disabled when the monster is not active.

Ammo

There are different types of ammo used in the game, and each behaves in its own way. The stock ammo type is a plasma ball.

The default shooting behavior releases a shell when the user taps anywhere on the screen (other than the pause button area). Given the texture of the shell, when it is released the tail must be oriented towards the bear at an angle equal to the one created by the bear and the target position. To get a better idea, take a look at the images below:

In the image on the left, the player tapped at the bottom of the screen. In the one on the right, the player tapped at the right edge of the screen. The shell accordingly rotates depending on the tapped position. To get a better understanding of how the rotation is performed, imagine the XY coordinate grid with the bear located at the intersection of the axes:

Thinking back to trigonometry, notice the triangle formed by the shell and the bear origin point:

Remember also the trig concept of quadrants:

The quadrant in which the shell is located determines how the angle is calculated and the tail rotated. If the shell is in quadrant 2 or 3, calculate the rotation radians by following this algorithm:

1. Find the hypotenuse from the X and Y components of the velocity, which can be calculated by finding the difference between the bear and the position of the tap. To do this, us the well-known Pythagorean theorem:

The hypotenuse formula can be deduced from the formula above and is the square root of the sum of the velocity component squares:

2. Once calculated, it is necessary to find the sine of the target angle. The value can be obtained by dividing the Y component of the velocity by the hypotenuse.

3. Find the angle by calculating the arcsine from the resultant sine value. This will return the final angle in radians.

That said, for shells launched in quadrants 1 or 4, there is one extra step that needs to be performed. In addition to the rotation value obtained from the steps listed above, the rotation should be incremented by the radian value of a 180-degree angle minus twice the rotation value previously obtained. This ensures that the tail is correctly flipped relative to the axis origin.

In C++, the implementation is done inside the AmmoItem class. For flexibility purposes, it triggers inside the Update loop—that way, it is possible to modify the trajectory of the shell and not worry about turning it again manually after launch:

if ((Velocity.x > 0 && Velocity.y > 0) || (Velocity.x > 0 && Velocity.y <0))
{
    Rotation = CalculateRadians(Velocity);
    Rotation += CalculateRadiansFromAngle(180) - 2 * Rotation;
}
else
{
    Rotation = CalculateRadians(Velocity);
}

CalculateRadians is the method that transforms the velocity components in a rotation value. It is located in the BasicMath.h helper:

inline float CalculateRadians(float2 velocity)
{
    float hypothenuse = sqrt(velocity.x * velocity.x + velocity.y * velocity.y);
    float sine = velocity.y / hypothenuse;
    float angle = asin(sine);
    return angle;
};

There is a potential problem with the implementation above. As the user taps on different parts of the screen, the X and Y components are different, each resulting in a different hypotenuse. As the target is set, the shell flies faster the further away from the bear the user taps. To avoid this, the triangle legs should be normalized to a near-constant value:

Take a look at the ShootAtTarget function in the Bear class:

void Bear::ShootAtTarget(float2 lastTargetTrace)
{
    OnPulledTrigger(Position.x, Position.y, 
    GetVelocityLegs(lastTargetTrace).x, GetVelocityLegs(lastTargetTrace).y, 
    CurrentDamage, IsFriendly, false, HostContainer->CurrentSpriteBatch);
}

Notice that the triangle legs are not passed as a raw value, but are proxied through GetVelocityLegs, which forces the resulting vector to be produced from a constant triangle with the velocity constant at 10 pixels per iteration:

float2 LivingEntity::GetVelocityLegs(float2 lastTargetTrace)
{
    float bottomLeg = 0.0f;
    float sideLeg = (Position.y - lastTargetTrace.y) / 100.0f;
    bottomLeg = (Position.x - lastTargetTrace.x) / 100.0f;
    float requiredVelocity = 10.0f;
    float hypothenuse = sqrt(bottomLeg * bottomLeg + sideLeg * sideLeg);
    float proportionalX = 0.0f;
    float proportionalY = 0.0f;
    proportionalX = (requiredVelocity * bottomLeg) / hypothenuse;
    proportionalY = (sideLeg < 0.0f ? -1 : 1) * sqrt(requiredVelocity * requiredVelocity - proportionalX * proportionalX);
    return float2(proportionalX,proportionalY);
}

For ammo that does not have a visual rotation dependency, such as the plasma ball, simply increment the rotation to make the thrown item continuously spin:

if (Type == PowerupType::PLASMA_BALL)
{
    if ((Velocity.x > 0 && Velocity.y > 0) || (Velocity.x > 0 && Velocity.y <0))
    {
        Rotation = CalculateRadians(Velocity);
        Rotation += CalculateRadiansFromAngle(180) - 2 * Rotation;
    }
    else
    {
        Rotation = CalculateRadians(Velocity);
    }
}
else
{
    Rotation += 0.2f;
}

Conclusions

FallFury mostly relies on sprite-based animations in which there are several textures cycled through in order to create the desired dynamic effect. These are not really hard to build, given that there is a possibility to integrate them in a timed loop. Be aware, however, that with slower machines the timing might be off and the time delta value between loops might be higher. In that case the sprite cycling will not be as smooth as it should be, which is why it’s important to perform animation testing on a variety of hardware with different OS loads.

In the previous article, you learned how to build level XML files in order to create playable levels. This article discusses how the internal parser works and how XML nodes become items on the game screen.  Check out the video for this article at http://channel9.msdn.com/Series/FallFury/Part-6-Rendering-Level-Elements.  For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles.

Project Interop – The C# Parser

Thanks to classes such as XmlSerializer and XDocument, parsing XML in .NET Framework is not a complicated task. Similarly, in the WinRT world there are alternatives such as Windows::Data::Xml::Dom. That being said, using the parser skeleton in C# this project leverages the existing codebase, adapting it to the specific level reading needs.

Start by creating a new WinRT project that is the part of the existing solution. Make sure that you create a Windows Runtime Component:

There are major differences between the .NET and WinRT stacks, especially when it comes to building code that can be invoked from any potential WinRT project type, as is the case here. Make sure that the output of the newly created project is a Windows Runtime Component (WinMD file):

The XMLReader project reads much more than the level data—it also manages internal XML metadata such as user scores. This article, however, focuses only on level-related aspects of the engine.

Reader is the one core class used here. It contains the ReadXml method, which receives the file name and the type of the XML file to be read. Regardless of the name, the XML file type determines the parsing rules, the root lookup location, and the produced output. The tier (level set) and individual level data are stored internally in the application folder, and the score metadata is outside the sandbox in the application data folder:

async Task<Object> ReadXml(string fileName, XmlType type)
{
        StorageFile file = null;
        StorageFolder folder = null;
    
        if (type == XmlType.LEVEL || type == XmlType.TIERS)
        {
                folder = Windows.ApplicationModel.Package.Current.InstalledLocation;
                folder = await folder.GetFolderAsync("Levels");
        }
        else if (type == XmlType.HIGHSCORE)
        {
                folder = ApplicationData.Current.LocalFolder;
                folder = await folder.GetFolderAsync("Meta");
        }
    
        file = await folder.GetFileAsync(fileName);
    
        Object returnValue = null;
        string data;
    
        using (IRandomAccessStream readStream = await file.OpenAsync(FileAccessMode.Read))
        {
                using (Stream inStream = Task.Run(() => readStream.AsStreamForRead()).Result)
                {
                        using (StreamReader reader = new StreamReader(inStream))
                        {
                                data = reader.ReadToEnd();
                        }
                }
        }
    
        if (type == XmlType.LEVEL)
        {
                returnValue = ExtractLevelData(data);
        }
        else if (type == XmlType.HIGHSCORE)
        {
                returnValue = ExtractScoreData(data);
        }
        else if (type == XmlType.TIERS)
        {
                returnValue = ExtractTierData(data);
        }
    
        return returnValue;
}

Before the level data is extracted, FallFury needs to know about the existing tiers pointing to those levels. That’s where ExtractTierData, an internal method that accepts a raw XML string and return a TierSet instance, comes in:

public sealed class TierSet
{
    public Tier[] Tiers { get; set; }
}

A Tier class has the following structure:

public sealed class Tier
{
    public string Name { get; set; }
    public string[] LevelNames { get; set; }
    public string[] LevelFiles { get; set; }
}

I am using the most fundamental types – arrays instead of generic collections, to maximize the portability of the code. The ExtractTierData method handles the XML-to-Object transformation:

TierSet ExtractTierData(string tierString)
{
        TierSet set = new TierSet();
    
        XDocument document = XDocument.Parse(tierString);
    
        set.Tiers = new Tier[document.Root.Elements().Count()];
    
        int tierCounter = 0;
    
        foreach (XElement element in document.Root.Elements())
        {
                Tier tier = new Tier();
                tier.Name = element.Attribute("name").Value;
        
                tier.LevelFiles = new string[element.Elements("level").Count()];
                tier.LevelNames = new string[element.Elements("level").Count()];
        
                
                int count = 0;
                foreach (XElement lElement in element.Elements("level"))
                {
                        tier.LevelFiles[count] = lElement.Attribute("file").Value;
                        tier.LevelNames[count] = lElement.Attribute("name").Value;
                        count++;
                }
        
                set.Tiers[tierCounter] = tier;
                tierCounter++;
        }
    
        return set;
}

I am running a coupled array block here, utilizing one array for level names and another for file locations. This is a very basic implementation of a key-value pair that depends on code-based coupling. ExtractTierData, however, is not exposed publicly and the C++ layer of FallFury will not access it because TierSet is not exposed through a compatible async method. Looking back at ReadXml, might seem like the answer to this problem. But a Task<Object> is not an interop-compatible type, which means I use ReadXmlAsync, which proxies the call through an IAsyncOperation:

public IAsyncOperation<Object> ReadXmlAsync(string filename, XmlType type)
{
    return (IAsyncOperation<Object>)AsyncInfo.Run((CancellationToken token) => ReadXml(filename, type));
}

AsyncInfo.Run starts a WinRT async operation and handles the returned result, regardless of the selected file path or type.

Level-based data is extracted in a similar manner to the tier data. The difference lies in the used internal models, as well as the node reading order:

Level ExtractLevelData(string levelString)
{
        XDocument document = XDocument.Parse(levelString);
    
        Level level = new Level();
        level.LevelMeta = new Meta();
     
        level.LevelMeta.Score = Convert.ToInt32(document.Root.Element("meta").Attribute("score").Value);
        level.LevelMeta.ButtonPrice = Convert.ToInt32(document.Root.Element("meta").Attribute("buttonPrice").Value)
        level.LevelMeta.LevelType = (LevelType)Convert.ToInt32(document.Root.Attribute("type").Value);
    
        int count = document.Root.Element("obstacles").Elements().Count();
        level.Obstacles = new Obstacle[count];
    
        count = 0;
        foreach (XElement element in document.Root.Element("obstacles").Elements())
        {
                Obstacle obstacle = new Obstacle();
                obstacle.HealthDamage = Convert.ToSingle(element.Attribute("healthDamage").Value);
                obstacle.InflictsDamage = Convert.ToBoolean(element.Attribute("inflictsDamage").Value);
                obstacle.Rotation = Convert.ToSingle(element.Attribute("rotation").Value);
                obstacle.Scale = Convert.ToSingle(element.Attribute("scale").Value);
                obstacle.X = Convert.ToSingle(element.Attribute("x").Value);
                obstacle.Y = Convert.ToSingle(element.Attribute("y").Value);
                obstacle.Type = (ObstacleType)Convert.ToInt32(element.Attribute("type").Value);
                level.Obstacles[count] = obstacle;
                count++;
        }
    
        count = document.Root.Element("monsters").Elements().Count();
        level.Monsters = new Monster[count];
    
        count = 0;
        foreach (XElement element in document.Root.Element("monsters").Elements())
        {
                Monster monster = new Monster();
                monster.CriticalDamage = Convert.ToSingle(element.Attribute("criticalDamage").Value);
                monster.Damage = Convert.ToSingle(element.Attribute("damage").Value);
                monster.DefaultAmmo = Convert.ToInt32(element.Attribute("defaultAmmo").Value);
                monster.MaxHealth = Convert.ToSingle(element.Attribute("maxHealth").Value);
                monster.X = Convert.ToSingle(element.Attribute("x").Value);
                monster.Y = Convert.ToSingle(element.Attribute("y").Value);
                monster.VelocityX = Convert.ToSingle(element.Attribute("velocityX").Value);
                monster.Lifetime = Convert.ToSingle(element.Attribute("lifetime").Value);
                monster.Type = (MonsterType)Convert.ToInt32(element.Attribute("type").Value);
                monster.Bonus = Convert.ToInt32(element.Attribute("bonus").Value);
                monster.Scale = Convert.ToSingle(element.Attribute("scale").Value);
        
                level.Monsters[count] = monster;
                count++;
        }
    
        count = document.Root.Element("buttons").Elements().Count();
        level.Buttons = new Button[count];
    
        count = 0;
        foreach (XElement element in document.Root.Element("buttons").Elements())
        {
                Button button = new Button();
        
                button.X = Convert.ToSingle(element.Attribute("x").Value);
                button.Y = Convert.ToSingle(element.Attribute("y").Value);
        
                level.Buttons[count] = button;
                count++;
        }
    
        count = document.Root.Element("powerups").Elements().Count();
        level.Powerups = new Powerup[count];
    
        count = 0;
        foreach (XElement element in document.Root.Element("powerups").Elements())
        {
                Powerup powerup = new Powerup();
        
                powerup.X = Convert.ToSingle(element.Attribute("x").Value);
                powerup.Y = Convert.ToSingle(element.Attribute("y").Value);
                powerup.Category = (PowerupCategory)Convert.ToInt32(element.Attribute("category").Value);
                powerup.Type = (PowerupType)Convert.ToInt32(element.Attribute("type").Value);
                powerup.Lifespan = Convert.ToSingle(element.Attribute("lifespan").Value);
                powerup.Effect = Convert.ToSingle(element.Attribute("effect").Value);
        
                level.Powerups[count] = powerup;
                count++;
        }
    
        Bear bear = new Bear();
        XElement bearElement = document.Root.Element("bear");
        bear.CriticalDamage = Convert.ToSingle(bearElement.Attribute("criticalDamage").Value);
        bear.Damage = Convert.ToSingle(bearElement.Attribute("damage").Value);
        bear.DefaultAmmo = Convert.ToInt32(bearElement.Attribute("defaultAmmo").Value);
        bear.MaxHealth = Convert.ToSingle(bearElement.Attribute("maxHealth").Value);
        bear.StartPosition = Convert.ToSingle(bearElement.Attribute("startPosition").Value);
        bear.Velocity = Convert.ToSingle(bearElement.Attribute("velocity").Value);
    
        level.GameBear = bear;
        
        return level;
}

As a result, the Level instance has an internal counterpart in the C++ project:

public sealed class Level
{
    public Bear GameBear { get; set; }
    public Meta LevelMeta { get; set; }
    public Obstacle[] Obstacles { get; set; }
    public Monster[] Monsters { get; set; }
    public Button[] Buttons { get; set; }
    public Powerup[] Powerups { get; set; }
}

From C# to C++ - Leveraging the WinRT Component

At this point, the C#-based WinRT component is complete and can be integrated into the C++ project. This can be done in two ways: either by adding a reference to the project itself or by adding a reference to the generated WinMD file. Both methods will ultimately produce the same output, but it’s easier to debug and modify coupled projects on the go, so I went with the first option.

To add a reference to an internal Windows Runtime Component project, right click on the C++ project in Solution Explorer and select Properties. You will see a dialog like this:

Select the Common Properties node in the tree on the left and open the Framework and References page:

In the sample screen capture above, the FallFury.XMLReader project is already added as a reference. For a new project, simply click on Add New Reference and add any compatible project or third-party extension. As soon as the reference is added, the publicly exposed methods can be accessed.

In DirectXPage.cpp I have a method called LoadLevelData that allows me to load the list of the registered levels from core.xml as well as build the visual tree for the menu items, whichallows players to select a level:

void DirectXPage::LoadLevelData()
{
    Coding4Fun::FallFury::XMLReader::Reader^ reader = ref new Coding4Fun::FallFury::XMLReader::Reader();
    Windows::Foundation::IAsyncOperation<Platform::Object^>^ result = reader->ReadXmlAsync("core.xml", Coding4Fun::FallFury::XMLReader::Models::XmlType::TIERS);
    result->Completed =
        ref new AsyncOperationCompletedHandler<Platform::Object^>(this, &DirectXPage::OnLevelLoadCompleted);
}

Following the normal asynchronous pattern, as well as the structure of the method exposed in the Reader class, I am calling ReadXmlAsync and getting the result in OnLevelLoadCompleted when the operation reaches its final stage. It is worth mentioning that the associated AsyncOperationCompletedHandler is invoked even when the reading fails; therefore, the invocation of that callback does not on its own mean that the necessary data is obtained:

Here is what happens when OnLevelLoadCompleted is called:

void DirectXPage::OnLevelLoadCompleted(IAsyncOperation<Platform::Object^> ^op, AsyncStatus s)
{
    if (s == AsyncStatus::Completed)
    {
        auto set = (Coding4Fun::FallFury::XMLReader::Models::TierSet^)op->GetResults();
        
        auto tiers = set->Tiers;
        
        int levelCounter = 0;
        
        for (auto tier = tiers->begin(); tier != tiers->end(); tier++)
        {
            StackPanel^ panel = ref new StackPanel();
            
            TextBlock^ levelTierTitle = ref new TextBlock();
            levelTierTitle->Text = (*tier)->Name;
            levelTierTitle->Style = (Windows::UI::Xaml::Style^)Application::Current->Resources->Lookup("LevelSelectTierItemText");
            levelTierTitle->RenderTransform = ref new TranslateTransform();
            panel->Children->Append(levelTierTitle);
            
            levelNames = (*tier)->LevelNames;
            auto levelFiles = (*tier)->LevelFiles;
            
            int max = levelNames->Length;
            
            for(int i = 0; i < max; i++)
            {
                MenuItem^ item = ref new MenuItem();
                item->Tag = levelCounter;
                item->Label = levelNames[i];
                item->HorizontalAlignment = Windows::UI::Xaml::HorizontalAlignment::Left;
                item->OnButtonSelected += ref new MenuItem::ButtonSelected(this, &DirectXPage::OnLevelButtonSelected);
                item->RenderTransform = ref new TranslateTransform();
                
                m_renderer->Levels.Insert(levelCounter,levelFiles[i]);
                
                panel->Children->Append(item);
                levelCounter++;
            }
            
            stkLevelContainer->Items->Append(panel);
        }
    }
}

Here, AsyncStatus can also be Cancelled or Error, so checking for Complete ensures that the expected result is processed further inside the context of the callback.

As the returned TierSet exposes the Tiers array, I am simply iterating through each of the existing items to create independent tier blocks, grouped in StackPanel elements, coupled with level-specific MenuItem instances that present the user with a choice of playing a given level. The level ID is carried in the Tag property and it will be used to identify the selected button when DirectXPage::OnLevelButtonSelected is triggered:

Reading Level-specific Data

Level data is read in the GamePlayScreen once the selected menu button passes the name and level identifiers. The LoadLevelXml method is called as the screen begins to load, preparing all assets for the start of the game session:

void GamePlayScreen::LoadLevelXML()
{
    Coding4Fun::FallFury::XMLReader::Reader^ reader = ref new Coding4Fun::FallFury::XMLReader::Reader();
    
    Platform::String^ LevelName = Manager->Levels.Lookup(StaticDataHelper::CurrentLevelID);
    
    Windows::Foundation::IAsyncOperation<Platform::Object^>^ result = 
            reader->ReadXmlAsync(LevelName, Coding4Fun::FallFury::XMLReader::Models::XmlType::LEVEL);
    
        result->Completed =
            ref new AsyncOperationCompletedHandler<Platform::Object^>(this, &GamePlayScreen::OnLevelLoadCompleted);
}

The selected ID passes from the menu screen to the game screen through CurrentLevelID, an intermediary value preserved in a StaticDataHelper class. The level file name is looked up based on the ID and then passed to ReadXmlAsync with the XmlType set to LEVEL. When the load completes, OnLevelLoadCompleted is invoked, and an additional helper class— LevelDataLoader—sets up the game components based on the received data:

void GamePlayScreen::OnLevelLoadCompleted(IAsyncOperation<Platform::Object^> ^op, AsyncStatus s)
{
        if (s == AsyncStatus::Completed)
        {
                InitializeSpriteBatch();    
        m_loader = ref new BasicLoader(Manager->m_d3dDevice.Get(), Manager->m_wicFactory.Get());
        CreateBear();
        
                LevelDataLoader^ loader = ref new LevelDataLoader((Coding4Fun::FallFury::XMLReader::Models::Level^)op->GetResults(), this);
        loader->SetupBear(GameBear);
        loader->SetupObstacles(m_obstacles);
        loader->SetupMonsters(m_monsters);
        loader->SetupButtons(m_buttons, m_buttonPrice);
        loader->SetupPowerups(m_powerups);
        m_currentLevelType = (LevelType) loader->CurrentLevel->LevelMeta->LevelType;
        
        StaticDataHelper::CurrentLevel = loader->CurrentLevel;
        StaticDataHelper::ButtonsTotal = loader->CurrentLevel->Buttons->Length;
        
        LoadTextures();
        
                CreateMonster();
                CreatePowerups();
        
        IsLevelLoaded = true;
        
        GameBear->TurnRight();
        m_particleSystem.CreatePreCachedParticleSets();
        LoadSounds();
        }
}

Each object is separated in its own method, such as SetupBear or SetupObstacles. SetupBear, for example, transforms the exposed managed Bear model to a native C++ Characters::Bear one:

void LevelDataLoader::SetupBear(Bear ^gameBear)
{
        gameBear->Position = float2(GetXPosition(CurrentLevel->GameBear->StartPosition), 0);    
    gameBear->MaxHealth = CurrentLevel->GameBear->MaxHealth;
    gameBear->CurrentHealth = CurrentLevel->GameBear->MaxHealth;
    gameBear->CurrentDamage = CurrentLevel->GameBear->Damage;
    gameBear->Velocity.y = CurrentLevel->GameBear->Velocity;
    gameBear->MaxCriticalDamage = CurrentLevel->GameBear->CriticalDamage;
    gameBear->Rotation = 0.0f;
}

Conclusion

Mixing C# and C++ components is not a complicated process. Nonetheless, it comes with specific restrictions and considerations, such as the format of the publicly exposed asynchronous calls. The above C#-based level loading engine highlights the fact that WinRT interoperability allows you to leverage the languages you know best in order to efficiently accomplish project tasks.

One of the goals set when developing FallFury was making the game extensible in regards to playable levels. Furthermore, I thought that it would make sense from the development and testing perspective to have levels as a completely separate entity that can be modified as necessary. For example, when a new power-up was introduced, I wanted to add an extra line in a level file and test it out. This was ultimately achieved by creating an XML-based level engine and in this article I will describe the level structure and design process.

Check out the video for this article at http://channel9.msdn.com/Series/FallFury/Part-5-Creating-Levels.  For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles.

The First Steps

When I started working on the level engine concept, I began designing the potential XML file structure for the level and ended up with the following requirements:

• Level Type Identifier – As different levels have different backgrounds and sound themes, there should be a way to mark a level type. There are currently four level types: dream, nightmare, space, and magic bean.
• The Starting Character Descriptor – When the game starts, the teddy bear has some initial, basic properties, such as maximum health level, horizontal position, and velocity.
• A Collection of Obstacles – For each level, obstacles are positioned differently, and it’s important to specify that. For some obstacles it might be desirable to disable the damage infliction component, while for others it might be good to maximize the damage caused by colliding with them. Also, there are multiple textures associated with different obstacle types, so I wanted to specify the obstacles to render regardless of the selected level type.
• A Collection of Monsters – Obstacles are not the only component that can damage the bear during gameplay. There are also monsters that can pop up and shoot at the main character. Similar to the bear, monsters represent a living entity and have some specific properties, such as the initial health, damage, starting position, velocity, and type.
• Buttons – These are the bonus point boosters in FallFury. The player collects as many of those as possible, and each of them should be individually positioned to form either a trail or a shape.
• Power-Ups – With the basic set of abilities, the bear is able to get some bonuses such as a cape that will speed-up his descent or a bubble that will protect him from incoming shells.

The first build of the level engine integrated into FallFuryused percentage-based relative values to position elements on the screen. Although this seemed like a good idea at the time, it became problematic because

• It required the level to be a fixed size, which restricted element addition and level extension.
• It caused problems with obstacles that needed to be scaled and therefore had a non-standard size.
• Small modifications were harder to make because minimal adjustments would throw off the relative position.

So, I switched to a pixel-based conditioning in which each position is relative to zero. With this in place, levels can be infinitely long (within the context of the machine’s rendering and memory capabilities)and extra elements can be more seamlessly added.

Additionally, levels need to be packaged together in individual sets, normally grouped by themes, without restriction. This is achieved with the help of an extra XML file,called core.xml, which keeps track of level tiers, and acts as a container that allows the developer to name and easily enable or disable specific levels .

The structure for the core.xml file looks like this:

<tiers>
  <tier name="GO, GO, TEDDY">
    <level name="mind travels" file="GoGoTeddy\mind.xml"></level>
    <level name="falling in" file="Nightmare\full_pilot.xml"></level>
    <level name="frontlines" file="Nightmare\the_beginning.xml"></level>
    <level name="the chase" file="Nightmare\chasing_monsters.xml"></level>
  </tier>
  <tier name="SECRET GARDEN">
    <level name="bean stalking" file="Garden\bean_stalking.xml"/>
    <level name="thorn apart" file="Garden\thorn_apart.xml"/>
  </tier>
  <!--<tier name="Obstacle ***TEST***">
    <level name="Nightmare 0" file="test\Obstacle\nightmare\0.xml" />
    <level name="Nightmare 1" file="test\Obstacle\nightmare\1.xml" />
    <level name="Bean 0" file="test\Obstacle\bean\0.xml" />
    <level name="Bean 1" file="test\Obstacle\bean\1.xml" />
    <level name="Dream 0" file="test\Obstacle\dream\0.xml" />
    <level name="Dream 1" file="test\Obstacle\dream\1.xml" />
  </tier>-->
  <!--<tier name="Death ***TEST***">
    <level name="Monster 0" file="test\death\0.xml" />
  </tier>-->
  <!--<tier name="Monster ***TEST***">
    <level name="0" file="test\Monsters\0.xml" />
    <level name="1" file="test\Monsters\1.xml" />
    <level name="2" file="test\Monsters\2.xml" />
    <level name="3" file="test\Monsters\3.xml" />
    <level name="4" file="test\Monsters\4.xml" />
    <level name="5" file="test\Monsters\5.xml" />
    <level name="6" file="test\Monsters\6.xml" />
    <level name="7" file="test\Monsters\7.xml" />
    <level name="8" file="test\Monsters\8.xml" />
    <level name="9" file="test\Monsters\9.xml" />
    <level name="10" file="test\Monsters\10.xml" />
  </tier>-->
  <!--<tier name="MEDALS *****TEST******">
    <level name="gold" file="test\Medals\gold.xml"></level>
    <level name="silver" file="test\Medals\silver.xml"></level>
    <level name="bronze" file="test\Medals\bronze.xml"></level>
  </tier>
  <tier name="Buttons ***TEST***">
    <level name="1" file="test\Buttons\single.xml" />
    <level name="Lots" file="test\Buttons\multiple.xml" />
  </tier>
  <tier name="Obstacles ***TEST***">
    <level name="Cape" file="test\PowerUps\0.xml" />
  </tier>-->
</tiers>

Tiers that are commented out are ignored and the included levels aren’t on the game list. Also, the paths indicated for each file attribute—for each individual tier—are relative to the game folder itself. There is no limit on the number of subfolders that can be included in the path. The above structure will render this level set:

The Level XML

Let’s now take a look at the layout of the level descriptor XML file:

<?xml version="1.0" encoding="utf-8" ?>
<level type="0">
  <meta score="0" buttonPrice="10"></meta>
  <bear maxHealth="100" startPosition="300" velocity="8.0" damage="11" criticalDamage="20" defaultAmmo="100" />
  <obstacles>
    <obstacle type="1" x="119" y="2300" inflictsDamage="true" healthDamage="5" rotation="3.14" scale="1" />
    <obstacle type="3" x="534.5" y="3000" inflictsDamage="true" healthDamage="5" rotation="0" scale="1" />
    <obstacle type="3" x="534.5" y="3300" inflictsDamage="true" healthDamage="5" rotation="0" scale="1" />
    <obstacle type="1" x="119" y="4200" inflictsDamage="true" healthDamage="5" rotation="3.14" scale="1" />
    <obstacle type="2" x="546" y="5000" inflictsDamage="true" healthDamage="5" rotation="0" scale="1" />
    <obstacle type="1" x="119" y="5800" inflictsDamage="true" healthDamage="5" rotation="3.14" scale="1" />
    <obstacle type="2" x="546" y="6600" inflictsDamage="true" healthDamage="5" rotation="0" scale="1" />
  </obstacles>
  <monsters>
    <monster lifetime="3000" scale=".2" velocityX="2" velocityY="2" type="0" x="460" y="19900" maxHealth="80" bonus="100" lives="0" damage="10" criticalDamage="8" defaultAmmo="50" />
    <monster lifetime="3000" scale=".2" velocityX="2" velocityY="2" type="1" x="460" y="25000" maxHealth="80" bonus="100" lives="0" damage="10" criticalDamage="8" defaultAmmo="50" />
    <monster lifetime="3000" scale=".2" velocityX="2" velocityY="2" type="2" x="460" y="34500" maxHealth="80" bonus="100" lives="0" damage="10" criticalDamage="8" defaultAmmo="50" />
    <monster lifetime="6000" scale=".4" velocityX="2" velocityY="2" type="3" x="460" y="41400" maxHealth="180" bonus="100" lives="0" damage="17" criticalDamage="8" defaultAmmo="50" />
  </monsters>
  <buttons>
    <button x="300" y="800" />
    <button x="360" y="800" />
    <button x="300" y="860" />
    <button x="360" y="860" />
    <button x="300" y="920" />
    <button x="360" y="920" />
    <button x="300" y="980" />
    <button x="360" y="980" />
  </buttons>
  <powerups>
    <powerup category="1" type="4" x="140" y="9200" effect="3" lifespan="4"></powerup>
    <powerup category="1" type="3" x="480" y="19800" effect="10" lifespan="6"></powerup>
    <powerup category="1" type="0" x="480" y="27500" effect="10" lifespan="6"></powerup>
    <powerup category="1" type="1" x="100" y="34000" effect="10" lifespan="6"></powerup>
    <powerup category="1" type="0" x="100" y="41200" effect="10" lifespan="6"></powerup>
  </powerups>
</level>

The opening level tag carries a type attribute. This is level theme flag. It can be set to one of the three four values:

• 0 – The Nightmare Theme
• 1 – The Magic Bean Theme
• 2 – The Dream Theme
• 3 – The Space Theme

You can see the design differences in the images below:

Remember, that obstacles and the level theme itself do not influence much other than the background and soundboard.

Once the type is specified, the meta tag brings up the score and buttonPrice attributes. If you for some reason want to create a level including an initial score, you can specify it here. And because buttons are fixed bonus assets that are all created equal, each of them carries a given bonus point weight. The score based on the collected buttons is calculated at the end of the game and relies on the value specified in the meta tag.

Obstacles

Next comes the obstacle collection, which is represented by the obstacles tag. This tag is required even if there are no obstacles on a given level. Simply use <obstacles /> as necessary. Each child node represents an instance of an obstacle that can be choosen from the following enum:

enum class ObstacleType
{
    OT_CLOUD = 0,
    OT_SPIKE_NIGHTMARE_LARGE = 1,
    OT_SPIKE_NIGHTMARE_MEDIUM = 2,
    OT_SPIKE_NIGHTMARE_SMALL = 3,
    OT_BEAN_A = 4,
    OT_BEAN_B = 5,
    OT_BEAN_C = 6,
    OT_BEAN_D = 7,
    OT_BEAN_E = 8,
    OT_SPACE_ROCKET = 9,
    OT_SPACE_COMET_A = 10,
    OT_SPACE_COMET_B = 11,
    OT_SPACE_SATELLITE = 12,
    OT_SPACE_UFO = 13,
    OT_SPACE_BALL = 14
};

Here is the complete table showing the appearance of each of them:

No matter how the obstacles are positioned, they will either be located in the visible area or displaced outside the viewport and not displayed. The ultimate position is taken from the obstacle size and is relative to the center of the texture. For example, if an obstacle texture is 400- pixels wide, the X-relative position should be set to 200. If the position differs from the starting one, however, the obstacle texture is cut to include the area that fits in the 768 pixel wide playable zone visible. There are no restrictions regarding the Y position.

To help in level creation, some obstacles have pre-defined left and right margins. For example:

• OT_BEAN_A: 269.5 (Left), 498.5 (right, with 3.14 rotation)
• OT_BEAN_B: 128.5 (left), 638.5 (right, with 3.14 rotation)
• OT_BEAN_C: 172 (left), 596 (right, with 3.14 rotation)
• OT_BEAN_D: 188.5 (left), 579.5 (right, with 3.14 rotation)
• OT_BEAN_E: 206.5 (left), 561.5 (right, with 3.14 rotation)

The inflictsDamage attribute determines whether the obstacle harms the main character. If it is set to false, the character will still make the sound of colliding with it but will not lose any health points. The primary use for this attribute is level testing.If it is set to true, the character will loose the amount of health points indicated by the healthDamage attribute.

The rotation and scale attributes can be used to flip and resize the texture as needed. Rotation is measured in radians, and the scale is a normalized value in which 1.0 represents 100% of the scale.

Monsters

As with obstacles, the monsters node should never be omitted from the file and should at least contain a placeholder: <monsters />. Unlike obstacles, however, monsters are dynamic and do not have fixed positions. Moreover, monsters have limited active time during gameplay. The first attribute,lifetime, determines the length of the fall during which the monster will be visible in the viewport. With the y attribute as the Y-based position at which the monster appears, at the y+lifetime position the monster simply flies away if not killed.

The scale attribute carries the same purpose as the one for the obstacle—normalized texture size relative to the size of the original image file. As such, the level designer does not have to worry about linking the width and height of the monster when resizing and can instead use a percentage-like value to scale the monster up or down, simultaneously modifying both the width and the height with zero stretching.

Nextup are velocityX and velocityY. These two attributes are used to set the motion velocity when the monster is already visible. Instead of being a static shooting entity, the enemy moves on a randomized zig-zag path at the bottom of the screen. The horizontal and vertical displacement—in pixels, per update cycle—is individually set through the values carried by the above-mentioned attributes. If necessary, this functionality can be disabled in the code-behind by assigning a fixed value for the vertical and horizontal movement for all monsters that are being loaded on a given level.

The monster type is an integer value that is translated in a value from the following enum (located in MonsterType.h):

enum class MonsterType
{
    MT_NIGHTMARE_A = 0,
    MT_NIGHTMARE_B = 1,
    MT_NIGHTMARE_C = 2,
    MT_MAGICBEAN_A = 3,
    MT_MAGICBEAN_B = 4,
    MT_MAGICBEAN_C = 5,
    MT_CANDYLAND_A = 6,
    MT_CANDYLAND_B = 7,
    MT_CANDYLAND_C = 8,
    MT_CANDYLAND_D = 9,
    MT_CANDYLAND_E = 10
};

Here is a table that shows the texture associated with each identifier:

Each monster has three separate textures associated with it. The three textures are cycled inside the update loop for each monster entity when the monster becomes visible, creating the movement effect:

Each monster currently shoots only one type of ammo: a red plasma ball. Be aware, however, that there is a preprogrammed condition in which the last monster in the XML file collection is automatically considered the final boss. This means the scale, maxHealth and lifetime properties must be manually adjusted to reflect the effect. Without doing so, the last monster will, regardless of the XML setting, switch in-game to a triple fireball shot that inflicts three times the damage indicated by the damage attribute:

Each monster can have limited ammo as set by the defaultAmmo attribute. In the case that the ammo is exhausted before the monster expires, the monster will continue its motion at the bottom of the screen without inflicting direct damage to the main character.

Buttons

These are the least complex entities and only carry an X and a Y position. Given those coordinates, relative to the left margin of the visible area, a button texture is rendered:

Once picked-up, the button counter is increased by one and the meta-score incremented by the value set in the meta tag at the beginning of the level XML as long as the feature is activated in the code-behind.

Power-ups

To make the game more fun, there are bonus elements that can be picked up by the bear in order to enhance its performance or protection. These elements are declared in the <powerups/> collection. First and foremost, it is important to declare whether the power-up is positive or negative. In the current version of FallFury, only positive power-ups are included. Nonetheless, the harness for negative ones is already integrated in the parser. Therefore, the category attribute should be set to 1 if the power-up has a positive effect and 0 for a negative effect. This value will only have an effect over the sound played when the bonus is collected.

The power-up type can be one of the following (enum located in PoweupType.h):

enum class PowerupType
{
    HEALTH = 0,
    HELMET = 1,
    PARACHUTE = 2,
    BUBBLE = 3,
    CAPE = 4,
    AXE = 5,
    BOOMERANG = 6,
    HAMMER = 7,
    KNIFE = 8,
    PLASMA_BALL = 9,
    CIRCLE = 10
};

The table below shows the power-up texture appearance. Behaviors are already defined in the game and influenced by only the effect and lifespan (seconds) attributes:

CIRCLE is a helper power-up that has no effect on the bear and is instead used as an additional texture overlay along with any other power-up in order to create a pulsating circle effect.

Conclusion

FallFury ships with a dozen of sample levels that showcase all of the elements described in the article. At the moment, a level editor is in the works, but it isn’t too complicated to build XML files manually. To do so, you need to consider the pixel-based locations and ensure that they’re all in the visible area—the game engine will automatically handle all other displacements and adjustments.

The Concept of a Swap Chain

Before I go into detail about the DirectX and XAML interop in FallFury, I want to cover one important aspect of DirectX development that you need to familiarize yourself with: the swap chain.

When your graphics adapter draws on the visual surface, you, as the user, see only minor potential redraws. Internally, however, the device switches buffers that reflect the displayed content, with each buffer representing a frame that has to be drawn. You can deduce from this that any swap chain has at least two buffers that it can switch between.

For example, if I want to display my character as being displaced by a specific amount of pixels, the buffer will at the outset present to me the character in its initial position, while the second buffer will be constructed in the background with the proper position adjustments. The first frame, made from the content from the first buffer, will be discarded, and then the second frame will be displayed, and so on. This process occurs at a very high speed that depends on the processing capabilities of the graphics adapter, so the user does not notice the swapping itself.

The most common swap chain is composed of two buffers—the screenbuffer and the secondary framebuffer.

SwapChainBackgroundPanel

DirectX interoperability with XAML simplifies a lot of routine tasks that would otherwise be handled with manual rendering procedures, such as a menu system or a simple game HUD. That being said, the way the XAML workflow is organized in a Direct2D project is quite different compared to a standard XAML Windows Store or Windows Phone application.

The core difference is that there is no navigational system per-se and the fundamental entity in a Direct2D (XAML) project that manages the XAML content is a SwapChainBackgroundPanel control. This control allows the developer to overlay XAML elements on top of the DirectX renders. It replaces the normal page-based layout with one in which it is sole container for every XAML control that has to be used in the application. This necessitates that you will have to organize the secondary elements in such a way that the correct set is displayed for the current application state.

For example, if the user is in the main menu, menu options as well as the game logo should be shown. When the user switches to the game mode, the HUD should appear and the menu should become hidden. Although both the menu and the HUD are a part of the same SwapChainBackgroundPanel, I will have to manually manage state and visibility changes.

Using the SwapChainBackgroundPanel also means that you will have to enforce specific graphic configuration rules in your application. One of them applies to setting up the swap chain. When you set up the scaling, it must be set to DXGI_SCALING_STRETCH:

DXGI_SWAP_CHAIN_DESC1 swapChainDesc = {0};
swapChainDesc.Width = static_cast<UINT>(m_renderTargetSize.Width); 
swapChainDesc.Height = static_cast<UINT>(m_renderTargetSize.Height);
swapChainDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
swapChainDesc.Stereo = false; 
swapChainDesc.SampleDesc.Count = 1;
swapChainDesc.SampleDesc.Quality = 0;
swapChainDesc.BufferUsage = DXGI_USAGE_RENDER_TARGET_OUTPUT;
swapChainDesc.BufferCount = 2;
swapChainDesc.Scaling = DXGI_SCALING_STRETCH;
swapChainDesc.SwapEffect = DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL;
swapChainDesc.Flags = 0;

The swap chain itself should be configured for composition, mixing the native DirectX buffer with the overlaid XAML. This is done by calling CreateSwapChainForComposition:

ThrowIfFailed(
    dxgiFactory->CreateSwapChainForComposition(
        m_d3dDevice.Get(),
        &swapChainDesc,
        nullptr,
        &m_swapChain
        )
    );

ComPtr<ISwapChainBackgroundPanelNative> panelNative;
ThrowIfFailed(
    reinterpret_cast<IUnknown*>(m_panel)->QueryInterface(IID_PPV_ARGS(&panelNative))
    );

ThrowIfFailed(
    panelNative->SetSwapChain(m_swapChain.Get())
    );

There really isn’t much additional configuration work beyond this point. Remember, that the XAML content will always be overlaid on top of the DirectX content; therefore, plan the game components accordingly.

The XAML Menu System

The menu is at the core of the FallFury experience. When designing it, I was inspired by how Dance Central built the user interaction mechanism, and I tried to implement a similar approach in which the user would have to slide the button to the right instead of simply tapping on it:

Each menu item is built around a custom MenuItem control (MenuItem.xaml):

<UserControl
    x:Class="Coding4Fun.FallFury.MenuItem"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:Coding4Fun.FallFury"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d"
    x:Name="menuItem"
    Margin="10, 0, 0, 10">

    <Grid 
        Height="65" 
        ManipulationMode="TranslateX"
        ManipulationCompleted="Grid_ManipulationCompleted"
        ManipulationDelta="Grid_ManipulationDelta" 
        x:Name="ControlContainer"
        PointerPressed="Grid_PointerPressed" 
        PointerReleased="Grid_PointerReleased">
        
        <Grid.Resources>
            <Storyboard x:Name="ArrowAnimator">
                <DoubleAnimation Storyboard.TargetName="ImageTranslateTransform"
                                 Storyboard.TargetProperty="X"
                                 From="0"
                                 To="20"
                                 Duration="0:0:0.4"
                                 RepeatBehavior="Forever"
                                 AutoReverse="True">
                </DoubleAnimation>
            </Storyboard>

        </Grid.Resources>
        
        <Grid.RowDefinitions>
            <RowDefinition Height="10" />
            <RowDefinition />
            <RowDefinition />
            <RowDefinition Height="10" />
        </Grid.RowDefinitions>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="10" />
            <ColumnDefinition />
        </Grid.ColumnDefinitions>
        <StackPanel Orientation="Horizontal" Grid.RowSpan="4" Grid.ColumnSpan="4">
            <!-- width is sent in code behind, have to get this dynamic ...  -->
            <Grid Width="350" x:Name="coverRectangle">
                <Rectangle Fill="#303030" />
                <Rectangle 
                    Fill="Red" 
                    x:Name="coverActiveRectangle" />
            </Grid>
            <Image x:Name="MenuImage" Source="ms-appx:///MenuItems/single_arrow.png" Margin="10,0,0,0" Stretch="Uniform">
                <Image.RenderTransform>
                    <TranslateTransform x:Name="ImageTranslateTransform"></TranslateTransform>
                </Image.RenderTransform>
            </Image>
        </StackPanel>
        <!---->

        <TextBlock 
            Text="{Binding ElementName=menuItem, RelativeSource={RelativeSource Self}, Path=Label}"
            Grid.RowSpan="4"
            Grid.ColumnSpan="4"
            Style="{StaticResource MenuItemText}">

        </TextBlock>
        
        <MediaElement x:Name="coreMenuMedia" Source="ms-appx:///Assets/Sounds/MenuTap.wav" AutoPlay="False"></MediaElement>
        <MediaElement x:Name="slideMenuMedia" Source="ms-appx:///Assets/Sounds/MenuSlide.wav" AutoPlay="False"></MediaElement>
    </Grid>
</UserControl>

There is a core storyboard that is used to perform an image bounce. Initial tests showed that that most users tend to follow the standard “click-and-go” pattern and even though the implemented button was a slider, they tried to click it at least once, expecting the game to take them to the next screen. To avoid this, I introduced a visual arrow indicator that bounces left and right to indicate the direction of the necessary slide.

When the sliding occurs, the overlay grid width is being adjusted relative to the manipulation. At the same time, the button changes from passive state (dark background):

to active state (red background):

Internally, the double-arrow image replaces the single-arrow one and is translated across the X-axis to create a bouncing effect:

void MenuItem::Grid_PointerPressed(Platform::Object^ sender, Windows::UI::Xaml::Input::PointerRoutedEventArgs^ e)
{
    BitmapImage^ image = ref new BitmapImage(ref new Uri("ms-appx:///MenuItems/double_arrow.png"));
    MenuImage->Source = image;
    
    coverActiveRectangle->Visibility = Windows::UI::Xaml::Visibility::Visible;
    ((Storyboard^)ControlContainer->Resources->Lookup("ArrowAnimator"))->Begin();
    
    coreMenuMedia->Play();
}

As the user drags the finger on the button, the red overlay size should be adjusted accordingly. The size adjustment can be easily tracked in the ManipulationDelta event handler for the focused grid:

void MenuItem::Grid_ManipulationDelta(Platform::Object^ sender, Windows::UI::Xaml::Input::ManipulationDeltaRoutedEventArgs^ e)
{
    double diff = e->Cumulative.Translation.X;
    
    if (diff > maxDeltaSize)
    {
        diff = maxDeltaSize;
    }
    else if (diff < 0)
    {
        diff = 0;
    }
    
    coverRectangle->Width = initialBarWidth + diff;
}

When the user action on the button is completed, the ManipulationCompleted event handler is triggered on the same overlay grid. If the relative drag hits the critical threshold, the action linked to the button should be invoked:

void MenuItem::Grid_ManipulationCompleted(Platform::Object^ sender, Windows::UI::Xaml::Input::ManipulationCompletedRoutedEventArgs^ e)
{
    float diff = e->Cumulative.Translation.X;
    
    if (diff > maxDeltaSize - 50) // slight buffer
    {
        slideMenuMedia->Play();
        OnButtonSelected(this, Label);
    }
    
    ResetElements();
}

There are also local sound effects that are played when the slider-button is tapped and moved to the end. Both are handled by separate MediaElement controls that avoid an internal sound file switch, calling instead the Play method as necessary. The OnButtonSelected event handler can be dynamically hooked in the application backend.

Each button also has a visible content area that can display a text label. It can be set through a DependencyProperty:

DependencyProperty^ MenuItem::_LabelProperty = 
    DependencyProperty::Register("Label", 
    Platform::String::typeid,
    MenuItem::typeid, 
    nullptr);

In its current configuration, the menu label can be also set in XAML:

<local:MenuItem x:Name="btnNewGame" Label="new game">
    <local:MenuItem.RenderTransform>
        <TranslateTransform></TranslateTransform>
    </local:MenuItem.RenderTransform>
</local:MenuItem>

When the button loses the focus, the state resets to the passive one, stopping the animation and resetting the arrow image to the single one:

void MenuItem::ResetElements()
{
    BitmapImage^ image = ref new BitmapImage(ref new Uri("ms-appx:///MenuItems/single_arrow.png"));
    MenuImage->Source = image;
    
    ((Storyboard^)ControlContainer->Resources->Lookup("ArrowAnimator"))->Stop();
    
    coverRectangle->Width = initialBarWidth;
    coverActiveRectangle->Visibility = Windows::UI::Xaml::Visibility::Collapsed;
}

That’s about as complex as the menu item control will get. The menu container itself can be any Grid or StackPanel control. The way the menu items are used across the game states, there is no need to have a separate unified container.

The Main XAML Container & State Changes

Going back to DirectXPage.xaml, there are several parts of the XAML layout that should be highlighted. First and foremost, the game curtains—the static parts of the screen that are being displayed instead of blacking out parts of the viewport that are not being used when the game runs in landscape mode. Because the game includes a rectangular frame instead of stretching the entire playable area to the size of the screen, there is an unknown amount of unallocated visual space on both the right and left sides of the frame itself:

<Grid HorizontalAlignment="Left" x:Name="containerA">
    <Rectangle Fill="#09bbe3" />

    <Rectangle Width="10" HorizontalAlignment="Left">
        <Rectangle.Fill>
            <LinearGradientBrush StartPoint="0,0" EndPoint="1,0">
                <GradientStop Color="#a000" Offset="0.0"></GradientStop>
                <GradientStop Color="#0000" Offset="1.0"></GradientStop>
            </LinearGradientBrush>
        </Rectangle.Fill>
    </Rectangle>
</Grid>

<Grid HorizontalAlignment="Right" x:Name="containerB">
    <Rectangle Fill="#09bbe3" />

    <Rectangle Width="10" HorizontalAlignment="Right">
        <Rectangle.Fill>
            <LinearGradientBrush StartPoint="1,0" EndPoint="0,0">
                <GradientStop Color="#a000" Offset="0.0"></GradientStop>
                <GradientStop Color="#0000" Offset="1.0"></GradientStop>
            </LinearGradientBrush>
        </Rectangle.Fill>
    </Rectangle>
</Grid>

Although these two grids have set alignments, the actual location on the screen will be set in code-behind because the size will also be re-calculated. Also, I need to make sure that the application is in the full display mode—if it is snapped, there is no need to display the curtains because the visible area will be reduced to an overlay grid.

The curtain resize and visibility are determined in the UpdateWindowSize method:

void DirectXPage::UpdateWindowSize()
{
    bool visibility = true;
    if (ApplicationView::Value == ApplicationViewState::Snapped)
            visibility = false;
    
    float margin = (m_renderer->m_renderTargetSize.Width - 768.0f) / 2.0f;
    if (margin < 2.0)
            visibility = false;
    
    if (visibility)
    {
        containerA->Width =  margin;
        containerA->HorizontalAlignment = Windows::UI::Xaml::HorizontalAlignment::Right;
        containerB->Width =  margin;
        containerB->HorizontalAlignment = Windows::UI::Xaml::HorizontalAlignment::Left;
        containerA->Visibility = Windows::UI::Xaml::Visibility::Visible;
        containerB->Visibility = Windows::UI::Xaml::Visibility::Visible;
    }
    else
    {
        containerA->Visibility = Windows::UI::Xaml::Visibility::Collapsed;
        containerB->Visibility = Windows::UI::Xaml::Visibility::Collapsed;
    }
}

The snippet above ensures that the curtains will only be displayed when there is extra, unused space in addition to the 768 pixels taken by the playable area.

Let’s take a look at state-specific content that is being displayed whenever the game switches states. Because of the nature of DirectX interaction, there is no way for me to hook to specific event handlers from the native loop. Therefore, I need to constantly check that the content displayed is associated with the current state.

This can be done with the help of the SwitchGameState method:

void DirectXPage::SwitchGameState()
{
    switch (m_renderer->CurrentGameState)
            {
            case GameState::GS_FULL_WIN:
                    {
                            grdCompleteWin->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            break;
                    }
            case GameState::GS_PLAYING:
                    {
                            Hud->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            break;
                    }
            case GameState::GS_MAIN_MENU:
                    {
                            stkMainMenu->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            break;
                    }
            case GameState::GS_GAME_OVER:
                    {
                            UpdateResults();
                            ResultPanel->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            grdGameOver->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            break;
                    }
            case GameState::GS_SUBMIT_SCORE:
                    {
                            grdSubmitScore->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            txtSubmitScoreView->Text = StaticDataHelper::Score.ToString();
                            break;
                    }
            case GameState::GS_TOP_SCORES:
                    {
                            grdTopScores->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            break;
                    }
            case GameState::GS_WIN:
                    {
                            UpdateResults();
                            grdWinner->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            ResultPanel->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            break;
                    }
            case GameState::GS_ABOUT_SCREEN:
                    {
                            grdAbout->Visibility =  Windows::UI::Xaml::Visibility::Visible;
                            break;
                    }
            case GameState::GS_LEVEL_SELECT_SINGLE:
                    {
                            animationBeginTime = 0;
            
                            stkLevelSelector->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            break;
                    }
            case GameState::GS_HOW_TO:
                    {
                            grdHowTo->Visibility = Windows::UI::Xaml::Visibility::Visible;
                            Storyboard^ howToInitialBoard = (Storyboard^)grdMain->Resources->Lookup("StoryboardChainA");
                            howToInitialBoard->Begin();
                            break;
                    }
        default:
                    break;
            }
    
            Storyboard^ loc = (Storyboard^)grdMain->Resources->Lookup("FadingOut");
            loc->Begin();
}

One thing you’ve probably noticed about the snippet above is the fact that there is no indicator showing that controls are being hidden when the state changes. Just calling SwitchGameState in the Update loop would cause multiple controls to be displayed at once. However, there is also the HideEverything method that goes through the visual tree and sets the Visibility to Collapsed for everything:

void DirectXPage::HideEverything()
{
    for (uint i = 0; i < grdMain->Children->Size; i++)
    {
        grdMain->Children->GetAt(i)->Visibility = Windows::UI::Xaml::Visibility::Collapsed;
    }
}

HUD Interaction

The Heads-Up Display (HUD) is used to alert the player about the current state of the game and the game character. In FallFury, it is used to display three indicators: how many buttons the character collected, how much time it took the character to get to the current level part, and current character health. It is also the container for the Pause button.

Here is the visual representation:

Here is the underlying XAML:

<Grid x:Name="Hud" VerticalAlignment="Top" Visibility="Collapsed">
    <Grid.RowDefinitions>
        <RowDefinition Height="80" />
        <RowDefinition Height="Auto" />
    </Grid.RowDefinitions>
    <Grid.ColumnDefinitions>
        <ColumnDefinition />
        <ColumnDefinition />
        <ColumnDefinition />
        <ColumnDefinition Width="80" />
    </Grid.ColumnDefinitions>

    <Rectangle Fill="Black" Grid.ColumnSpan="4" />

    <Button
        Grid.Column="0"
        x:Name="btnPause"
        Click="btnPause_Click"
        Style="{StaticResource PauseButton}">
        <Image Source="ms-appx:///Assets/HUD/pauseButton.png" Stretch="None" /
    </Button>

    <StackPanel Grid.Column="1" Orientation="Horizontal">
        <Image Source="ms-appx:///Assets/HUD/buttonHud.png" Stretch="None" />

        <TextBlock
            x:Name="txtButtons"
            Text="0"
            Style="{StaticResource hudResult}"/>
    </StackPanel>

    <StackPanel Grid.Column="2" Orientation="Horizontal">
        <Image Source="ms-appx:///Assets/HUD/clockHud.png" Stretch="None" />

        <TextBlock
            x:Name="txtTimer"
            Text="00:00"
            Style="{StaticResource hudResult}"/>
    </StackPanel>

    <Image
        Grid.Column="3"
        Source="ms-appx:///Assets/HUD/heartHud.png"
        Stretch="None" />

    <Grid
        Grid.Column="3"
        Grid.Row="1">
        <Rectangle Fill="Black" Opacity=".8"/>
        <controls:HealthBar x:Name="healthBar"></controls:HealthBar>
    </Grid>
</Grid>

As I mentioned before, the DirectX layer cannot directly interact with the XAML layer. Therefore, there needs to be an intermediary binding class. As the game progress changes, the UpdateHUD method is called, taking a reference to the current DirectX screen and reading the game data from the character and the StaticDataHelper class, which is the container for the time elapsed:

void DirectXPage::UpdateHud(GamePlayScreen^ playScreen)
{
    healthBar->Update(playScreen->GameBear->CurrentHealth, playScreen->GameBear->MaxHealth);
    
    txtButtons->Text = StaticDataHelper::ButtonsCollected.ToString();
    
    // Find a better built-in string formatting code
    if (m_renderer->CurrentGameState == GameState::GS_PLAYING && !(StaticDataHelper::IsPaused) && playScreen->IsLevelLoaded)
    {
        StaticDataHelper::SecondsTotal += m_timer->Delta;
        
        txtTimer->Text = StaticDataHelper::GetTimeString((int)StaticDataHelper::SecondsTotal);
    }
}

As seen above, part of the HUD is taken by a health indicator control—HealthBar. It is a simple composite element made out of two overlaying rectangles, one of which is resized as the bear health changes:

<UserControl
    x:Class="Coding4Fun.FallFury.Controls.HealthBar"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:FallFury"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d"
    d:DesignHeight="410"
    d:DesignWidth="20" Height="410" Width="20">
    
    <Grid>
        <StackPanel VerticalAlignment="Top" Margin="0, 5, 0,20">
            <Rectangle Height="400" Style="{StaticResource HealthRectangle}" />
        </StackPanel>

        <StackPanel VerticalAlignment="Top" Margin="0, 5, 0, 10">
            <Rectangle x:Name="OverlayStacker" Height="0" Style="{StaticResource HealthOverlayRectangle}" />
        </StackPanel>
    </Grid>
</UserControl>

Being of a fixed size, it is fairly easy to calculate the health-to-damage ratio and display that as the size of the overlaid rectangle:

DependencyProperty^ HealthBar::_MaxHealthProperty = 
    DependencyProperty::Register("MaxHealth", 
    double::typeid,
    HealthBar::typeid, 
    nullptr);

DependencyProperty^ HealthBar::_CurrentHealthProperty = 
    DependencyProperty::Register("CurrentHealth", 
    double::typeid,
    HealthBar::typeid, 
    nullptr);

void HealthBar::Update(double currentHealth, double maxHealth)
{
    CurrentHealth = currentHealth;
    MaxHealth = maxHealth;
    
    if (CurrentHealth >= 0)
    {
        OverlayStacker->Height = 400.0 - ((400.0 * CurrentHealth) / MaxHealth);
    }
}

Conclusion

Using XAML as a part of a DirectX application does not require the developer to do a massive overhaul of the infrastructure. That said, be mindful when deciding whether to use a hybrid XAML application, as it is much easier to integrate a SwapChainBackgroundPanel with the underlying DirectX configuration from the ground up instead of trying to do so when the DirectX component is completed.

The Background

Each game screen in FallFury has a moving background that creates the illusion of a fall. The way the screen is designed, it simulates vertical parallax scrolling, as the background moves faster than the overlaid objects. There is a simple way to make background movement possible without actually having to replicate parts of the texture and render them all over again. Take a look at this image, showcasing the process:

First of all, two textures are loaded that, connected at the bottom, create the illusion of a single texture. In the beginning, texture A takes the entire screen and texture B is positioned directly underneath it, with a non-existing gap between them. To initiate the scrolling, texture A is being displaced vertically by an arbitrary number of pixels, and texture B follows it at the same pace. As texture B reaches the zero point (top of the viewport), texture A is no longer visible, therefore it is displaced vertically to be below texture B. This cycle can be repeated as many times as necessary during gameplay as well as while the user is in the menu.

Let’s take a look at the code that makes it possible, starting with the main menu screen:

First of all, you need to be aware that every game screen is automatically inheriting the properties and capabilities of base class GameScreenBase. This is the class the offers the foundation both for basic texture loading and movement:

protected private:
    // Core background textures
    Microsoft::WRL::ComPtr<ID3D11Texture2D>                    m_backgroundBlockA;
    Microsoft::WRL::ComPtr<ID3D11Texture2D>                    m_backgroundBlockB;

void MoveBackground(float velocity);

It also offers the core texture containers for the overlaid elements that are present in most screens, such as clouds:

// Overlayed clouds
Microsoft::WRL::ComPtr<ID3D11Texture2D> m_overlayA;
Microsoft::WRL::ComPtr<ID3D11Texture2D> m_overlayB;

The overlay movement is inherently dependent on the base background displacement and can be adjusted relative to the initial velocity. Let’s take a look at GameScreenBase.cpp, specifically at MoveBackground:

void GameScreenBase::MoveBackground(float velocity)
{
    if (m_backgroundPositionA <= -BACKGROUND_MIDPOINT)
        m_backgroundPositionA = m_backgroundPositionB + (BACKGROUND_MIDPOINT * 2);

    if (m_backgroundPositionB <= -BACKGROUND_MIDPOINT)
        m_backgroundPositionB = m_backgroundPositionA + (BACKGROUND_MIDPOINT * 2);

    m_backgroundPositionA -= velocity;
    m_backgroundPositionB -= velocity;
}

The BACKGROUND_MIDPOINT value is relative to the height of the background texture. As we are working with a variable screen size, given that tablets and desktops do not have the same resolution, the movement has to be adjusted accordingly. One way, however, to ensure the proper texture positioning would be to place it in relation to its previous instance. Hence, this snippet in UpdateWindowSize:

BACKGROUND_MIDPOINT = 1366.0f / 2.0f;
m_backgroundPositionA = BACKGROUND_MIDPOINT;
m_backgroundPositionB = m_backgroundPositionA * 3;

Because overlays are not necessarily a part of every screen, I am not including the MoveOverlay method in the base class. Let’s now take a look at MenuScreen.cpp. Take a look at the Load method and you will see several lines that prepare the background and overlays:

m_loader = ref new BasicLoader(Manager->m_d3dDevice.Get(), Manager->m_wicFactory.Get());

m_loader->LoadTexture("Assets\\Backgrounds\\generic_blue_a.png", &m_backgroundBlockA, nullptr);
m_loader->LoadTexture("Assets\\Backgrounds\\generic_blue_b.png", &m_backgroundBlockB, nullptr);
m_loader->LoadTexture("Assets\\Backgrounds\\cloud_overlay_a.png", &m_overlayA, nullptr);
m_loader->LoadTexture("Assets\\Backgrounds\\cloud_overlay_b.png", &m_overlayB, nullptr);

CurrentSpriteBatch->AddTexture(m_backgroundBlockA.Get());
CurrentSpriteBatch->AddTexture(m_backgroundBlockB.Get());
CurrentSpriteBatch->AddTexture(m_overlayA.Get());
CurrentSpriteBatch->AddTexture(m_overlayB.Get());

The BasicLoader class was imported from the Direct3D sprite sample. A call to LoadTexture will read the data from a DDS or PNG file and output the data in an ID3D11Texture2D object. Once loaded, the texture is added to the SpriteBatch instance associated with the screen, also declared as a part of GameScreenBase.

Depending on the screen, this procedure might have to be done for multiple textures, as you will see later in this article. Each page also has two timed loops—RenderScreen and Update. RenderScreen is responsible for taking everything from the SpriteBatch instance and showing it to the user. If you’ve used the XNA SpriteBatch before, you are aware that you need to start the drawing cycle by calling Begin and end it by calling End. The same applies here:

CurrentSpriteBatch->Begin();

CurrentSpriteBatch->Draw(
    m_backgroundBlockA.Get(),
    float2(Manager->m_windowBounds.Width / 2, m_backgroundPositionA),
    PositionUnits::DIPs,
    m_screenSize,
    SizeUnits::Pixels);

CurrentSpriteBatch->Draw(
    m_backgroundBlockB.Get(),
    float2(Manager->m_windowBounds.Width / 2 ,m_backgroundPositionB),
    PositionUnits::DIPs,
    m_screenSize,
    SizeUnits::Pixels);

if (m_showBear->IsLoaded)
    m_showBear->Render();

if (m_showMonster->IsLoaded)
    m_showMonster->Render();

CurrentSpriteBatch->Draw(
    m_overlayA.Get(),
    float2(Manager->m_windowBounds.Width/2, m_backgroundPositionA),
    PositionUnits::DIPs,
    m_screenSize,
    SizeUnits::Pixels);

CurrentSpriteBatch->Draw(
    m_overlayB.Get(),
    float2(Manager->m_windowBounds.Width / 2 ,m_backgroundPositionB),
    PositionUnits::DIPs,
    m_screenSize,
    SizeUnits::Pixels);

CurrentSpriteBatch->End();

The current overloaded Draw call gets the following items:

• The texture object
• The position where the object has to be drawn on the screen
• The way the object is positioned (can also be a normalized value or a pixel value)
• The size of the texture (stretching may occur)
• The type of the size value (in this case, I am using pixels, but can also be normalized)

The order in which the Draw calls are arranged determines the order of objects drawn on the screen. The calls on top will place texture objects at the bottom of the rendering stack, and the calls at the end will place the objects at the top of the stack.

The Gameplay Screen Background & Overlays

Let’s move on to the arguably most critical screen in the project, GamePlayScreen.cpp. There are a few nuances that differentiate it from any other screen—in particular, the background loading routine.

For every game, there is a different level type that is being used. With each level type, there is a different combination of a background and the overlay. Currently, there are four supported level types:

• Space
• Nightmare
• Magic Bean
• Dream

When the proper level is detected and the metadata is loaded from the associated XML file (more about this process later on in the series), the level textures are loaded into memory:

switch (m_currentLevelType)
{
    case LevelType::SPACE:
    {
        m_loader->LoadTexture("Assets\\Backgrounds\\generic_dark_blue_a.png", &m_backgroundBlockA, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\generic_dark_blue_b.png", &m_backgroundBlockB, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\star_overlay_a.png", &m_overlayA, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\star_overlay_b.png", &m_overlayB, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\galaxy_overlay_a.png", &m_overlayGalaxyA, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\galaxy_overlay_b.png", &m_overlayGalaxyB, nullptr);
        
        CurrentSpriteBatch->AddTexture(m_overlayGalaxyA.Get());
        CurrentSpriteBatch->AddTexture(m_overlayGalaxyB.Get());
        break;
    }
    case LevelType::NIGHTMARE:
    {
        m_loader->LoadTexture("Assets\\Backgrounds\\generic_red_a.png", &m_backgroundBlockA, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\generic_red_b.png", &m_backgroundBlockB, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\cloud_overlay_a.png", &m_overlayA, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\cloud_overlay_b.png", &m_overlayB, nullptr);
        break;
    }
    case LevelType::MAGIC_BEANS:
    {
        m_loader->LoadTexture("Assets\\Backgrounds\\generic_blue_a.png", &m_backgroundBlockA, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\generic_blue_b.png", &m_backgroundBlockB, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\cloud_overlay_a.png", &m_overlayA, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\cloud_overlay_b.png", &m_overlayB, nullptr);
        
        break;
    }
    case LevelType::DREAM:
    {
        m_loader->LoadTexture("DDS\\Levels\\Dream\\TEST_backgroundDream_01.dds", &m_backgroundBlockA, nullptr);
        m_loader->LoadTexture("DDS\\Levels\\Dream\\TEST_backgroundDream_02.dds", &m_backgroundBlockB, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\cloud_overlay_a.png", &m_overlayA, nullptr);
        m_loader->LoadTexture("Assets\\Backgrounds\\cloud_overlay_b.png", &m_overlayB, nullptr);
        break;
    }
}

This way, no unnecessary textures are loaded. A screen-based level flag allows me to control the incoming assets. At this point, the level environment does not change dynamically, so it is safe to assume that the textures should be loaded on a per-level basis.

The RenderScreen method is called in the same manner as in the menu screen, with the moving background and overlays located at the bottom of the rendering stack:

CurrentSpriteBatch->Draw(
    m_backgroundBlockA.Get(),
    float2(Manager->m_windowBounds.Width / 2, m_backgroundPositionA),
    PositionUnits::DIPs,
    m_screenSize,
    SizeUnits::Pixels);

CurrentSpriteBatch->Draw(
    m_backgroundBlockB.Get(),
    float2(Manager->m_windowBounds.Width / 2, m_backgroundPositionB),
    PositionUnits::DIPs,
    m_screenSize,
    SizeUnits::Pixels);

CurrentSpriteBatch->Draw(
    m_overlayA.Get(),
    float2(Manager->m_windowBounds.Width / 2, m_overlayPositionA),
    PositionUnits::DIPs,
    float2(768.0f, 1366.0f),
    SizeUnits::Pixels);

CurrentSpriteBatch->Draw(
    m_overlayB.Get(),
    float2(Manager->m_windowBounds.Width / 2, m_overlayPositionB),
    PositionUnits::DIPs,
    float2(768.0f, 1366.0f),
    SizeUnits::Pixels);

Character Movement

As you are now aware of the basic screen structure and how the basic rendering process is built, let’s take a look at how the main game character moves on the screen. Falling down, the teddy bear also needs to move left and right to ensure that he is able to pick up power-ups and buttons as well as avoid obstacles and enemy ammo.

There are several important considerations here. The most important one is to not assume that the user will have a specific input device. Potential movement controllers include a keyboard, the mouse, the touch screen and the accelerometer. In the most common scenarios, the desktop machines will not have an accelerometer, and the tablet computers will not have a constantly attached keyboard and a mouse. In FallFury, I decided to leverage all potential input engines and let the user choose the best option for himself.

When the current GamePlayScreen instance loads, I am attempting to get access to the system accelerometer device:

m_systemAccelerometer = Windows::Devices::Sensors::Accelerometer::GetDefault();

m_systemAccelerometer is of type Windows::Devices::Sensors::Accelerometer and is declared in GamePlayScreen.h. If an accelerometer is detected, I need to bind it to a ReadingChanged event handler that will give me the current G-force transformed into X, Y, and Z displacement:

if (m_systemAccelerometer != nullptr)
{
    m_systemAccelerometer->ReadingChanged += ref new TypedEventHandler<Accelerometer^, AccelerometerReadingChangedEventArgs^>
            (this, &GamePlayScreen::AccelerometerReadingChanged);
}

ReadingChanged itself does not participate in updating the position for the character, but rather passes the current values to m_xAcceleration, which is later used in the Update method:

void GamePlayScreen::AccelerometerReadingChanged(_In_ Accelerometer^ accelerometer, _In_ AccelerometerReadingChangedEventArgs^ args)
{
    if (StaticDataHelper::IsAccelerometerEnabled)
    {
        auto currentOrientation = DisplayProperties::CurrentOrientation;
        float accelValue;
        
        if (currentOrientation == DisplayOrientations::Portrait)
                    accelValue = args->Reading->AccelerationY;
        else if (currentOrientation == DisplayOrientations::PortraitFlipped)
                    accelValue = -args->Reading->AccelerationY;
        else if (currentOrientation == DisplayOrientations::Landscape)
                    accelValue = args->Reading->AccelerationX;
        else if (currentOrientation == DisplayOrientations::LandscapeFlipped)
                    accelValue = -args->Reading->AccelerationX;
        else
                    accelValue = 0.0f;
        
        if (StaticDataHelper::IsAccelerometerInverted)
                    m_xAcceleration = -accelValue;
        else
                    m_xAcceleration = accelValue;
    }
}

The reason for this lies in the fact that ReadingChanged is triggered at a much lower rate than the Update loop. If the character position would be adjusted through the core accelerometer event handler, the result would be choppy (“step-by-step”) movement instead of a smooth transition.

Notice, also, that depending on the screen orientation, I need to get the acceleration either on the X- or Y-axes. Since an accelerometer is detected, I am assuming that the device that’s being used is a tablet. Therefore, it can have auto-rotate enabled, which means that the reference axis (horizontal) might change depending on how the user holds the device. DisplayProperties::CurrentOrientation can give me the current orientation, whether portrait (Y acceleration value) or landscape (X acceleration value):

As with many other titles, the user may invert the movement based on the accelerometer reading. For example, if the device is tilted to the right, the character will move to the left, and vice-versa. The effect is easily achieved by negating the current reading, regardless of the axis it is relying on.

Obviously, as the character moves, there should be boundaries that restrict the movement within the context of the current game screen. To get the correct screen bounds, GameScreenBase, the foundation class for every screen, sets four properties: LoBoundX, HiBoundX, LoBoundY and HiBoundY:

LoBoundX = (rWidth - 768.0f) / 2.0f;
HiBoundX = LoBoundX + 768.0f;

LoBoundY = 0;
HiBoundY = LoBoundY + rHeight;

LoBoundX is the leftmost limit for the horizontal playable area and HiBoundX is the rightmost limit for the same horizontal area. LoBoundY and HiBoundY are responsible for carrying the limits for the vertical space. All four boundary values are relative to the screen size:

To consistently get the correct X boundary no matter the screen size, the playable area is set to a fixed width of 768 pixels. The rest of the screen is accordingly cancelled out and the remaining area split in two. The original X point (zero) is added to the curtain size and that way there is the leftmost X boundary. The Y boundaries are simply obtained from the screen height, as there are no gameplay experience restrictions in that domain.

The bear position is updated in the Update loop in the game screen. There are several parameters that need to be adjusted, such as the vertical velocity, the bear rotation on tilt and the horizontal displacement.

When the level just starts, the bear falls faster until he reaches the standard static Y point. Because of screen size differences, this should not be done by comparing the pixel distance but rather by utilizing a ratio built from the current bear Y position and the high Y boundary:

if ((GameBear->Position.y / HiBoundY) < 0.19f)
{
    GameBear->Position.y += GameBear->Velocity.y * (3.2f);
}

Assuming that the game is not paused, and thus the background is moving, the bear Y position should be adjusted relative to the current velocity set for the background scrolling. Because of how parallax scrolling works, the bear velocity has to be higher than the initial screen movement. Therefore, it is multiplied by a fixed value independent of the level played:

if (!m_isBackgroundMoving)
{
    if (GameBear->Position.y > m_screenSize.y)
    {
        Manager->CurrentGameState = GameState::GS_GAME_OVER;
    }
    else
    {
        GameBear->Position.y += GameBear->Velocity.y * 1.5f;
    }
}

In the statement above, there is also an extra condition that verifies whether the bear is below the low Y boundary. If it is, then the game is over. This is imposed by a game animation that takes the bear off the Y limits, meaning that the bear is dead.

Bear rotation is based on the current X acceleration, but the bear shouldn’t rotate 360 degrees. To limit the rotation, there is a rotation threshold set (value taken in radians) that is being checked before rotating the character:

float compositeRotation = GameBear->Rotation - (float)m_xAcceleration / 10.0f;
if (compositeRotation < m_rotationThreshold && compositeRotation > -m_rotationThreshold)
    GameBear->Rotation = compositeRotation;

The composite value lets me verify the perspective rotation without actually assigning it to the bear. If it is within the imposed threshold, only then will the bear rotation be adjusted.

Using the composite value approach is also efficient when performing the horizontal displacement adjustment. The initial value is composed of the current position and the X-based acceleration multiplied by a dynamic multiplier value to create the inertia effect:

float compositePosition = GameBear->Position.x + ((float)m_xAcceleration * m_accelerationMultiplier);

if (m_xAcceleration < 0)
    compositePosition -= 100.0f;
else
    compositePosition += 100.0f;

if (Manager->IsWithinScreenBoundaries(float2(compositePosition, GameBear->Position.y)))
{
    GameBear->Position.x += (float)m_xAcceleration * m_accelerationMultiplier;
}
else
{
    if (GameBear->Position.x > HiBoundX)
    {
        GameBear->Position.x = HiBoundX - 180.0f;
    }
    else if (GameBear->Position.x < LoBoundX)
    {
        GameBear->Position.x = LoBoundX + 180.0f;
    }
}

If the bear is attempting to hit a “wall” (screen limit), its position is set back to the one barely prior to the limit. By doing this, I am avoiding locking the character on the side of the screen.

There is something in this snippet above that you might not be aware of—a reference to Manager->IsWithinScreenBoundaries. This method belongs to the ScreenManager (ScreenManager.cpp) class—a utility class that ensures the proper screen is displayed depending on the current game mode, and also allows control of mouse actions and boundary checks:

bool ScreenManager::IsWithinScreenBoundaries(float2 position)
{
    if (position.x < CurrentGameScreen->LoBoundX || position.x > CurrentGameScreen->HiBoundX || position.y < CurrentGameScreen->LoBoundY || position.y > CurrentGameScreen->HiBoundY)
            return false;
    else
            return true;
}

You saw how the movement is accomplished with the help of the accelerometer. Let’s take a look at how a keyboard can be used to do the same thing. There is a HandleKeyInput method, exposed through the GamePlayScreen class. As a matter of fact, HandleKeyInput is wired into the GameScreenBase class, therefore if a specific combination needs to be handled outside the context of the game play screen, it can be:

void GamePlayScreen::HandleKeyInput(Windows::System::VirtualKey key)
{
    if (key == Windows::System::VirtualKey::Right)
    {
        if (GameBear->IsWithinScreenBoundaries(GameBear->Size.x, 0.0f, GetScreenBounds()))
        {
            GameBear->Direction = TurningState::RIGHT;
            
            m_xAcceleration = 0.8f;
            
            if (GameBear->Rotation >= -m_rotationThreshold)
                            GameBear->Rotation -= 0.02f;
        }
    }
    else if (key == Windows::System::VirtualKey::Left)
    {
        if (GameBear->IsWithinScreenBoundaries(-GameBear->Size.x, 0.0f, GetScreenBounds()))
        {
            GameBear->Direction = TurningState::LEFT;
            
            m_xAcceleration = -0.8f;
            
            if (GameBear->Rotation <= m_rotationThreshold)
                            GameBear->Rotation += 0.02f;
        }
    }
}

As the accelerometer is no longer influencing the rotation or displacement, both indicators have to be manipulated through key presses. There are still standard thresholds in place to limit the potential incorrect movement, but the idea remains the same. This method is being called from the XAML page, which is overlaid on top of the DirectX renders:

<SwapChainBackgroundPanel
    x:Class="Coding4Fun.FallFury.DirectXPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:Coding4Fun.FallFury" x:Name="XAMLPage"
    xmlns:controls="using:Coding4Fun.FallFury.Controls"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d" Loaded="OnLoaded" KeyDown="OnKeyDown" LayoutUpdated="XAMLPage_LayoutUpdated">

void DirectXPage::OnKeyDown(Platform::Object^ sender, Windows::UI::Xaml::Input::KeyRoutedEventArgs^ e)
{
    m_renderer->CurrentGameScreen->HandleKeyInput(e->Key);
}

When the keyboard is not available, movement can be controlled with the help of a mouse. Again, with standard event handlers, I am simply using OnPointerMoved:

void GamePlayScreen::OnPointerMoved(Windows::UI::Core::PointerEventArgs^ args)
{
    if (StaticDataHelper::IsMouseEnabled)
    {
        if (GameBear != nullptr)
        {
            m_touchCounter++;
            if (GameBear->IsWithinScreenBoundaries(GameBear->Size.x, 0.0f, GetScreenBounds()))
            {
                m_xAcceleration = (args->CurrentPoint->RawPosition.X - GameBear->Position.x) / m_screenSize.x;
            }
        }
    }
}

By using this method, the bear will accelerate to the point where the mouse cursor is located, having a higher velocity the further it is located from the cursor. Once again, this creates the inertia visualization.

Conclusion

Concluding Part 3 of the series, remember that because Windows 8 is not a tablet-only OS, some users might have different input devices. Try to accommodate as many of those as possible.

The next article in this series will focus on the XAML overlay used in FallFury.

FallFury not only relies on standard C++/C# and XAML code, but also on shaders. This article is intended for developers who are not aware of what shaders are and want to know how to use them in their projects. I will talk about creating shaders, as well as the shaders used in my project.

Check out the video for this article at http://channel9.msdn.com/Series/FallFury/Part-2-Shaders.  For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles.

Code on GPU

In simple terms, shaders are small programs that are executed on the Graphical Processing Unit (GPU) instead of the Central Processing Unit (CPU). In recent years, we’ve seen a major spike in the capabilities of graphic devices, allowing hardware manufacturers to design an execution layer tied to the GPU, therefore being able to target device-specific manipulations to a highly optimized unit. Shaders are not used for simple calculations, but rather for image processing. For example, a shader can be used to adjust image lighting or colors.

Modern GPUs give access to the rendering pipeline that allows developers to execute arbitrary image processing code. This is a step forward from a fixed-function pipeline that was present in older GPUs, where image processing tasks were integrated into the hardware unit and were only able to perform a limited set of actions, such as transforms. Unlike the rendering pipeline, the fixed-function pipeline was not programmable, therefore the developers were often tied to the hardware they used for offering specific game effects, in some cases having to resort to software-based adjustments.

Types of Shaders

There are different types of image manipulations that can be performed on a given input, and so there are different types of shaders designed to handle that. Currently, we can highlight three main shader types:

• Vertex shaders – because what the user sees on the screen is not really three-dimensional, but rather a three-dimensional simulation in a 2D space, vertex shaders translate the coordinates of a vector in 3D space in relation to the 2D frame. Vertex shaders are executed one time per vector passed to the GPU. Typically, vectors carry data related to their position and the coordinate of the bound texture as well as color. A vertex shader is able to manipulate all these properties, but there is never a situation where a new vertex is created as a result of the execution.
• Pixel shaders – these programs are executed on the GPU in relation to every passed pixel, working on a much lower level. For example, if you want specific pixels adjusted for lighting or 3D bump mapping, a pixel shader can provide the desired effect for a surface. Rarely, there are situations where only a few pixels should be re-adjusted at once. Pixel shaders are often run with an input of millions of pixels, resulting in complex effects.
• Geometry shaders – these shaders are the next progression from vertex shaders, introduced with DirectX 10. The developer is able to pass specific primitives as input and either have the output represent the modified version of what was passed to the program or have new primitives, such as triangles, be generated as a result. Geometry shaders are always executed on post-vertex processing in the rendering pipeline. When vertex shader execution is completed, geometry shaders step in, if present. Geometry shaders can be used to refine the level of detail of a specific object. For example, when an object is closer or further away from the viewport camera, the mesh would have to be refined to minimize the rendering load.

FallFury uses all these shader types as a part of the game.

Enough Talk, Let’s Code

Now that you are aware of what shaders are, let’s write some sample code and test it. It is worth mentioning that shaders are not written in a standard high-level language, but rather in a language defined by the environment the shader is used in. For Direct3D, this is the High-Level Shader Language, or HLSL. It is somewhat similar to C, but with some specific nuances.

Start by creating a test Direct3D Windows Store application:

If you create and run this sample application, you will see that the output of it is a simple 3D spinning cube:

You probably also noticed that there are two shaders that are a component part of the newly created solution: SimplePixelShader.hlsl and SimpleVertexShader.hlsl. Take a look inside the pixel shader:

struct PixelShaderInput
{
    float4 pos : SV_POSITION;
    float3 color : COLOR0;
};

float4 main(PixelShaderInput input) : SV_TARGET
{
    return float4(input.color,1.0f);
}

First of all, there is a PixelShaderInput struct[DW1] . It represents a single pixel – it has a float4 field that represents the position of the pixel and a float3 field that represents the RGB pixel color. The field itself is also marked by a predefined type, such as SV_POSITION or COLOR. This is a string that determines the use of the field. You can read more about the shader semantics here.

The strings that are prefixed with SV_ are representing system-value semantics. Those have special meaning in the pipeline during the processing stages. In the pixel shader above, SV_POSITION will always mean the pixel position.

Look at what is being returned from the pixel shader—instead of the standard float3 color indicator, you are now returning a float4, which couples the existing value, input.color, with a 1.0f float value that represents the alpha-channel value. Remember, that inside shaders the color is clamped between 0 and 1 instead of the standard 255-value limit.

Since the cube rendering mechanism is already in place, let’s experiment with the pixel shader a bit. You’ve already got the float4 color representation:

You can set any of these values to 1.0 to return the solid color, so let’s do that. Let’s render the cube green. Modify the return statement to be this:

return float4(0.0, 1.0, 0.0, 1.0);

At this point, if you will run the program you will get a rendering very similar to this:

This is a good start.. Now, let’s now take a look at the vertex shader in the project. By default, you get this:

cbuffer ModelViewProjectionConstantBuffer : register(b0)
{
    matrix model;
    matrix view;
    matrix projection;
};

struct VertexShaderInput
{
    float3 pos : POSITION;
    float3 color : COLOR0;
};

struct VertexShaderOutput
{
    float4 pos : SV_POSITION;
    float3 color : COLOR0;
};

VertexShaderOutput main(VertexShaderInput input)
{
    VertexShaderOutput output;
    float4 pos = float4(input.pos, 1.0f);
    
    pos = mul(pos, model);
    pos = mul(pos, view);
    pos = mul(pos, projection);
    output.pos = pos;
    
    // Pass through the color without modification.
    output.color = input.color;
    
    return output;
}

There are a couple of differences here compared to the above mentioned pixel shader:

• cbuffer ModelViewProjectionConstantBuffer – represents a constant buffer that contains three matrices, to which vertices are related—the model, view and projection matrices. Constant buffers are interesting structures that have been optimized to allow block-wise updates, where multiple constants are grouped in one boxed unit and can be updated simultaneously instead of having a one-by-one iterative update cycle.
• struct VertexShaderInput – vertices are passed one-by-one to the shader, and this specific structure represents the input. Notice that it carries the 3D representation of the vertex position as well as its RGB color value.
• struct VertexShaderOutput – the final processed output, which now carries the position in relation to all three matrices mentioned above as well as the processed color. Notice that the fourth value is the camera distance in the demo.
• VertexShaderOutput main(VertexShaderInput input) – this is the main function that performs the vertex processing. Initially, it performs the position adjustment relative to the camera and then uses mul, a built-in function that multiplies matrices, to represent the current vertices in the projected space.

So let’s play around with what we have in stock. Take a look at the main function and how initially, the float3-based position is transformed to contain the camera distance:

float4 pos = float4(input.pos, 1.0f);

This position is normalized in relation to the current viewport. Afterwards, the vertex is transformed in the current 3D space (relative to three matrices – model, view, and projection):

pos = mul(pos, model);
pos = mul(pos, view);

pos = mul(pos, projection);

output.pos = pos;

As with the pixel shader, these are some very basic manipulations. Let’s flip the cube upside-down. To do this, we need to apply a rotation transformation. Here is an interesting piece of advice—when working with manipulations in shaders, make sure that you know basic matrix operations.

For example, to perform a simple 2D rotation, you need to use the rotation matrix:

But since we’re in 3D space, we not only have the X and Y coordinates, which relate to the matrix above, but also the Z coordinate. Therefore, a different method should be applied. There are three fundamental matrices that can be used to rotate an object around the three possible axes:

For now, we want to rotate the cube along the X-axis. To do this, inside the main function, we need to declare a constant that represents the rotation angle, in radians:

const float angle = 1.3962634;

Looking at the matrix above, we need to find out the sine and cosine of the given angle. When working with vertex shaders, HLSL offers a built-in function, called sincos that is able to get both values from an input value:

float cosLength, sinLength;
sincos(angle, sinLength, cosLength);

Now you need to declare the rotation matrix. Again, HLSL comes with some built-in capabilities to declare matrices and set their values:

float3x3 xAxisRotation = { 
    1.0, 0.0, 0.0, // Row 1
    0, cosLength, -sinLength, // Row 2
    0, sinLength, cosLength}; // Row 3

The standard format for matrix declaration in this case follows the following pattern:

Matrix<type,size> localName

For matrix multiplication, we can once again leverage the mul function:

float3 temporaryPosition;
temporaryPosition = mul(input.pos,xAxisRotation);

Now we can normalize the position to a float4 value that also includes the camera distance:

float4 pos = float4(temporaryPosition, 1.0f);

The rest of the transformation procedures can be taken directly from the original shader, multiplying the float4 position by the model, view, and projection matrices. Your entire main function should now look like this:

VertexShaderOutput main(VertexShaderInput input)
{
    VertexShaderOutput output;

    const float angle = 1.3962634;
    float cosLength, sinLength;
    sincos(angle, sinLength, cosLength);

    float3x3 xAxisRotation = { 
        1.0, 0.0, 0.0, // Row 1
        0, cosLength, -sinLength, // Row 2
        0, sinLength, cosLength}; // Row 3

    float3 temporaryPosition;
    temporaryPosition = mul(input.pos,xAxisRotation);
    float4 pos = float4(temporaryPosition, 1.0f);

    // Transform the vertex position into projected space.
    pos = mul(pos, model);
    pos = mul(pos, view);
    pos = mul(pos, projection);
    output.pos = pos;

    // Pass through the color without modification.
    output.color = input.color;

    return output;
}

The current angle is set to 80 degrees, or 1.3962634 radians. If you run the rendering test application, the result you will get will be similar to this:

Now, let’s look at how shaders are handled on the DirectX side.

Shaders in DirectX

Open CubeRenderer.cpp. This is the source file where internal shaders are read and passed to the device to be executed. Notice one interesting aspect of the process – the shader file contents are being read first and then passed to m_d3dDevice->CreateVertexShader. m_d3dDevice is a pointer to a virtual adapter that can be used to create device-specific resources. CreateVertexShader creates a vertex shader from an already compiled shader. You can notice it from the fact that the data is read not from the initial .hlsl file, but rather from the .cso (Compiled Shader Object):

auto loadVSTask = DX::ReadDataAsync("SimpleVertexShader.cso");

As with any program, before it is passed onto the execution layer, it has to be compiled first. Visual Studio 2012 comes with a bundled HLSL compiler, fxc—the Effect Compiler Tool. The default behavior for a shader is to be compiled with the .cso file extension in the same output directory as the project, but this can be changed by setting the Object File Name in the shader properties:

You can read more about the shader compilation process here.

Looking back at the sample vertex shader that was created as a part of the project, you will notice that there is a specific input layout set for the incoming object:

struct VertexShaderInput
{
    float3 pos : POSITION;
    float3 color : COLOR0;
};

The virtual adapter is not aware of the structure layout. Therefore, the developer needs to explicitly create an internal D3D11_INPUT_ELEMENT_DESC array that will contain the type of information passed to the shader:

const D3D11_INPUT_ELEMENT_DESC vertexDesc[] = 
{
    { 
        "POSITION", 
        0, 
        DXGI_FORMAT_R32G32B32_FLOAT, 
        0, 
        0, 
        D3D11_INPUT_PER_VERTEX_DATA, 
        0 },
    {
        "COLOR", 
        0, 
        DXGI_FORMAT_R32G32B32_FLOAT, 
        0, 
        12, 
        D3D11_INPUT_PER_VERTEX_DATA, 
        0 },
};

Once the description is complete, the input needs to be assembled for processing. That’s where CreateInputLayout comes into play:

DX::ThrowIfFailed(
    m_d3dDevice->CreateInputLayout(
        vertexDesc,
        ARRAYSIZE(vertexDesc),
        fileData->Data,
        fileData->Length,
        &m_inputLayout
        )
    );

You are basically passing a shader signature to the input assembler that will perform all the consequent processing. A similar process is applied for the existing pixel shader, with the main difference being the fact that instead of CreateVertexShader, CreatePixelShader is called:

Passing Custom Parameters to Shaders

Going back to our vertex shader where we performed the rotation relative to the X-axis, you probably noticed that the angle is hard-coded and is applied to each vector in the same way. This is rarely the case, as the angle would normally come from inside the game itself, often responding to internal behavior, such as character movement or action.

To pass a parameter to the shader, you need to first of all redefine the input structure. Let’s add an angle carrier to the VertexShaderInput struct in SimpleVertexShader.hlsl:

struct VertexShaderInput
{
    float3 pos : POSITION;
    float3 color : COLOR0;
    float angle : TRANSFORM0;
};

This alone won’t do anything. Go the entry point function (main) and make sure that the angle constant is set to input.angle:

const float angle = input.angle;

Now the shader is ready, but you also need to let your game know that the vertex shader has a modified input layout. To do this, go to CubeRenderer.cpp and find the D3D11_INPUT_ELEMENT_DESC array that defines the input layout for incoming vertex shaders. Simply add an identifying item to the existing array:

const D3D11_INPUT_ELEMENT_DESC vertexDesc[] = 
{ 
    { "POSITION", 0, DXGI_FORMAT_R32G32B32_FLOAT, 0, 0, D3D11_INPUT_PER_VERTEX_DATA, 0 }, 
    { "COLOR", 0, DXGI_FORMAT_R32G32B32_FLOAT, 0, 12, D3D11_INPUT_PER_VERTEX_DATA, 0 }, 
    { "TRANSFORM", 0, DXGI_FORMAT_R32_FLOAT, 0, 24, D3D11_INPUT_PER_VERTEX_DATA, 0 } 
};

The first parameter is the pre-defined semantic used in the shader. The second parameter defines the input index. After that, use DXGI_FORMAT_R32_FLOAT to indicate that the value passed will be a standard float. Pay close attention when you specify the input offset. Notice that the two FLOAT3 values, POSITION and COLOR, take 12 bytes each —a float takes 32 bits (4 bytes), and you have a triplet, therefore the offset for TRANSFORM should be 24 bytes (two times 12 bytes from previous indicators).

Now you need to modify the internal vector descriptors that are being passed to the rendering pipe. In the sample project those are created with the help of the VertexPositionColor class, located in the CubeRenderer.h. Add a simple float field to the existing struct, so it looks like this:

struct VertexPositionColor 
{ 
    DirectX::XMFLOAT3 pos; 
    DirectX::XMFLOAT3 color; 
    float angle; 
};

In CubeRenderer.cpp, find the cubeVerticles array. For each created VertexPositionColor you are now able to add a third value representing the rotation:

auto createCubeTask = (createPSTask && createVSTask).then([this] () { 
    VertexPositionColor cubeVertices[] = 
    { 
        {XMFLOAT3(-0.5f, -0.5f, -0.5f), XMFLOAT3(0.0f, 0.0f, 0.0f), 1.38f}, 
        {XMFLOAT3(-0.5f, -0.5f, 0.5f), XMFLOAT3(0.0f, 0.0f, 1.0f), 1.38f}, 
        {XMFLOAT3(-0.5f, 0.5f, -0.5f), XMFLOAT3(0.0f, 1.0f, 0.0f), 1.38f}, 
        {XMFLOAT3(-0.5f, 0.5f, 0.5f), XMFLOAT3(0.0f, 1.0f, 1.0f), 1.38f}, 
        {XMFLOAT3( 0.5f, -0.5f, -0.5f), XMFLOAT3(1.0f, 0.0f, 0.0f), 1.38f}, 
        {XMFLOAT3( 0.5f, -0.5f, 0.5f), XMFLOAT3(1.0f, 0.0f, 1.0f), 1.38f}, 
        {XMFLOAT3( 0.5f, 0.5f, -0.5f), XMFLOAT3(1.0f, 1.0f, 0.0f), 1.38f}, 
        {XMFLOAT3( 0.5f, 0.5f, 0.5f), XMFLOAT3(1.0f, 1.0f, 1.0f), 1.38f}, 
    };

Now you will be able to pass parameters to the vertex shader without having to manually modify the shader itself.

Shaders in FallFury

There are several shaders used in FallFury, which must be selected depending on the Direct3D feature level available on the machine. A feature level determines what a video adapter can do in terms of rendering—even though Direct3D is a unified framework, it still heavily relies on how individual graphic cards can perform.

There are 5 shaders used internally for texture rendering. Specifically, there is a pixel shader, a replication vertex shader, an instancing vertex shader and geometry shaders. This could be familiar if you worked with Microsoft DirectX sprite samples before:

Given the platform limitations, geometry shaders can only be used on devices supporting Direct3D Feature Level 10. When FallFury detects that the feature level is lower than that, it has to fall back on either the instancing or the replication shaders. Some devices, such as the Microsoft Surface, are at feature level 9.1. Therefore, the replication render technique must be enforced.

Following the Microsoft DirectX Sprite Sample, it is fairly easy to simply select the correct render technique once the feature level is detected with GetFeatureLevel:

auto featureLevel = m_d3dDevice->GetFeatureLevel(); 
if (featureLevel >= D3D_FEATURE_LEVEL_10_0) 
{ 
    m_technique = RenderTechnique::GeometryShader; 
} 
else if (featureLevel >= D3D_FEATURE_LEVEL_9_3) 
{ 
    m_technique = RenderTechnique::Instancing; 
} 
else 
{ 
    m_technique = RenderTechnique::Replication; 
    if (capacity > static_cast<int>(Parameters::MaximumCapacityCompatible)) 
    { 
        // The index buffer format for feature-level 9.1 devices may only be 16 bits. 
        // With 4 vertices per sprite, this allows a maximum of (1 << 16) / 4 sprites. 
        throw ref new Platform::InvalidArgumentException(); 
    } 
}

Depending on the selected render technique, which is determined on application startup, the proper input layout is selected for each shader and is used to control the rendering pipe.

Different graphic adapters also support different shader models. You need to account for those, and before running the application, the HLSL compiler will take on the task of determining whether a proper shader component is supported by the given model. In the shader properties, make sure that you specify the used shader model, as well as the type of the shader:

Without this, you are bound to experience compile time errors that will prevent you from launching the application or deploying it to different devices.

Conclusion

Shaders are not an easy subject. I highly recommend reading The Cg Tutorial, published online for free by nVidia. Even though it describes the Cg shader language, it is virtually identical to HLSL and you will not have any problems using the practices you learned there in building HLSL shaders.

FallFury is a 2D platformer in which the player controls a falling bear, trying to avoid obstacles, dodge missiles, and destroy monsters as the bear falls. The project incorporates several of the new Windows 8 APIs, including the accelerometer and touch as well as integrations with core OS capabilities such as settings and share charms. Additionally, the project leverages the most exacting addition to the Visual Studio development environment—hybrid application development with XAML, C++, and DirectX.

Check out the video for this article at http://channel9.msdn.com/Series/FallFury/Part-1-Introduction.  For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles.

Design & Idea

From the outset, Rick Barraza and I decided that since our target audience was composed of both kids and adults, the main character had to be familiar to both groups. Teddy bears turned out to be the best choice. Rick spent a day creating tens of potential bear drawings—out of which I had to choose one:

As the bear’s fall progresses, the character encounters a variety of obstacles dependent on the level type and complexity. Those obstacles should of course be avoided, so as the user tilts the device, the character in the game moves in the associated direction.

The design of the project took a week, and during this time the following items were determined and conceptualized. Considerations for both tablet and desktop environments directed our decisions:

· The main character layout.

· The way the game progresses as the character falls down.

· How the user interacts with the game in a wide variety of possible scenarios.

· What the game screens look like.

· What the menu system interaction looks like.

· How the game looks in different screen modes and on different device types.

· Some of the bonuses that the main character can pick up during free fall.

· What happens when the user progresses through the game and iterates through levels.

· What is shared and how this is accomplished.

As the game ideas were outlined, Arturo Toledo, the designer behind ux.artu.tv, was brought on to create the game assets. Then the core design decisions were made and we jumped into the development process.

Beginning the development

You will need to download and install Microsoft Visual Studio 2012 Express for Windows 8 in order to be able to follow the steps that I am describing in this series. The development has to be done on Windows 8, because the end result is a Windows Store application that relies on Windows 8 APIs. Though specific hardware is not required, both ARM and x86 devices will work well, so whether you have a Microsoft Surface RT or Samsung Series 7 slate, you will be able to test the code when you have the opportunity.

To get started, open Visual Studio 2012 and select the C++ project types. You will notice that there are several options you can choose from. You want to create a Windows Store application, so choose the appropriate category. Windows Store applications run in a sandbox, outside the boundaries of the standard .NET runtime.

Next, select the Direct2D (XAML) app type from the project list. This is a new application type introduced in Visual Studio 2012 that allows developers to combine native DirectX graphics with XAML overlays. Do not be confused by the fact that there is a Direct2D in the name—you can still invoke DirectX capabilities supported in the WinRT sandbox:

At this point you might be wondering, why choose a hybrid application instead of a fully-native project? The reason behind this decision is that is allows the developer to focus more on fine-tuning the gameplay instead of creating the UI core in pure DirectX. Because XAML is a part of the game, we can create dynamic UI layouts for the HUD, settings charm and menus without touching the graphical backbone. It is possible to do the same directly through DirectX and was a perfect approach for FallFury, considering the time constraints and the fact that I needed minimal UI overlays. Most of the graphics were already processed through the DirectX pipeline, so I did not have to invest significant resources and time into designing low-level structures for the interactivity layer.

FallFury uses XAML for the following:

• Menus – depending on the screen, the user is able to trigger a number of actions. For example, when the game starts, the user might select the New Game option or decide to take a look at the About screen. In Paused mode, the menu is used to resume the game, adjust settings, or possibly skip a level.
• Game HUD (score indicator, pick-up indicator, health indicator) – during the gameplay, the user is interested in keeping track of where the character is and what is the state of it. The game HUD is shown in active game mode.
• Settings charm extensions – the way the Settings charm works, the OS provides the core harness to hook to the Settings popup. Once shown, it is up to the developer to provide a multitude of options that customize the application behavior; any additional popups shown on selection should be designed individually in XAML.
• User notification – when something happens that can potentially affect the gameplay, the user should be alerted. The core framework provides the capabilities to use a MessageDialog, but in some cases it might not be enough. For example, if new levels are available for download, the user might want to check those out in a custom popup including full previews rather than just text.

When you create the project, a default infrastructure that prints text on the screen—both through Direct2D and XAML—is available. Direct2D is a subset of DirectX APIs and facilitates hardware-accelerated 2D graphics processing. It is used to create basic geometry elements and text. Since I am here working mostly with XAML, I am not going to cover Direct2D in-depth in this article:

I will go into more details regarding the XAML and DirectX interaction model later in the series, but for now, take a look at which parts of the project you have available. First and foremost, you probably notice the combination of both C++ source and header files and DirectXPage.xaml. Starting with Visual Studio 2012, you are now able to create XAML applications in C++. So even if you are an experienced C++ developer who never worked with the Extensible Application Markup Language, you can create the product core in your familiar environment and either import existing XAML structures or delegate the XAML writing to a designer.

If you open the XAML page, you will notice one significant change that you haven’t experienced in standard XAML applications, such as WPF or Silverlight for Windows Phone:

<SwapChainBackgroundPanel x:Name="SwapChainPanel" PointerMoved="OnPointerMoved" PointerReleased="OnPointerReleased">
    <TextBlock x:Name="SimpleTextBlock" HorizontalAlignment="Center" FontSize="42" Height="72" Text="Hello, XAML!" Margin="0,0,0,50"/>
</SwapChainBackgroundPanel>

A SwapChainBackgroundPanel is a component that lets the developer overlay XAML on top of the core DirectX-based experience. Look, for example, at the FallFury main menu as the game runs in landscape mode:

The menu buttons, the label and the side curtains are designed and rendered entirely in XAML. The clouds in the background, as well as the teddy bear, are rendered directly through the DirectX stack. The end-user does not notice any difference in the way these elements interact or are displayed. From a development perspective, however, there are several conditions that must be met.

There can only be one instance of SwapChainBackgroundPanel per app. Therefore, you can have only one overlaid XAML controls set. This does not mean that you can’t have multiple controls, but it implies that to do so you have to implement a control management flow that handles content adaptation. For example, if I invoke the pause state in the game, I don’t show the HUD but rather the screen-specific controls that let me resume or abandon the game and the PAUSE label. This switch needs to be handled on both the DirectX and XAML because a state change affects what is shown on the screen and what behaviors are tracked. As you will see through this series, this is not too hard to implement with a helper class that will store the global game state, that can be accessed from anywhere in the game.

When using SwapChainBackgroundPanel, remember that XAML is in all cases overlaid on top of the DirectX renders. So, no matter what controls you are using, those will always be placed on top of what DirectX shows to the user. For more details about how DirectX and XAML interoperate, check out this MSDN article.

Assets for the game

As with any other game, there is not only code involved in production—there are also sound and graphical assets that create a unique experience for the user. FallFury includes a wide variety of graphical assets designed by Toledo Design as well as audio created by David Walliman.

It is important that all graphical resource requirements are established at the very beginning of development. As I mentioned in the Design section of this article, I had to put together a list of all the game screens, power-ups, obstacles, backgrounds, character states and possible particles that were generated from a texture. That way, when the designer started creating the assets, all components blended together well and their styles were compatible with the vision of the game.

While working on the assets, Arturo Toledo created multiple variations of the same set up that showed how assets integrate in different game conditions:

As you follow this series, you will not have to create your own game assets—we at Coding4Fun decided to provide all graphical and audio assets, which you can download at http://fallfury.codeplex.com/.  We not only provided you with the final PNG and DDS files, but also with the raw assets that can be used in Microsoft Expression Design and Adobe Illustrator. Let’s take a look at what is in the package.

You will notice that the project Assets folder is split in several subfolders. All these assets are used mostly in the XAML layer or in game conditions where texture/sound processing is not necessary. There is also an additional asset folder I will discuss later.

• Backgrounds –the level backgrounds, such as the blue, purple or red sky, as well as the overlays, such as clouds that move simultaneously with the backgrounds. Each background/overlay combination is assigned to a specific level type.
• Bear – some of the bear elements that are displayed at different times in the game, such as when the game is over or when the player wins the entire level set.
• HUD – basic elements that are displayed during the game.
• Icons – the application icons, in a variety of sizes, required for a Windows Store application.
• Medals – winning players are awarded a medal. It can be golden, silver, or bronze.
• Misc – you get some branding elements as well as some graphics that are used in combination with other game items, such as particles.
• Monsters – monsters are used in the “How To” part of the game.
• Music – the long tracks used in different levels and screens.
• Sounds – short sound clips used when different game behaviors in the game are triggered. For example, when the bear gets hit, he lets out a brief cry.

All graphical assets mentioned here are PNG resources. The way I structured the game, some elements are larger than the others and I needed to take some measures to cut down on the package size. PNGs are fairly well-sized without much quality loss compared to raw images. At the beginning of the development process, all graphics were stored in DDS files.

DirectDraw Surface, or DDS, is a file format developed by Microsoft that helps developers optimize some of the graphics by avoiding re-compression performance loss. One of the benefits of using DDS files is the fact that whenever they are processed by the GPU, the amount of memory taken by them is the same as the file size of the DDS file itself. Usually, for 2D games the compression-based performance loss is not necessarily noticeable. For PNG files used on the DirectX stack, resources are allocated to decompress the texture. Not so for DDS files.

Depending on the case, DDS files can be generated on the fly. FallFury preserves relatively small textures in DDS format and larger ones, such as backgrounds, in PNG. It’s the best of both worlds. DDS files can be generated by a tool bundled with the DirectX SDK called dxtex (usually located in C:\Program Files (x86)\Microsoft DirectX SDK (June 2010)\Utilities\bin\x64):

There is also texconv that allows you to create DDS components from the command line, but we’ll explore that later in the series.

All DDS assets are located in the DDS folder in the solution. All that’s there are assets related to levels, such as obstacles, and assets related to character behavior, such as bear states and used weapons and powerups.

Unlike graphic assets that can be mocked from the very start, audio assets are a bit harder to come up with, mainly because at that point you need to be sure what the game will be like at the end. Music and effects should go together with the overall game feel. There are two types of audio components in FallFury—the music and the action-based sound effects. Music is played constantly, whether in the game or on a game screen such as the main menu, unless disabled by the user.

The way the audio engine works in FallFury, files are handled differently depending on where in the game they are used. All music is stored in MP3 files and the short sound effects are stored in uncompressed WAV files.

As a part of the new project that you are creating, replicate the folder structure for the Assets and DDS folders and add them to the solution. The way C++ project references work, you might want to switch to the “Show All Files” mode in the Solution Explorer:

Then, simply copy the folders to the solution folder itself. That way you will have all these resources as a part of the project itself and not just links to external files.

Source Control

It’s a really good idea to use source control. There are multiple options available at no cost, such as CodePlex, Assembla, and Team Foundation Service. You need source control for multiple reasons. The most important reason of them all, however, is that code will break. There were multiple situations where I changed parts of the project and all of a sudden some components stopped working. With the project hooked to a source control system, all I needed was to do a quick rollback to the previous check-in and I was good to go.

A good practice I learned from Clint Rutkas is performing atomic check-ins. When something goes wrong, it is much easier to go back to the check-in where only 20 or 30 code lines were modified from what is currently in the stack, compared to going back to the solution where you will be missing two or more entire source files.

Modular Design & Prototyping

As you are following this project creation from scratch, notice how the entire code base is modular and interchangeable. If I decide to create a new game screen, I can do so easily by inheriting from an existing base class that provides the basic harness. If I want to replace a character model, I can do so by modifying a single class without breaking the entire interaction model.

Prototyping is also a big part of FallFury and it is key that no time is wasted working on features that will have to be entirely replaced or re-written. A good example to this scenario happened just as I started writing the project code. I noticed that the Settings charm required some XAML work for secondary popups that extended from the OS-invoked layer. As I was working on a pure DirectX application, I had to create the XAML in code-behind and that was one of the experiences that could’ve been avoided in a hybrid application. During the prototyping stage it is easy to spot potential integration problems and later change parts of the project to work better together. It is much more complicated to replace project components when the core is wired-in than when you have small parts that independently show how a part of the game works.

Hardware

As you are about to start writing large amounts of code, you are probably wondering whether actual hardware is needed to test the application. The way FallFury was designed, you will be able to run it on any Windows 8 compatible machine, whether a desktop computer or a tablet. If the game is being run on the desktop, I assume that you probably do not have access to an accelerometer or a touch display. If you are running the game on the tablet, you probably do not have a physical keyboard constantly attached to it. These facts ultimately affect how you experience the game itself. From a development perspective, it is important to have multiple types of target hardware available, because the game might behave differently depending on the device configuration. But it is not required to follow this article series. As long as you have a Windows 8 machine, you are good to go.

Conclusion

Now onto 11 more articles on how to build Fall Fury!

For a complete, offline version of this series, you may download a nicely formatted PDF of all the articles!

Prior to DirectX 11.1, stereoscopic 3D required specific hardware and a custom API written for that hardware.  With DX11.1, stereoscopic 3D has been "democratized."  Any GPU that supports DirectX 11.1 can be connected to any device which supports stereoscopic 3D, be it a projector, an LCD TV, or anything else.  Plug it in and DirectX does the rest.

From the software side of things, any DX11.1 application can determine if the connected display supports stereoscopic 3D, and choose to render itself separately for the player's left and right eye.

To showcase this feature, we decided to build a very simple game that would give the illusion of depth, but be easy to explain and play.  What's easier than Pong?  So, we built the world's most over-engineered game of 3D Pong named Maelstrom.

Software

Each player setup consists of two applications: the DirectX 11.1 game written in C++, and the touch-screen controller written in C#/XAML.  Both are Windows Store applications.  Since this is a two player game, there are two instances of each application running, one per player.  All for applications are networked together using StreamSockets from the Windows Runtime.  The two controllers and player two's DirectX game connect to player one's DirectX game, which acts as the "master".  Controller input is sent here, and, once the ball and paddle positions are calculated, the data is drawn for player one and sent to player two which draws the world from the other player's perspective.

Direct3D Application

Getting Started with stereoscopic DirectX11, C++, and XAML

If you have never worked with DirectX before, it can be a little overwhelming at first. And even if you have worked with it some in the past, targeting the new Windows 8 ecosystem, along with C++ and XAML have added some additional changes in how you may have designed your solution previously.

Fortunately, the Windows Dev Center for Windows Store Apps has some great samples to get you started, and we took full advantage of them to get to speed. For a great, simple example of how to leverage the new stereoscopic feature in Direct3D 11.1, we started with Direct3D Stereoscopic Sample which shows the basic adjustments to the Render loop for toggling your virtual cameras. However, to see a great example of a simple game structure that also leverages stereoscopic rendering where available, the tutorial found at Walkthrough: a simple Windows Store game with DirectX is invaluable. Further in this article, we will dive deeper into the specifics of stereoscopic rendering in our game.

One thing to note, if you follow the link in the above Walkthrough to the original project, it will take you to a C++ only implementation of the game. Now, of course, all the DirectX game objects such as the paddle, puck and walls are all rendered using D3D. However, for HUD (Heads up Display) elements, this C++ only sample also uses DirectX exclusively. If you are coming from a managed code background, this will definitely seem like unnecessary overhead. That is because this C++ only sample was created after last year's BUILD conference in 2011 and C++ and DirectX still did not play well with XAML.

However, a few months later, the ability to nest DirectX content in a XAML project became available for true hybrid style solutions (see the article DirectX and XAML interop - Windows Store apps using C++ and DirectX for more information). After this feature was added, the simple Shooter Game referenced above had its HUD logic rewritten in XAML and posted up to Dev Center as XAML DirectX 3D shooting game sample, which shows both stereoscopic support, a simple Game Engine structure in C++ and XAML integration. At this point, we had all the starter code we needed to start writing our own game.

Game Engine

We modified the base sample to accommodate our needs.  We created specific GameObjects, such as Paddle, Puck, etc. to add the behaviors we needed.  We also added an Update and Render method to the base GameObject so that, for every frame, we could do any calculations required, and then draw the object to the screen.  This is very similar to how XNA sets up its game engine.

Game Constants

Because we were tweaking a variety of values like colors, sizes, camera locations, etc., we created a GameConstants.h header file which contains nothing but these types of values in a single location.  This made it very easy for us to quickly try out various tweaks and see the results on the next run.  Using namespaces helped keep the code a bit more manageable here as well.  Here’s a quick snippet of that file:

namespace GameConstants
{
    // bounds of the arena
    static const DirectX::XMFLOAT3 MinBound    = DirectX::XMFLOAT3( 0.0f,  0.0f,  0.0f);
    static const DirectX::XMFLOAT3 MaxBound    = DirectX::XMFLOAT3(19.0f, 10.0f, 90.0f);

    // game camera "look at" points
    static const DirectX::XMFLOAT3 LookAtP1    = DirectX::XMFLOAT3(9.5f, 5.0f, 90.0f);
    static const DirectX::XMFLOAT3 LookAtP2    = DirectX::XMFLOAT3(9.5f, 5.0f,  0.0f);

    // Waiting Room camera positions
    static const DirectX::XMFLOAT3 WaitingEyeP1                = DirectX::XMFLOAT3(GameConstants::MaxBound.x/2, GameConstants::MaxBound.y/2, GameConstants::MaxBound.z - 12.0f);
    static const DirectX::XMFLOAT3 WaitingEyeP2                = DirectX::XMFLOAT3(GameConstants::MaxBound.x/2, GameConstants::MaxBound.y/2, GameConstants::MinBound.z + 12.0f);
    static const DirectX::XMFLOAT3 WaitingEyeMjpegStation    = DirectX::XMFLOAT3(GameConstants::MaxBound.x/2, GameConstants::MaxBound.y/2, GameConstants::MinBound.z + 9.6f);

    // game camera eye position
    static const DirectX::XMFLOAT3 EyeP1        = DirectX::XMFLOAT3(GameConstants::MaxBound.x/2, GameConstants::MaxBound.y/2, GameConstants::MinBound.z - 6.0f);
    static const DirectX::XMFLOAT3 EyeP2        = DirectX::XMFLOAT3(GameConstants::MaxBound.x/2, GameConstants::MaxBound.y/2, GameConstants::MaxBound.z + 6.0f);

    static const float Paddle2Position    = MaxBound.z - 5.0f;

    namespace PaddlePower
    {
        // power level to light paddle at maximum color
        static const float Max                    = 9.0f;

        // max paddle power color...each component will be multiplied by power factor
        static const DirectX::XMFLOAT4 Color    = DirectX::XMFLOAT4(0.2f, 0.4f, 0.7f, 0.5f);
    
        // factor to multiply mesh percentage based on power
        static const float MeshPercent            = 1.2f;
    };
    
    // time to cycle powerups
    namespace Powerup
    {
        namespace Split
        {
            static const float Time                        = 10.0f;
            static const float NumTiles                    = 4;
            static const DirectX::XMFLOAT4 TileColor    = DirectX::XMFLOAT4(0.1f, 0.4f, 1.0f, 1.0f);
            static const float TileFadeUp                = 0.20f;
            static const float TileDuration                = 2.10f;
            static const float TileFadeDown                = 0.20f;
            static const float TileMeshPercent            = 2.0f;
            static const float TileDiffusePercent        = 2.0f;
        };
    };
}

Stereoscopic 3D

Direct3D must be initialized properly to support stereoscopic displays.  When the swap chains are created, an additional render target is required, such that one render target is for the left eye, and one render target is for the right eye.  Direct3D will let you know if a stereoscopic display is available, so you can create the swap chain and render targets appropriately.

With those in place, it’s simply a matter of rendering your scene twice, once per eye…that is, once per render target.

For our game this was very simple.  Our in-game camera contains two projection matrices, one representing the view from the left eye, and one from the right eye.  These are calculated when the projection parameters are set.

void Camera::SetProjParams(
    _In_ float fieldOfView,
    _In_ float aspectRatio,
    _In_ float nearPlane,
    _In_ float farPlane
    )
{
    // Set attributes for the projection matrix.
    m_fieldOfView = fieldOfView;
    m_aspectRatio = aspectRatio;
    m_nearPlane = nearPlane;
    m_farPlane = farPlane;
    XMStoreFloat4x4(
        &m_projectionMatrix,
        XMMatrixPerspectiveFovLH(
            m_fieldOfView,
            m_aspectRatio,
            m_nearPlane,
            m_farPlane
            )
        );

    STEREO_PARAMETERS* stereoParams = nullptr;
    // Update the projection matrix.
    XMStoreFloat4x4(
        &m_projectionMatrixLeft,
        MatrixStereoProjectionFovLH(
            stereoParams,
            STEREO_CHANNEL::LEFT,
            m_fieldOfView,
            m_aspectRatio,
            m_nearPlane,
            m_farPlane,
            STEREO_MODE::NORMAL
            )
        );

    XMStoreFloat4x4(
        &m_projectionMatrixRight,
        MatrixStereoProjectionFovLH(
            stereoParams,
            STEREO_CHANNEL::RIGHT,
            m_fieldOfView,
            m_aspectRatio,
            m_nearPlane,
            m_farPlane,
            STEREO_MODE::NORMAL
            )
        );
}

Depending on which eye we are rendering, we grab the appropriate projection matrix and pass it down to the vertex shader, so the final scene is rendered offset for the proper eye.

Collision Detection

If you are just starting to move into 3D modeling and programming, one of the trickier aspects of your game can be collision detection and response. Maelstrom uses primitives for all of the game elements, so our collision code was able to be a bit more straightforward compared to complex mesh collisions, but understanding a few core math concepts is still critical to grasp what the code is doing.

Fortunately, DirectX provides us with an DirectX Math Library that is able to do the serious heavy lifting, so the main complexity comes from framing the problem and learning how to apply the library.

For example, In our situation we had up to three very fast moving spheres and needed to check for wall collisions and then handle to appropriate bounce, since some of the walls would also be angled. In a 2D game, a collision detection between a sphere and an axis line is very easy. If the distance between a circle and the line is less than or equal to the radius of the sphere, they are touching. On every frame, you move your circle based on its velocity and do your collision test again. But even here, your solution may not be that easy for two reasons.

First, what if the line is angled and not lying flat on the X or Y axis? You have to find the point on the line based on the line's angle that is closest to the sphere to do your distance calculations. And if you then want it to bounce, you have to rotate the velocity of the circle by the line's angle, calculate your bounce, and then rotate back. And that's just rotated walls in 2D. When you move up to 3D, you have to take into account the surface normal  (which way the 3D plane is facing) in your calculations.

The second complexity that we needed to account for and which pops up in either 2D or 3D collision detection is travel between frames. In other words, if your ball is travelling very fast, it may have completely travelled through your collision boundary in between frames and you wouldn't notice it if you are only doing a distance / overlap check as outlined above. In our case, the pucks had the ability of travelling very fast with a speed boost, so we needed a more robust solution. Therefore, instead of implementing a simple sphere plane intersection test, we needed to create a line of motion that represented where the ball ended on the previous frame and where it currently is after it's new velocity is added to it's position. That line then needs to first be tested to see if it crosses a WallTile. If it does cross, then we know an collision has occurred between frames. We then solve for the time (t) between frames the Sphere would have first made contact to know the exact point of impact and calculate the appropriate "bounce off" direction.

The final code for a puck (or moving sphere) and wallTile collision test looks like this:

bool GameEngine::CheckWallCollision(Puck^ puck)
{
    bool isIntersect = false;
    bool wallCollision = false;

    for(unsigned int i = 0; i < m_environmentCollisionWalls.size(); i++)
    {
        WallTile^ wall = m_environmentCollisionWalls[i];
        float radius = puck->Radius();
        float signedRadius = puck->Radius();
        float contactTime = 0.0f;
        XMVECTOR contactPlanePoint = XMVectorSet(0.0f, 0.0f, 0.0f, 0.0f);
        XMVECTOR contactPuckPosition = XMVectorSet(0.0f, 0.0f, 0.0f, 0.0f);

        bool intersectsPlane = false;

        // Determine the velocity of this tick by subtracting the previous position from the proposed current position.
        // in the previous update() cycle, puck->Position() = puck->OldPosition() + ( puck->velocity * timerDelta ).
        // Therefore, this calculated velocity for the current frame movement differs from the stored velocity
        // since the stored velocity is independent of each game tick's timerDelta.
        XMVECTOR puckVectorVelocity = puck->VectorPosition() - puck->OldVectorPosition();
        float D = XMVectorGetX( XMVector3Dot( wall->VectorNormal(), wall->VectorPosition() ) );

        // Determine the distance of the puck to the plane of the wall.
        float dist = XMVectorGetX( XMVector3Dot(wall->VectorNormal(), puck->OldVectorPosition() )) - D;
        signedRadius = dist > 0 ? radius : -radius;

        // if the distance of the puck to the plane is already less than the radius, the oldPosition() was intersecting already
        if ( fabs(dist) < radius )
        {
            // The sphere is touching the plane.
            intersectsPlane = true;
            contactTime = 0.0f;
            contactPuckPosition = puck->OldVectorPosition();
            contactPlanePoint = puck->OldVectorPosition() + wall->VectorNormal()*XMVectorSet(signedRadius,signedRadius,signedRadius,1.0f);
        }
        else
        {
            // See if the time it would take to cross the plane from the oldPosition() with the current velocity falls within this game tick.
            // puckVelocityNormal is the amount of force from the velocity exerted directly toward the plane.
            float puckVelocityNormal = XMVectorGetX(XMVector3Dot(wall->VectorNormal(), puckVectorVelocity ));
            // if the puckvVelocityNormal times the distance is less than zero, a plane intersection will occur
            if ( puckVelocityNormal * dist < 0.0f )
            {
                // determine the contactTime, taking into account the shell of the sphere ( position() + radius )
                // is what will make contact, not the position alone.
                contactTime =  (signedRadius - dist) / puckVelocityNormal;
                // if the contact time is bewteen zero and one, the intersection has occured bewteen oldPosition() and position()
                if ( contactTime > 0.0f && contactTime < 1.0f )
                {
                    intersectsPlane = true;
                    // this is the position of the puck when its shell makes contact on the plane
                    contactPuckPosition = puck->OldVectorPosition() + XMVectorScale(puckVectorVelocity, contactTime);
                    // this is the position on the plane where the shell touches.
                    contactPlanePoint = contactPuckPosition - XMVectorScale(wall->VectorNormal(), signedRadius);
                }
            }
        }

        // If the puck has contacted the wall plane, determine if the point of contact falls within the wall boundary for true contact.
        if (intersectsPlane)
        {
            float Kr = 1.0f; // Kr is the coefficient of restitution. At 1.0, we have a totally elastic bounce with no dampening. At Kr = 0.0, the ball would stop at the wall.

            // Make sure the puck velocity and wall normal are facing each other
            float impact = XMVectorGetX ( XMVector3Dot ( wall->VectorNormal(), puck->VectorVelocity()) );
            if (impact < 0.0f)
            {
                wallCollision = true;

                //// bounce the vector off the plane
                XMVECTOR VectorNormal = XMVector3Dot(wall->VectorNormal(), puck->VectorVelocity())*wall->VectorNormal();
                XMVECTOR VectorTangent = puck->VectorVelocity() - VectorNormal;

                puck->Velocity(VectorTangent - (XMVectorScale(VectorNormal, Kr)));
                puck->Position(contactPuckPosition);

                int segment = (int)(puck->Position().z / GameConstants::WallSegmentDepth);
                segment = max(min(segment, GameConstants::NumWallSegments-1), 0);

                auto tiles = m_wallTiles[segment];
                WallTile^ tile = tiles[i];
                if(tile->GetPowerup() == Powerup::Split)
                    SplitPucks();

                break;
            }
        }
    }

    return wallCollision;
}

Drawing Maelstrom

To draw the game, we wanted to use some advanced techniques. We decided to go with a light pre-pass deferred rendering pipeline with normal mapping. That’s a lot of jargon but it isn’t all that complicated once you know what the jargon means, so let’s break it down.

When you draw something in 3D, there are three things that come together to determine the final color of each pixel on the screen: meshes, materials, and lights. A mesh is a collection of triangles that make up a game object (such as a wall tile in Maelstrom). On its own, a mesh is just a bunch of dots and lines. A material makes a mesh look like something. It could be as simple as a solid color but usually it’s a texture and sometimes it’s more (the wall tiles in Maelstrom use both a texture and a normal map to define their material properties). Lastly, lights transform materials by determining how bright they should appear and what sort of tint, if any, they should have. Without lights you would either have complete darkness or you would have flat lighting (where everything has a uniform brightness and adding a tint color would uniformly tint everything on the screen).

Forward Rendering vs. Deferred Rendering vs. Light Pre-Pass Rendering

The simplest approach to drawing 3D graphics is something called forward rendering. With forward rendering, drawing consists of rendering the mesh and calculating its material and all the lights that affect the material all at the same time. The more lights you add, the more complicated your shaders become since you have to determine whether each light affects the material and if so how much. (Ok, so there’s also multi-pass forward rendering, but that has its own problems – more passes mean longer render times and thus a lower frame rate – and we wanted to keep the descriptions simple).

In the last 5 years, many games started using a technique called deferred rendering. In classic deferred rendering, there are two rendering passes. The first pass renders the positions, normals, and material values of all the meshes in the scene to something called a G-Buffer (two or more render targets); nothing is actually drawn to the screen in this first pass. The second pass uses the data from the G-Buffer (which tells us everything we need to know about the geometry that appears at each screen pixel) and combines it with the lights to create the final image that you see. By doing this, we decouple geometry and lighting. This makes it possible to add more lights to the scene with a much smaller performance impact than in forward rendering since we don’t need to create a really complex pixel shader to handle all the lights (single-pass forward rendering) or draw the geometry over and over again for each light (multi-pass forward rendering).

There are drawbacks to classic deferred rendering though. Even a minimal G-Buffer takes up quite a bit of memory and the more different types of materials you want to support, the larger the G-Buffer will need to be. Wolfgang Engel, an XNA/DirectX MVP, came up with a variation on deferred rendering which he called Light Pre-Pass Rendering. This is a three pass technique. We once again use a G-Buffer, but in this case it is smaller than the classic deferred rendering G-Buffer and can even be squeezed down to a single render target which makes it viable for graphics hardware which does not support drawing to multiple render targets at the same time.

The G-Buffer is created in the first pass by rendering all the scene geometry. It only needs to store normals and the geometry’s world position. We stored the world position of the geometry at that screen position in one render target and its normal at that screen position in second render target for simplicity.

The next pass draws the lights to a light accumulation buffer. The buffer starts out entirely dark and each light that is rendered adds brightness (and tint, if any) to the light buffer. These lighting calculations take into account the normal and world position of the geometry that is at each screen position, drawing the values from the G-Buffer, such that each light only affects the pixels it is supposed to have an impact on. In Maelstrom we ended up only using point lights (spheres of light that fade out as you get further from the light’s position), but you can use any kind of light you can imagine (spot lights and directional lights are the two other common light types). Adding more lights has a very low impact on rendering time and this kind of lighting tends to be much easier for the designer to work with since there’s no need for him or her to understand HLSL or even any complicated C++ in order to add, remove, reposition, or otherwise change any lights.

The final pass draws the geometry a second time. This time, though, all the lighting calculations are done so all we do here is just render the meshes with their appropriate materials, adjust the color values and intensities from the material based on the light buffer value, and we’re done. Each rendering style (forward, deferred, and light pre-pass) has its own benefits and drawbacks, but in this case light pre-pass was a good solution and choosing it let us show how a state-of-the-art graphics technique works.

Normal Mapping

We also incorporated normal mapping. Normal mapping makes us of a special texture (a normal map) in addition to the regular texture that a material has. Normals are values used in lighting calculations to determine how much a particular light should affect a particular pixel. If you wanted to draw a brick wall, you would typically create two triangles that lined up to form a rectangle and you would apply a texture of a brick wall to them as their material. The end result of that doesn’t look very convincing though since unlike a real brick wall there are no grooves in the mortared area between each brick since our brick and mortar is just a flat texture applied to flat triangles. We could fix this by changing from two triangles to a fully modeled mesh with actual grooves, but that would add thousands of extra vertices which would lower the frame rate.

So instead we use a normal map, which fakes it. One of the reasons that the two triangles + a brick wall texture approach doesn’t look right is because the lighting doesn’t behave correctly when compared to a real brick wall (or to a fully modeled mesh of a real brick wall). The normals point straight out perpendicular from the face of the rectangle whereas if we had the fully modeled mesh with actual grooves, the surface normals would only point straight out on the bricks themselves and they would curve along the mortared areas such that the lighting calculations would end up giving us the right levels of light and dark depending on the location and direction of the light. That’s where a normal map comes in. The normal map (which you can generate using a plugin for Adobe Photoshop or GIMP or by modeling a real brick wall in 3DSMax, Maya, or Blender which you then “bake” a normal map from) allows us to get the same lighting effect as we would with a fully modeled mesh while still keeping the simple two triangle + a brick wall texture approach that gives us really good performance for our game. There are limits to the effectiveness of normal mapping (you can’t use it to fake anything too deep and it doesn’t hold up as well if the camera can get really close to the object) but in Maelstrom it allowed us to keep the walls as simple triangles (like the two triangles + a brick wall texture example above) while making it seem like there were geometric grooves in the wall. Here’s a before and after screenshot using normal mapping:

Post-Processing Effects

We also used several post-processing effects. The first was the bloom effect. Bloom is an effect that analyzes a rendered image, identifies parts that are above a certain brightness threshold, and makes those areas brighter and adds a peripheral glow to them as well, giving it a look and feel that is similar to a neon sign or to the light cycles in the movie Tron. Here’s the same shot as above with the addition of bloom:

We also made use of two different damage effects. Whenever the player took damage, we had a reddish tinge around the edge of the screen. This was simply a full screen overlay texture that is actually white but is tinted red by the shader. It is alpha-blended over the final rendered scene and fades out over the course of a couple of seconds. Rather than fading out linearly, we use a power curve which helps to sell the effect as being more complicated than it really is.

Lastly we added in some damage particles. The particles themselves were created using a geometry shader. The vertex shader took in a series of points in world space and passed these points along to the geometry shader. The geometry shader expanded these points into two triangles by generating the missing vertices and applying the world-view-projection transformation matrix to transform the positions from world coordinates to homogeneous coordinates so that they can then be rasterized correctly by D3D and the resulting pixels passed along to the pixel shader. Once again we used a simple texture with alpha blending to simulate much more complicated geometry than we were actually drawing. In this case we also made use of a texture atlas (an image made up of smaller images) which, in conjunction with the randomizer we used to generate the initial vertices for the particles, allowed us to have several different particle textures. Like with the power curve for the damage texture, the texture atlas allowed us to make the particles seem more complex than they really were. It also let us show off the use of a geometry shader, a feature that was added in DirectX 10 and requires DirectX 10 or higher hardware.

Audio

All audio was done using the XAudio2 API.  Thankfully, we were able to get a huge head start by using some of the code from the sample project we started from.  The audio engine sets up the very basics of XAudio2, and then wraps that with a simpler API for the rest of the application to call.

We don’t have many sound effects, so we on startup, we load all sounds effects and music cues into a std::map, keyed on a SoundCue enum.  Sounds are loaded using the Media Foundation classes, and the resulting byte data of the sound (and some format information) are stored in our SoundEffect class.

void AudioEngine::Initialize()
{
    m_audio = ref new Audio();
    m_audio->CreateDeviceIndependentResources();

    m_mediaReader = ref new MediaReader();

    // Impacts
    m_soundMap[SoundCue::BallLaunch]    = LoadSound("Sounds\\Impacts\\BallLaunch.wav");
    m_soundMap[SoundCue::Buzz]          = LoadSound("Sounds\\Impacts\\Buzz.wav");
    m_soundMap[SoundCue::Impact1]       = LoadSound("Sounds\\Impacts\\Impact1.wav");
    m_soundMap[SoundCue::Impact2]       = LoadSound("Sounds\\Impacts\\Impact2.wav");

...

}

SoundEffect^ AudioEngine::LoadSound(String^ filename)
{
    Array<byte>^ soundData = m_mediaReader->LoadMedia(filename);
    auto soundEffect = ref new SoundEffect();
    soundEffect->Initialize(m_audio->SoundEffectEngine(), m_mediaReader->GetOutputWaveFormatEx(), soundData);
    return soundEffect;
}

When the game needs to play a sound, it simply calls the PlaySound method, passing in the cue to play, and the volume to play it at.  PlaySound keys into the sound map, getting the associated SoundEffect, and plays it.

void AudioEngine::PlaySound(SoundCue cue, float volume, bool loop)
{
    m_soundMap[cue]->Play(volume, loop);
}

MJPEG Cameras

To achieve the effect of seeing the opponent in stereoscopic 3D, we strapped two Axis M1014 network cameras side-by-side.  Using Brian’s MJPEG Decoder library, with a special port to Windows Runtime, individual JPEG frames were pulled off each camera, and then applied to a texture at the back of the arena.  The image from the left camera is drawn when DirectX renders the player’s left eye, and the frame from the right camera is drawn when DirectX renders the right eye.  This is a cheap and simple way to pull off live stereoscopic 3D.

void MjpegCamera::Update(GameEngine^ engine)
{
    if(m_decoderLeft != nullptr)
        UpdateTexture(m_decoderLeft->CurrentFrame, &textureLeft);

    if(m_decoderRight != nullptr)
        UpdateTexture(m_decoderRight->CurrentFrame, &textureRight);

    Face::Update(engine);
}

void MjpegCamera::Render(_In_ ID3D11DeviceContext *context, _In_ ID3D11Buffer *primitiveConstantBuffer, _In_ bool isFirstPass, int eye)
{
    if(eye == 1 && textureRight != nullptr)
        m_material->SetTexture(textureRight.Get());
    else if(textureLeft != nullptr)
        m_material->SetTexture(textureLeft.Get());

    GameObject::Render(context, primitiveConstantBuffer, isFirstPass);
}

With the distance between the cameras being about the distance of human eyes (called the intra-axial distance), the effect works pretty well!

Tablet/Controller

The Tablet controller is the touch screen that lets the player control their 3D paddle in the Game Console app. For this part of the game system, there wasn't a reason to dive deep into DirectX and C++ since the controller is neither stereoscopic or visually intense, so we kept things simple with C#.

Since the controller would also serve as our attract screen in the podium to entice potential players, we wanted to have the wait screen do something eye-catching. However, if you are moving from C# in WPF to C# and XAML in WinRT and are used to taking advantage of some of the more common "memory hoggish UX hacks" from WPF, you'll quickly find them absent in WinRT! For example, we no longer have OpacityMask, non-rectangular clipping paths or the ability to render a UIElement to a Bitmap. Our bag of UX tricks may be in need of an overhaul. However, what we do get in C# / XAML for WinRT is Z rotation, which is something we've had in Silverlight but I personally have been begging for in WPF for a long time.

Therefore, the opening animation in the controller is a procedurally generated effect that rotates PNG "blades" in 3D space, creating a very compelling effect. Here is how it works. The Blade user control is a simple canvas that displays one of a few possible blade images. The Canvas has a RenderTransform to control the scale and rotation and a PlaneProjection which allows us to rotate the blade graphic in X, Y and Z space.

<Canvas>
    <Canvas.RenderTransform>
        <ScaleTransform x:Name="scale" ScaleX="1.0" ScaleY="1.0" CenterX="0" CenterY="0"/>
    </Canvas.RenderTransform>
    <Canvas.Projection>
        <PlaneProjection x:Name="projection" CenterOfRotationX="0" CenterOfRotationY="0" CenterOfRotationZ="0" RotationX="00" RotationY="0" RotationZ="0"/>
    </Canvas.Projection>
    <Image x:Name="img0" Width="300" Height="100" Canvas.Left="-4" Canvas.Top="-12" Source="Images/s0.png"/>
</Canvas>

Each Blade is added dynamically to the Controller when the Tablet application first loads, stored in a List to have it's Update() method called during the CompositionTarget.Rendering() loop.

protected override void OnNavigatedTo(NavigationEventArgs e)
{
    canvas_blades.Children.Clear();
    _blades.Clear();
    
    for (int i = 0; i < NumBlades; i++)
    {
        Blade b = new Blade { X = 950.0, Y = 530.0 };
        int id = _rand.Next(0, 5);
        b.SetBlade(id);
        b.Speed = .1 + id * .1;
        SeedBlade(b);
        _blades.Add(b);
        canvas_blades.Children.Add(b);
    }
}

void CompositionTarget_Rendering(object sender, object e)
{
    if(_inGame)
    {
        paddle.Update();
    }
    else if(_isClosing)
    {
        foreach (Blade b in _blades)
            b.UpdateExit();
    }
    else
    {
        foreach (Blade b in _blades)
            b.Update();
    }
}

Since each Blade has been assigned an individual speed and angle of rotation along all three axis, they have a very straightforward Update function. The reason we keep the rotation values between -180 and 180 during the spinning loop is to make it easier to spin them out zero when we need them to eventually leave the screen.

public void Update()
{
   _rotX += Speed;
   _rotZ += Speed;
   _rotY += Speed;

   if (_rotX > 180) _rotX -= 360.0;
   if (_rotX < -180) _rotX += 360.0;
   if (_rotY > 180) _rotY -= 360.0;
   if (_rotY < -180) _rotY += 360.0;
   if (_rotZ > 180) _rotZ -= 360.0;
   if (_rotZ < -180) _rotZ += 360.0;

   projection.RotationX = _rotX;
   projection.RotationY = _rotY;
   projection.RotationZ = _rotZ;
}

public void UpdateExit()
{
   _rotX *= .98;
   _rotZ *= .98;
   _rotY += (90.0 - _rotY) * .1;

   projection.RotationX = _rotX;
   projection.RotationY = _rotY;
   projection.RotationZ = _rotZ;
}

Network

To continue the experiment of blending C# and C++ code, the network communication layer was written in C# as a Windows Runtime component.  Two classes are key to the system: SocketClient and SocketListener.  Player one’s game console starts a SocketListener to listen for incoming connections from each game controller, as well as player two’s game console.  Each of those use a SocketClient object to make the connection.

In either case, once the connection is made, the client and the listener sit and wait for data to be transmitted.  Data must be sent as an object which implements our IGamePacket interface.  This contains two important methods: FromDataReaderAsync and WritePacket.  These methods serialize and deserialze the byte data to/from an IGameState packet of whatever type is specified in the PacketType property.

namespace Coding4Fun.Maelstrom.Communication
{
    public enum PacketType
    {
        UserInputPacket = 0,
        GameStatePacket
    }
    
    public interface IGamePacket
    {
        PacketType Type { get; }
        
        IAsyncAction FromDataReaderAsync(DataReader reader);
        void WritePacket(DataWriter writer);
    }
}

The controllers write UserInputPackets to the game console, consisting of X,Y positions of the paddle, as well as whether the player has tapped the screen to begin.

public sealed class UserInputPacket : IGamePacket
{
    public PacketType Type { get { return PacketType.UserInputPacket; } }
    
    public UserInputCommand Command { get; set; }
    public Point3 Position { get; set; }
}

Player one’s game console writes a GameStatePacket to player' two’s game console, which consists of the positions of each paddle, each ball, the score, and which tiles are lit for the ball splitter power up.  Player two’s Update and Render methods use this data to draw the screen appropriately.

Hardware

The hardware layer of this project is responsible for two big parts.  One is a rumble effect that fires every time the player is hit, and the other is a lighting effect that changes depending on the game state.

As all good programmers do, we reused code from another project.  We leveraged the proven web server from Project Detroit for our Netduino, but with a few changes.  Here, we had static class “modules” which knew how to talk to the physical hardware, and “controllers” which handled items like a player scoring, game state animations, and taking damage.  Because the modules are static classes, we can have them referenced in multiple classes without issue.

NETMF Web Server

When a request comes in, we perform the requested operation, and then return a new line character to verify we got the request. If you don’t return any data, some clients will actually fire a second request which then can cause some odd behaviors.  The flow is as follows:

1. Parse the URL
2. Get the target controller
3. Execute the appropriate action

private static void WebServerRequestReceived(Request request)
{
    var start = DateTime.Now;
    Logger.WriteLine("Start: " + request.Url + " at " + DateTime.Now);
    
    try
    {
        var data = UrlHelper.ParseUrl(request.Url);
        
        var targetController = GetController(data);
        
        if (targetController != null)
        {
            targetController.ExecuteAction(data);
        }
    }
    catch (Exception ex0)
    {
        Logger.WriteLine(ex0.ToString());
    }
    
    request.SendResponse(NewLine);
    
    Logger.WriteLine("End: " + request.Url + " at " + DateTime.Now + " took: " + (DateTime.Now - start).Milliseconds);
}

public static IController GetController(UrlData data)
{
    if (data.IsDamage)
        return Damage;
    
    if (data.IsScore)
        return Score;
    
    if (data.IsGameState)
        return GameState;
    
    // can assume invalid
    return null;
}

Making It Shake

We used a Sparkfun MP3 trigger board, a subwoofer amplifier, and bass rumble plates to create this effect.  The MP3 board requires power, and two jumpers to cause the MP3 to play.  It has an audio jack that then gets plugged into the amplifier which powers the rumble plates.

From here, we just needed to wire a ground to the MP3 player’s ground pin, and the target pin on the MP3 player to a digital IO pin on the Netduino.  In the code, we declare it as an OutputPort and give it an initial state of true.  When we get a request, we toggle the pin on a separate thread.

private static readonly OutputPort StopMusic = new OutputPort(Pins.GPIO_PIN_D0, true);
private static readonly OutputPort Track1 = new OutputPort(Pins.GPIO_PIN_D1, true);
// .. more pins

public static void PlayTrack(int track)
{
    switch (track)
    {
        case 1:
            TogglePin(Track1);
            break;
            // ... more cases
        default:
            // stop all, invalid choice
            TogglePin(StopMusic);
            break;
    }
}

public static void Stop()
{
    TogglePin(StopMusic);
}

private static void TogglePin(OutputPort port)
{
    var t = new Thread(() =>
    {
            port.Write(false);
            Thread.Sleep(50);
            port.Write(true);
    });
    
    t.Start();
}

Lighting Up the Room

For lighting, we used some RGB Lighting strips.  The strips can change a single color and use a PWM signal to do this.  This is different than the lighting we used in Project Detroit which allowed us to individually control each LED and used SPI to communicate.  We purchased an RGB amplifier to allow a PWM signal to power a 12 volt strip.  We purchased ours from US LED Supply and the exact product was RGB Amplifier 4A/Ch for interfacing with a Micro-Controller (PWM/TTL Input).

We alter the Duty Cycle to shift the brightness of the LEDs and do this on a separate thread.  Below is a stripped down version of the lighting hardware class.

public static class RgbStripLighting
{
    private static readonly PWM RedPwm = new PWM(Pins.GPIO_PIN_D5);
    private static readonly PWM GreenPwm = new PWM(Pins.GPIO_PIN_D6);
    private static readonly PWM BluePwm = new PWM(Pins.GPIO_PIN_D9);
    
    private const int ThreadSleep = 50;
    private const int MaxValue = 100;
    
    const int PulsePurpleIncrement = 2;
    const int PulsePurpleThreadSleep = 100;
    
    private static Thread _animationThread;
    private static bool _killThread;
    
    #region game state animations
    public static void PlayGameIdle()
    {
        AbortAnimationThread();
        
        _animationThread = new Thread(PulsePurple);
        _animationThread.Start();
    }
    #endregion
        
            
    private static void PulsePurple()
    {
        while (!_killThread)
        {
            for (var i = 0; i <= 50; i += PulsePurpleIncrement)
            {
                SetPwmRgb(i, 0, i);
            }
            
            for (var i = 50; i >= 0; i -= PulsePurpleIncrement)
            {
                SetPwmRgb(i, 0, i);
            }
            
            Thread.Sleep(PulsePurpleThreadSleep);
        }
    }
    
    private static void AbortAnimationThread()
    {
        _killThread = true;
        
        try
        {
            if(_animationThread != null)
                    _animationThread.Abort();
        }
        catch (Exception ex0)
        {
            Debug.Print(ex0.ToString());
            Debug.Print("Thread still alive: ");
            Debug.Print("Killed Thread");
        }
        
        _killThread = false;
    }
    
    private static void SetPwmRgb(int red, int green, int blue)
    {
        // typically, 0 == off and 100 is on
        // things flipped however for the lighting so building this in.
        
        red = MaxValue - red;
        green = MaxValue - green;
        blue = MaxValue - blue;
        
        red = CheckBound(red, MaxValue);
        green = CheckBound(green, MaxValue);
        blue = CheckBound(blue, MaxValue);
        
        RedPwm.SetDutyCycle((uint) red);
        GreenPwm.SetDutyCycle((uint) green);
        BluePwm.SetDutyCycle((uint) blue);
        
        Thread.Sleep(ThreadSleep);
    }
    
    public static int CheckBound(int value, int max)
    {
        return CheckBound(value, 0, max);
    }
    
    public static int CheckBound(int value, int min, int max)
    {
        if (value <= min)
            value = min;
        else if (value >= max)
            value = max;
        
        return value;
    }
}

Conclusion

We built this experience over the course of around 4 to 5 weeks.  It was our first DirectX application in a very long time, and our first C++ application in a very long time.  However, we were able to pick up the new platform and language changes fairly easily and create a simple, yet fun game in that time period.

Attributions

• Music track - "FatLoad- The Bullet(no master)" by FreaK NeoS
• Sound effects + music edition - David Wallimann
• DirectX shaders - Michael McLaughlin

In early January, we were tasked with creating a unique, interactive experience for the SXSW Interactive launch party with Frog Design. We bounced around many ideas, and finally settled on a project that Rick suggested during our first meeting: boxing robots controlled via Kinect.

The theme of the opening party was Retro Gaming, so we figured creating a life size version of a classic tabletop boxing game mashed up with a "Real Steel"-inspired Kinect experience would be a perfect fit. Most importantly, since this was going to be the first big project of the new Coding4Fun team, we wanted to push ourselves to create an experience that needed each of us to bring our unique blend of hardware, software, and interaction magic to the table under an aggressively tight deadline.

Hardware

The BoxingBots had to be fit a few requirements:

1. They had to be fun
2. They had to survive for 4 hours, the length of the SXSW opening party
3. Each robot had to punch for 90 seconds at a time, the length of a round
4. They had to be life-size
5. They had to be Kinect-drivable
6. They had to be built, shipped, and reassembled for SXSW

Creating a robot that could be beaten up for 4 hours and still work proved to be an interesting problem. After doing some research on different configurations and styles, it was decided we should leverage a prior project to get a jump start to meet the deadline. We repurposed sections of our Kinect drivable lounge chair, Jellybean! This was an advantage because it contained many known items, such as the motors, motor controllers, and chassis material.  Additionally, it was strong and fast, it was modular, and the code to drive it was already written.

Jellybean would only get us part of the way there, however.  We also had to do some retrofitting to get it to work for our new project. The footprint of the base needed to shrink from 32x50 inches to 32x35 inches, while still allowing space to contain all of the original batteries, wheels, motors, motor controllers, switches, voltage adapters. We also had to change how the motors were mounted with this new layout, as well as provide for a way to easily "hot swap" the batteries out during the event. Finally, we had to mount an upper body section that looked somewhat human, complete with a head and punching arms.

 
Experimenting with possible layouts

The upper body had its own challenges, as it had to support a ton of equipment, including:

• Punching arms
• Popping head
• Pneumatic valves
• Air manifold
• Air Tank(s)
• Laptop
• Phidget interface board
• Phidget relay boards
• Phidget LED board
• Xbox wireless controller PC transmitter / receiver
• Chest plate
• LEDs
• Sensors to detect a punch

Brian and Rick put together one of the upper frames

Punching and Air Tanks

We had to solve the problem of getting each robot to punch hard enough to register a hit on the opponent bot while not breaking the opponent bot (or itself). Bots also had to withstand a bit of side load in case the arms got tangled or took a side blow. Pneumatic actuators provided us with a lot of flexibility over hydraulics or an electrical solution since they are fast, come in tons of variations, won't break when met with resistance, and can fine tuned with a few onsite adjustments.

To provide power to the actuators, the robots had two 2.5 gallon tanks pressurized to 150psi, with the actuators punching at ~70psi.  We could punch for about five 90-second rounds before needing to re-pressurize the tanks.  Pressurizing the onboard tanks was taken care of by a pair of off-the-shelf DeWalt air compressors.

The Head

It wouldn’t be a polished game if the head didn’t pop up on the losing bot, so we added another pneumatic actuator to raise and lower the head, and some extra red and blue LEDs. This pneumatic is housed in the chest of the robot and is triggered only when the game has ended.

To create the head, we first prototyped a concept with cardboard and duct tape. A rotated welding mask just happened to provide the shape we were going for on the crown, and we crafted each custom jaw with a laser cutter.  We considered using a mold and vacuum forming to create something a bit more custom, but had to scrap the idea due to time constraints.

Sensors

Our initial implementation for detecting punches failed due to far too many false positives. We thought using IR distance sensors would be a good solution since we could detect a “close” punch and tell the other robot to retract the arm before real contact. The test looked promising, but in practice, when the opposite sensors were close, we saw a lot of noise in the data. The backup and currently implemented solution was to install simple push switches in the chest and detect when those are clicked by the chest plate pressing against them.

Power

Different items required different voltages. The motors and pneumatic valves required 24V, the LEDs required 12V and the USB hub required 5V. We used Castle Pro BEC converters to step down the voltages. These devices are typically used in RC airplanes and helicopters.

Shipping

So how does someone ship two 700lb robots from Seattle to Austin? We did it in 8 crates. . The key thing to note is that the tops and bottoms of each robot were separated. Any wire that connected the two parts had to be able to be disconnected in some form. This affected the serial cords and the power cords (5V, 12V, 24V).

Software

The software and architecture went through a variety of iterations during development. The final architecture used 3 laptops, 2 desktops, an access point, and a router. It's important to note that the laptops of Robot 1 and Robot 2 are physically mounted on the backs of each Robot body, communicating through WiFi to the Admin console. The entire setup looks like the following diagram:

Admin Console

The heart of the infrastructure is the Admin Console. Originally, this was also intended to be a scoreboard to show audience members the current stats of the match, but as we got further into the project, we realized this wouldn't be necessary. The robots are where the action is, and people's eyes focus there. Additionally, the robots themselves display their current health status via LEDs, so duplicating this information isn't useful. However, the admin side of this app remains.

Sockets

The admin console is the master controller for the game state and utilizes socket communication between it, the robots, and the user consoles. A generic socket handler was written to span each computer in the setup. The SocketListener object allows for incoming connections to be received, while the SocketClient allows clients to connect to those SocketListeners. These are generic objects, which must specify objects of type GamePacket to send and receive:

public class SocketListener<TSend, TReceive> where TSend : GamePacket                                                
                                             where TReceive : GamePacket, new()

GamePacket is a base class from which specific packets inherit:

public abstract class GamePacket
{
    public byte[] ToByteArray() 
    {   
        MemoryStream ms = new MemoryStream();           
        BinaryWriter bw = new BinaryWriter(ms);
                
        try
        {           
            WritePacket(bw);
        }
        catch(IOException ex)
        {
            Debug.WriteLine("Error writing packet: " + ex);
        }
        
        return ms.ToArray();
    }
        
    public void FromBinaryReader(BinaryReader br)
    {        
        try
        {    
            ReadPacket(br);       
        }
        catch(IOException ex)       
        {
            Debug.WriteLine("Error reading packet: " + ex);
        }
    }
    
    public abstract void WritePacket(BinaryWriter bw);
    public abstract void ReadPacket(BinaryReader br);
}

For example, in communication between the robots and the admin console, GameStatePacket and MovementDescriptorPacket are sent and received. Each GamePacket must implement its own ReadPacket and WritePacket methods to serialize itself for sending across the socket.

Packets are sent between machines every "frame". We need the absolute latest game state, robot movement, etc. at all times to ensure the game is functional and responsive.

As is quite obvious, absolutely no effort was put into making the console "pretty". This is never seen by the end users and just needs to be functional. Once the robot software and the user consoles are started, the admin console initiates connections to each of those four machines. Each machine runs the SocketListener side of the socket code, while the Admin console creates four SocketClient objects to connect to each those. Once connected, the admin has control of the game and can start, stop, pause, and reset a match by sending the appropriate packets to everyone that is connected.

Robot

The robot UI is also never intended to be seen by an end user, and therefore contains only diagnostic information.

Each robot has a wireless Xbox 360 controller connected to it so it can be manually controlled. The UI above reflects the positions of the controller sticks and buttons. During a match, it's possible for a bot to get outside of our "safe zone". One bot might be pushing the other, or the user may be moving the bot toward the edge of the ring. To counter this, the player's coach can either temporarily move the bot, turning off Kinect input, or force the game into "referee mode" which pauses the entire match and turns off Kinect control on both sides. In either case, the robots can be driven with the controllers and reset to safe positions. Once both coaches signal that the robots are reset, the admin can then resume the match.

Controlling Hardware

Phidget hardware controlled our LEDs, relays, and sensors. Getting data out of a Phidget along with actions, such as opening and closing a relay, is shockingly easy as they have pretty straightforward C# APIs and samples, which is why they typically are our go-to product for projects like this.

Below are some code snippets for the LEDs, relays, and sensor.

LEDs – from LedController.cs

This is the code that actually updates the health LEDs in the robot's chest. The LEDs were put on the board in a certain order to allow this style of iteration. We had a small issue of running out of one color of LEDs so we used some super bright ones and had to reduce the power levels to the non-super bright LEDs to prevent possible damage:

private void UpdateLedsNonSuperBright(int amount, int offset, int brightness)
{
    for (var i = offset; i < amount + offset; i++)
    {
        _phidgetLed.leds[i] = brightness / 2;
    }
}

private void UpdateLedsSuperBright(int amount, int offset, int brightness)
{
    for (var i = offset; i < amount + offset; i++)
    {
        _phidgetLed.leds[i] = brightness;
    }
}

Sensor data – from SensorController.cs

This code snippet shows how we obtain the digital and analog inputs from the Phidget 8/8/8 interface board:

public SensorController(InterfaceKit phidgetInterfaceKit) : base(phidgetInterfaceKit)
{    
    PhidgetInterfaceKit.ratiometric = true;
}

public int PollAnalogInput(int index)
{
    return PhidgetInterfaceKit.sensors[index].Value;
}

public bool PollDigitalInput(int index)
{
    return PhidgetInterfaceKit.inputs[index];
}

Relays – from RelayController.cs

Electrical relays fire our pneumatic valves. These control the head popping and the arms punching. For our application, we wanted the ability to reset the relay automatically. When the relay is opened, an event is triggered and we create an actively polled thread to validate whether we should close the relay. The reason why we actively poll is someone could be quickly toggling the relay. We wouldn't want to close it on accident. The polling and logic does result in a possible delay or early trigger for closing the relay, but for the BoxingBots the difference of 10ms for a relay closing is acceptable:

public void Open(int index, int autoCloseDelay)
{   
    UseRelay(index, true, autoCloseDelay);
}

public void Close(int index)
{
    UseRelay(index, false, 0);
}

private void UseRelay(int index, bool openRelay, int autoCloseDelay)
{
    AlterTimeDelay(index, autoCloseDelay);
    PhidgetInterfaceKit.outputs[index] = openRelay;
}

void _relayController_OutputChange(object sender, OutputChangeEventArgs e)
{
    // closed
    if (!e.Value)
        return;

    ThreadPool.QueueUserWorkItem(state =>
    {                                          
        if (_timeDelays.ContainsKey(e.Index))
        {                                              
            while (_timeDelays[e.Index] > 0)                       
            {
                Thread.Sleep(ThreadTick);           
                _timeDelays[e.Index] -= ThreadTick;
            }                                            
        }
                                            
        Close(e.Index);
                                        
    });
}

public int GetTimeDelay(int index)
{
    if (!_timeDelays.ContainsKey(index))
        return 0;
    return _timeDelays[index];
}

public void AlterTimeDelay(int index, int autoCloseDelay)
{
    _timeDelays[index] = autoCloseDelay;
}

User Console

Since the theme of the party was Retro Gaming, we wanted to go for an early 80's Sci-fi style interface, complete with starscape background and solar flares! We wanted to create actual interactive elements, though, to maintain the green phosphor look of early monochrome monitors. Unlike traditional video games, however, the screens are designed not as the primary focus of attention, but rather to help calibrate the player before the round and provide secondary display data during the match. The player should primarily stay focused on the boxer during the match, so the interface is designed to sit under the players view line and serve as more of a dashboard during each match.

However, during calibration before each round, it is important to have the player understand how their core body will be used to drive the Robot base during each round. To do this, we needed to track an average of the joints that make up each fighter's body core. We handled the process by creating a list of core joints and a variable that normalizes the metric distances returned from the Kinect sensor into a human-acceptable range of motion:

private static List<JointType> coreJoints = newList<JointType>( 
    newJointType[] {
        JointType.AnkleLeft,
        JointType.AnkleRight,
        JointType.ShoulderCenter,
        JointType.HipCenter
    });
private const double RangeNormalizer = .22;
private const double NoiseClip = .05;

And then during each skeleton calculation called by the game loop, we average the core positions to determine the averages of the players as they relate to their playable ring boundary:

public staticMovementDescriptorPacket AnalyzeSkeleton(Skeleton skeleton)
{           
    // ...

    CoreAverageDelta.X = 0.0;
    CoreAverageDelta.Z = 0.0;
    foreach (JointType jt in CoreJoints)
    {
        CoreAverageDelta.X += skeleton.Joints[jt].Position.X - RingCenter.X;
        CoreAverageDelta.Z += skeleton.Joints[jt].Position.Z - RingCenter.Z;
    }

    CoreAverageDelta.X /= CoreJoints.Count * RangeNormalizer;
    CoreAverageDelta.Z /= CoreJoints.Count * RangeNormalizer;

    // ...

    if (CoreAverageDelta.Z > NoiseClip || CoreAverageDelta.Z < -NoiseClip)
    {
        packet.Move = -CoreAverageDelta.Z;
    }

    if (CoreAverageDelta.X > NoiseClip || CoreAverageDelta.X < -NoiseClip)
    {
        packet.Strafe = CoreAverageDelta.X;
    }
}

In this way, we filter out insignificant data noise and allow the player's average core body to serve as a joystick for driving the robot around. Allowing them to lean at any angle, the move and strafe values are accordingly set to allow for a full 360 degrees of movement freedom, while at the same time not allowing any one joint to unevenly influence their direction of motion.

Another snippet of code that may be of interest is the WPF3D rendering we used to visualize the skeleton. Since the Kinect returns joint data based off of a center point, it is relatively easy to wire up a working 3D model in WPF3D off of the skeleton data, and we do this in the ringAvatar.xaml control.

In the XAML, we simply need a basic Viewport3D with camera, lights, and an empty ModelVisual3D container to hold or squares. The empty container looks like this:              

<ModelVisual3D x:Name="viewportModelsContainer2">                   
    <ModelVisual3D.Transform>
        <Transform3DGroup>           
            <RotateTransform3D x:Name="bodyRotationCenter" CenterX="0" CenterY="0" CenterZ="0">      
                <RotateTransform3D.Rotation>             
                    <AxisAngleRotation3D x:Name="myAngleRotation" Axis="0,1,0" Angle="-40"/>                 
                </RotateTransform3D.Rotation>           
            </RotateTransform3D> 
        </Transform3DGroup> 
    </ModelVisual3D.Transform>
</ModelVisual3D>

In the code, we created a generic WPF3DModel that inherits from UIElement3D and is used to store the basic positioning properties of each square. In the constructor of the object, though, we can pass a reference key to a XAML file that defines the 3D mesh to use:

public WPF3DModel(string resourceKey)
{
    this.Visual3DModel = Application.Current.Resources[resourceKey] as Model3DGroup;
}

This is a handy trick when you need to do a fast WPF3D demo and require a certain level of flexibility. To create a 3D cube for each joint when ringAvatar is initialized, we simply do this:

private readonly List<WPF3DModel> _models = new List<WPF3DModel>();
private void CreateViewportModels()
{           
    for (int i = 0; i < 20; i++)
    { 
        WPF3DModel model = new WPF3DModel("mesh_cube");
        viewportModelsContainer2.Children.Add(model);

        // ...

        _models.Add(model);
    }

    // ...
}

And then each time we need to redraw the skeleton, we loop through the skeleton data and set the cube position like so:

if (SkeletonProcessor.RawSkeleton.TrackingState == SkeletonTrackingState.Tracked)
{
    int i = 0;  

    foreach (Joint joint in SkeletonProcessor.RawSkeleton.Joints)
    {
        if (joint.TrackingState == JointTrackingState.Tracked)
        {
                                        
            _models[i].Translate(                                        
                joint.Position.X * 8.0,                                    
                joint.Position.Y * 10.0,
                joint.Position.Z * -10.0);

            i++;
        }
    }

    // ...

}

There are a few other areas in the User Console that you may want to further dig into, including the weighting for handling a punch as well dynamically generating arcs based on the position of the fist to the shoulder. However, for this experience, the User Console serves as a secondary display to support the playing experience and gives both the player and audience a visual anchor for the game.

Making a 700lb Tank Drive like a First Person Shooter

The character in a first person shooter (FPS) video game has an X position, a Y position, and a rotation vector. On an Xbox controller, the left stick controls the X,Y position. Y is the throttle (forward and backward), X is the strafing amount (left and right), and the right thumb stick moves the camera to change what you're looking at (rotation).  When all three are combined, the character can do things such as run around someone while looking at them.

In the prior project, we had existing code that worked for controlling all 4 motors at the same time, working much like a tank does, so we only had throttle (forward and back) and strafing (left and right). Accordingly, we can move the motors in all directions, but there are still scenarios in which the wheels fight one another and the base won't move. By moving to a FPS style, we eliminate the ability to move the wheels in an non-productive way and actually make it a lot easier to drive.

Note that Clint had some wiring "quirks" with polarity and which motor was left VS right, he had to correct in these quirks in software :

public Speed CalculateSpeed(double throttleVector, double strafeVector, double rotationAngle)
{
    rotationAngle = VerifyLegalValues(rotationAngle);
    rotationAngle = AdjustValueForDeadzone(rotationAngle, AllowedRotationAngle, _negatedAllowedRotationAngle);

    // flipped wiring, easy fix is here  
    throttleVector *= -1;
    rotationAngle *= -1;
    
    // miss wired, had to flip throttle and straff for calc
    return CalculateSpeed(strafeVector + rotationAngle, throttleVector, strafeVector - rotationAngle, throttleVector);
}

protected Speed CalculateSpeed(double leftSideThrottle, double leftSideVectorMultiplier, double rightSideThrottle, double rightSideVectorMultiplier) 
{ /* code from Jellybean */ }

Conclusion

The Boxing Bots project was one of the biggest things we have built to date. It was also one of our most successful projects. Though it was a rainy, cold day and night in Austin when the bots were revealed, and we had to move locations several times during setup to ensure the bots and computers wouldn't be fried by the rain, they ran flawlessly for the entire event and contestants seemed to have a lot of fun driving them.

Introduction

This app is based on an old hack from back to at least 1997 that uses the HP Job control language to change the text on the LCD status display. Some background on this hack can be found here: http://www.irongeek.com/i.php?page=security/networkprinterhacking. There are various versions of the hack code out there, and typically they all work the same way: you specify the address of the printer and the message to send, open a TCP connection to the printer over port 9100, and then send a command to update the display.

This app is a variation of that hack. It’s a tray application that periodically checks the stock price for a company and then sends a formatted message of the stock symbol and price to a specified printer.

To get the current stock price, we retrieve the data from Yahoo! through finance.yahoo.com. The data comes back in CSV format. To save a step in parsing the CSV columns, we use YQL, the Yahoo! Query Language. Yahoo! created YQL to provide a SQL-like API for querying data from various online web services. YQL! can return XML or JSON data, and we’ll take the XML and use LINQ to parse the data.

How to Use the App

The first time you run the app, the main form will appear and you'll be able to enter in the stock symbol and the IP address of your printer. Click the “Get Printer” button to view a dialog listing the available printers connected on port 9100.

There are two checkboxes. The first one is labeled “Start with Windows”. When this setting is saved, the following code is executed to tell Windows whether to start the app when user logs in:

C#

private void StartWithWindows(bool start) 
{ 
    using (RegistryKey hkcu = Registry.CurrentUser) 
    { 
        using (RegistryKey runKey = hkcu.OpenSubKey(@"Software\Microsoft\Windows\CurrentVersion\Run", true)) 
        { 
            if (runKey == null) 
                return;
 
            if (start) 
                runKey.SetValue(wpfapp.Properties.Resources.Code4FunStockPrinter, Assembly.GetEntryAssembly().Location); 
            else 
            { 
                if (runKey.GetValue(wpfapp.Properties.Resources.Code4FunStockPrinter) != null) 
                    runKey.DeleteValue(wpfapp.Properties.Resources.Code4FunStockPrinter); 
            } 
        } 
    } 
}

VB

Private Sub StartWithWindows(ByVal start As Boolean)
    Using hkcu As RegistryKey = Registry.CurrentUser
        Using runKey As RegistryKey = hkcu.OpenSubKey("Software\Microsoft\Windows\CurrentVersion\Run", True)
            If runKey Is Nothing Then
                Return
            End If

            If start Then
                runKey.SetValue(My.Resources.Code4FunStockPrinter, System.Reflection.Assembly.GetEntryAssembly().Location)
            Else
                If runKey.GetValue(My.Resources.Code4FunStockPrinter) IsNot Nothing Then
                    runKey.DeleteValue(My.Resources.Code4FunStockPrinter)
                End If
            End If
        End Using
    End Using
End Sub

The enabled checkbox is used so that you can pause the sending of the stock price to the printer without having to exit the app. When you press the “Start” button, you are prompted to save any changed settings and the app hides the main form, leaving just the system tray icon. While the app is running, it will check the stock price every 5 minutes. If the price has changed, it tells the printer to display the stock symbol and price on the display.

A DispatcherTimer object is used to determine when to check the stock price. It’s created when the main form is created and will only execute the update code when the settings have been defined and enabled.

If an unexpected error occurs, the DispatcherUnhandledException event handler will log the error to a file and alert the user:

C#

void App_DispatcherUnhandledException(object sender, System.Windows.Threading.DispatcherUnhandledExceptionEventArgs e) 
{ 
    // stop the timer 
    _mainWindow.StopPrinterHacking();

    // display the error 
    _mainWindow.LogText("Sending the stock prince to the printer was halted due to this error:" + e.Exception.ToString()); 

    // display the form 
    ShowMainForm(); 

    // Log the error to a file and notify the user 
    Exception theException = e.Exception; 
    string theErrorPath = Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData) + "\\PrinterDisplayHackError.txt"; 

    using (System.IO.TextWriter theTextWriter = new System.IO.StreamWriter(theErrorPath, true)) 
    { 
        DateTime theNow = DateTime.Now; 
        theTextWriter.WriteLine(String.Format("The error time: {0} {1}", theNow.ToShortDateString(), theNow.ToShortTimeString())); 
        while (theException != null) 
        { 
            theTextWriter.WriteLine("Exception: " + theException.ToString()); 
            theException = theException.InnerException; 
        } 
    } 

    MessageBox.Show("An unexpected error occurred. A stack trace can be found at:\n" + theErrorPath); 
    e.Handled = true; 
}

VB

Private Sub App_DispatcherUnhandledException(ByVal sender As Object, ByVal e As System.Windows.Threading.DispatcherUnhandledExceptionEventArgs)
    ' stop the timer
    _mainWindow.StopPrinterHacking()

    ' display the error
    _mainWindow.LogText("Sending the stock prince to the printer was halted due to this error:" & e.Exception.ToString())

    ' display the form
    ShowMainForm()

    ' Log the error to a file and notify the user
    Dim theException As Exception = e.Exception
    Dim theErrorPath As String = Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData) & "\PrinterDisplayHackError.txt"
    Using theTextWriter As System.IO.TextWriter = New System.IO.StreamWriter(theErrorPath, True)
        Dim theNow As Date = Date.Now
        theTextWriter.WriteLine(String.Format("The error time: {0} {1}", theNow.ToShortDateString(), theNow.ToShortTimeString()))
        Do While theException IsNot Nothing
            theTextWriter.WriteLine("Exception: " & theException.ToString())
            theException = theException.InnerException
        Loop
    End Using
    MessageBox.Show("An unexpected error occurred.  A stack trace can be found at:" & vbLf & theErrorPath)
    e.Handled = True
End Sub

The User Interface

The application currently looks like this:

Pressing the “Get Printer” button opens a dialog that looks like this:

The UI was designed with WPF and uses the basic edit controls as well as a theme from the WPF Themes project on CodePlex. On the main form, the stock symbol, printer IP address, and the check boxes using data bindings to bind each control to a custom setting are defined in the PrinterHackSettings class.

The settings are defined in a class descended from ApplicationSettingsBase. The .NET runtime will read and write the settings based on the rules defined here.

The big RichTextBog in the center of the form is used to display the last 10 stock price updates. The app keeps a queue of the stock price updates, and when the queue is updated it’s sent to the RichTextBox with the following code:

C#

public void UpdateLog(RichTextBox rtb) 
{ 
    int i = 0; 
    TextRange textRange = new TextRange(rtb.Document.ContentStart, rtb.Document.ContentEnd); 
    textRange.Text = string.Empty; 
    foreach (var lg in logs) 
    { 
        i++; 
        TextRange tr = new TextRange(rtb.Document.ContentEnd, rtb.Document.ContentEnd) { Text = String.Format("{0} : ", lg.LogTime.ToString("hh:mm:ss")) }; 
        tr.ApplyPropertyValue(TextElement.ForegroundProperty, Brushes.DarkRed); 
        tr = new TextRange(rtb.Document.ContentEnd, rtb.Document.ContentEnd) { Text = lg.LogMessage + Environment.NewLine }; 
        tr.ApplyPropertyValue(TextElement.ForegroundProperty, Brushes.Black); 
    } 
    if (i > 10) 
        logs.Dequeue(); 

    rtb.ScrollToEnd(); 
}

VB

Public Sub UpdateLog(ByVal rtb As RichTextBox)
    Dim i As Integer = 0

    For Each lg As LogEntry In logs
        i += 1
        Dim tr As New TextRange(rtb.Document.ContentEnd, rtb.Document.ContentEnd) With {.Text = String.Format("{0} : ", lg.LogTime.ToString("hh:mm:ss"))}
        tr.ApplyPropertyValue(TextElement.ForegroundProperty, Brushes.Red)

        tr = New TextRange(rtb.Document.ContentEnd, rtb.Document.ContentEnd) With {.Text = lg.LogMessage & Environment.NewLine}
        tr.ApplyPropertyValue(TextElement.ForegroundProperty, Brushes.White)
    Next lg

    If i > 10 Then
        logs.Dequeue()
    End If

    rtb.ScrollToEnd()
End Sub

Displaying a notification trace icon

WPF does not provide any functionality for running an app with just an icon in the notification area of the taskbar. We need to tap into some WinForms functionality. Add a reference to the System.Windows.Form namespace to the project. In the App.xaml file, add an event handler to the Startup event. Visual Studio will wire up an Application.Startup event in the code behind file. We can use that event to add a WinForms.NotifyIcon and wireup a context menu to it:

C#

private void Application_Startup(object sender, StartupEventArgs e) 
{ 
    _notifyIcon = new WinForms.NotifyIcon(); 
    _notifyIcon.DoubleClick += notifyIcon_DoubleClick; 
    _notifyIcon.Icon = wpfapp.Properties.Resources.Icon; 
    _notifyIcon.Visible = true; 

    WinForms.MenuItem[] items = new[] 
    { 
        new WinForms.MenuItem("&Settings", Settings_Click) { DefaultItem = true } , 
        new WinForms.MenuItem("-"), 
        new WinForms.MenuItem("&Exit", Exit_Click) 
    }; 

    _notifyIcon.ContextMenu = new WinForms.ContextMenu(items); 
    _mainWindow = new MainWindow(); 

    if (!_mainWindow.SettingsAreValid()) 
        _mainWindow.Show(); 
    else 
        _mainWindow.StartPrinterHacking(); 
}

VB

Private Sub Application_Startup(ByVal sender As Object, ByVal e As StartupEventArgs)
    _notifyIcon = New System.Windows.Forms.NotifyIcon()
    AddHandler _notifyIcon.DoubleClick, AddressOf notifyIcon_DoubleClick

    _notifyIcon.Icon = My.Resources.Icon
    _notifyIcon.Visible = True

    Dim items() As System.Windows.Forms.MenuItem = {New System.Windows.Forms.MenuItem("&Settings", AddressOf Settings_Click) With {.DefaultItem = True}, New System.Windows.Forms.MenuItem("-"), New System.Windows.Forms.MenuItem("&Exit", AddressOf Exit_Click)}

    _notifyIcon.ContextMenu = New System.Windows.Forms.ContextMenu(items)

    _mainWindow = New MainWindow()

    If Not _mainWindow.SettingsAreValid() Then
        _mainWindow.Show()
    Else
        _mainWindow.StartPrinterHacking()
    End If
End Sub

Getting the Stock Information

From the Yahoo Financial site, you get can download a CSV file for any specified stock. Here's a web site that documents the format needed to get the right fields: http://www.gummy-stuff.org/Yahoo-data.htm. We want to return the stock symbol and the last traded price. That works out to be “s” and “l1”, respectively.

If you open the following URL with a browser, a file named quotes.csv will be returned:

http://download.finance.yahoo.com/d/quotes.csv?s=MSFT&f=sl1

You should get a file like this:

The first field is the stock symbol and the second is the last recorded price. You could just read that data and parse out the fields, but we can get the data in more readable format.

Yahoo! has a tool called the YQL Console that will you let you interactively query against Yahoo! and other web service providers. While it's overkill to use on a two column CSV file, it can be used to tie together data from multiple services.

To use our MSFT stock query with YQL, we format the query like this:

select * from csv where url='http://download.finance.yahoo.com/d/quotes.csv?s=MSFT&f=sl1' and columns='symbol,price'

You can see this query loaded into the YQL Console here.

When you click the “TEST” button, the YQL query is executed and the results displayed in the lower panel. By default, the results are in XML, but you can also get the data back in JSON format.

Our result set has been transformed into the following XML:

<?xml version="1.0" encoding="UTF-8"?> 
<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="1" yahoo:created="2012-08-23T02:36:06Z" yahoo:lang="en-US"> 
    <results> 
        <row> 
            <symbol>MSFT</symbol> 
            <price>30.54</price> 
        </row> 
    </results> 
</query>

This XML document can be easily parsed in the application code. The URL listed below “THE REST QUERY” on the YQL page is the YQL query encoded so that it can be sent as a GET request. For this YQL query, we use the following URL:

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20csv%20where%20url%3D'http%3A%2F%2Fdownload.finance.yahoo.com%2Fd%2Fquotes.csv%3Fs%3DMSFT%26f%3Dsl1'%20and%20columns%3D'symbol%2Cprice'

This is the URL that our application uses to get the stock price. Notice the MSFT in bold face—we replace that hard coded stock symbol with a format item and just use String.Format() to generate the URL at run time.

To get the stock price from our code, we can wrap this with the following method:

C#

public string GetPriceFromYahoo(string tickerSymbol) 
{ 
    string price = string.Empty; 
    string url = string.Format("http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20csv%20where%20url%3D'http%3A%2F%2Fdownload.finance.yahoo.com%2Fd%2Fquotes.csv%3Fs%3D{0}%26f%3Dsl1'%20and%20columns%3D'symbol%2Cprice'", tickerSymbol); 

    try 
    { 
        Uri uri = new Uri(url); 
        HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri); 
        HttpWebResponse resp = (HttpWebResponse)req.GetResponse(); 
        XDocument doc = XDocument.Load(resp.GetResponseStream()); 
        resp.Close(); 
        var ticker =    from query in doc.Descendants("query") 
                        from results in query.Descendants("results") 
                        from row in query.Descendants("row") 
                        select new { price = row.Element("price").Value }; 

        price = ticker.First().price; 
    } 
    catch (Exception ex) 
    { 
        price = "Exception retrieving symbol: " + ex.Message; 
    } 
    return price; 
}

VB

Public Function GetPriceFromYahoo(ByVal tickerSymbol As String) As String
    Dim price As String

    Dim url As String = String.Format("http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20csv%20where%20url%3D'http%3A%2F%2Fdownload.finance.yahoo.com%2Fd%2Fquotes.csv%3Fs%3D{0}%26f%3Dsl1'%20and%20columns%3D'symbol%2Cprice'", tickerSymbol)

    Try
        Dim uri As New Uri(url)

        Dim req As HttpWebRequest = CType(WebRequest.Create(uri), HttpWebRequest)
        Dim resp As HttpWebResponse = CType(req.GetResponse(), HttpWebResponse)

        Dim doc As XDocument = XDocument.Load(resp.GetResponseStream())

        resp.Close()

        Dim ticker = From query In doc.Descendants("query") , results In query.Descendants("results") , row In query.Descendants("row") _
                     Let xElement = row.Element("price") _
                     Where xElement IsNot Nothing _
                     Select New With {Key .price = xElement.Value}

        price = ticker.First().price
    Catch ex As Exception
        price = "Exception retrieving symbol: " & ex.Message
    End Try

    Return price
End Function

While this code makes the readying of a two column CSV file more complicated than it needs to be, it makes it easier to adapt this code to read the results for multiple stock symbols and/or additional fields.

Getting the List of Printers

We are targeting a specific type of printer: those that use the HP PJL command set. Since we talk to these printers over port 9100, we only need to list the printers that listen on that port. We can use Windows Management Instrumentation (WMI) to list the printer TCP/IP addresses that are using port 9100.  The WMI class Win32_TCPIPPrinterPort can be used for that purpose, and we’ll use the following WMI query:

Select Name, HostAddress from Win32_TCPIPPrinterPort where PortNumber = 9100

This returns the list of port names and addresses on your computer that are being used over port 9100. Take that list and store it in a dictionary for a quick lookup:

C#

static public Dictionary<string, IPAddress> GetPrinterPorts()
{ 
    var ports = new Dictionary<string, IPAddress>(); 

    ObjectQuery oquery = new ObjectQuery("Select Name, HostAddress from Win32_TCPIPPrinterPort where PortNumber = 9100"); 
    ManagementObjectSearcher mosearcher = new ManagementObjectSearcher(oquery); 

    using (var searcher = new ManagementObjectSearcher(oquery)) 
    { 
        var objectCollection = searcher.Get(); 
        foreach (ManagementObject managementObjectCollection in objectCollection) 
        { 
            var portAddress = IPAddress.Parse(managementObjectCollection.GetPropertyValue("HostAddress").ToString()); 
            ports.Add(managementObjectCollection.GetPropertyValue("Name").ToString(), portAddress); 
        } 
    } 

    return ports; 
}

VB

Public Shared Function GetPrinterPorts() As Dictionary(Of String, IPAddress)
    Dim ports = New Dictionary(Of String, IPAddress)()

    Dim oquery As New ObjectQuery("Select Name, HostAddress from Win32_TCPIPPrinterPort where PortNumber = 9100")

    Dim mosearcher As New ManagementObjectSearcher(oquery)

    Using searcher = New ManagementObjectSearcher(oquery)
        Dim objectCollection = searcher.Get()

        For Each managementObjectCollection As ManagementObject In objectCollection
            Dim portAddress = IPAddress.Parse(managementObjectCollection.GetPropertyValue("HostAddress").ToString())
            ports.Add(managementObjectCollection.GetPropertyValue("Name").ToString(), portAddress)
        Next managementObjectCollection
    End Using

    Return ports
End Function

Next, we get the list of printers that this computer knows about. We could do that through WMI, but I decided to stay closer to the .NET Framework and use the LocalPrintServer class. The GetPrintQueues method returns a collection of print queues of the type PrintQueueCollection. We can then iterate through the PrintQueueCollection and look for all printers that have a port name that matches the names returned by the WMI query. That gives code that looks like this:

C#

public class LocalPrinter 
{ 
    public string Name { get; set; } 
    public string PortName { get; set; } 
    public IPAddress Address { get; set; } 
} 

static public List<LocalPrinter> GetPrinters() 
{ 
    Dictionary<string, IPAddress> ports = GetPrinterPorts(); 
    EnumeratedPrintQueueTypes[] enumerationFlags = { EnumeratedPrintQueueTypes.Local }; 

    LocalPrintServer printServer = new LocalPrintServer(); 
    PrintQueueCollection printQueuesOnLocalServer = printServer.GetPrintQueues(enumerationFlags); 

    return (from printer in printQueuesOnLocalServer 
            where ports.ContainsKey(printer.QueuePort.Name) 
            select new LocalPrinter() 
            { 
                Name = printer.Name, 
                PortName = printer.QueuePort.Name, 
                Address = ports[printer.QueuePort.Name] 
            }).ToList(); 
}

VB

Public Class LocalPrinter
    Public Property Name() As String
    Public Property PortName() As String
    Public Property Address() As IPAddress
End Class

Public Shared Function GetPrinters() As List(Of LocalPrinter)
    Dim ports As Dictionary(Of String, IPAddress) = GetPrinterPorts()

    Dim enumerationFlags() As EnumeratedPrintQueueTypes = { EnumeratedPrintQueueTypes.Local }

    Dim printServer As New LocalPrintServer()

    Dim printQueuesOnLocalServer As PrintQueueCollection = printServer.GetPrintQueues(enumerationFlags)

    Return ( _
            From printer In printQueuesOnLocalServer _
            Where ports.ContainsKey(printer.QueuePort.Name) _
            Select New LocalPrinter() With {.Name = printer.Name, .PortName = printer.QueuePort.Name, .Address = ports(printer.QueuePort.Name)}).ToList()
End Function

Sending the Stock Price to the Printer

The way to send a message to a HP display is via a PJL command. PJL stands for Printer Job Language. Not all PJL commands are recognized by every HP printer, but if you have an HP laser printer with a display, the command should work. This should work for any printer that is compatible with HP’s PJL command set. For the common PJL commands, HP has an online document here.

We will be using the “Ready message display” PJL command. All PJL commands will start and end with a sequence of bytes called the “Universal Exit Language” or UEL. This sequence tells the printer that it’s about to receive a PJL command. The UEL is defined as

<ESC>%-12345X

The format of the packet sent to the printer is be "UEL PJL command UEL". The Ready message display format is

@PJL RDYMSG DISPLAY=”message”[<CR>]<LF>

To send the command that has the printer display “Hello World”, you would send the following sequence:

<ESC>%-12345X@PJL RDYMSG DISPLAY=”Hello World”[<CR>]<LF><ESC>%-12345X[<CR>]<LF>

We wrap this up in a class called SendToPrinter and the good stuff gets executed in the Send method, as listed below:

C#

public class SendToPrinter 
{ 
    public string host { get; set; } 
    public int Send(string message) 
    { 
        IPAddress addr = null; 
        IPEndPoint endPoint = null; 

        try 
        { 
            addr = Dns.GetHostAddresses(host)[0]; 
            endPoint = new IPEndPoint(addr, 9100); 
        } 
        catch (Exception e) 
        { 
            return 1; 
        } 

        Socket sock = null; 
        String head = "\u001B%-12345X@PJL RDYMSG DISPLAY = \""; 
        String tail = "\"\r\n\u001B%-12345X\r\n"; 
        ASCIIEncoding encoding = new ASCIIEncoding(); 

        try 
        { 
            sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.IP); 
            sock.Connect(endPoint); 
            sock.Send(encoding.GetBytes(head)); 
            sock.Send(encoding.GetBytes(message)); 
            sock.Send(encoding.GetBytes(tail)); 
            sock.Close(); 
        } 
        catch (Exception e) 
        { 
            return 1; 
        } 

        int bytes = (head + message + tail).Length; 
        return 0; 
    } 
}

VB

Public Function Send(ByVal message As String) As Integer
    Dim endPoint As IPEndPoint = Nothing

    Try
        Dim addr As IPAddress = Dns.GetHostAddresses(Host)(0)

        endPoint = New IPEndPoint(addr, 9100)
    Catch
        Return 1
    End Try

    Dim startPJLSequence As String = ChrW(&H1B).ToString() & "%-12345X@PJL RDYMSG DISPLAY = """
    Dim endPJLSequence As String = """" & vbCrLf & ChrW(&H1B).ToString() & "%-12345X" & vbCrLf

    Dim encoding As New ASCIIEncoding()

    Try
        Dim sock As New Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.IP)

        sock.Connect(endPoint)

        sock.Send(encoding.GetBytes(startPJLSequence))
        sock.Send(encoding.GetBytes(message))
        sock.Send(encoding.GetBytes(endPJLSequence))

        sock.Close()
    Catch
        Return 1
    End Try

    Return 0
End Function

The Installer

The installer for this app was written with WiX, Windows Installer XML. WiX is an open source project created by Rob Mensching that lets you build Windows Installer .msi and .msm files from XML source code. I used the release candidate of WiX 3.6, but any recent version should work. Of course, you don’t need an installer if you build the app yourself.

Setting InstalScope to “perUser” designates this package as being a per-user install. Adding the property “WixAppFolder” and set to “WixPerUserFolder” tells WiX to install this app under %LOCALAPPDATA% instead of under %ProgramFiles%. This eliminates the need for the installer to request elevated rights and the UAC prompt:

<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"> 
    <Product Id="*" Name="Coding4Fun Printer Display Hack" Language="1033" Version="1.0.0.0" Manufacturer="Coding4Fun" UpgradeCode="e0a3eed3-b61f-46da-9bda-0d546d2a0622"> 
    <Package InstallerVersion="200" Compressed="yes" InstallScope="perUser" /> 
    <Property Id="WixAppFolder" Value="WixPerUserFolder" />

Because we are not touching any system settings, I eliminated the creation of a system restore point at the start of the installation process. This greatly speeds up the installation of the app, and is handled by adding a property named MSIFASTINSTALL with the value of “1”:

<Property Id="MSIFASTINSTALL" Value="1" />

I modified the UI sequence to skip over the end user license agreement. There is nothing to license here and no one reads EULAs anyways. To do this, I needed to download the WiX source code and extract a file named WixUI_Mondo.wxs. I added it to the installer project and renamed it to WixUI_MondoNoLicense.wxs. I also added a checkbox to the exit dialog to allow the user to launch the app after it been installed:

<Property Id="WIXUI_EXITDIALOGOPTIONALCHECKBOXTEXT" Value="Launch Printer Display Hack" /> 
<Property Id="WixShellExecTarget" Value="[#exe]" /> 
<CustomAction Id="LaunchApplication" BinaryKey="WixCA" DllEntry="WixShellExec" Impersonate="yes" /> 
<UI> 
    <UIRef Id="WixUI_MondoNoLicense"/> 
    <Publish Dialog="ExitDialog" Control="Finish" Event="DoAction" Value="LaunchApplication"> 
        WIXUI_EXITDIALOGOPTIONALCHECKBOX = 1 and NOT Installed 
    </Publish> 
</UI>

When you build the installer, it generates two ICE91 warning messages. An ICE91 warning occurs when you install a file or shortcut into a per-user only folder. Since we have explicitly set the InstallScope to “perUser”, we can safely ignore these two warnings. If you hate warning messages, you can use the tool settings for the installer project to suppress ICE91 validation checks:

Conclusion

I have had various versions of this app running in my office for over a year. It’s been set to show our current stock price on the main printer in the development department. It’s fun to watch people walk near the printer just to check out the current stock price.

If you want to try this out, the download link for the source code and installer is at the top of the article!

About The Author

I am a senior R&D engineer for Tyler Technologies, working on our next generation of school bus routing software. I also am the leader of the Tech Valley .NET Users Group (TVUG). You can follow me at @anotherlab and check out my blog at anotherlab.rajapet.net. I would list my G+ address, but I don’t use it. I started out with a VIC-20 and been slowly moving up the CPU food chain ever since.

I would like to thank Brian Peek on the Coding4Fun team for his encouragement and suggestions and for letting me steal large chunks of the UI code from his TweeVo project .

Introduction

In our previous article, we examined what an Arduino shield is, how to build a simple custom shield and discussed how to quickly identify shields that are good candidates for a Netduino adaptation versus shields that may not be.

In this article, we’ll take a popular Arduino Logger Shield produced by Adafruit and we’ll interface it with a Netduino / Plus microcontroller

The Arduino Logger Shield is an excellent one to start with because it offers immediate benefits to a Netduino / Plus user:

• Time-keeping
• SD card storage
• Two user-controllable LEDs
• A small prototyping area
• An onboard 3.3v voltage regulator for clean analog readings and power decoupling

In our C# data logging application, we'll interact with the time keeper, the SD card storage and its 'card detect' pin, the two LEDs as well as a temperature sensor (not included with the shield).

Before diving into the details associated with the hardware, you may want to take a look at the C# objects representing the hardware:

public static readonly string SdMountPoint = "SD";
public static OutputPort LedRed = new OutputPort(Pins.GPIO_PIN_D0, false);
public static OutputPort LedGreen = new OutputPort(Pins.GPIO_PIN_D1, false);
public static InputPort CardDetect = new InputPort(Pins.GPIO_PIN_D3, true, Port.ResistorMode.PullUp);
public static readonly Cpu.Pin ThermoCoupleChipSelect = Pins.GPIO_PIN_D2;
public static DS1307 Clock;
public static Max6675 ThermoCouple;

and their initialization:

public static void InitializePeripherals() {
    LedGreen.Write(true);
    Clock = new DS1307();
    ThermoCouple = new Max6675();
    InitializeStorage(true);
    InitializeClock(new DateTime(2012, 06, 14, 17, 00, 00));
    ThermoCouple.Initialize(ThermoCoupleChipSelect);
    TemperatureSampler = new Timer(new TimerCallback(LogTemperature), null, 250, TemperatureLoggerPeriod);
    LedGreen.Write(false);
}

The SD card, represented by the SdMountPoint string, communicates with the application over SPI. The presence of the SD card in the reader is determined through the CardDetect input pin.

The LEDs are simple outputs that we'll turn ON / OFF as the peripherals gets initialized and file I/Os take place with the SD card.

The clock communicates with the application over the I2C protocol. The clock's most important functions are accessed through the Set() and Get() methods respectively used to set the time once and to get updated time stamps afterward.

The thermocouple communicates over SPI with the application. It exposes a Read() method which caches a raw temperature sample accessed through the Celsius and Fahrenheit properties.

Note: the Netduino Plus already features a built-in microSD card reader, in which case, having another one on the shield is not really needed. Except for this hardware difference, everything else discussed within this article applies equally to the regular Netduino and the Netduino Plus.

Interfacing with the Arduino Logger shield’s hardware

Adafruit is pretty good about making usable products and generally provides Arduino libraries to use with their hardware. Indeed, the Arduino Logger Shield is well documented and comes with two C++ libraries: SD which implements a FAT file system and supporting low-level SD card I/O functions. RTCLib which wraps the I2C interface required to communicate with the DS1307 real time clock.

The SD Card Interface

Let’s deal with the SD card reader and the file system first: a quick review of SD.h reveals two C++ classes:

• class File : public Stream {} exposing standard read, write, seek, flush file access functions.
• class SDClass {} exposing storage management such as file and directory operations.

This is good news because the .NET Micro Framework on the Netduino already supports file streams and directory management through the use of the .NET MF System.IO assembly. This assembly comes with the .NET MF SDK port to the Netduino.

By the same token, interfacing with an SD card is provided by an assembly built by Secret Labs named SecretLabs.NETMF.IO which comes with the Netduino SDK.

SecretLabs.NETMF.IO provides two functions for 'mounting' and 'un-mounting' an SD card device and the associated FAT file system so that it can be made usable by the .NET MF through assemblies such as System.IO.

It's important to note that the SecretLabs.NETMF.IO assembly must not be deployed with an application targeting the Netduino Plus: on boot, the .NET Micro Framework implementation specific to the Netduino Plus automatically detects and mounts the SD card if one is present in its microSD card reader. This functionality is redundant with the MountSD / Unmount functions provided by the SecretLabs.NETMF.IO assembly which is only needed on Netduino SKUs without a built-in SD card reader.

How does the .NET MF interact with the SD card through the shield?

At this point, it's a good time to review the Arduino Logger Shield's pin-out and the shield's schematics:

As we know from our previous article, pins D10-D13 map to the SPI interface and pins A4-A5 map to the I2C interface of the Netduino. On the shield's schematics, the SPI interface leads us to the SD & MMC section of the diagram, connected through a 74HC125N logic-level shifter chip indicated as IC3A-D.

The role of the logic-level shifter is to ensure that logic voltages supplied to the SD card do not exceed 3.3v, even if they come from a microcontroller using 5v logic levels, such as the Arduino. When using an SD card with a Netduino, a level-shifter is not required since all logic levels run at 3.3v on the AT91SAM7x chip but it doesn't interfere with any I/O operations either when the voltage is already 3.3v.

The SD card reader in itself is just a passive connector, giving access to the controller built into the SD card. It also provides a mechanical means (i.e. switches) of detecting the presence of a card in the reader (see JP14 pin 1) as well as detecting if the card is write-protected (see JP14 pin 2). We'll make use of the card detection pin in the sample temperature logging application later on.

For background on how SD cards work, the following application note "Secure Digital Card Interface for the MSP430" is excellent and much easier to digest than the extensive 'simplified' SD card protocol specifications provided on the SD Card Association site. The following table taken from the "Secure Digital Card Interface for the MSP430" shows the pin out of an SD card and the corresponding SPI connections:

An SD standard-compliant card can support 3 distinct access modes, each one providing different performance characteristics:

• SD 1-bit protocol: synchronous serial protocol with one data line, one clock line and one line for commands. The full SD card protocol command set is supported in 1-bit mode.
• SD 4-bit protocol: this mode is nearly identical to the SD 1-bit mode, except that the data is multiplexed over 4 data lines, yielding up to 4x the performance of SD 1-bit mode. The full SD card protocol command set is supported in 4-bit mode.
• SPI mode: provide a standard SPI bus interface (/SS, MOSI, MISO, SCK). In SPI mode, the SD card only supports a subset of the full SD card protocol but it is sufficient for implementing a fully functional storage mechanism with a file system.

As you might have guessed, the .NET Micro Framework on the Netduino makes use of the SD card in SPI mode (see \DeviceCode\Drivers\BlockStorage\SD\SD_BL_driver.cpp). The block-oriented SD card I/Os are abstracted thanks to the FAT file system provided by the System.IO assembly (see \DeviceCode\Drivers\FS\FAT\FAT_FileHandle.cpp and FAT_LogicDisk.cpp).

The role of the SecretLabs.NETMF.IO assembly on the Netduino (or its built-in equivalent on the Netduino Plus) is to initialize the SD card in SPI mode during the 'mounting' process by sending the proper set of commands as defined in the SD Card protocol.

In the C# code of the AdafruitNetduinoLogger sample application, which we will review as a whole later on in the code walkthrough section, the following function takes care of the SD card initialization:

public static void InitializeStorage(bool mount) {
    try {
        if (mount == true) {
            StorageDevice.MountSD(SdMountPoint, SPI.SPI_module.SPI1, Pins.GPIO_PIN_D10);
        } else {
            StorageDevice.Unmount(SdMountPoint);
        }
    } catch (Exception e) {
        LogLine("InitializeStorage: " + e.Message);
        SignalCriticalError();
    }
}

Once mounted, the file system is accessed through System.IO calls such as this:

using (var tempLogFile = new StreamWriter(filename, true)) {
    tempLogFile.WriteLine(latestRecord);
    tempLogFile.Flush();
}

Using the StreamWriter class in this context made sense for writing strings as used in the sample application:

However, there are many other file I/O classes available in System.IO that may be better suited depending on the scenario.

The DS1307 real time clock

Our next step is to examine the interface with the DS1307 real time clock (RTC). We'll start by extracting the most important parts of the DS1307 datasheet and reviewing how it's wired up on the shield's schematics.

DS1307 features

• Real-Time Clock (RTC) Counts Seconds, Minutes, Hours, Date of the Month, Month, Day of the week, and Year with Leap-Year Compensation Valid Up to 2100
• 56-Byte, Battery-Backed, General-Purpose RAM with Unlimited Writes
• I2C Serial Interface
• Programmable Square-Wave Output Signal
• Automatic Power-Fail Detect and Switch Circuitry
• Consumes Less than 500nA in Battery-Backup Mode with Oscillator Running

Note: If you need to measure the time something takes in milliseconds, a time granularity that the DS1307 clock does not provide, you can use the Utility functions provided by the .NET Micro Framework like this:

var tickStart = Utility.GetMachineTime().Ticks;

// <...code to be timed...>

var elapsedMs = (int)((Utility.GetMachineTime().Ticks - tickStart) / TimeSpan.TicksPerMillisecond);

This timing method relies on the CPU's internal tick counter and is not 100% accurate due to the overhead of the .NET MF itself but may be sufficient in most scenarios. In addition, the internal tick counter rolls over every so often, something that should be taken into account in production code.

DS1307 register map

Accessing the clock's features comes down reading and writing to and from a set of registers as described on page 8 of the datasheet.

Page 9 of the DS1307 datasheet provides more details about the square wave generation function of the clock, which we will not be using here. The generated square wave signal is available on the shield through connector JP14 on pin 3 as you can see on the schematics below and can be used to provide a slow but reliable external clock signal to another device such as a microcontroller.

DS1307 I2C bus address

The final piece of the puzzle needed before we can use the DS1307 is the device's address on the I2C data bus and its maximum speed (specified at 100 KHz on page 10 of the datasheet). The device address is revealed on page 12 as being 1101000 binary (0x68) along with the two operations modes (Slave Receiver and Slave Transmitter) of the clock. The 8th bit of the address is used by the protocol to indicate whether a 'read' or a 'write' operation is requested.

Note: I2C devices sometime make use of 10-bit addresses. If you aren't familiar with the I2C data bus, you should read the section of the datasheet starting on page 10 which provides a good foundation for understanding how I2C generally works.

It can be summarized as follows:

• I2C is a 2-wire serial protocol with one bidirectional data line referred to as SDA and one clock line, referred to as SCL.
• The I2C bus is an open-drain bus (i.e. devices pull the bus low to create a '0' and let go of the bus to create a '1'). To achieve this, I2C requires a pull-up resistor on the SCL and SDA lines between 1.8K ohms and 10K ohms. I2C devices do not need to provide pull-ups themselves if the bus already has them.
• The I2C master (i.e. the Netduino microcontroller) always provides the clock signal, generally between 100 KHz (or lower) for standard speed devices or 400 KHz for high-speed devices. There's also a 'Fast Mode Plus' allowing for speeds up to 1MHz on devices supporting it. There can be more than one master on the bus even though this is uncommon.
• An I2C device can have a 7-bit or 10-bit address, allowing for multiple I2C devices to be used on the same bus.
• I2C read and write operations are transactions initiated by the I2C master targeting a specific device by address. Some I2C slave devices can notify their master that they need to communicate using a bus interrupt.
• A transaction is framed by 'start' and 'stop signals, with each byte transferred requiring an acknowledgement signal.

At this point, we have all the pieces needed to communicate with the RTC using I2C transactions.

Using the I2C protocol with the .NET Micro Framework

On the Arduino, the library used with the shield to communicate with the DS1307 is a C++ library called RTClib. The header of the library declares a DateTime class, similar in functionality to the standard .NET Micro Framework DateTime class provided by System in the mscorlib assembly. We'll use the standard .NET MF data type to work with the clock instead.

The next declared class is RTC_DS1307 which implements the driver for the DS1307 chip using the Wire library to wrap the I2C protocol. The .NET Micro Framework also supports the I2C protocol through to the Microsoft.SPOT.Hardware assembly. Here again, we'll use the .NET MF implementation of I2C in order to communicate with the clock. However, the I2C transaction patterns implemented by the C++ driver can still provide a useful guide for writing a C# driver for the DS1307 when you don't know where to begin just based on the datasheet.

For instance, the following functions taken from RTClib.cpp shows the call sequence used with the Wiring API to address the date and time registers of the clock:

int i = 0; //The new wire library needs to take an int when you are sending for the zero register

void RTC_DS1307::adjust(const DateTime& dt) {
    Wire.beginTransmission(DS1307_ADDRESS);
    Wire.write(i);
    Wire.write(bin2bcd(dt.second()));
    Wire.write(bin2bcd(dt.minute()));
    Wire.write(bin2bcd(dt.hour()));
    Wire.write(bin2bcd(0));
    Wire.write(bin2bcd(dt.day()));
    Wire.write(bin2bcd(dt.month()));
    Wire.write(bin2bcd(dt.year() - 2000));
    Wire.write(i);
    Wire.endTransmission();
}

DateTime RTC_DS1307::now() {
    Wire.beginTransmission(DS1307_ADDRESS);
    Wire.write(i);
    Wire.endTransmission();
    Wire.requestFrom(DS1307_ADDRESS, 7);
    uint8_t ss = bcd2bin(Wire.read() & 0x7F);
    uint8_t mm = bcd2bin(Wire.read());
    uint8_t hh = bcd2bin(Wire.read());
    Wire.read();
    uint8_t d = bcd2bin(Wire.read());
    uint8_t m = bcd2bin(Wire.read());
    uint16_t y = bcd2bin(Wire.read()) + 2000;
    return DateTime (y, m, d, hh, mm, ss);
}

The final class is RTC_Millis, a utility class converting time data into milliseconds, effectively providing the functionality of the DateTime.Millisecond property on the .NET MF.

Having assessed that the functionality of RTClib only handles date and time registers and knowing the role of the other clock registers, we can proceed with implementing a complete DS1307 C# driver, supporting the square wave and RAM functions, using the native I2C protocol support of the .NET Micro Framework.

The driver starts by defining key constants matching the clock registers according to the datasheet:

[Flags]
// Defines the frequency of the signal on the SQW interrupt pin on the clock when enabled
public enum SQWFreq { SQW_1Hz, SQW_4kHz, SQW_8kHz, SQW_32kHz, SQW_OFF };

[Flags]
// Defines the logic level on the SQW pin when the frequency is disabled
public enum SQWDisabledOutputControl { Zero, One };

// Real time clock I2C address
public const int DS1307_I2C_ADDRESS = 0x68;

// Start / End addresses of the date/time registers
public const byte DS1307_RTC_START_ADDRESS = 0x00;
public const byte DS1307_RTC_END_ADDRESS = 0x06;

// Start / End addresses of the user RAM registers
public const byte DS1307_RAM_START_ADDRESS = 0x08;
public const byte DS1307_RAM_END_ADDRESS = 0x3f;

// Square wave frequency generator register address
public const byte DS1307_SQUARE_WAVE_CTRL_REGISTER_ADDRESS = 0x07;

// Start / End addresses of the user RAM registers
public const byte DS1307_RAM_START_ADDRESS = 0x08;
public const byte DS1307_RAM_END_ADDRESS = 0x3f;

// Total size of the user RAM block
public const byte DS1307_RAM_SIZE = 56;

Next the driver defines an I2C device object representing the clock:

// Instance of the I2C clock
protected I2CDevice Clock;

In the class constructor, the I2C clock device is initialized, specifying its address and speed in KHz:

public DS1307(int timeoutMs = 30, int clockRateKHz = 50) {
    TimeOutMs = timeoutMs;
    ClockRateKHz = clockRateKHz;
    Clock = new I2CDevice(new I2CDevice.Configuration(DS1307_I2C_ADDRESS, ClockRateKHz));
}

The driver retrieves the date and time from the clock through a Get function returning a DateTime object.

public DateTime Get() {
    byte[] clockData = new byte [7];

    // Read time registers (7 bytes from DS1307_RTC_START_ADDRESS)
    var transaction = new I2CDevice.I2CTransaction[] {
    I2CDevice.CreateWriteTransaction(new byte[] {DS1307_RTC_START_ADDRESS}),
        I2CDevice.CreateReadTransaction(clockData)
    };

    if (Clock.Execute(transaction, TimeOutMs) == 0) {
        throw new Exception("I2C transaction failed");
    }

    return new DateTime(
        BcdToDec(clockData[6]) + 2000, // year
        BcdToDec(clockData[5]), // month
        BcdToDec(clockData[4]), // day
        BcdToDec(clockData[2] & 0x3f), // hours over 24 hours
        BcdToDec(clockData[1]), // minutes
        BcdToDec(clockData[0] & 0x7f) // seconds
    );
}

Let's break it down:

• A 7-byte array is allocated which will receive the raw date and time data registers, starting at address DS1307_RTC_START_ADDRESS (0x00) and ending at DS1307_RTC_END_ADDRESS (0x06).
• An I2C transaction object is allocated, comprising two parameters:
  • A 'write' transaction object telling the DS1307 device which register address to start reading data from. In this case, this is DS1307_RTC_START_ADDRESS (0x00), the very first time-keeping register.
  • A 'read' transaction object specifying where the clock's time-keeping data registers will be stored, implicitly defining the total number of bytes to be read and acknowledged.
• Clock.Execute is the function calling into the .NET MF I2C interface to run the prepared transactions. The second parameter specifies a time out value expressed in milliseconds before the transaction fails, resulting in a generic exception being thrown.
• When the transactions succeed, a DateTime object is instantiated with the 7 time-keeping registers returned by the 'read' transaction. Each register is converted from Binary Coded Decimal form to decimal form using a custom utility function:

protected int BcdToDec(int val) {
    return ((val / 16 * 10) + (val % 16));
}

Conversely, the driver provides a Set function to update the clock's time-keeping registers. Because the driver doesn't expect a response from the DS1307 in this scenario, the I2C transaction is write-only. The fields of the DateTime parameter corresponding to the time -keeping registers are converted from decimal form to BCD form and stuffed in a 7-byte array before executing the transaction.

public void Set(DateTime dt) {
    var transaction = new I2CDevice.I2CWriteTransaction[] {
        I2CDevice.CreateWriteTransaction(new byte[] {
            DS1307_RTC_START_ADDRESS,
            DecToBcd(dt.Second),
            DecToBcd(dt.Minute),
            DecToBcd(dt.Hour),
            DecToBcd((int)dt.DayOfWeek),
            DecToBcd(dt.Day),
            DecToBcd(dt.Month),
        DecToBcd(dt.Year - 2000)} )
    };

    if (Clock.Execute(transaction, TimeOutMs) == 0) {
        throw new Exception("I2C write transaction failed");
    }
}

The rest of the functions provided by the C# driver implement the other DS1307 features, such as

• SetSquareWave
• Halt
• SetRAM
• GetRAM
• The [] operator used to access a specific clock register
• WriteRegister

In all case, these functions are wrappers around the 'read' and 'write' I2C transaction model, involving the appropriate DS1307 registers as defined in the datasheet. 

Using the Adafruit Arduino Logger Shield as a temperature logger

To illustrate the points discussed so far, we'll use the Adafruit Arduino Logger shield with a Netduino and a MAX6675 thermocouple amplifier for the purpose of recording ambient temperature samples at ten second intervals.

Each record includes a date, a time and the temperature expressed in Celsius and Fahrenheit. The records are written to daily files in CSV format for easy export to a spreadsheet, making the application easily adaptable for acquiring data from different sensors:

Device Connections

Instead of permanently soldering the temperature sensor to the prototyping area of the shield, female / female jumper wires were used to make connections between the shield's own pin headers as well as the thermocouple's male pin headers.

The following table enumerates these connections:

Reading temperature using an Adafruit Max6675 Thermocouple amplifier breakout board

The Max6675 thermocouple amplifier chip on the breakout board is a read-only SPI device. When the CS pin (SPI /SS) of the device is asserted with a 1ms delay before reading, the chip returns a 12-bit value on its DO pin (SPI MISO) corresponding to the temperature measured by a K-type Thermocouple wire. The resulting C# driver for the Max6675 is short:

using System;
using Microsoft.SPOT;
using Microsoft.SPOT.Hardware;

namespace Maxim.Temperature{
    public class Max6675 : IDisposable {
        protected SPI Spi;

        public void Initialize(Cpu.Pin chipSelect) {
            Spi = new SPI(
            new SPI.Configuration(
            chipSelect, false, 1, 0, false, true, 2000, SPI.SPI_module.SPI1)
            );
        }

        public double Celsius {
            get { return RawSensorValue * 0.25; }
        }

        public double Farenheit {
            get { return ((Celsius * 9.0) / 5.0) + 32; }
        }

        protected UInt16 RawSensorValue;
        protected byte[] ReadBuffer = new byte[2];
        protected byte[] WriteBuffer = new byte[2];

        public void Read() {
            RawSensorValue = 0;
            Spi.WriteRead(WriteBuffer, ReadBuffer);
            RawSensorValue |= ReadBuffer[0];
            RawSensorValue <<= 8;
            RawSensorValue |= ReadBuffer[1];

            if ((RawSensorValue & 0x4) == 1) {
                throw new ApplicationException("No thermocouple attached.");
            }

            RawSensorValue >>= 3;
        }

        public void Dispose() {
            Spi.Dispose();
        }

        ~Max6675() {
            Dispose();
        }
    }
}

Temperature logger application walkthrough

Let's review the key parts of the temperature logging application code and how it interacts with the devices connected to the shield.

public static readonly string SdMountPoint = "SD";

Defines an arbitrary string used to refer to the SD card when using StorageDevice.MountSD and StorageDevice.Unmount functions.

public static readonly int TemperatureLoggerPeriod = 10 * 1000; // milliseconds

Defines the interval between temperature samples.

public static OutputPort LedRed = new OutputPort(Pins.GPIO_PIN_D0, false);

Defines an output connected to pin D0 controlling the state of the red LED on the shield.

public static OutputPort LedGreen = new OutputPort(Pins.GPIO_PIN_D1, false);

Defines an output connected to pin D1 controlling the state of the green LED on the shield.

public static InputPort CardDetect = new InputPort(
    Pins.GPIO_PIN_D3,
    true,
    Port.ResistorMode.PullUp);

Defines an input connected to pin D3 used to determine if an SD card is inserted in the SD socket.

public static ManualResetEvent ResetPeripherals = new ManualResetEvent(false);

Defines a manual reset event object that will be used in the main application loop to determine when to re-initialize the shield's peripherals.

public static readonly Cpu.Pin ThermoCoupleChipSelect = Pins.GPIO_PIN_D2;

Defines D2 as the SPI chip select pin connected to the Max6675 Thermocouple board.

public static Timer TemperatureSampler;

Defines an instance of a timer object which will drive temperature sampling.

public static DS1307 Clock;

Defines an instance of the DS1307 real time clock driver.

public static Max6675 ThermoCouple;

Defines an instance of the Max6675 thermocouple driver.

public static ArrayList Buffer = new ArrayList();

Defines an array list instance which will be used as a temporary buffer when the SD card is removed from its socket.

The application's main loop is only concerned about the state of the peripherals:

• It initializes the devices connected to the shield
• It waits indefinitely for a signal indicating that a hardware error occurred
• It disposes of the current device instances and starts over

public static void Main() {
    while (true) {
        InitializePeripherals();
        ResetPeripherals.WaitOne();
        ResetPeripherals.Reset();
        DeInitializePeripherals();
    }
}

InitializePeripherals indicates that it is working by controlling the green LED on the shield. Its role is focused on object creation and initialization.

public static void InitializePeripherals() {
    LedGreen.Write(true);
    Clock = new DS1307();
    ThermoCouple = new Max6675();
    InitializeStorage(true);
    InitializeClock(new DateTime(2012, 06, 14, 17, 00, 00));
    ThermoCouple.Initialize(ThermoCoupleChipSelect);
    TemperatureSampler = new Timer(
    new TimerCallback(LogTemperature),
    null,
    250,
    TemperatureLoggerPeriod);
    LedGreen.Write(false);
}

If the initialization of a peripheral fails, the shield will quickly blink its LEDs, indefinitely:

public static void SignalCriticalError() {
    while (true) {
        LedRed.Write(true);
        LedGreen.Write(true);
        Thread.Sleep(100);
        LedRed.Write(false);
        LedGreen.Write(false);
        Thread.Sleep(100);
    }
}

The clock initialization function only sets the clock date and time when it is unable to find a file named 'clockSet.txt' on the SD card, ensuring that the initialization of the DS1307 only happens once in the InitializePeripherals function or until the file is deleted.

public static void InitializeClock(DateTime dateTime) {
    var clockSetIndicator = SdMountPoint + @"\clockSet.txt";

    try {
        if (File.Exists(clockSetIndicator) == false) {
            Clock.Set(dateTime);
            Clock.Halt(false);
            File.Create(clockSetIndicator);
        }
    } catch (Exception e) {
        LogLine("InitializeClock: " + e.Message);
        SignalCriticalError();
    }
}

The LogTemperature function is the callback invoked by the Timer object every 10 seconds. The function indicates that it is working by turning the red LED on the shield ON and OFF.

public static void LogTemperature(object obj) {
    LedRed.Write(true);
}

The function reads the current time from the clock with Clock.Get() and takes a temperature sample with ThermoCouple.Read().

var tickStart = Utility.GetMachineTime().Ticks;
var now = Clock.Get();
ThermoCouple.Read();
var elapsedMs = (int)((Utility.GetMachineTime().Ticks - tickStart) / TimeSpan.TicksPerMillisecond);

Then, it concatenates a string containing the date, time and temperature expressed in Celsius and Fahrenheit, with each field separated by commas.

var date = AddZeroPrefix(now.Year) + "/" + AddZeroPrefix(now.Month) + "/" + AddZeroPrefix(now.Day);
var time = AddZeroPrefix(now.Hour) + ":" + AddZeroPrefix(now.Minute) + ":" + AddZeroPrefix(now.Second) + ":" + AddZeroPrefix(elapsedMs);
var celsius = Shorten(ThermoCouple.Celsius.ToString());
var farenheit = Shorten(ThermoCouple.Farenheit.ToString());
var latestRecord = date + "," + time + "," + celsius + "," + farenheit;

To make the data more manageable, daily temperature files are created as needed, each one starting with the column headers expected for parsing the values in CSV format.

var filename = SdMountPoint + BuildTemperatureLogFilename(now);
if (File.Exists(filename) == false) {
    using (var tempLogFile = new StreamWriter(filename, true)) {
        tempLogFile.WriteLine("date,time,celsius,fahrenheit");
    }
}

The temperature sampling application lets the user remove the SD card from its socket so that the CSV files can be moved over to a PC for processing without losing data in the meantime. In order to do this, the application checks the state of the 'Card Detect' pin before attempting file system I/Os.

When the SD card is not present, the latest temperature record is preserved in the array list buffer until the SD card is put back in its socket. The array list data is then flushed to storage.

if (CardDetect.Read() == false) {
    using (var tempLogFile = new StreamWriter(filename, true)) {
        if (Buffer.Count != 0) {
            foreach (var bufferedLine in Buffer) {
                tempLogFile.WriteLine(bufferedLine);
            }
            Buffer.Clear();
        }
        tempLogFile.WriteLine(latestRecord);
        tempLogFile.Flush();
    }
} else {
    LogLine("No card in reader. Buffering record.");
    Buffer.Add(latestRecord);
}

The temperature logging function expects to run out of memory if the array list buffer grows too large, in which case, all the records get purged. Other memory management strategies could be used to mitigate data loss in this case. However, this depends entirely on the requirements of the data logging application and is out of scope for this discussion.

catch (OutOfMemoryException e) {
    LogLine("Memory full. Clearing buffer.");
    Buffer.Clear();
}

The temperature logging function also handles file system exceptions caused by the removal of the SD card and reacts by signaling the ResetPeripherals event. In turn, this lets the application's main loop know that the peripherals, and most specifically the SD card, need to be recycled and initialized again in order to recover from the error.

catch (IOException e) {
    LogLine("IO error. Resetting peripherals.");
    Buffer.Add(latestRecord);
    ResetPeripherals.Set();
}

Conclusion

In this article, we took a shield designed for the Arduino and learned how to critically review the Arduino code libraries supporting it, drawing parallels with features offered by the .NET Micro Framework. This process allowed us to identify areas in the Arduino code which were not necessary to port over to C# such as SD card and file system handlers. It also allowed us to see the similarities in the way the Arduino and the Netduino handle I2C communications.

Most importantly, we also learned the importance of reviewing a device's schematics and component datasheets to ensure that important features have not been omitted and potentially incorrectly implemented when considering using an unknown library: in the case of RTClib, we saw that the implementation was limited to the basic date and time functions of the DS1307, leaving out other useful features such as the clock's built-in RAM and the square wave generation functions.

In our next article, we'll take on a much more complex shield and we will learn how to analyze Arduino libraries in depth before porting them from C/C++ to C#.

Bio

Fabien is the Chief Hacker and co-founder of Nwazet, a start-up company located in Redmond WA, specializing in Open Source software and embedded hardware design. Fabien's passion for technology started 30 years ago, creating video games for fun and for profit. He went on working on mainframes, industrial manufacturing systems, mobile and web applications. Before Nwazet, Fabien worked at MSFT for eight years in Windows Core Security, Windows Core Networking and Xbox. During downtime, Fabien enjoys shooting zombies and watching sci-fi.

 
Photo courtesy of John Boxall

Introduction

When the Arduino Duemilanove microcontroller appeared in 2005, it featured a set of female pin headers exposing most of the pins of the ATmega168 for easy hacking and for connecting accessory boards known as 'Shields'. The purpose of a shield is to provide new plug-and-play functionality to the host microcontroller, such as circuit prototyping, motion control, sensor integration, network and radio communication, or gaming interfaces, without worrying too much about the hardware implementation details. Seven years after the birth of the original Arduino, new shields keep coming out and are being cataloged on http://shieldlist.org/, a testament to the versatility of the design. It is also simple to build a DIY shield when nothing out there will meet your needs or when you want to understand how the shield concept works from the ground up.

In their infinite wisdom, Secret Labs, the makers of the Netduino, adopted the same Arduino form factor and header pin out to let .NET Micro Framework users tap into the vast array of Arduino shields on the market. Yet, there's always one hurdle that needs to be overcome: in order to give access to their hardware functions easily, many Arduino shields provide sample sketches (a.k.a. C/C++ Wiring programs) and libraries also written in C/C++ designed to be used within the Arduino IDE. Unfortunately, Arduino sketches and libraries aren't compatible with the Visual Studio environment and the .NET runtime: they need to be adapted in order to make use of the shield's hardware.

The goal of this series of articles is to attempt to demystify what an Arduino shield is. To do this, we'll start by building one from scratch. Then, we'll present a generic approach for selecting Netduino-compatible Arduino shields and adapting Arduino code to interface with the .NET Micro Framework.

Selecting Arduino shields for use with Netduino / Plus

Arduino shields using hardware components interfaced over protocols such as SPI, I2C, 1-Wire or serially over a UART are generally good candidates for a Netduino adaptation because the .NET Micro Framework supports these protocols natively and can do bulk I/O transfers, yielding good performance.

However, if a shield requires bit-banging data on GPIOs (i.e. communicating data serially, such as SPI, I2C, RS-232, in software by toggling GPIO lines individually instead of using dedicated hardware interfaces) or has very low I/O latency requirements, it may not be practical or even possible to use the shield on a Netduino / Plus.

The reason for this is simple: the Arduino runs native code and can access hardware registers directly with a latency orders of magnitude lower than what the .NET Micro Framework can achieve on a Netduino / Plus where toggling GPIOs means going through multiple framework layers while running interpreted code. The maximum raw I/O toggling frequency on a Netduino / Plus clocked at 48 MHz was measured around ~8.4KHz which is slow compared to the few MHz achievable on an Arduino clocked at 16 MHz.

Fortunately, there are possible workarounds and strategies

• Whenever an Arduino library bit-bangs SPI data on GPIOs, regardless of the pins used to do it, it can be replaced by using the standard SPI interface on the Netduino / Plus.
• The Netduino / Plus has significantly more RAM available than the Arduino and can transfer larger SPI data buffer in a single shot, dramatically reducing I/O latency. For instance, driving a display shield would greatly benefit from this method where a frame is cached in RAM on the Netduino / Plus then sent in one shot over SPI to the hardware display driver.

The bottom line is that it is always a good idea to review how the Arduino code interfaces with a given shield before proceeding with a purchase or starting a conversion project. This review process is also facilitated by http://shieldlist.org which shows which pins are being used to drive a shield. For example, the Adafruit Industries Logger Shield uses hardware SPI (pins D10-D13) for the SD card and I2C (pins A4-A5) for the DS1307 real time clock, making it a perfect adaptation candidate for the Netduino / Plus as we'll see in our next article.

It's also very important to know what features are available on the .NET Micro Framework running on the Netduino / Plus: some things come standard with the .NET Micro Framework which normally require add-ons libraries on the Arduino such as file system and networking support.

Finally, it is wise to search code repositories such as CodePlex, BitBucket and GitHub for hardware drivers and libraries before undertaking writing one in C# from scratch based on component datasheets. This extends to the Netduino and the TinyCLR community forums where many people have posted drivers in discussion threads.

Building a shield from scratch to meet specific requirements

Recently, we were asked if it was possible to connect our Touch Display module designed for a Netduino Go! to a Netduino / Plus. By doing a bit of research and seeing how the driver for the Touch Module works, this can be done by building a simple custom shield composed of a few parts connecting the Netduino pin headers to a Netduino Go! socket breakout board like so:

High-Level Software Interface

To frame the rest of this article, the communication interface with the shield is done using SPI and a GPIO used for synchronization. The communication protocol details are wrapped in the Nwazet.Go.Display.Imaging.VirtualCanvas class which needs to be initialized with the SPI and GPIO parameters matching the pin out of the shield:

var canvas = new VirtualCanvas(TouchEventHandler, WidgetClickedHandler);

canvas.Initialize(
    displaySpi: SPI.SPI_module.SPI1,
    displayChipSelect: Pins.GPIO_PIN_D9,
    displayGPIO: Pins.GPIO_PIN_D8,
    speedKHz: 5000);

From that point on, all interactions with the shield are done through Draw* methods followed by a call to the Execute() method which handles sending and receiving SPI data packets. The VirtualCanvas class also provides methods for creating widgets such as buttons, receiving touch screen events and setting the screen orientation.

For example, the following code snippet draws and updates a progress bar and uses one of the proportional fonts built into the display:

public static void Main() {
    var canvas = new VirtualCanvas();
    canvas.Initialize(
        displaySpi: SPI.SPI_module.SPI1,
        displayChipSelect: Pins.GPIO_PIN_D9,
        displayGPIO: Pins.GPIO_PIN_D8,
        speedKHz: 5000);
    var fontInfo = new DejaVuSans9().GetFontInfo();
    canvas.DrawFill(0xFFFF);
    canvas.DrawString(5, 144, (ushort)BasicColor.Black, fontInfo.ID, "Progress");
    for (var progress = 1; progress <= 100; progress++) {
        canvas.DrawProgressBar(
            70, 140,
            75, 12,
            RoundedCornerStyle.All,
            RoundedCornerStyle.All,
            (ushort)BasicColor.Black,
            (ushort)GrayScaleValues.Gray_128,
            (ushort)GrayScaleValues.Gray_30,
            (ushort)BasicColor.Green,
            progress);
        Thread.Sleep(100);
        canvas.Execute();
    }
}

Netduino Go! Socket Pin Out

In order to build the shield, we need to start with mapping out the connections between the Go! bus socket and the header pins of the shield.

The Go! socket pin out was designed by Secret Labs to be compatible with Gadgeteer modules supporting the following socket types: S (SPI), U (UART), X (3 GPIO) as documented in the .NET Gadgeteer Mainboard Builder's Guide v 1.8, page 6. This makes our little shield compatible with a Gadgeteer board as long as it is connected to an S socket. As far as other .NET Micro Framework boards are concerned, they should also work fine if they can drive an SPI slave device at 5 MHz, the minimum SPI clock frequency required by the Touch Display module.

Gadgeteer Socket Types

Netduino Go! Socket

The hardware specifications of the Netduino / Plus tells us that the Netduino GPIOs correspond exactly to the Arduino Uno GPIOs:

• digital pins 0-1: UART 1 RX, TX
• digital pins 2-3: UART 2 RX, TX
• digital pins 5-6: PWM, PWM
• digital pins 7-8: UART 2 RTS, CTS
• digital pins 9-10: PWM, PWM
• digital pins 11-13: SPI MOSI, MISO, SPCK
• analog pins 4-5: I2C SDA, SCL

With this information, we can proceed with the creation of schematics to bridge the gap between the two worlds.

Shield schematics

How does this custom shield work?

The Netduino Go! bus is designed around the SPI protocol (see Go! socket pins 6-10) which the Netduino / Plus can speak easily, making a Netduino Go! module or a S-type Gadgeteer module appear like any other SPI device to a Netduino / Plus.

You will notice that the following pins aren't connected on the Go! socket for the sake of simplification:

• Pin 2 (5v power): the Touch Display module operates on 3.3v, therefore the 5v power supply is not needed but could be connected to the 5v header of the Netduino to be fully compliant.
• Pin 4 and 5 (UART): the Touch Display module does not make use of the serial interface during normal operation, so these lines can remain disconnected. However, to be fully compliant, Pin 4 (RX) and 5 (TX) on the Go! socket could be respectively connected to Netduino pin D1 (TX) and D0 (RX).
• The fact that these lines aren't connected in this specific scenario would cause another module requiring a UART and/or 5v to not work obviously.

While the SPI part is straight forward, there's a bit more to the Go! bus electrical specification that must be taken into account: the Netduino Go! bus also uses its GPIO pin on boot (see Go! socket pin 3) to control the behavior of modules based on ARM Cortex chips: when the GPIO pin is held HIGH on module power-up, the ARM Cortex chip enters its bootloader mode and waits to be flashed with new firmware over the Go! bus UART (see Go! socket pins 4-5). The ARM Cortex bootloader remains active until the ARM Cortex chip is reset or power-cycled.

This is important to know for two reasons:

1. The Touch Display module is powered by an ARM Cortex M3 chip.
2. On boot, a Netduino / Plus sets all of its I/O pins HIGH, preventing the Touch Display from booting its firmware, which is undesirable. The reason for the I/O pins being set HIGH is documented on page 5 of the ATMEL AT91SAM7X datasheet which states that the pins on Parallel IO Controller A and B (PA0 - PA30 and PB0 - PB30) are pulled-up input at reset. The pins stay in that HIGH state until they're reconfigured by the Tiny CLR and the C# application itself.

For this reason, the custom shield is designed to control the power supply of the Touch Display module through a general-purpose 2N4403 PNP transistor (see T1_PNP on the shield schematics) connected to pin D7: it is only when the base of the transistor is asserted LOW that power flows to the Touch Display module, countering the behavior of the Netduino / Plus on boot. In addition, the Netduino / Plus controls the state of the Go! bus GPIO pin through D8, asserting it LOW before powering up the Touch Display module.

About the 2N4403 transistor: any other PNP transistor, or a P-Channel MOSFET, capable of switching at least 150mA continuously, the maximum current consumption of the touch display with some headroom, would have worked equally as well. The 2N4403 used here is more than adequate:

In the context of this shield, the PNP transistor is used as a switch, either fully ON or OFF, as opposed to being used as an amplifier. To this end, a 1K ohm resistor is used between the base of the transistor and Netduino / Plus pin D7 (POWER_CTRL) to ensure that the transistor is fully saturated but also limiting the maximum current sunken through pin D7 and potential damage being done to the ATMEL chip. The 10K ohm pull-up resistor placed between the base of the transistor and the 3.3v power rail ensures that the transistor will always be turned off by default, even when pin D7 is floating (i.e. not specifically declared as an Input or as an Output and driven HIGH or LOW by the C# application).

The PowerUpDisplay function summarizes the interaction with the module through the shield on boot:

// Pin connected to the transistor controlling the display's power supply
public static OutputPort PowerTransistor = new OutputPort(Pins.GPIO_PIN_D7, true);

public static void PowerUpDisplay() {
    // Ensure that the GPIO pin in low to prevent the display module to start in bootloader mode
    var goBusGPIO = new OutputPort(Pins.GPIO_PIN_D8, false);
    // Power up the display module
    PowerTransistor.Write(false);
    // Always wait 250ms after power-up to ensure that the display module is fully initialized before sending commands
    Thread.Sleep(250);
    goBusGPIO.Dispose();
}

Refactoring a Netduino Go! C# driver to work on a regular Netduino / Plus

The final step to get the Touch Display module working outside of its natural environment involved refactoring its C# driver (VirtualCanvas.cs), eliminating the idiosyncrasies specific to Netduino Go! bus. Go! modules are supposed to implement the GoModule interface as defined by the Netduino Go! SDK. Because the regular Netduino / Plus SDK has no knowledge of this interface, it's necessary to strip out any references and API calls specific to the Go! framework, leaving only what's absolutely required to communicate with the Touch Display module over SPI and maintaining the disposable nature of the class. For more details on Go! SDK, please refer to the Netduino Go! community forums and the source code of the existing module drivers as it is currently the best source of documentation.

This step is akin to the work required when adapting an Arduino library to run on the .Net Micro Framework, only in this case, the task is easier since we're dealing with a C# driver to begin with instead of going from C/C++ to C#.

Starting from the top of VirtualCanvas.cs, the following changes were made in the Netduino / Plus version of the driver:

• Removed
  • using GoBus;
• Replaced
  • public class VirtualCanvas : GoModule
  • with: public class VirtualCanvas : IDisposable
• Replaced
  • public void Initialize(GoSocket socket, uint speedKHz = 25000)
  • with: public void Initialize(SPI.SPI_module displaySpi, Cpu.Pin displayChipSelect, Cpu.Pin displayGPIO, uint speedKHz = 25000). The core of the function simply initializes the SPI interface.
• Replaced
  • protected override void Dispose(bool disposing)
  • with: public void Dispose() also removing the following calls:
    • SetSocketPowerState(false); this call is used by the GoModule object model to control when a socket is powered ON / OFF and is no longer applicable when the VirtualCanvas class is not derived from GoModule.
    • base.Dispose(disposing); by the same token, deriving from GoModule requires overriding the behavior of the Dispose function and calling the base class at the end, something that is no longer needed now that the class implements the regular IDisposable interface.
• Moved
  • GoBusIrqEvent.Reset(); from WaitUntilGoBusIrqIsAsserted()
  • to: Execute(Synchronicity sync = Synchronicity.Synchronous) taking into account the higher latency of the Netduino / Plus

The rest of the code remained unchanged. The Netduino / Plus Touch Display driver and test application for this custom shield can be downloaded from the Netudino Helpers project on CodePlex in the \Samples\Coding4Fun directory.

Conclusion

With nearly 300 known Arduino shields on the market, there's a wide variety of plug-and-play functionality waiting to be leveraged by the Netduino community, sometime with little or no software work required. When all else fails, building a shield from scratch to get a desired feature onto a Netduino / Plus is a relatively simple process only requiring reading component datasheets and some patience with a soldering iron or a small breadboard. In our next article, we'll put these guidelines to the test with a simple and popular off-the-shelf data logger shield made by Adafruit Industries.

Appendix

To build the shield interfacing with the [nwazet Touch Display, you will need the following parts:

Custom shield Eagle schematics

• Custom shield

Bill of materials (~$15 for the shield w/o the [nwazet Touch Display module)

• 1 [nwazet Touch Display module
• 1 10cm FCI connection cable
• 1 Adafruit DIY shield for Arduino
• 1 ProtoAdvantage IDC Cable Breakout Board
• 1 PNP Transistor 2N4403
• 1 1K ohm resistor
• 1 10K ohm resistor
• 1 22 AWG solid core wire
• 1 0.32" diameter 60/40 rosin-core solder

Eagle libraries

• Adafruit Eagle library (Netduino shield part)
• [nwazet Eagle library (Go Socket part)

Netduino Go! Wiki

• Netduino GO! Module Builder's Guide (documents the Go! bus specifications)

Bio

Fabien is the Chief Hacker and co-founder of Nwazet, a start-up company located in Redmond WA, specializing in Open Source software and embedded hardware design. Fabien's passion for technology started 30 years ago, creating video games for fun and for profit. He went on working on mainframes, industrial manufacturing systems, mobile and web applications. Before Nwazet, Fabien worked at MSFT for eight years in Windows Core Security, Windows Core Networking and Xbox. During downtime, Fabien enjoys shooting zombies and watching sci-fi.

Overview

The external lighting system for Project Detroit is run on a web server on a Netduino Plus, which uses the .NET Micro Framework (NETMF). To get an accelerated start, we leveraged code from the CodePlex project Netduino Helpers to control the AdaFruit LPD8806 LED strip, and the web server base code from the Netduino forums. They have been tweaked slightly for our project.

Controllers, Routes, RESTful oh my!

We wanted the NETMF web server to have the exact same routes as the full-blown REST service layer in Detroit. With compiler conditionals, this allowed the same code to use the same routes on both the full .NET stack and NETMF. Since we couldn't use the ASP.NET MVC framework here, we had to make something close to it.

Controllers

Controllers implement an interface with a method named ExecuteAction. When GetController is called, we return the correct controller but cast as the interface IController. This lets us figure out what controller should be executing the request and continue to work generically in the context of the request connection thread. If new functionality is needed, we create a new controller that inherits from IController and update GetController:

private static void RequestReceived(Request request)
{
    // url comes in as /foo/bar/12345?style=255
    //
     // /CONTROLLER/ACTION/
    //
    // LIGHTING SAMPLE URL: /0/1/?r=255&g=255&b=255&zone=EnumAsInt
    // /0/... 0 = NetduinoControllerType.Lighting
    var validResponse = false;
    try
    {
        Logger.WriteLine("Start: " + request.Url + " at " + DateTime.Now);
        var urlInParts = request.Url.TrimStart(UriPathSeparator).Split(new[] {'?'}, 2);
        string[] uriSubParts = null;
        string[] queryStringSubParts = null;
        if (urlInParts.Length > 0)
            uriSubParts = urlInParts[0].Split(UriPathSeparator);
                
        if (urlInParts.Length > 1)
            queryStringSubParts = urlInParts[1].Split(QueryStringSeparator);
        if (uriSubParts != null && uriSubParts.Length > 1)
        {
            var targetController = GetController(uriSubParts[0]);
            if (targetController != null)
            {
                var action = (uriSubParts.Length >= 2) ? uriSubParts[1] : string.Empty;
                var result = targetController.ExecuteAction(action, queryStringSubParts);
                validResponse = true;
                request.SendResponse(result.ToString());
                Logger.WriteLine("Result: " + result + " from " + request.Url + " at " + DateTime.Now);
            }
        }
        if (!validResponse)
        {
            SendIndexHtml(request);
        }
    }
    catch(Exception ex0)
    {
        Logger.WriteLine(ex0.ToString());
    }
}

Routes, strings and enums on NETMF

NETMF can't parse an enum from a string, which is something that can be done in the full .NET Framework. But what does this actually mean? It means the routes for the NETMF web server are a bit unreadable. Here is the same route but with the parsing issue exposed:

Detroit's REST service route with ASP.NET MVC:
http://…/Lighting/Solid/?zone=all&r=255&g=0&b=0

Detroit's NETMF route:
http://…/0/3/?zone=0&r=255&g=0&b=0

This issue made direct device debugging a bit harder.

Parsing the Action and query string

Parsing the route is broken into two parts. The first is determines what controller to target. For Project Detroit, that meant the lighting system or the rear glass system. Once we know the controller, we still need to parse the query string and action into something more useful:

#if NETMF
private static readonly char[] QueryStringParamSeparator = new[] { '=' };
public static LightingData ParseQueryStringValues(string action, params string[] queryStringParameters)
#else
public static LightingData ParseQueryStringValues(string action, Dictionary<string, object> queryStringParameters)
#endif
{
    var data = new LightingData {LightingZone = LightingZoneType.All, LightingAction = ParseLightingActionType(action)};
#if NETMF
    if (queryStringParameters != null)
    {
        foreach (var queryString in queryStringParameters)
        {
            var valueKey = queryString.Split(QueryStringParamSeparator);
            if (valueKey.Length != 2)
                continue;
            var key = valueKey[0];
            var value = valueKey[1];
#else
        foreach(var key in queryStringParameters.Keys)
        {
            var value = queryStringParameters[key].ToString();
#endif
            switch (key.ToLower())
            {
                case ZoneHuman:
                case ZoneComputer:
                    data.LightingZone = ParseLightingZoneType(value);
                    break;
                // more stuff
            }
        }
#if NETMF
    }
#endif
    // more stuff
            
    return data;
}

Turning on the lights

Individually addressable LED strips allow us to tell each individual LED what color to be. To adjust the colors of the LED lights, we loop through all the LEDs, set their color value, and then call LedRefresh() on our LED strip. There's no need to set all of them, only the ones being updated:

private static void SolidAnimationWorker()
{
    var leds = GetLedsToIlluminate();
    var dataCopy = _data;
    for (var i = 0; i < leds.Length; i++)
        SetLed(leds[i], dataCopy.Red, dataCopy.Green, dataCopy.Blue);
    LedRefresh();
}

Zones / Animation Threads

A solid color is boring, but we can use procedural animations to add some flair. The current code supports the following animations:

• Solid - A single solid color.
• Fill - Goes from black to the desired color. The color stays lit. Much like filling a glass of liquid.
• Random - LEDs are set to a random value.
• Pulse - A glowing effect from bright to black.
• Snake - A one way lighting effect with a fading tail that moves left to right.
• Sweep - This effect was inspired by KITT and a Cylon robot and is similar to a snake pattern. The effect can also handle being split in two while maintaining a continual line for items such as the grill or the wheels.
• Police - Each side flashes a red and blue light.

Since we know the LED strip is a single strand, we can be clever with offsets and create groupings. By having a dedicated animation thread in each zone, we can have the grill of the car in Police Mode while the bottom fades to red and the vents fill to blue at different refresh rates.

To run an animation, we need to first kill the old animation thread, spin up a new thread, mark LEDs in the "All LED" zone as do not touch, and execute the new pattern. Without a way to mark the LEDs as "bad" in the "All" zone, the two animation threads would compete for setting the color and result in a flickering effect. Other zones don't have to worry about this since there is zero overlap of LEDs: 

public bool ExecuteAction(string action, params string[] queryStringParameters)
{
    var data = LightingDataHelper.ParseQueryStringValues(action, queryStringParameters);
            
    SignalToEndAnimationThread(data.LightingZone);
    int counter = 0;
    while (counter++ < 5 && IsThreadAlive(data.LightingZone))
        Thread.Sleep(10);
    if (IsThreadAlive(data.LightingZone))
        AbortAnimationThread(data.LightingZone);
    // moving data bucket into public so new threads can leverage it
    _data = data;
    if (_data.LightingAction == LightingActionType.Police || _data.LightingAction == LightingActionType.Sweep)
    {
        _data.LightingZone = LightingZoneType.Grill;
        if (IsThreadAlive(_data.LightingZone))
            AbortAnimationThread(_data.LightingZone);
    }
    // cleaning up just incase
    if (data.LightingZone != LightingZoneType.All)
    {
        for (var i = 0; i < _zoneAllAnimation.Length; i++)
        {
            var leds = GetLedsToIlluminate();
            for (var z = 0; z < leds.Length; z++)
            {
                // flagging item as bad in animation lib
                if (_zoneAllAnimation[i] == leds[z])
                    _zoneAllAnimation[i] = -1;
            }
        }
    }
    switch (_data.LightingAction)
    {
        case LightingActionType.Solid:
            Debug.Print("Solid");
            ExecuteThread(SolidAnimationWorker);
            break;
        // more patterns
    }
    return true;
}

We can now do something more interesting, like Police Mode:

private static void PoliceModeAnimationWorker()
{
    var leds = GetLedsToIlluminate();
    var dataCopy = _data;
    var length = leds.Length;
    var half = length / 2;
    var index = 0;
    while (IsThreadSignaledToBeAlive(dataCopy.LightingZone))
    {
        int redIntensity = (index == 0) ? 255 : 0;
        int blueIntensity = (index == 0) ? 0 : 255;
        for (var i = 0; i < half; i++)
            SetLed(leds[i], redIntensity, 0, 0);
        for (var i = half; i < length; i++)
            SetLed(leds[i], 0, 0, blueIntensity);
        LedRefresh();
        index++;
        index %= 2;
        Thread.Sleep(100);
    }
}

Improvements

Hindsight is 20/20. Nothing is ever perfect and there are of course items we'd love to change. At the start, the goal was to make this fairly generic and not Project Detroit "aware." As you can see by looking at the source, that wasn't accomplished due to time constraints.

We would use a different LED system for two reasons: data transfer speed and power. The longer the LED strip is, the longer it takes to address an item further down the line. Project Detroit, having 15 meters of lighting in it, was impacted by this issue. It works well, just not as well as we hoped. The voltage requirements of the LEDs are another consideration. They are 5V, while a car uses 12V. Had we used 12V LEDs, our wiring would have been far simpler and wouldn't have required multiple transformers to get the required amperage needed at the correct voltage.

Conclusion

We think the lighting solution, coupled with the time and material constraints, worked out pretty well.  Project Detroit was a ton of fun to build out and working with West Coast Customs was the chance of a lifetime.

Key Design Decisions

It’s important to keep in mind that this car was built for a TV show with a set schedule. As a result, there are a number of unique design decisions that came into play.

Schedule

Working backwards, the reveal for the car was set for Monday November 28, 2011 at the Microsoft Store in Bellevue, Washington. We started the project in early August, which gave us approximately 12 weeks for research, development, vehicle assembly, and testing. This was by far the #1 design decision as any ideas or features for the car had to be implemented by the reveal date.

Off the Shelf Parts

Another key design decision was to, where possible, use off-the-shelf hardware and software in order to allow  interested developers to build and reuse some of the subsystems for their own car (at least the ones that don’t require welding). For example, instead of buying pricey custom sized displays for the instrument cluster or passenger display, we used stock Samsung Series 7 Slate PCs and had West Coast Customs do the hard work of building a custom dash to hold the PC.

Hardware and Networking

The car is packed with a variety of computers and networking hardware.

• Instrument Cluster Slate – This slate is on the driver's side and manages the instrument cluster application and the On-Board Diagnostic (OBD) connection to read telemetry data from the car.
• Passenger Slate – This slate, which is built into the passenger's side, runs a custom Windows 8 application (see Passenger slate below).
  
• Laptop 1 – This laptop runs the REST service to control different parts of the car, the Kinect socket service for the front Kinect, and the user message service to display messages on the rear glass while driving.
• Laptop 2 – This laptop runs the Heads Up Display (HUD) service, the Kinect socket service for the back Kinect, the OBD-II database, and Azure services.
• Windows Phone – A Nokia Lumia 800 connects via WiFi and a custom Windows Phone 7 application (See Windows Phone application below).
• Xbox 360 – The Xbox 360 displays on either the passenger HUD or the rear glass display.
• Networking – A NETGEAR N600/WNDR3700 wireless router provides wired and wireless access for everything in the car, which is used in conjunction with a Verizon USB network card plugged into a Cradle Point MBR900 to provide an always-on 3G/4G LTE internet connection. The slates, laptops, and Xbox 360 are connected via CAT5e cable, while the Windows Phone 7 connects via WiFi.

Note: One of the limitations of the Kinect SDK is that if you have multiple Kinects plugged into one PC, only one of those Kinects can do skeletal tracking at a time (color/depth data works just fine). Because of this, we decided to have a dedicated laptop plugged into the front Kinect and another laptop plugged into the back Kinect in order to allow front and back skeletal tracking at the same time. If we'd not used simultaneous skeletal tracking, we could have combined all of the systems onto a single laptop.

Architecture

Here is a quick overview of the application architecture.

REST Service Layer

The REST Service Layer allowed different systems talk to one another. More importantly, it allowed different services to control hardware they normally wouldn't be able to access.

1. Thin client approach
   The solution we chose was to have all the services that control different parts of the car reside on the laptops and have client applications like the Windows Phone application send REST commands to execute an action so the service layer would execute the request.
   
   
2. REST-enable hardware
   Controlling hardware should be invisible to the consuming clients. For example, hardware that requires USB communication would be impossible to control with a Windows Phone. The service layer allowed us to control hardware in a way that was invisible to the end user.
   
   
3. Helper Libraries
   To simplify communication with the service layer, we built a set of helper classes to abstract out repetitive tasks like JSON serialization/deserialization, URI building, etc. For example, to get the list of car horn “ringtones”, the client application can call HornClient.List() to get back a list of available ringtone filenames. To set the car horn, the client calls HornClient.Set(filename), and to play the car horn, it then calls HornClient.Play(filename). The Helper libraries were built to work on Windows 7, Windows 8, and Windows Phone 7.

OBD-II

We have already released an article and library on the OBD-II portion of the car.  In short, OBD-II stands for On-Board Diagnostics. Hooking into this port allows one to query for different types of data from the car, which we use to get the current speed, RPMs, fuel level, etc. for display in the Instrument Cluster and other locations. OBD can do far more than this, but it's all we needed for our project. Please see the linked articles for further details on the OBD-II library itself.

For the car, because only one application can open and communicate with a serial port at one time, we created a WCF service that polls the OBD-II data from the car and GPS data from a Microsoft Streets & Trips GPS locator, and returns it to any application that queries the service.

For the OBD library, we used a manual connection to poll different values at different intervals. For values critical to driving the car—like RPM, speed, etc.—we polled for the values as quickly as the car could return them. With other values that weren’t critical to driving the car—like the fuel level, engine coolant temperature, etc.—we polled at a 1-2 second interval. For GPS, we subscribed to the LocationChanged event, which would fire when the GPS values changed.

Rather than creating a new serial port connection for every WCF request for OBD data, we created a singleton service that is instantiated when the service first runs. Accordingly, there is only one object in the WCF service that represents the last OBD and GPS data returned, which is obtained by the continual reading of the latest OBD data using the OBD library as described above. This means that calls to the WCF service ReadMeasurement method didn’t actually compute anything, but instead serialized the last saved data and returned it via the WCF service.

Since WCF supports multiple protocols, we implemented HTTP and TCP and ensured that any WCF service options we chose worked on Windows Phone, which, for example, can only use basic HTTP bindings.

To enable the ability to change the programming model later and to simplify the polling of the service, we built a helper library for Windows and Windows Phone that abstracts all the WCF calls.

The code below creates a new ObdService class and signs up for an event when the measurement has changed. The Start method does a couple of things: it lets you set the interval that you want to poll the ObdService, in this case every second (while the instrument cluster needs fast polling, the database logger can poll once a second). It also determines what IP address the service is hosted at (localhost), the protocol (HTTP or TCP), and whether to send “demo mode” data. Since one of the main ways the car is showcased is when it’s stopped on display, “demo mode” sends fake data, instead of always returning 0's for MPH, RPM, etc., so people can see what the instrument cluster would look like in action.

_service = new ObdService();
_service.ObdMeasurementChanged += service_ObdMeasurementChanged;
_service.Start(new TimeSpan(0, 0, 0, 0, 1000), localhost, Protocol.Http, false);

void service_ObdMeasurementChanged(object sender, ObdMeasurementChangedEventArgs e)
{
  Debug.Writeline("MPH=” + e.Measurement.MilesPerHour);
}

OBD-II Database & Azure Services

To record and capture the car telemetry data like MPH, RPM, engine load, and throttle (accelerator) position, as well as location data (latitude, longitude, altitude, and course), we used a SQL Server Express database with a simple, flat Entity Framework model, shown below. The primary key, the ObdMeasurementID is a GUID that is returned via the ObdService.  Just like above, the database logger subscribes to the ObdMeasurementChanged event and receives a new reading at the time interval set in the Start() method.

The Windows Azure data model uses Azure Table Services instead of SQL Server. The data mapping is essentially the same since both have a flat schema.

For Azure Table Storage, in addition to the schema above, you also need a partition key and a row key. For the partition key, we used a custom TripID (GUID) to represent a Trip. When the car is turned on/off a new TripID is created. That way we could group all measurements for that particular trip and do calculations based on that trip, like the average miles per gallon, distance traveled, fastest speed, etc. For the row key, we used a DateTimeOffset and a custom extension method, ToEndOfDays() that provides a unique numerical string (since Azure's row key is a string type) that subtracts the time from the DateTime.Max value. The result is that the earlier a DateTime value, the larger the number.

Example:

Time=5/11/2012 9:14:09 AM, EndOfDays=2520655479509478223 //larger
Time=5/11/2012 9:14:11 AM, EndOfDays=2520655479482804811 //smaller

Since they are ordered in reverse order, with the most recent date/time being the first row, we can write an efficient query to pull just the first row to get the current latitude/longitude without needing to scan the entire table for the last measurment.

public override string RowKey
{
    get
    {
        return new DateTimeOffset(TimeStamp).ToEndOfDays();
    }
    set
    {
        //do nothing
    }
}

public static class DateTimeExtensions
{
  public static string ToEndOfDays(this DateTimeOffset source)
  {  
    TimeSpan timeUntilTheEnd = DateTimeOffset.MaxValue.Subtract(source);
    return timeUntilTheEnd.Ticks.ToString();
  }

  public static DateTimeOffset FromEndOfDays(this String daysToEnd)
  {
    TimeSpan timeFromTheEnd = newTimeSpan(Int64.Parse(daysToEnd));
    DateTimeOffset source = DateTimeOffset.MaxValue.Date.Subtract(timeFromTheEnd);
    return source;
  }
}

To upload data to Azure, we used a timer-based background uploader that would check to see if there was an internet connection, and then filter and upload all of the local SQL Express rows that had not been submitted to Azure using the Submitted boolean database field. On the Azure side, we used an ASP.NET MVC controller to submit data. The controller deserializes the data into a List<MeasurementForTransfer> type, it adds the data to a blob, and adds the blob to a queue as shown below.

A worker role (or many) will then read items off the queue and the new OBD measurement rows are placed into Azure Table Storage.

public ActionResult PostData()
{
    try
    {
        StreamReader incomingData = new StreamReader(HttpContext.Request.InputStream);

        string data = incomingData.ReadToEnd();
        JavaScriptSerializer oSerializer =
            new JavaScriptSerializer();

        List<MeasurementForTransfer> measurements;
        measurements = oSerializer.Deserialize(data, typeof(List<MeasurementForTransfer>)) as List<MeasurementForTransfer>;

        if (measurements != null)
        {
            CloudBlob blob = _blob.UploadStringToIncoming(data);
            _queue.PushMessageToPostQueue(blob.Uri.ToString());
            return new HttpStatusCodeResult(200);
        }
        ...
    }
}

Instrument Cluster

Much of this is also covered in our previously released OBD-II library where the instrument cluster application is included as a sample. This is a WPF application that runs on a Windows 7 slate. It contains three different skins designed by 352 Media—a 2012 Mustang dashboard, a 1967 Mustang dashboard, and a Metro-style dashboard—each of which can be "swiped" through. This application queries the OBD-II WCF service described above as quickly as it can to retrieve speed, RPM, fuel level, and other data for display to the driver. The gauges are updated in real-time just as a real dashboard instrument cluster would behave.

HUD

The HUD (or Heads Up Display) application runs on one of the two Windows 7 computers in the car. This is a full-screen application that is output via a projector to a series of mirrors and a projection screen. This is then reflected onto the front glass of the windshield of the car. To install these, we altered the physical car's body and created brackets to mount mirrors and the projectors. In the picture on the left, you can see the dashboard's structural member pivoted outward. You can see the 12" section we removed and added in the base plate to allow light to be reflected through to the windshield. Bill Steele helped design and implement the physical HUD aspect into the car.

The HUD application has several different modes.  The mode is selected from the Windows Phone application.

• POI / Mapping – This uses Bing Maps services. The phone or Windows 8 passenger application can choose one of a select group of categories (Eat, Entertain, Shop, Gas). Once selected, the REST service layer is contacted and the current choice is persisted. The HUD is constantly polling the service to know what the current category is, and when it changes, the HUD switches to an overhead map display with the closest locations of that category displayed, along with your always updated current GPS position and direction. The list of closest items in the category is requested every few seconds from the Bing Maps API and the map is updated appropriately.
  
• Car telemetry - In the car telemetry mode, the OBD data from the WCF service described above is queried and displayed on the screen.  This can be though of as an overall car "status" display with the speed, RPMs, real-time MPG, time, and weather information.
• Weather – We use the World Weather Online API to get weather data for display on the HUD. This API allows queries for weather based on a latitude and longitude, which we have at all times. A quick call to the service gives us the current temperature and a general weather forecast, which we display as an icon next to the temperature in the lower-left portion of the screen.
  
• Kinect – Using our Kinect Service, with the standard WPF client code, we can display the rear camera on the HUD to help the driver when backing up. See the Kinect Service project for more information on how this works and to use the service in an application of your own.

Windows Phone Application

One of the main ways to control the vehicle is through the Windows Phone application. 

The first pivot of the app allows the user to lock, unlock, start the car, and set off the alarm.  This is done through the Viper product from Directed Electronics.

The second pivot contains the remaining ways that a user can interact with the car.

• Kinect – This uses the Kinect service much in the way the HUD does. It can display both the front and rear cameras as well as allow the user to listen to an audio clip and send it up to the car while applying a voice changing effect.
  
• Voice Effect – When the Talk button is pressed, the user can record their voice via the microphone. When released, the audio data is packaged in a simple WAV file and uploaded to the REST service. The user can select from several voice effects, such as Chipmunk and Deep. On the service side, that WAV file is modified with the selected effect and then played through the PA system. The code in this section of the app is very similar to the Coding4Fun Skype Voice Changer. We use NAudio and several pre-made effects to process the WAV file for play.
• Lighting – This controls the external lighting for the car. The user can select a zone, an animation, and a color to apply. Once selected, this is communicated through the REST service to the lighting controller.
  
• Messaging – This presents a list of known pictures and videos for the user. The selection is sent to the car through the REST service and displayed on the projector that is pointed at the rear window, allowing following drivers to see the image, video, or message. 
• Point Of Interest – As described earlier, this is the way the user can turn on the Point of Interest map on the HUD.  Selecting one of the four items sends the selection to the REST service where it is persisted. The polling HUD will know when the selection is changed and display the map interface as shown above.
• Telemetry – This is a replica of the instrument cluster that runs on the Windows 7 slate. OBD data is queried via the WCF service, just like the slate, and displayed on the gauges, just like the slate.
   
• Projection Screen – This will raise and lower the projection screen on the rear of the car.
• Horn – This displays a list of known horn sound effects that live on the REST service layer. Selecting any of the items will send a command through the REST to play that sound file on the external sound system of the car. This selected audio file would play when the horn was pressed in the car.
• Settings – Internal settings for setting up hardware and software for the car.

Passenger Application

The passenger interface runs on a Samsung Series 7 slate running the Windows 8 Consumer Preview. This interface has a subset of the functionality provided by the Windows Phone application, but communicates through the same REST service. From this interface, the passenger can set the car horn sound effect, view the front and back Kinect cameras, select a Point of Interest category to be displayed on the HUD, and select the image, video or message that will be displayed on the rear window.

External Car Lighting

The external lighting system was controlled by a web server running on a Netduino Plus using a Sparkfun protoshield board to simplify wiring, and allow for another shield to be used. The actual lights were Digital Addressable RGB LED w/ PWM. We'll also have a more in-depth article on this system on Coding4Fun shortly.

The car is broken down into different zones—grill, wheels, vents, etc. It also has a bunch of pre-defined procedural animation patterns that have a few adjustable parameters that allow for things like a snake effect, a sensor sweep, or even a police pattern. Each zone has its own thread which provides the ability to have multiple animation patterns going at the same time. When a command is received, the color, pattern, zone, and other data is then processed.

Here is a basic animation loop pattern.

private static void RandomAnimationWorker()
{
    var leds = GetLedsToIlluminate();
    var dataCopy = _data;
    var r = new Random();

    while (IsThreadSignaledToBeAlive(dataCopy.LightingZone))
    {
        for (var i = 0; i < leds.Length; i++)
            SetLed(leds[i], r.Next(255), r.Next(255), r.Next(255));

        LedRefresh();
        Thread.Sleep(dataCopy.TickDuration);
    }
}

Rear Projection Window

The rear projection system consists of two 4” linear actuators, a linear actual controller, the NETMF web server from above, a Seeed Studio Relay Shield, the back glass of a 1967 Ford Mustang, some rear projection film, a low profile yet insanely bright projector that accepts serial port commands, and a standard USB to serial adapter.

The REST service layer toggles the input of the projector based on the selected state.  This would allow us to go from the HDMI output of an Xbox 360 to the VGA output of the laptop.  While doing this, the REST layer sends a command to the NETMF web server to either raise or lower the actuators.

Here is the code for the NETMF to control the raising and lowering the glass:

public static class Relay
{
    // code for for Electronic Brick Relay Shield
    static readonly OutputPort RaisePort = new OutputPort(Pins.GPIO_PIN_D5, false);
    static readonly OutputPort LowerPort = new OutputPort(Pins.GPIO_PIN_D4, false);

    const int OpenCloseDelay = 1000;

    public static bool Raise()
    {
        return ExecuteRelay(RaisePort);
    }

    public static bool Lower()
    {
        return ExecuteRelay(LowerPort);
    }

    private static bool ExecuteRelay(OutputPort port)
    {
        port.Write(true);
        Thread.Sleep(OpenCloseDelay);
        port.Write(false);

        return true;
    }
}

Messaging System

This is a WPF application that leveraged the file system on the computer to communicate between the REST service layer and itself.  Visually, it shows the message/image/video in the rear view mirror but it actually does two other tasks, it operates our car horn system and plays the recorded audio output from the phone.

• Displaying Messages
  To display images, we poll the REST service every second for an update.  Depending on the return type, we either display a TextBlock element or a MediaElement.
  
  
• Detecting and Playing Car Horn
  When someone presses the horn in the car, it is detected by a Phidget 8/8/8 wired into a Digital Input. In-between the car horn and the Phidget, there is a relay as well. This isolates the voltage coming from the horn and solves a grounding issue. We then feed back two wires from that relay and put one into the ground and the other into one of the digital inputs. In the application, we listen to the InputChange event on the Phidget and play / stop the audio based on the state.
  
  
• Detecting new recorded audio from the phone and car horn changes
  When someone talks into the phone or selects a new car horn, the REST service layer places that audio file into a predetermined directory. The Messaging service then uses a FileSystemWatcher to detect when this file is added. The difference between the car horn detection and recorded audio is the recorded audio will play once it is done writing to the file system.

External PA System

To interact with people, we installed an external audio PA or Public Address system. This system is hooked into the laptop that is connected to the car horn, and can play audio data from the phone. Having a PA system that is as simple as an audio jack that plugs into a PC enabled us to have different ringtones for the car horn and to talk through the car using Windows Phone.

Conclusion

After months of planning and building, the Project Detroit car was shown to the world on an episode of Inside West Coast Customs. Though it was a ton of work, the end product is something we are all proud of. We hope that this project inspires other developers to think outside the box and realize what can be done with some off-the-shelf hardware, software, and passion.