Today, let's look at a new Coding4Fun Kinect project and one that could open up some very interesting ways to build awesome apps.

Image the Kinect (and PC) in one room and your app in another (room, building, state, country). Say something that can run on the machine that has the Kinect connected to it but send the camera, depth, skeleton and audio to an app running on another computer or device, like a Windows Phone 7...

You don't need to imagine, because there's the...

Coding4Fun Kinect Service

The Coding4Fun Kinect Service allows you to stream Kinect color, depth, skeleton, and audio from one PC to another PC or a Windows Phone via sockets. Both server and client libraries are available to send and receive the data. Please review the included WPF and Windows Phone samples for a quick explanation of how to use the libraries. Also check out the Documentation tab for more information and sample usage.

This project requires the Kinect for Windows SDK v1.0. If you want to learn how to use the Kinect for Windows SDK, head over to the Channel 9 and view the Quickstart series.

...

Project Information URL: http://kinectservice.codeplex.com/

Project Download URL: http://kinectservice.codeplex.com/releases/view/81291

Project Source URL: http://kinectservice.codeplex.com/SourceControl/list/changesets

Runs with v1? Yes

Here's the C# Solution;

And the VB (yes, there's a full VB.Net version) Solution;

Here's a snip of code from the C# Phone Sample;

void client_ColorFrameReady(object sender, ColorFrameReadyEventArgs e)
{
    if(e.ColorFrame.BitmapImage != null)
        this.Color.Source = e.ColorFrame.BitmapImage;
}

void client_DepthFrameReady(object sender, DepthFrameReadyEventArgs e)
{
    if(_outputBitmap == null)
    {
        this._outputBitmap = new WriteableBitmap(
            e.DepthFrame.ImageFrame.Width, 
            e.DepthFrame.ImageFrame.Height);

        this.Depth.Source = this._outputBitmap;
    }

    byte[] convertedDepthBits = this.ConvertDepthFrame(e.DepthFrame.DepthData, e);

    Buffer.BlockCopy(convertedDepthBits, 0, _outputBitmap.Pixels, 0, convertedDepthBits.Length);
    _outputBitmap.Invalidate();
}

void client_AudioFrameReady(object sender, AudioFrameReadyEventArgs e)
{
    _kinectSound.SubmitBuffer(e.AudioFrame.AudioData);

    if(_kinectSound.State != SoundState.Playing)
        _kinectSound.Play();
}

private void client_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    Skeleton skeleton = (from s in e.SkeletonFrame.Skeletons
                             where s.TrackingState == SkeletonTrackingState.Tracked
                             select s).FirstOrDefault();

    if(skeleton == null)
        return;

    SetEllipsePosition(headEllipse, skeleton.Joints[(int)JointType.Head]);
    SetEllipsePosition(leftEllipse, skeleton.Joints[(int)JointType.HandLeft]);
    SetEllipsePosition(rightEllipse, skeleton.Joints[(int)JointType.HandRight]);
}

Contact Information:

• Blog: http://www.brianpeek.com/
• Twitter: @BrianPeek

Terrain Rendering

Terrain Rendering is a game technology code sample that demonstrates how to render large-scale terrains in real time by efficiently distributing the tasks between the CPU and the GPU. This article provides an overview of the terrain-rendering application and includes a link to the free code.

Introduction

This sample demonstrates how to render large-scale terrains on Intel® microarchitecture codename Sandy Bridge in real time by efficiently distributing the tasks between the CPU cores and the processor graphics unit. The sample pre-processes an input height map into a hierarchical quadtree representation which is used to render the terrain with adaptively selected level of detail (LOD). The adaptive simplified triangulation calculated during the pre-processing is compactly encoded to save runtime processing and memory space. LOD construction is asynchronously performed by the CPU cores while rendering is done by the processor graphics unit.

Application

Terrain Rendering is an application using DXUT and Microsoft DirectX* 11 with D3D_FEATURE_LEVEL_10_0. The application handles all rendering, user interaction and GUI. Upon initialization, the application loads all models, allocates resources and compiles shaders. On the first run, the application pre-calculates triangulation, which can take some time (up to one minute), and stores it on the disk. On subsequent runs, the application loads the data from the disk.

Overview

Terrain rendering demonstrates how to render large-scale terrains on Intel microarchitecture codename Sandy Bridge in real time by efficiently distributing the tasks between the CPU cores and the processor graphics unit. The terrain rendering can be optimized by constructing simplified triangulation, which is adaptive to the terrain surface characteristics. Such triangulation contains more primitives in sharp regions with high-frequency details and allocates a small number of large triangles for flat areas. This significantly reduces the total triangle count while providing almost the same visual quality (compare fig. 1 and 2).

Full-resolution triangulation (fig. 1) is more than 5x redundant compared to adaptive (fig. 2), which results in nearly 3x lower performance with almost the same visual quality.

Pre-computing adaptive triangulation is a complex and computation-intensive task. Doing this at runtime could require a significant amount of time and would lead to perceptible stalls or delays. To solve this problem, the application pre-computes adaptive triangulation for the whole terrain at the pre-process stage and stores the resulting data on the disk in a compact representation (see section 4). For example, the whole encoded triangulation for an 8192×8192 terrain consumes approximately 6 MB (compare this to 128 MB required to store the height map). At the runtime stage, this data is used to efficiently construct the triangulation.

...

SNB Terrain

This sample demonstrates how to render large-scale terrains on Intel® microarchitecture code name Sandy Bridge in real time by efficiently distributing the tasks between the CPU cores and the processor graphics unit. The input height map is preprocessed into a hierarchical quad tree representation which is used to render the terrain with adaptively selected level of detail (LOD). The adaptive simplified triangulation is compactly encoded to save runtime processing and memory space. LOD construction is asynchronously performed by the CPU cores while rendering is done by the processor graphics unit, which provides stable frame rate.

System Requirements

Hardware:

• CPU: Intel® Advanced Vector Extensions (Intel® AVX)-enabled x86 (Intel® microarchictecture code name Sandy Bridge or better suggested)
• GFX: uses Microsoft DirectX* 11 graphics API on Microsoft DirectX* 10 (or better) hardware
• OS: Microsoft Windows* 7 SP1 (for Intel AVX support)
• MEM: 2 GB of RAM or better

Software:

Toolkits Supported:

• Microsoft DirectX* SDK (June 2010 release or later)
• Microsoft Windows* SDK May 2010

Compilers Supported:

• Microsoft Visual Studio* 2010 (Intel AVX support required) or Intel® C++ Compiler version 11

Libraries Required:

• Microsoft* C Run-Time Libraries (CRT) 2008/2010

Getting this to compile and run on my notebook wasn't hard, once I got the DirectX SDK installed. Here's some snaps of it running on my notebook;

Here's a snap of the Solution;

So what good is this? I'm not really sure...lol. I just thought it was pretty cool and fun, to be able to "fly" around the terrain, to play with the settings and know that I've got the source for how it was generated. That said, the best part was the original CodeProject article with the details and depth it went into. If you've ever wondered about terrain retendering, this is an article you might want to check out...

If you have, or download, beta 2 based code/project and it doesn't compile, one of your first stops should be this post collection.

Kinect for Windows – Code Migration from Beta2 to v1.0 (C#/VB)

There have been a number of significant changes and improvements in our APIs since Beta2. This set of documents attempts to detail the changes to facilitate code migration from beta 2 to v1.

API Change Details

[these links take you to a seperate post with details]

• Namespace and Assembly Name changes
• Runtime type (Rename to KinectSensor, refactoring)
• ColorImage API changes
• DepthImage API changes
• Skeleton API changes
• Mapping API changes (Skeleton -> Depth, Depth -> Color)
• Audio API changes
• Speech API changes

Adapting to API Changes

In order to adapt to this change, C#/VB developers should:

• Backup code projects (if not using source control, such as TFS)
• Uninstall Beta 2 SDK (including speech runtime 10.x components).
• Install Kinect for Windows SDK v1.
• Migrate code to use v1 APIs and best practices via one of the migration techniques below
• If you need help with the answer, go the Kinect for Windows forums. Search for other similar issues first. If needed, create a new issue. Please prefix the title of the forum post with “Migration issue:”. (for example: “Migration Issue: Runtime.Sensors replacement?”)
• If you figured out the answer, and want to suggest improvements to the details and techniques in these documents, please leave a comment (We generally will respond to that feedback, but won’t show the comments, as it may get unwieldy).

Where to go for help

Code Migration Techniques (C#/VB)

There are three major approaches to migration

1) RECOMMENDED: Use a migration reference assembly to assist with learning about API changes

2) RECOMMENDED: Go back to the examples you may have started from, find the newest versions of those, and transfer your improvements to this new version.

3) NOT RECOMMENDED: Change reference assembly, (recompile -> search for info -> change -> adapt -> repeat)

...

Project Information URL: http://robrelyea.wordpress.com/2012/02/01/k4w-code-migration-from-beta2-to-v1-0-managed/

Contact Information:

• Blog: http://robrelyea.wordpress.com/
• Twitter: @rrelyea

Now that the Kinect for Windows SDK is in a v1 state and we can start officially deploying Kinect for Windows SDK based applications, one of the things we're going to be facing is just that, deployment and setup issues. Rob Relyea has kicked off helping us with just that, providing suggestions and tips...

Kinect Apps – ensuring Kinect Runtime is installed

Kinect for Windows 1.0 enabled apps should ensure that the Kinect Runtime is installed wherever the app is installed.

[Note: Kinect for Windows 1.0's latest public preview is beta 2. Parts of this blog post may be applicable to beta 2, but is primarily focused on the final v1.0 version, coming February 1st. Since v1.0 is not yet released, information I give here may change when it does release. I also am filtering this information to ensure that I am not giving away details that we don't yet want to release.]

Guidelines

App installers should install its dependencies, including Kinect Runtime – CRITICAL

Kinect for Windows 1.0 will have a KinectRuntime-v1.0-Setup.exe that your app installer MUST chain install. In addition to installing Kinect specific software (drivers + runtime), KinectRuntime-v1.0-Setup.exe will ensure the following dependencies are installed:

• VCRT x86 and/or x64
• .NET 4 client profile (or later 4.x versions like .NET 4.5)
• Microsoft Speech Runtime v11 x86 and/or x64

Running an app should provide a decent user experience if Kinect Runtime isn’t installed

...

Project Information URL: http://robrelyea.wordpress.com/2012/01/11/kinect-apps-ensuring-kinect-runtime-is-installed/

public partial class App : Application
    {
        protected override void OnStartup(StartupEventArgs e)
        {
            if (IsKinectRuntimeInstalled)
            {
                //go ahead and load the StartupUri as defined in App.xaml
                base.OnStartup(e);
            }
            else
            {
                MessageBox.Show(Assembly.GetExecutingAssembly().FullName + " is not able to excecute."
                + "An important dependency should have been installed by its setup program: Microsoft Kinect Runtime 1.0");
            }
        }

        public bool IsKinectRuntimeInstalled {
            get
            {
                bool isInstalled;
                try
                {
                    TestForKinectTypeLoadException();
                    isInstalled = true;
                }
                catch (FileNotFoundException)
                {
                    isInstalled = false;
                }
                return isInstalled;
            }
        }

        // This Microsoft.Kinect.dll based type, must be isolated in its own method
        // as the CLR will attempt to load the Microsoft.Kinect.dll assembly it when this method is executed.
        private void TestForKinectTypeLoadException()
        {
#pragma warning disable 219 //ignore the fact that status is unused code after this set.
            var status = KinectStatus.Disconnected;
#pragma warning restore 219
        }
    }

Contact Information:

• Blog: http://robrelyea.wordpress.com/
• Twitter: @rrelyea

This week we're highlighting another framework that while only available in binary form (through of course the demos and samples include their source) is also free, has been around a while longer, and has a much broader deployment reach, including not only Windows Phone 7, but also XBox 360, Windows and even the Zune!

DPSF (Dynamic Particle System Framework)

DPSF (Dynamic Particle System Framework) is a tried and tested, free programmer's tool for creating custom particle systems in XNA quickly and easily.

Incorporate particle effects into your project within a matter of minutes by using the provided Default Classes.

Unlike other particle system APIs / libraries, DPSF is flexible and allows you to code your own custom behaviors into the particle system; You are not limited to only using the parameters provided by the framework. You can create and control your own particle properties to make just about any effect you can imagine; If you can code it, you can create it in DPSF.

Upload particle systems you create to the DPSF forums and download particle systems created by others.

Check out the demo videos to see some of the things you can do with DPSF, or go ahead and download DPSF and try it out for yourself.

If you use DPSF in your project, be sure to post a link to your project on the the DPSF forums.

Features

Here is a list of some of the features DPSF provides:

• A single API for multiple platforms: supports 2D and 3D particles for Windows, Xbox 360, Windows Phone 7, and Zune.
• Easily integrates with graphics engines, including Synapse Gaming's SunBurn engine.
• Full API documentation is provided in the help file, as well as in the online help documentation.
• Tutorials and their source code are provided in the installer. The tutorials (without source code) are also available in the online help documentation.
• Allows particle systems to be created in minutes by using the Default Particle Systems provided; Just set values for the built-in parameters, such as position, velocity, acceleration, rotation, external force, start/end color, etc.
• The Default Particle Systems may be extended, allowing you to provide any extra required functionality. Want your particles to have a weight property that determines how fast they accelerate? Or do you want your particles to follow a specific path or pattern? You can code the behavior to make it happen!
• Templates to start from are provided to make creating new particle systems quick and easy.
• You write the particle system code, giving you full control over the particle system and its particles, allowing you to create any type of particle system effects you desire. Your imagination is the limit.
• Easy to integrate into existing projects; just add a reference to a dll file.
• Use the built-in Effects (i.e. shaders) as well as custom Effects.
• Modify the default Effect (i.e. shaders) to create new custom Effects quickly and easily.
• Particle System Managers are provided to make updating and drawing many particle systems easy.
• An Animations class is provided for easily creating animated particles.
• Easily create a sequence of images, tile-sets, or animated gifs displaying particle system animations.
• Particle systems may be implemented as DrawableGameComponents if required.

Check out these videos;

DPSF show's its maturity in a number of ways. First, there's good documentation.

Secondly there's a nice set of tutorials;

And finally the demo app is pretty awesome, showing off many of the capabilities of the framework. The demo ran for me the first time, with only a reference fix-up required;

All I needed to do was point that DPSF reference to the folder where I installed the framework and all was well.

One of my favorites (not sure why, maybe because the dog is cute...lol) is the Photo demo.

This demo shows off some of the cool things that DPSF can do. From 3D features;

To different ways the photo can be "particle'ized"

And yes, you get the source for this demo and particle systems.

If you are looking around for a mature particle system, one with a price and license that's hard to beat, you run, not walk, and check out DPSF.

If you follow Pete, it probably won't surprise you to know he's gotten his hands on a PIX-6T4 and has shared his recent experience building the kit and writing his first "real" game for it.

Assembling the PIX-6T4 Netduino-powered Hand-Held Game System

I recently picked up a PIX-6T4 build by Fabien Royer (with games by Fabien Royer and Bertrand Le Roy). This is a 64 pixel, two joystick/button, monophonic sound hand-held game device based around the Netduino Mini from Secret Labs. You create games in C# using Visual studio.

Disclaimer: I work for Microsoft and I enjoy working in the .NET Micro Framework as well as C++ on other microcontrollers. I purchased this product on my own, at full price; this is not a sample or review unit. Presumably, I got the same package of goodies everyone else gets.

Unboxing

The kit came in a regular USPS shipping box inside which were four bags. Two bags had components, and two had joysticks.

Conclusion

I'm impressed with what Fabien has come up with here, and the stock games he and Bertrand have done are just perfect. I think the board and enclosure could use a little more design to make it more compact and also more hand-friendly, but overall, I think this is an excellent way to get into Netduino programming using something fun and exciting. Also, because of the display technology, you are constrained to creating games with very simple graphics (just LEDs) so, by necessity, you avoid that common barrier to entry. Sometimes constraint is a good thing.

Congratulations Fabien and Bertrand!

My First Real PIX-6T4 Game: Sixty4Racer

After assembling my Netduino-powered PIX-6T4, I wanted to go and write a simple game. This post describes the construction of that game, including all source code.

Concept

When you have 64 monochrome red pixels, you need to keep the graphics simple. I decided on a game inspired by the classic Atari River Raid game. This is essentially a vertical scrolling game where you need to dodge obstacles with your boat. Variations included things like Spy Hunter on the C64 and many many others. Most of those games also involved shooting and enemies, but that's a but more complex than you can reasonably do on this board. I won't enable moving walls like Laser Gates, but I'll leave things open enough not to make it impossible to do that in the future.

The game had to be small enough that I could figure out the API, and then design, code, and blog about it in a single evening after my kids went to bed The PIX-6T4 is fun, but I have way too many projects on my backlog to be able to devote any significant time to it (here's a taste: a ShapeOko CNC mill, an AVR MIDI->CV Converter, the final touches on the MIDI Thru Box, several MFOS Synth Modules, Several Gadgeteer Board Concepts, a Win8 XAML book, chapters to review in my Silverlight 5 book, and much much more). In fact, that was one of the big selling points of this device: simple gameplay and quick to develop for. Combined with the great library Fabien designed, and my past experience with Netduino and, more specifically, C#, and this should be an evening project.

Screen Design

Back in the 80s, in 7th grade, I used to design single-color sprites for the commodore 64. The sprites themselves were 3 bytes wide, with each pixel represented as a single bit in the byte. I used to define them on graph paper, but alas, the notebooks I filled with sprites and BASIC listings have long since disappeared.

...

...

First Iteration: Creating the scrolling playfield

I called my project PeteBrown.Sixty4Racer. Just as in the previous post, I copied over the Program.cs file from another project and used that as the start. Please read my previous post to see what references you need and whatnot.

The first class I created was the one that manages the creation of the screens.

...

Second Iteration: Adding in the player

The PIX-6T4 libraries have built-in the concept of a PlayerMissile. This is a single pixel on the playfield. It may move, so it has X and Y speed. You can show or hide it, so it has Visibility. And most importantly, it has collision detection with other PlayerMissile instances. For our game, we're not going to use that, since we're looking for collision detection with the background. So, a little manual detection is in order.

...

Third Iteration: Polishing

The first thing I realized was that it was really hard to make out the player pixel in the sea of red. That's to be expected on a monochrome display at 8x8 resolution. The approach I came up with to make it a bit easier is to simply flicker the player pixel. Each time the game loop executes, I toggle the visibility of the ship PlayerMissile to give it a nice seizure-inducing flicker.

...

Final Steps

The final things to do are to create the manifest file and bitmap which will be used on the SD card. I'll need to check with Fabien to see what the exact format of the .bin file is, but I suspect it's just the 8 bytes of data formatted like all the other bitmap data in this application. I'm also not sure if he has a nice little app to write that data out, or convert from a bitmap, or something else. I ended up just using a hex editor to recreate the pattern from one of the images I created in the font editor.

...

What You Can Do

This came is completely free and open source. While I'd love credit for the initial work, it's not a requirement. Go ahead and do whatever you'd like with the source and have a blast

Here's a video of the game in action.

In short, if you have a hankering to get your soldering iron hot and build your own hand-held game console, one that you can also write your own games for, the PIX-6T4, and this guidance from Peter, has your name all over it...

WPF Sample Screenshots

Windows Phone Sample Screenshots

Clearly, the Mini does not have 512 pins and so we use Persistence of Vision to control the LEDs in the cube. The result is that seven pins on the Netduino Mini can control 512 LEDs.

At the end of this article you should be able to create a cube capable of the following:

Bill of Materials

The following items are also required:

• 5V Power capable of delivering 2A
• Solder (I used about 15 metres over the life of this project).
• One 30cm x 30cm piece of wood
• Drill and wood drill bits (4mm – 5mm)
• Some cardboard and tape
• A fair amount of patience
• Good attention to detail

You may need to shop around for the LEDs. I went to my preferred supplier and was quoted 18.5 pence each. By going on eBay I managed to negotiate the price down to 2.5 pence each including delivery. You are buying in bulk so don’t be afraid to ask for a good bulk price. Also, buy a few more than you need in case you have a faulty part or break a few.

Building the Cube

This is probably the most time consuming part of the project requiring a lot of patience and testing. The aim of the project is to convert these 512 blue LEDs:

Into this:

Shaping the LEDs

The exact dimensions of the cube depend upon the length of the legs on the LEDs. The legs are soldered together with the cathodes of the LEDs forming a horizontal plane and the anodes vertically connecting the layers. Each of the layers are configured into a common cathode display.

Take one of the LEDs and bend the cathode at a point as close to the body of the LED as possible. The cathode should be at 90 degrees to the anode and parallel to the flat base of the LED. This allows the horizontal legs to be soldered together to form the common cathode.

Shaping the anode is a little more difficult. The anodes connect the horizontal layers and so need to be shaped to ensure that each LED in the layer is directly above the corresponding LED in the layer below. Otherwise, each layer will be slightly offset from the one beneath it.

To achieve this, bend the anode in the opposite direction as described for the cathode. Then, still working with the anode, start about 2mm along the leg. Finally, bend the top 2-3mm through and fashion an angle at 90 degrees pointing back to the LED body. The end result should look like this:

The horizontal leg is the cathode and the vertical leg is the anode. When these are connected it should look like this:

Notice how the shaped anode allows the LEDs to be placed above each other.

Now repeat this process for the remaining 511 LEDs.

Building a Layer

Our next task is to connect the LEDs together. This is best achieved by using a template for the individual layers. Use a piece of wood with a grid of holes drilled into it as the template.

Measure the distance from the center of the now horizontal cathode to the end of the leg and subtract about 2mm. This will tell you how far apart the LEDs are in the horizontal plane. The 2mm will be used to overlap with the neighboring LED and will connect the cathodes. Now drill an 8 x 8grid of holes, each of which are distance of your choosing apart.  Here they are 20mm apart.

The holes need to be large enough to hold the LEDs securely but should not be so small that the LEDs are wedged into the hole and difficult to remove.

Take eight of the LEDs and place them along the top row with the cathodes all pointing to either the right or the left. If all are pointing to the right, the cathodes of seven LEDs will overlap slightly with the next LED to the right. The rightmost cathode will go off into space. Solder the cathodes together.

Now let’s add the remaining LEDs in the layer. I started on the left because I hold the soldering iron in my right hand. Take another seven LEDs and place these under the top row and down the far left column. The cathode of each first LED should overlap the cathode of the LED in the top layer. Solder these together.

Repeat with the remaining columns. At this point, you should have a horizontal row of LEDs connected together with eight strings of seven LEDs hanging from it. Eventually you should have something looking like this:

Now test the layer using a power supply and current limiting resistor. A 5V supply and a 68 Ohm resistor are adequate. Ground the cathode of the LED pointing off into space. Now touch each leg of the LEDs in turn with the positive output of the supply (through the current limiting resistor, of course). Each LED should light up and as you touch the anode.

One final bit of soldering is needed to add a stiffening wire to the layer. Cut and strip a piece of wire. The wire should be long enough to cross the entire layer. Place the stripped wire on the cathodes of the LEDs at the bottom of the string and solder it on to each.

At this point you will have one complete layer. Remove the layer from the template. This should be done carefully so that you do not put too much stress on the joints. Gently lifting it up with a screwdriver should help. Don’t hurry. Put this layer to one side and repeat another seven times.

Connecting the Layers - The Cube Appears

Now that all of the layers are built, test them again. This repeated test may save you future pain. Just imagine how difficult it will be to fix a bad joint in the middle of the cube. Check that the LEDs are still connected and none of the joints were broken when layers were removed from the template.

Now drop one of the layers back into the template. What we now need to do is to place a second layer on top of the first so that the anodes of the layer in the template touch the anodes of the LEDs in the upper layer. Once in place, we need to solder the anodes together, and so need a way of supporting the top layer whilst connecting the LEDs. A strip of cardboard can accomplish this. Cut the cardboard into strips (making sure the cardboard is the height needed to support the layer) and bind the strips together with tape. Two of these strips should be enough to support the layer. Here’s how it looked when several layers had already been connected:

Now that you have the layer supported, solder the anode of each LED in the top layer to the anode of the corresponding LED in the layer directly beneath it. Once this has been done, test each LED. Connect the cathode of the bottom layer to ground and touch each of the legs on the top layer in turn with the positive supply (going through the current limiting resistor). The LED on the bottom layer should light up. Repeat for each LED in the layer. Move the cathode to the top layer and repeat the test—this time the LED on the top layer should light. Again, repeat for each LED in the layer.

Now add the remaining layers. Just for safety, test every layer in the cube as it is built up. This repeated testing sounds like a big pain, but trust me it's worth it. At this point you will have a cube of LEDs looking something like this:

Now trim the cathodes that are still flying off into space.

Building the Controller Board

The controller board will allow any of the 512 LEDs in the cube to be turned on by a Netduino Mini using only seven pins.

Persistence of Vision

Connecting the anode to a positive supply while grounding the layer in which the LED is located can turn on any LED. You can also do this with more than one LED in a layer, and so within a layer can turn on 0 to 64 LEDs. To light the entire cube, we need to switch on each layer in turn while doing so fast enough to give the impression of static image. This is where the principle of Persistence of Vision comes into play.

The basic algorithm is as follows:

• Layer = 0
• Repeat
  • Connect the anodes of the desired LEDs in Layer to positive supply
  • Connect Layer to ground (this turns the layer on)
• Move to next Layer

If we do this fast enough, the human eye will see a single image as when watching TV or a movie.

Our basic building blocks for the controller are as follows:

• Microcontroller (a Netduino Mini) to run the whole show
• Hardware to select the layer to be turned on (74HC238 and TIP122 transistors)
• Hardware to turn on the LEDs (Shift registers)

Shift Registers

The shift registers determine which LEDs are turned on. The board has a series of eight cascaded 74HC595 registers. This allows us to control 64 LEDs (8 chips x 8 outputs). The following schematic shows how two of these registers should be wired together:

The above should be repeated until you have eight shift registers cascaded.

The output from the register is 5V and will give more than enough power to burn out an LED, so we need to put a current limiting resistor in the circuit. A 68 Ohm resistor is required for the LEDs in this project—make sure you verify the value you choose against the LEDs used.

Putting this together gives the following layout:

Each socket holds a 74HC595 shift register. The connections are identical for each register with the data cascaded into the next register. So if we look at the bottom left socket you will see the following:

• Below the socket there is a connector that allows the output to be connected to the LEDs in the cube.
• Above the connector are the resistors that limit the current flowing through the LEDs.
• The socket above that will hold the 74HC595 shift register.
• To the left of the socket is a 100nF capacitor. This smooths out the power supplied to the shift register and is connected between the power input and ground and placed as close to the IC as possible.

The following colors have been used:

• Red = Power
• Blue = SPI and cascaded data
• Yellow = RCK
• White = SPI clock

Note that the connectors and current limiting resistor are slightly offset as output 0 is on pin 15 and this is routed on the underside of the board. The remaining connections are a direct one to one path from the pin to the resistor/connector.

The microcontroller uses the Serial Peripheral Interface (SPI) bus to tell the shift registers which LED to turn on. The data from the registers is cascaded and so we can store 64 bits of information (1 bit for each LED in a layer). With logic 1 turning a particular LED on and 0 turns the LED off.

Layer Switching

The layer switching logic allows the controller to connect any one of the layers to ground (using a common cathode). Coupling this with the LED selection logic above allows us to turn on any one of the LEDs in the cube. This is achieved by using a transistor as a switch. The TIP122 was selected because it is capable of sinking 2A. This may seem like a lot considering the 25mA for each LED, but remember that we potentially have 64 LEDs being turned on at once. This means we may be drawing 1.6A of current. If you use a different LED, you will need to verify that the shift registers, power supply, and the transistor are capable of dealing with the amount of power you will be drawing.

The schematic for the layer switching looks like this:

The 74HC238 has three input signals. These represent a binary number 0-7. The chip converts this number into eight output lines. 0 turns on line 0, 1 turns on line 1 etc. The output from each line (0 through 7) is then fed in the base of a TIP122 transistor. This turns on the appropriate layer by connecting the layer through to ground.

One line for the Netduino Mini controls enable line on the 72HC238 chip. This allows us to turn all of the outputs off whilst a new value is being loaded into the chip. This line is used to make sure that the transitions between the states are “invisible.” Without this there is the chance that the viewer may see a flickering effect when new values are loaded into both the 74HC595s and the 74HC238.

The Completed Controller Board

The completed controller board looks something like this:

Note that there are a pair of connectors to the top right and bottom left of the Netduino Mini. The pair at the top right breakout the COM1 port. The two at the bottom left allow grounding of an FTDI lead (connected to the controller board) and also one socket that is not connected. This allows the 5V lead on the FTDI connector to have a place to live and not be in danger of touching something it shouldn’t.

And on the underside:

Connecting the Cube and Controller

The final task (from a hardware point of view) is to connect the cube to the controller board. I tried both ribbon cable and alarm cable. The alarm cable was a little more difficult to connect but was flexible. The ribbon cable was easier to work with but was not as flexible. The principle is the same whichever you chose.

Place the cube on a flat surface (the anodes touching the surface) with one face of the cube facing you. The connections should be made so that the lower back left corner is co-ordinate (0, 0, 0). The co-ordinates increase moving to the right, towards you and up. So looking at the controller board above, shift register 0 connects to the LEDs farthest away from you with output 0 from the register connecting to the LED to the far left. Cut and make the 8 cables according to this pattern varying the lengths to suit the location of the controller board with respect to the cube. Each cable will need a single eight-way socket on one end with the other end connected to the appropriate LED.

The layer selection logic should be connected using a similar cable. Each layer should be connected to a TIP122 with layer 0 being the bottom layer.

The cube will then look like this:

Connecting it up to the controller:

I found it easier to be consistent and wire each plug and LED identically, and so all of the connections above have a black wire to the right of the connector. It helps with connecting things up later.

If we have everything connected then we only need one more thing...

Software

The software running the cube needs to perform two main tasks:

• Work out which LEDs are turned on
• Run the display (cube)

These two tasks need to be performed at the same time (or so fast that they appear to run at the same time). Luckily the .NET Micro Framework has a built in mechanism to allow us to do this—threading. Threading allows us to do this by running two tasks interleaved. So task 1 will run for a while, the system will then switch and run task 2 for a while, then back to task 1 and so on.

To do this, the software is split into two parts, the main program that decides what to display and a separate class that runs the display.

LEDCube Class

This class has only one purpose in life, to output data to the controller board and so “run” the display. It is a relatively simple class containing the following methods:

• Constructor
• Buffer update method
• Display Buffer method

The constructor sets everything up by instantiating an instance of the SPI class and setting the buffer (which contains the data to be displayed) to be empty, effectively clearing the cube:

public LEDCube()
{
    config = new SPI.Configuration(SPI_mod: SPI.SPI_module.SPI1,
                                   ChipSelect_Port: Pins.GPIO_PIN_20,
                                   ChipSelect_ActiveState: false,
                                   ChipSelect_SetupTime: 0,
                                   ChipSelect_HoldTime: 0,
                                   Clock_IdleState: true,
                                   Clock_Edge: true,
                                   Clock_RateKHz: 100);    spi = new SPI(config);
    buffer = new byte[64];
    for (int index = 0; index < buffer.Length; index++)
    {
        buffer[index] = 0;
    }
}

The buffer is 64 bytes of data, 8 rows of 8 bytes. Each byte corresponds to a vertical layer in the cube.

The UpdateBuffer method allows the calling program to change what is displayed in the cube. A little control is needed here to ensure that the pattern displayed in the cube is consistent, and so locking is used to ensure that the buffer cannot be updated part way through a display cycle:

public void UpdateBuffer(byte[] newValues)
{
    lock (buffer)
    {
        for (int index = 0; index < buffer.Length; index++)
        {
            buffer[index] = newValues[index];
        }
    }
} 

The final method in this class is the method spawned off into it’s own thread:

public void DisplayBuffer()
{
    while (true)
    {
        lock (buffer)
        {
            byte[] displayData = new byte[8];            for (int row = 0; row < 8; row++)
            {
                int offset = row * 8;                for (int index = 0; index < 8; index++)
                {
                    displayData[index] = buffer[offset + index];
                }                enable.Write(true);
                bit0.Write((row & 1) != 0);
                bit1.Write((row & 2) != 0);
                bit2.Write((row & 4) != 0);
                spi.Write(displayData);
                enable.Write(false);
            }
        }
    }
}

This method again uses the lock statement to lock the buffer. This means we cannot update the buffer until a full cube of data has been displayed. The method takes a block of eight bytes representing a layer and then writes this to the shift registers using SPI. Note that the spi.Write is embedded in the write to the enable line. This ensures that all of the layers are turned off whilst we are updating the shift registers.

Wrapping all of this in the LEDCube class means that we now have a very simple class where we can spawn the DislpayBuffer method into its own thread.

Main Program

The main program sets the cube up and then controls what is actually shown in the cube. The first thing we need to do is to set up a buffer to store the cube display and then spawn off the DisplayBuffer into its own thread.

private static LEDCube cube = new LEDCube();
private static byte[] newFrame = new byte[64];
ClearCube();
Cube.UpdateBuffer(newFrame);
Thread display = new Thread(new ThreadStart(cube.DisplayBuffer));
display.Start(); 

The ClearCube method is trivial and simply sets all of the bytes in newFrame to zero. At this point we have a cube with a whole lot of nothing going on. The next thing we need to do is to add some effects.

Making it Rain

Now that we have the mechanism to control the cube, we simply need to work out what we want to display. The rain effect in the opening video illustrates the majority of the techniques used to control the cube.

The rain algorithm is as follows:

• Set up the cube with a number of rain drops
• Repeat
  • Count the drops in the bottom layer
  • Move drops down one layer
  • Add the number of drops that have disappeared back into the top layer

To do this we will create two methods:

• AddDrops
• Rain

private static void AddDrops(int count, int plane = -1)
{
    for (int drops = 0; drops < count; drops++)
    {
        bool findingSpace = true;
        while (findingSpace)
        {
            int x = rand.Next() % 8;
            int y = rand.Next() % 8;
            int z;            if (plane == -1)
            {
                z = rand.Next() % 8;
            }
            else
            {
                z = plane;
            }            int position = (z * 8) + y;
            byte value = (byte) ((1 << x) & 0xff);
            if ((newFrame[position] & value) == 0)
            {
                newFrame[position] |= value;
                findingSpace = false;
            }
        }
    }
}

AddDrops simply adds a specified number of rain drops to the buffer. It also makes sure that if asked for 10 it will always add 10 by checking to see if one exists in the location it wanted to use (check out the findingSpace variable):

private static void Rain(int noDrops, int cycles)
{
    ClearCube();
    AddDrops(noDrops);
    cube.UpdateBuffer(newFrame);
    for (int currentCycle = 0; currentCycle < cycles; currentCycle++)
    {
        int bitCount = 0;
        for (int plane = 0; plane < 8; plane++)
        {
            byte value = 1;
            for (int bit = 0; bit < 8; bit++)
            {
                if ((newFrame[56 + plane] & value) > 0)
                {
                    bitCount++;
                }
                value <<= 1;
            }
        }
        for (int plane = 7; plane > 0; plane--)
        {
            for (int currentByte = 0; currentByte < 8; currentByte++)
            {
                newFrame[(plane * 8) + currentByte] = newFrame[((plane - 1) * 8) + currentByte];
            }
        }
        for (int b = 0; b < 8; b++)
        {
            newFrame[b] = 0;
        }
        AddDrops(bitCount, 0);
        cube.UpdateBuffer(newFrame);
        Thread.Sleep(75);
    }
}

Rain adds the specified number of drops to the newFrame, shows the drops, and then moves them all down by one horizontal plane. Before doing this, it counts how many drops are going to fall out of the cube (the number in the bottom horizontal plane). After all the drops have moved down one plane it adds the drops which have disappeared out of the bottom of the cube back in at the top in random positions.

We can display this by adding the following to the main program:

while (true)
{
    Rain(10, 60000);
} 

Full source code at http://netduinoledcontrol.codeplex.com/.

If you add your own effects, remember to consider the rating of the equipment and parts used. Turning on a full horizontal layer consumes 1.6A for the LEDs alone. Things can get warm/hot if every LED in the cube is turned on at the same time so avoid such effects.

Conclusion

When I started out on this project I was not sure how fast the Netduino Mini could control the cube. It has proven that it is up to the task of keeping the display running whilst also being able to work on the upcoming “frame.” If you have gone this far and built a cube then how about expanding it:

• Add some filters and a microphone and make the display respond to its environment
• Connect the cube to a PC using Bluetooth
• Add an MP3 player module and write software to synchronise the display to the music

Where will you take it?

About The Author

Mark Stevens is a software engineer and self-confessed technology addict. Mark started in the computer industry in the 1980s and at the age of 16 wrote a disk operating system for a then state of the art 8-bit microprocessor (despite the fact the machine already had one). He did it for fun and has been hooked on IT ever since then. Mark currently writes line of business applications for Win32, ASP.NET, and Silverlight platforms. He has recently taken up microelectronics as a hobby and has since discovered that his wife does not like the smell of solder. Marks’ current exploits, thoughts, and contact information can be found on his blog.

Initial Mashup

The project was divided into three major components:

The desktop client – a Kinect-controlled 3D environment. It is only partially controlled by the motion sensor, since other responsibilities are delegate to the phone. Because it is a proof-of-concept and not a full-sized game, WPF is the primary sub-framework.

The mobile client – a Windows Phone application that also is in charge of partially controlling the 3D environment on the desktop client. We decided to use Silverlight in favor of XNA because we didn't want complex graphics and all we needed was the bird's eye view of the playable area.

The server – one of the most interesting (and issue-generating) components. This was the link between the desktop and the mobile client. Initially we planned on using a WCF service (and we actually did for a while), but then we decided to use sockets for better performance..

Developing the communication layer

Since we decided to remove support for WCF and focus on sockets, we needed to figure out a way to send appropriate data to both clients. Given the fact that the amounts of data that are being sent across clients are not really big, it seemed a good idea to make the server broadcast a dual data set—a collection of indicators that are used by both the phone and desktop clients. It will be the client's responsibility to select the required value.

Here is the general structure of a data package broadcasted by the server:

We tried to keep it as simple as possible. The desktop client is going to process the camera direction sent by the phone, and the phone will adjust the character position according to the movement generated by the Kinect-driven desktop client.

When the clients (both desktop and mobile) send data to the server, they identify themselves through a prefix. For example, the desktop client sends the following string to the server:

The c prefix identifies the sender as the desktop client (computer). The phone sends the camera angle, led by the p prefix (phone):

Before going any further, I must mention that socket communications on Windows Phone are fairly generic, so the code we decided to use is close to the Sockets Sample on MSDN. We added a custom string parser that makes sure that the proper data is read between devices.

The server

Our server is entirely based on socket interactions through the TCP layer. It is a .NET-based console application that is listening to incoming connections (limited to two) and is broadcasting the received content, synchronizing both the desktop and the mobile clients. It is a "dummy" server and so doesn't perform any data processing—it simply receives the data from one client and then passes it to another.

Here is how the listening process is organized:

public static void StartListening()
{

    byte[] dataBuffer = new Byte[StateObject.BUFFER_SIZE];
    IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Any, PORT_NUMBER);
    Socket listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);

    try
    {
        listener.Bind(localEndPoint);
        listener.Listen(2);

        while (true)
        {
            completed.Reset();
            Console.WriteLine("Async transmission started.");
            listener.BeginAccept(new AsyncCallback(AcceptCallback),listener);
            completed.WaitOne();
        }

    }
    catch (Exception e)
    {
        Debug.WriteLine(e.ToString());
    }
}

The received byte array is later transformed to a standard string that is parsed according to the pattern described above. The parser detects the prefix and assigns the correct values depending on the sender.

private static void ParseContent(string content)
{
    var values = content.Split('|');
    if (content.StartsWith("c"))
    {
        position = int.Parse(values[1]);
    }
    else if (content.StartsWith("p"))
    {
        angle = int.Parse(values[1]);
    }
}

The difference in received data is present because each client is designed to affect only a part of the gameplay, and character movement cannot be directed by a desktop client or by the mobile application on its own. The mobile client reports the angle of character rotation, and the desktop client controls general movement progress and bonus pickups.

Once all values are read, the server broadcasts them back:

private static void Send(Socket handler)
{
    // The generalized string broadcasted by the server does not contain a platform
    // identifying prefix (p or c)
    string data = string.Format("{0}|{1}", angle, position);

    byte[] byteData = Encoding.UTF8.GetBytes(data);

    Console.WriteLine("DATA LENGTH [SENT] = {0} bytes. DATA = {1}", data.Length, data);

    handler.BeginSend(byteData, 0, byteData.Length, 0, new AsyncCallback(SendCallback), handler);
}

This is pretty much it when it comes to defining the server that handles cross-platform communications. It can be installed on a machine other than the one where the desktop client is running, as long as all of them work on either the same subnet or a network in which the IP addresses are static and have the invoked port open.

The desktop client

The desktop client was a bit of a graphical “fortress.” Rick was building the entire 3D environment while Adam and I were building the mobile application and the server interoperability layer.

We used the newly launched Kinect SDK to access the connected Kinect and it was initialized the moment the main window is loaded:

void SetupKinect()
{
    nui = new Runtime();

    try
    {
        nui.Initialize(RuntimeOptions.UseSkeletalTracking);
    }
    catch (InvalidOperationException)
    {
        MessageBox.Show("Runtime initialization failed. Please make sure Kinect device is plugged in.");
        return;
    }

    nui.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);
}

Runtime is the standard device class that identifies the Kinect currently connected to the PC. As long as we were using just one sensor, we did not need to provide a numeric index when instantiating the Runtime class itself.

Since we are only using skeletal tracking—the desktop client is responsible for moving the character forward on a detected jump—the Runtime instance is initialized with a RuntimeOptions.UseSkeletalTracking argument. The event handler is invoked once we detect a tracked skeleton:

void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    SkeletonFrame skeletonFrame = e.SkeletonFrame;

    foreach (SkeletonData data in skeletonFrame.Skeletons)
    {
        if (SkeletonTrackingState.Tracked == data.TrackingState)
        {

            phL = new Point(data.Joints[JointID.HandLeft].Position.X, data.Joints[JointID.HandLeft].Position.Y);
            phR = new Point(data.Joints[JointID.HandRight].Position.X, data.Joints[JointID.HandRight].Position.Y);
            pCore = new Point(data.Joints[JointID.HipCenter].Position.X, data.Joints[JointID.HipCenter].Position.Y);

            coreDeep = data.Joints[JointID.HipCenter].Position.Z;

            leftDeep = data.Joints[JointID.HandLeft].Position.Z - coreDeep;
            rightDeep = data.Joints[JointID.HandRight].Position.Z - coreDeep;

            handL.x = 640 + 600 * phL.X;
            handL.y = 350 - 300 * phL.Y;
            handR.x = 640 + 600 * phR.X;
            handR.y = 350 - 300 * phR.Y;

            bodyCore.x = 640 + 30 * pCore.X;
            bodyCore.y = 350 - 300 * pCore.Y;

            follow.x = bodyCore.x;
        }
    }
}

The Kinect sensor monitors six skeletons at once, but not all of them are tracked for movement. That's the reason to check the SkeletonTrackingState before displaying the joint indicators on the screen.

The joint indicators themselves are declared in the XAML layout in the main page:

<Canvas x:Name="touch_screen" Width="1280" Height="720" Background="#00FF0000">
    <local:BodyMarker x:Name="handL"/>
    <local:BodyMarker x:Name="handR"/>
    <local:BodyMarker x:Name="follow" x="0" y="0"/>
    <local:BodyMarker x:Name="bodyCore" x="0" y="0"/>
</Canvas>

The BodyMarker class is in fact a custom UserControl. It is easier to manage and later extend instances of a customizable unit rather than a simple ellipse:

<UserControl x:Class="KinectoPhone.Desktop.BodyMarker"
             xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
             xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
             xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
             xmlns:d="http://schemas.microsoft.com/expression/blend/2008" >
    <Canvas>
        <Ellipse x:Name="bkg" Width="10" Height="10" Canvas.Left="-5" Canvas.Top="-5"/>
    </Canvas>
</UserControl>

In the skeleton tracking code above, we are not only tracking the joint position, we're also adjusting the character position based on the center hip location in space as well as detecting the jump by coordinating joint positions by the Z-axis.

The data is sent via an instance of the same AsynchronousClient I used on the server side, but with small modifications. I implemented a short verification mechanism that ensures the fact that I am using an IPv4 address as the endpoint will be functional with only IPv4 and not IPv6 addresses for the time being:

public AsynchronousClient()
{
    // The worker that is constantly syncing the state of the
    // desktop client with the server.
    backgroundWorker.DoWork += new DoWorkEventHandler(backgroundWorker_DoWork);

    IPHostEntry ipHost = Dns.GetHostEntry("192.168.173.1");

    var ipv4address = (from c in ipHost.AddressList where c.AddressFamily == AddressFamily.InterNetwork select c).First();

    if (ipv4address != null)
    {
        IPAddress ipAddress = (IPAddress)ipv4address;
        remoteEndPoint = new IPEndPoint(ipAddress, port);

        InitializeClient();
    }
}

The BackgroundWorker is managing the constant synchronization building the location and meta string that is being sent with updated character position and activity data from the desktop client:

void backgroundWorker_DoWork(object sender, DoWorkEventArgs e)
{
    if (!client.Connected)
    {
        client.Shutdown(SocketShutdown.Both);
        client.Close();
        InitializeClient();
    }

    DesktopData cd = (DesktopData)e.Argument;
    Send(client, string.Format("c|{0}", cd.Position));
    sendDone.WaitOne();

    // Receive the response from the remote device.
    Receive(client);
    receiveDone.WaitOne();
}

When the desktop client receives data, it only reads information regarding the camera angle. Therefore, that’s the only indicator we needed to set from the outside:

void aClient_ResponseReceived(object sender, ResponseReceivedEventArgs e)
{
    Dispatcher.Invoke(new Action(delegate
    {

        var values = e.Response.Split('|');

        cameraTargetRotation = Convert.ToDouble(values[0]);

    }), DispatcherPriority.Normal);
}

Phone client

The phone client has only one responsibility—adjust the camera angle as the user tilts the device. We used standard accelerometer-based capabilities to track the movement of the device in space.

The app uses a different implementation of the AsynchronousClient called SocketHandler. It works well to link the phone to the server. For example, take a look at the handler that processes the received input:

private void ProcessReceive(SocketAsyncEventArgs e)
{
    if (e.SocketError == SocketError.Success)
    {
        // Received data from server
        string dataFromServer = Encoding.UTF8.GetString(e.Buffer, 0, e.BytesTransferred);

        Socket sock = e.UserToken as Socket;
        sock.Shutdown(SocketShutdown.Send);
        sock.Close();
        clientDone.Set();

        // Respond to the client in the UI thread to tell him that data was received
        System.Windows.Deployment.Current.Dispatcher.BeginInvoke(() =>
        {
            ResponseReceivedEventArgs args = new ResponseReceivedEventArgs();
            args.response = dataFromServer;
            OnResponseReceived(args);
        });

    }
    else
    {
        clientDone.Set();
        throw new SocketException((int)e.SocketError);
    }
}

Due to threading limitations, we had to use the dispatcher class to notify the UI about the fact that all data from the server was received and that it should be transformed to the position of the character on the map.

In the main page constructor, we are initializing the SocketHandler with the same values for the server as we did in the desktop client. Additionally, we're wiring an event handler that will handle the received string. Unlike ProcessReceive above, the wired event handler actually parses the string for required values and accordingly takes the appropriate action:

SocketHandler handler = new SocketHandler("192.168.1.6", 13001);
handler.ResponseReceived += new ResponseReceivedEventHandler(handler_ResponseReceived);

void handler_ResponseReceived(object sender, ResponseReceivedEventArgs e)
{
    previousPos = currentPos;

    string[] data = e.response.Split('|');
    int.TryParse(data[1], out currentPos);

    if (currentPos != previousPos)
    {
        targetIndex = currentPos;
    }
}

The data is sent back to the server with the help of a DispatcherTimer instance running every 300 milliseconds:

void timer_Tick(object sender, EventArgs e)
{
    handler.SendData(string.Format("p|{0}", curTriangleAngle.ToString("0")));
}

When the accelerometer detects a change in the current value, we make sure that the object is rotated in the right direction by checking the sign of the reading:

void accel_ReadingChanged(object sender, AccelerometerReadingEventArgs e)
{
    Dispatcher.BeginInvoke(new Action(() =>
    {
        previousY = currentY;
        currentY = e.Y;

        if (e.Y > .7 || e.Y < -.7)
        {
            if (e.Y > .7 && !isTurnDirty)
            {
                isTurnDirty = true;
                Turn(1);
            }
            else if (e.Y < -.7 && !isTurnDirty)
            {
                isTurnDirty = true;
                Turn(-1);
            }
        }
        else
        {
            isTurnDirty = false;
        }
    
    }));
}

void Turn(double direction)
{
    currentTurnTime = DateTime.Now;
    TimeSpan diffResult = currentTurnTime.Subtract(lastTurnTime);
    if (diffResult.TotalSeconds > 1)
    {
        lastTurnTime = currentTurnTime;
        targetTriangleAngle += direction * 90.0;
    }
}

The turn time is checked so that the character on the device screen doesn’t constantly rotate. Since the accelerometer is very sensitive and without a time limit, a slight tilt will have unexpected consequences on the rotation behavior.

How to test

Virtual Router is a great way to test the set-up. Virtual Router allows the server computer to become a WiFi hotspot so that any connected device will be automatically registered as a part of the local subnet.

You will need:

• A computer with a WiFi card (most laptops have them anyway)

Make sure that you have the WiFi card enabled and launch Virtual Router Manager. Set a new network name, as well as specify a password. It is worth mentioning that this network will act like a standard wireless network, so make sure you secure it with a strong password if you don't want it accessed from the outside.

Once you enable the hotspot, you can run the desktop client on the same machine. Connect the phone to the newly created wireless network and make sure that the proper IP is set in the code-behind for both the desktop client and the Windows Phone app.

The IP for the server machine can be found by running the ipconfig command in the console.

The IP address for the Windows Phone device that is connected to the hotspot is shown in the main Virtual Router Manager window.

In case the router cannot be started, make sure that connection sharing is enabled for your main connection and that it points to the Miniport interface.

Once the network is correctly set up, I would highly recommend launching the project components in the following order:

1. Server
2. Desktop client
3. Phone client

Conclusion

With the latest 7.1 SDK update for Windows Phone as well as the 7.5 (Mango) OS, it is fairly easy to set up a Kinect-to-WP communication layer through sockets. With small modifications and performance optimizations we can also send Kinect imagery (captured through the depth and video sensors) directly to the phone without major delays. The field of third-party games that interoperate with Kinect and run on mobile devices is yet to be fully developed, but we already made the first steps in that direction.

About the author

Dennis Delimarsky is a Windows Phone Development MVP. He enjoys technical writing as well as building cool stuff around the .NET platform. Besides Coding4Fun, he also blogs at Den by default. He can be contacted at ddelimarsky@live.com.

Introduction

Working with mobile phones gives you access to a completely different form factor, as well as a completely different set of functionality compared to web and desktop development environments. This project demonstrates GPS, camera, isolated storage, map display, web services, and advanced map features. Use it as a starting point for another project, or just learn from the techniques shown here.

You don’t need a Windows Phone to learn from this sample, but GPS simulation is only available in the emulator for the 7.1 beta developer tools, and interacting with the camera on the emulator is no fun at all, so it’s definitely a limited debugging/testing experience. You need to be a registered phone developer in order to debug on a physical phone, but the price ($99) is definitely reasonable if you end up releasing even a single paid app.

Go to create.msdn.com, then click Download the free tools to download the Windows Phone Developer Tools (or use the direct download link provided above this Introduction section). This code is written for the Windows Phone Developer Tools 7.0, but has also been tested with Windows Phone SDK 7.1 Beta 2.  Either way, it’s is a mostly online install, and it’s pretty big so expect it to take some time. Even if you don’t have any development tools, this will give you Visual Studio Express, Blend, and XNA Game Studio. If you have the full-version tools already, it will add new templates.

Project Basics

Upon launching the app, there’s not much to see.  You can touch the car icon in the Map region in order to set your car’s parking location.  To make things easier, you can also touch the camera icon to take a picture of your parking spot or nearby landmark to make it easier to find later.  The data is saved between runs so you don’t need to leave it running.

The application takes advantage of a small set of pages:

• MainPage: Summary screen to see the map and photo in the same place.  Essentially, this uses two Image controls and a Map control.  The upper region shows a small map if you’ve set your parking spot.  It also caches a small version of the map in case you can’t get a GPS or data connection when you need to find your car.
• MapSet: Allows the user to set their car’s position on a map.  It starts out centered on the current location and the user can swipe around to place their car.
• MapPage: A larger map with panning/zooming support.  This is the one you see after locking in the parking spot.  You can jump to your current position, the car’s position, or generate a route between them.
• Media: A large view of the photo taken of the parking spot.
• SettingsPage: Choice of imperial (miles) or metric (kilometers) for distance measurements, and map style (aerial or street).  These are all data-bound to simplify managing the values.

It also uses a few IValueConverter classes to take data from my view model and make it look right in the UI:

DistanceConverter

Take a double value (assumed in meters), converts to miles if necessary, and appends the unit (km/mi).  Notice that the code references the WheresTheCarSettings object.  This is used to determine the measurement mode/unit.  This should probably be passed in as a parameter, since this limits the portability for other projects, but I’ll leave that as an exercise for the reader!

public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
{
    if (value == null ) return "";

    double val = Double.Parse(value.ToString());
	string unit;
    if( WheresTheCarSettings.Instance.MeasurementMode == MeasurementMode.Imperial)
    {
        unit = "miles";
        val = (val*3.2808399)/5280.0;
    }
    else
    {
        unit = "km";
        val /= 1000.0;
    }

    return string.Format("{0:0.00} {1}", val, unit);
}

ImageConverter

Takes an image path, checks for it in isolated storage, otherwise assumes it’s in the application XAP file, and then returns an BitmapImage for display.  This is a pretty handy converter since the Source can’t be set to isolated storage otherwise.  The Cache is very important since accessing isolated storage is more expensive than RAM access.  Also, notice that the Dictionary object is holding WeakReference instances pointing to the image.  This allows the garbage collector to trim the cache if memory gets tight.  Be sure to always use the IsAlive property so you know if the object’s been released.

public static Dictionary Cache { get; private set; }

static ImageConverter()
{
    Cache = new Dictionary();
}

#region IValueConverter Members

public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
{
    if (value != null && !string.IsNullOrEmpty(value.ToString()))
    {
        string fn = value.ToString();

        if (Cache.ContainsKey(fn))
            if (Cache[fn].IsAlive)
                return Cache[fn].Target;
            else
                Cache.Remove(fn);

        try
        {
            BitmapImage bs = null;
            using (var myStore = IsolatedStorageFile.GetUserStoreForApplication())
            {
                if (myStore.FileExists(fn))
                {
                    using (var photoStream = myStore.OpenFile(fn, FileMode.Open))
                    {
                        bs = new BitmapImage();
                        bs.SetSource(photoStream);
                    }
                }
                else
                    bs = (new BitmapImage(new Uri(fn, UriKind.Relative)));

                if( parameter == null || parameter.ToString() != "no-cache")
                    Cache.Add(fn, new WeakReference(bs));
                return bs;
            }
        }
        catch
        {
            // Errors will be handled silently
        }
    }

	// If all else fails...
	return null;
}

ValueToVisibilityConverter

Returns Visibility.Collapsed or Visibility.Visible depending on whether a GeoCoordinate value is non-null and valid.  This converter performs the simple action of making an element invisible if no coordinate is set.  Very simple, but it makes for much more elegant code.

public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
{
    var geo = value as GeoCoordinate;
	return (Utility.IsCoordNullOrUnknown(geo)) ? Visibility.Collapsed : Visibility.Visible;
}

Location

On Windows Phone, you determine your geographic location by using the GeoCoordinateWatcher class. When you instantiate it, you can specify an accuracy of Default or High. High accuracy comes at the cost of battery life. You can also set your movement threshold. The higher the threshold, the less often you’ll get position events. Call Start to wait for the sensor to turn on, or TryStart for asynchronous usage.

The GeoCoordinateWatcher raises an event for status changes such as starting or no data (StatusChanged), and an event to indicate a location change (PositionChanged). Locations are reported as GeoCoordinate objects, and include latitude, longitude, speed, bearing, and more. All distance measurements are in meters.

Note that like any GPS, the phone won’t be able to get a lock in underground parking, or even in many multilevel parking ramps. There’s not much you can do about this, but hopefully the photo comes in handy then!

Maps

Displaying a map on a Windows Phone application is really easy. The controls that come with the toolkit include a Map control. Simply drop this control onto a page and you are just about ready to go. The only other required step is to register as a developer on Bing. You can find out more information about the steps to follow to obtain your key and register your application here.  This gets you a credential key that you plug into the MAP_CREDENTIALS property of the RouteHelper class of the supplied code.  Note that without this, you won’t even be able to view maps, let alone generate routes. By doing this, you will get a map centered on Latitude = 0, and Longitude = 0. This puts you in the Atlantic Ocean west of South Africa. Without doing anything else you can pan and zoom the map.

Unfortunately, due to a bug in the current build of the controls, you can’t create the Map control in XAML on multiple pages of the same app.  The maps in Where’s My Car are created in the code-behind.  A little ugly, but it works.  Hopefully this will be resolved in Mango.  In XAML, it’s as easy as a one line control declaration to add a map:

<Map CredentialsProvider="XXXXXX" />

You can set properties to hide the Bing logo (LogoVisibility="Collapsed"), to hide the copyright (CopyrightVisibility="Collapsed"), to set an explicit zoom level (ZoomLevel="15") and much more. You might notice that I set the zoom bar to be visible in the emulator, but not on the phone. Without having a touchscreen laptop, I wouldn’t be able to zoom the map in the emulator otherwise. My code-based map creation looks this this:

return new Map
{

    CredentialsProvider = new ApplicationIdCredentialsProvider(MapHelper.MAP_CREDENTIALS),
    ZoomLevel = 15,
    Mode = s.UseAerialMode ? (MapMode) new AerialMode() : new RoadMode(),
    ZoomBarVisibility = (Environment.DeviceType == DeviceType.Emulator) ? Visibility.Visible : Visibility.Collapsed,
    CopyrightVisibility = Visibility.Collapsed,
    LogoVisibility = Visibility.Collapsed,
    Margin = new Thickness(0),
};

Another important part of working with a navigation-based map is to be able to set pushpins to indicate the locations of things (in this case, you and your car). The built-in pushpin visualization is fine, but not very special. Changing the look-and-feel is as simple as modifying the template for the Pushpin control.  Here, we change it to use a bitmap image:

<ControlTemplate x:Key="CarPushpinTemplate" TargetType="mpc:Pushpin">
    <Image Source="/images/carPushpin.png" Stretch="None" Margin="-24, 0, 0, -24" />
</ControlTemplate>

When you declare a Pushpin, just set its Template property, and add it as a child of the Map control. A Pushpin control knows where to place itself by its Location property. Note that if the location is null, the pushpin will just lurk in the upper-left corner. Make sure that you hide it when the location isn’t set. I do this with the ValueToVisibilityConverter covered above.  To use it, just set the Visibility property to the same Location property while using my ValueToVisibilityConverter to handle the conversion.

Routing

The Map control has no ability to perform routing, but it supports adding child controls. I’ve already discussed the Pushpin control as a child. The other important child control is MapLayer.  MapLayer acts as a container for other controls. By setting the Location properties of the nested items (in latitude/longitude), everything is displayed at the right location, regardless of panning or zooming the map (no scaling is done however).

In order to obtain the routing data, the user must first set their parking location point. The other point is the current phone location. The actual routing happens using the Bing web service. Generate a service reference pointing to the definition (WSDL):

http://dev.virtualearth.net/webservices/v1/routeservice/routeservice.svc?wsdl

Then, creating the request entails setting the Credentials property, and optionally setting RouteOptions. For walking directions, set Mode to TravelMode.Walking (though this still uses roads). The Waypoints collection property holds Waypoint objects containing a label along with a Location object with Altitude/Latitude/Longitude properties. For the best performance, I use the CalculateRouteAsync call to get off the UI thread as quickly as possible.

In the response, you get the travel distance and a collection of points that make up the route. Create a MapPolyLine instance, and then based on the Waypoints collection create and add GeoCoordinate objects to the Locations collection. The MapPolyLine object then gets added to a MapLayer object for rendering along with the map. The final step is to use the Result.Summary.BoundingRectangle property to center and size the map in order to show the whole route in one view.

Taking and Storing Pictures

Taking pictures is pretty easy. The built-in camera can be triggered using a CameraCaptureTask object. Just call the Show method and the camera will appear. It’s up to the user to actually take the picture. Note that there is no way to automatically take the photo, and there is at the moment no preview feature.  This will change in Mango, but for now it works well enough for our needs.

An important thing to keep in mind is that showing the camera task causes your app to be tombstoned (deactivated). This means that you need to save any relevant state before triggering it. After the user either takes the photo or cancels it, the app is reactivated. The Completed event of the CameraCaptureTask fires, so be sure to subscribe to the event in the constructor of the page. The Completed event includes a PhotoResult event class. From this you can check the TaskResult property to see if there is actually a photo ready, and use the ChosenPhoto property to get a stream containing the photo data. You can get the stream to process or write to storage.  Here we will open isolated storage and open the “parking.jpg” file, then call the SaveJpeg method of the WriteableBitmap class.

For the sake of efficiency, I prefer to resize the image to a maximum of 1024 width or 768 height, rather than loading a minimum 5MP image each time. The resize is very quick, and results in much less data transfer from isolated storage to memory.

Settings

Settings are kept track of in a custom class, WheresTheCarSettings.  This class has INotifyPropertyChanged properties and Load/Save methods to keep things together.  Saving is done with a data contract rather than raw serialization.  This is an amazingly easy way to load and save an object, and it’s very resilient against changes.  The INotifyPropertyChanged interface is used so disconnected parts of an application (between objects or objects and UI) can be notified of changes to a given object.  This happens because the PropertyChanged event fires whenever a property’s value changes, thus giving related code a chance to react.

#region MeasurementMode (INotifyPropertyChanged Property)
private MeasurementMode _measurementMode;
[DataMember]
public MeasurementMode MeasurementMode
{
	get { return _measurementMode; }
	set
	{
		_measurementMode = value;
		RaisePropertyChanged("MeasurementMode");
	}
}
#endregion

.

Next Steps

The application is fully functional as-is. It’s been in the Marketplace for a few months now.  There are a number of complaints about not being able to get a GPS lock, or the route not being optimized for walking.  These are true and valid complaints, but it’s difficult to do much about them.  This is where good alerts and status indicators are important!  Also, it could be useful to add a screen to see the walking route in a step-by-step view (already available in the route response). Integrating parking meter information might also be useful, though I’ve already implemented that in a different application—coming soon!

Conclusion

Being able to work with the location and camera enables some great scenarios that aren’t really available in other form factors. Get started by downloading the Windows Phone Developer Tools to see just how easy it is.

About the Author

Arian Kulp is a software developer living in Western Oregon. He creates samples, screencasts, demos, labs, and articles, speaks at programming events, and enjoys hiking and other outdoor adventures with his family.

If you want to learn how to extend the engine, check out the Coding4Fun article!

Screenshots