That's why today's post caught my eye. It's not "code" per-se (though there is a little C#!) instead it's a easy, simple and fairly short introduction to Unity3D.

Introduction to Unity3D

Unity3D is a powerful cross-platform 3D engine and a user-friendly development environment. Learn how Unity3D can help you create games in this article!

What is Unity3D?

Unity3D is a powerful cross-platform 3D engine and a user friendly development environment. Easy enough for the beginner and powerful enough for the expert; Unity should interest anybody who wants to easily create 3D games and applications for mobile, desktop, the web, and consoles.

The article starts off talking about cost, installation and basic getting started stuff.

Through to scripting and publishing;

If you've heard of Unity3D before but didn't know how to get started with it, this is the perfect article for you...

Kinect-enabled shopping cart is smarter than the shopper

Microsoft has just demoed a Kinect-enabled prototype shopping cart designed to take the guesswork out of your trip to the grocery store. All of the guesswork. In fact, all you are required to do is upload a shopping list and place items in the cart.

The cart utilizes the Kinect and other sensors to read what you are placing in the cart and check whether the item matches your pre-planned menu and any dietary constrains. If you've blindly placed an item in the cart that doesn't match with your downloaded list, forget about it. The Kinect Cart will tell gently — and slightly disapprovingly — verbally remind you of your mistake.

As you are scanning your items you are also being rung up so (in theory) forget the long check out lines. Just grab a bag if you haven't brought your own.

Here's the thing. Not only is the cart smart enough to check your list, scan your items, ring you out and converse with you — it is motorized. That means the cart will follow you as you walk down the aisle.

...

Project Information URL: http://dvice.com/archives/2012/02/kinect-enabled-1.php

Kinect imaging lets surgeons keep their focus

THE surgeon enters the operating theatre, covered in sterile blue scrubs. Machines beep and hiss. Nurses wait, tools at the ready: scalpel, forceps, bandage, Xbox... Xbox?

On Tuesday last week, a surgeon at Guy's and St Thomas' hospital in London began trials of a new device that uses an Xbox Kinect camera to sense body position. Just by waving his arms the surgeon can consult and sift through medical images, such as CT scans or real-time X-rays, while in the middle of an operation.

Maintaining a sterile environment in the operating room is paramount, but scrubbing in and out to scroll through scan images mid-operation can be time-consuming and break a surgeon's concentration or sense of flow. Depending on the type of surgery, a surgeon will stop and consult medical images anywhere from once an hour to every few minutes. To avoid leaving the table, many surgeons rely on assistants to manipulate the computer for them, a distracting and sometimes frustrating process.

...

"You usually think of Kinect in a game-like scenario where you can jump around and move your hands as wide as possible, but surgeons are not allowed to reach such a large area," says Gerardo Gonzalez of Microsoft Research in Cambridge, UK, who helped develop the system in conjunction with surgeons from Guy's and St Thomas' and King's College London.

So Gonzalez and colleagues developed a set of gestures that a surgeon can perform in a constrained space, while standing at the operating table. For the most common actions - rotating the 3D model or placing a marker on the image - the team designed one-handed gestures that combine with voice commands, leaving the other hand free for operating. To position a marker, for example, the surgeon simply points at the image to activate a cursor and says, "place marker". Other functions, such as panning or zooming, require two hands.

Project Information URL: http://www.newscientist.com/article/mg21428655.200-kinect-imaging-lets-surgeons-keep-their-focus.html

Kinect in the operating theater

I’m often asked where I see Kinect going next and much like the Kinect Effect advert, I put that question back on the questioner. One of the answers I give is we see it going into more and more places outside of the living room – places like schools and hospitals. In fact in the advert, there is a brief moment where you see Kinect in an operating theater.

That’s not happening tomorrow, it’s happening today, at Sunnybrook hospital in Toronto. Earlier this week, Discovery Channel Canada dropped in on Sunnybrook and got a demo of how Kinect is being used.

Project Information URL: http://blogs.technet.com/b/next/archive/2012/02/24/kinect-in-the-operating-theater.aspx

Overview

The external lighting system for Project Detroit is run on a web server on a Netduino Plus, which uses the .NET Micro Framework (NETMF). To get an accelerated start, we leveraged code from the CodePlex project Netduino Helpers to control the AdaFruit LPD8806 LED strip, and the web server base code from the Netduino forums. They have been tweaked slightly for our project.

Controllers, Routes, RESTful oh my!

We wanted the NETMF web server to have the exact same routes as the full-blown REST service layer in Detroit. With compiler conditionals, this allowed the same code to use the same routes on both the full .NET stack and NETMF. Since we couldn't use the ASP.NET MVC framework here, we had to make something close to it.

Controllers

Controllers implement an interface with a method named ExecuteAction. When GetController is called, we return the correct controller but cast as the interface IController. This lets us figure out what controller should be executing the request and continue to work generically in the context of the request connection thread. If new functionality is needed, we create a new controller that inherits from IController and update GetController:

private static void RequestReceived(Request request)
{
    // url comes in as /foo/bar/12345?style=255
    //
     // /CONTROLLER/ACTION/
    //
    // LIGHTING SAMPLE URL: /0/1/?r=255&g=255&b=255&zone=EnumAsInt
    // /0/... 0 = NetduinoControllerType.Lighting
    var validResponse = false;
    try
    {
        Logger.WriteLine("Start: " + request.Url + " at " + DateTime.Now);
        var urlInParts = request.Url.TrimStart(UriPathSeparator).Split(new[] {'?'}, 2);
        string[] uriSubParts = null;
        string[] queryStringSubParts = null;
        if (urlInParts.Length > 0)
            uriSubParts = urlInParts[0].Split(UriPathSeparator);
                
        if (urlInParts.Length > 1)
            queryStringSubParts = urlInParts[1].Split(QueryStringSeparator);
        if (uriSubParts != null && uriSubParts.Length > 1)
        {
            var targetController = GetController(uriSubParts[0]);
            if (targetController != null)
            {
                var action = (uriSubParts.Length >= 2) ? uriSubParts[1] : string.Empty;
                var result = targetController.ExecuteAction(action, queryStringSubParts);
                validResponse = true;
                request.SendResponse(result.ToString());
                Logger.WriteLine("Result: " + result + " from " + request.Url + " at " + DateTime.Now);
            }
        }
        if (!validResponse)
        {
            SendIndexHtml(request);
        }
    }
    catch(Exception ex0)
    {
        Logger.WriteLine(ex0.ToString());
    }
}

Routes, strings and enums on NETMF

NETMF can't parse an enum from a string, which is something that can be done in the full .NET Framework. But what does this actually mean? It means the routes for the NETMF web server are a bit unreadable. Here is the same route but with the parsing issue exposed:

Detroit's REST service route with ASP.NET MVC:
http://…/Lighting/Solid/?zone=all&r=255&g=0&b=0

Detroit's NETMF route:
http://…/0/3/?zone=0&r=255&g=0&b=0

This issue made direct device debugging a bit harder.

Parsing the Action and query string

Parsing the route is broken into two parts. The first is determines what controller to target. For Project Detroit, that meant the lighting system or the rear glass system. Once we know the controller, we still need to parse the query string and action into something more useful:

#if NETMF
private static readonly char[] QueryStringParamSeparator = new[] { '=' };
public static LightingData ParseQueryStringValues(string action, params string[] queryStringParameters)
#else
public static LightingData ParseQueryStringValues(string action, Dictionary<string, object> queryStringParameters)
#endif
{
    var data = new LightingData {LightingZone = LightingZoneType.All, LightingAction = ParseLightingActionType(action)};
#if NETMF
    if (queryStringParameters != null)
    {
        foreach (var queryString in queryStringParameters)
        {
            var valueKey = queryString.Split(QueryStringParamSeparator);
            if (valueKey.Length != 2)
                continue;
            var key = valueKey[0];
            var value = valueKey[1];
#else
        foreach(var key in queryStringParameters.Keys)
        {
            var value = queryStringParameters[key].ToString();
#endif
            switch (key.ToLower())
            {
                case ZoneHuman:
                case ZoneComputer:
                    data.LightingZone = ParseLightingZoneType(value);
                    break;
                // more stuff
            }
        }
#if NETMF
    }
#endif
    // more stuff
            
    return data;
}

Turning on the lights

Individually addressable LED strips allow us to tell each individual LED what color to be. To adjust the colors of the LED lights, we loop through all the LEDs, set their color value, and then call LedRefresh() on our LED strip. There's no need to set all of them, only the ones being updated:

private static void SolidAnimationWorker()
{
    var leds = GetLedsToIlluminate();
    var dataCopy = _data;
    for (var i = 0; i < leds.Length; i++)
        SetLed(leds[i], dataCopy.Red, dataCopy.Green, dataCopy.Blue);
    LedRefresh();
}

Zones / Animation Threads

A solid color is boring, but we can use procedural animations to add some flair. The current code supports the following animations:

• Solid - A single solid color.
• Fill - Goes from black to the desired color. The color stays lit. Much like filling a glass of liquid.
• Random - LEDs are set to a random value.
• Pulse - A glowing effect from bright to black.
• Snake - A one way lighting effect with a fading tail that moves left to right.
• Sweep - This effect was inspired by KITT and a Cylon robot and is similar to a snake pattern. The effect can also handle being split in two while maintaining a continual line for items such as the grill or the wheels.
• Police - Each side flashes a red and blue light.

Since we know the LED strip is a single strand, we can be clever with offsets and create groupings. By having a dedicated animation thread in each zone, we can have the grill of the car in Police Mode while the bottom fades to red and the vents fill to blue at different refresh rates.

To run an animation, we need to first kill the old animation thread, spin up a new thread, mark LEDs in the "All LED" zone as do not touch, and execute the new pattern. Without a way to mark the LEDs as "bad" in the "All" zone, the two animation threads would compete for setting the color and result in a flickering effect. Other zones don't have to worry about this since there is zero overlap of LEDs: 

public bool ExecuteAction(string action, params string[] queryStringParameters)
{
    var data = LightingDataHelper.ParseQueryStringValues(action, queryStringParameters);
            
    SignalToEndAnimationThread(data.LightingZone);
    int counter = 0;
    while (counter++ < 5 && IsThreadAlive(data.LightingZone))
        Thread.Sleep(10);
    if (IsThreadAlive(data.LightingZone))
        AbortAnimationThread(data.LightingZone);
    // moving data bucket into public so new threads can leverage it
    _data = data;
    if (_data.LightingAction == LightingActionType.Police || _data.LightingAction == LightingActionType.Sweep)
    {
        _data.LightingZone = LightingZoneType.Grill;
        if (IsThreadAlive(_data.LightingZone))
            AbortAnimationThread(_data.LightingZone);
    }
    // cleaning up just incase
    if (data.LightingZone != LightingZoneType.All)
    {
        for (var i = 0; i < _zoneAllAnimation.Length; i++)
        {
            var leds = GetLedsToIlluminate();
            for (var z = 0; z < leds.Length; z++)
            {
                // flagging item as bad in animation lib
                if (_zoneAllAnimation[i] == leds[z])
                    _zoneAllAnimation[i] = -1;
            }
        }
    }
    switch (_data.LightingAction)
    {
        case LightingActionType.Solid:
            Debug.Print("Solid");
            ExecuteThread(SolidAnimationWorker);
            break;
        // more patterns
    }
    return true;
}

We can now do something more interesting, like Police Mode:

private static void PoliceModeAnimationWorker()
{
    var leds = GetLedsToIlluminate();
    var dataCopy = _data;
    var length = leds.Length;
    var half = length / 2;
    var index = 0;
    while (IsThreadSignaledToBeAlive(dataCopy.LightingZone))
    {
        int redIntensity = (index == 0) ? 255 : 0;
        int blueIntensity = (index == 0) ? 0 : 255;
        for (var i = 0; i < half; i++)
            SetLed(leds[i], redIntensity, 0, 0);
        for (var i = half; i < length; i++)
            SetLed(leds[i], 0, 0, blueIntensity);
        LedRefresh();
        index++;
        index %= 2;
        Thread.Sleep(100);
    }
}

Improvements

Hindsight is 20/20. Nothing is ever perfect and there are of course items we'd love to change. At the start, the goal was to make this fairly generic and not Project Detroit "aware." As you can see by looking at the source, that wasn't accomplished due to time constraints.

We would use a different LED system for two reasons: data transfer speed and power. The longer the LED strip is, the longer it takes to address an item further down the line. Project Detroit, having 15 meters of lighting in it, was impacted by this issue. It works well, just not as well as we hoped. The voltage requirements of the LEDs are another consideration. They are 5V, while a car uses 12V. Had we used 12V LEDs, our wiring would have been far simpler and wouldn't have required multiple transformers to get the required amperage needed at the correct voltage.

Conclusion

We think the lighting solution, coupled with the time and material constraints, worked out pretty well.  Project Detroit was a ton of fun to build out and working with West Coast Customs was the chance of a lifetime.

Rob Relyea, a Principal Program Manager on the Kinect for Windows team joins us again on Channel 9 to discuss all of the new features with the 1.5 release of the Kinect for Windows SDK.

Kinect for Windows Blog - Kinect for Windows Runtime and SDK Version 1.5 Released

I am pleased to announce that today we have released version 1.5 of the Kinect for Windows runtime and SDK. Additionally, Kinect for Windows hardware is now available in Hong Kong, Korea, Singapore, and Taiwan. Starting next month, Kinect for Windows hardware will be available in 15 additional countries: Austria, Belgium, Brazil, Denmark, Finland, India, the Netherlands, Norway, Portugal, Russia, Saudi Arabia, South Africa, Sweden, Switzerland and the United Arab Emirates. When this wave of expansion is complete, Kinect for Windows will be available in 31 countries around the world. Go to our Kinect for Windows website to find a reseller in your region.

We have added more capabilities to help developers build amazing applications, including:

• Kinect Studio, our new tool which allows developers to record and play back Kinect data, dramatically shortening and simplifying the development lifecycle of a Kinect application. Now a developer writing a Kinect for Windows application can record clips of users in the application’s target environment and then replay those clips at a later time for testing and further development.
• A set of Human Interface Guidelines (HIG) to guide developers on best practices for the creation of Natural User Interfaces using Kinect.
• The Face Tracking SDK, which provides a real-time 3D mesh of facial features—tracking the head position, location of eyebrows, shape of the mouth, etc.
• Significant sample code additions and improvements. There are many new samples in both C++ and C#, plus a “Basics” series of samples with language coverage in C++, C#, and Visual Basic.
• SDK documentation improvements, including new resources as well as migration of documentation to MSDN for easier discoverability and real-time updates.

We have continued to expand and improve our skeletal tracking capabilities in this release:

• Seated Skeletal Tracking is now available....
• Skeletal Tracking is supported in Near Mode, including both Default and Seated tracking modes. ...

We have made performance and data quality enhancements, which improve the experience of all Kinect for Windows applications using the RGB camera or needing RGB and depth data to be mapped together (“green screen” applications are a common example):

• Performance for the mapping of a depth frame to a color frame has been significantly improved, with an average speed increase of 5x.
• Depth and color frames will now be kept in sync with each other. The Kinect for Windows runtime continuously monitors the depth and color streams and corrects any drift.
• RGB Image quality has been improved in the RGB 640x480 @30fps and YUV 640x480 @15fps video modes. The image quality is now sharper and more color-accurate in high and low lighting conditions.

New capabilities to enable avatar animation scenarios, which makes it easier for developers to build applications that control a 3D avatar, such as Kinect Sports.

• Kinect for Windows runtime provides Joint Orientation information for the skeletons tracked by the Skeletal Tracking pipeline.
• The Joint Orientation is provided in two forms: A Hierarchical Rotation based on a bone relationship defined on the Skeletal Tracking joint structure, and an Absolute Orientation in Kinect camera coordinates.

..."

Kinect for Windows SDK Developer Downloads

The Kinect for Windows SDK enables developers to create applications using C++, C# or Visual Basic, which support gesture and voice recognition using the Kinect for Windows sensor and a PC or embedded device.

Step 1: Set-up Kinect for Windows SDK

Includes drivers for using Kinect sensor on a computer running Windows 7, Windows 8 Consumer Preview, and Windows Embedded Standard 7. In addition, the download includes application programming interfaces (APIs) and device interfaces.
Version 1.5, updated 05/21/2012, 221 MB, English.

DOWNLOAD LATEST SDK

View Release Notes

Explore the features

Step 2: Set-up Kinect for Windows Developer Toolkit

Contains source code samples, Kinect Studio, Face Tracking SDK, and other resources to simplify developing applications using the Kinect for Windows SDK.
Version 1.5, updated 05/21/2012, 60 MB, English.

DOWNLOAD TOOLKIT

View Release Notes

Cocoon

Cocoon is a framework to support the development of .Net Windows 8 Metro-style applications, in particular those that link to web services. It simplifies accessing, displaying and editing data using standard Metro controls, and allows easy application of the MVVM pattern.

For more details, and news on the latest releases visit http://andyonwpf.blogspot.com

Current Features

• A navigation framework with,
  • Attribute based page and view model discovery using MEF
  • Automatic creation and wiring up of page and view model
  • Passing of parameters to view models upon activation
  • Persistence of navigation and state upon application suspension

• A data framework with,
  • Support for observable collections and data virtualization
  • An extensible framework for the retrieval, processing and display of data from the cloud

Documentation

• Introduction
• Base Types
  • ObservableVector<T>
  • VirtualizingVector<T>
• Cocoon Navigation Framework
  • Defining pages and view models
  • More to follow...
• Cocoon Data Framework (IDataListSource<T> and data lists)
  • Introduction
  • Example Code
  • SimpleDataListSource<T>
  • PagedDataListSource<T>
  • VirtualizingDataList<T>

Cocoon – A new framework for Windows 8 Development

To assist with Windows 8 Metro-style applications I would like to announce the Cocoon framework (http://cocoon.codeplex.com/). This aims to build on top of the learning from the Chrysalis project to provide a simple way to develop Metro-style applications in a manner that fits naturally with the MVVM design pattern.

It is likely that many Metro-style apps will be connected applications, downloading data from across the internet in response to user navigation. Modern applications are expected to do this in a seamless manner, retrieving data on demand and populating the user interface in response. In some ways this is at odds with the web API programming model that is based upon individual request-response calls, with paging used to retrieve large datasets. One of the first targets of the Cocoon framework is to simplify this process – bridging the world of stateless web API calls, with the “fast and fluid” interfaces expected by users.

 Navigation in Cocoon – MVVM for Metro-style apps

Navigation in Cocoon

Although not limited to this, the Cocoon framework is designed to work great with the Model-View-ViewModel (MVVM) pattern that has become common when designing XAML based applications. One problem when using this pattern is that you need some way of associating a view with a view-model. Often this is done by navigating directly to a page, and using a “ViewModelLocator” to identify and wire up the respective view-model.

In Cocoon a slightly different approach is used. Here, rather than navigating directly to a view, you navigate to a named page. Although the behaviour is extensible, by default Cocoon will use MEF (the built in composition framework included in .Net) to locate both the view and view-model associated with this page name, create and initialise instances of these and wire them together.

...

The Navigation Manager

In order to navigate between pages, Cocoon includes a navigation manager that can be accessed by importing the INavigationManager interface through MEF (simplified by the fact that all views and view models in Cocoon are themselves are composed by MEF).

Getting Started with Cocoon Navigation – The Cocoon Bootstrapper

Since the Cocoon navigation framework is built on top of the Managed Extensibility Framework (MEF), the application is composed with the required dependencies as the user navigates through the application. MEF is a composition framework that is built into the .Net framework, that enables the construction of loosely-coupled, easily maintainable and testable applications to be composed automatically at run time. An introductory guide for those who are not familiar with MEF is available here.

What I have not covered previously however was how to navigate to the application’s first page when it is launched,

Launching Navigation Using The Cocoon Bootstrapper

To simplify the initialization of Cocoon based applications the framework includes the ‘CocoonBootstrapper’ class that can be used at startup. This manages the configuration of MEF for the most common scenarios, initialization of services and activation of the application.

...

Here's a snip of the sample app running.

Note, to get the sample app to run, you'll need your own Flickr API Key.

Here's a snap of the Solution;

And some snaps of the code, showing a little bit of how MEF is used.

If you've thought, "I love MEF and I wonder if I can use it to compose Metro Style App's, without messing up navigation" or looking for a new data handling framework or just want to check out a cool framework, Cocoon, and all its source, is ready for you...

From a slide deck and great link round-up my Pete Brown, to watching him geek out for an hour to a getting started with FEZ Hydra Basic Kit post by Ian Lee, Sr. it's all here. If at the end of this, your "I wanna build something" itch isn't asking for attention...

The .NET Micro Framework and .NET Gadgeteer

At the South Florida Code Camp last week, I presented the ".NET Micro Framework and .NET Gadgeteer". The primary demo was the Gadgeteer diaper monitor, but I also showed off hardware, wired some LEDs to the Netduino etc.

Pete Brown Geeks out on .NET Gadgeteer and Netduino

This past week, my friend and fellow Microsoftie, Pete Brown, ran a neat online symposium for the Microsoft Patterns & Practices group. As part of the event, Pete did an hour-long session introducing the .NET Micro Framework, and many of the cool things you can do with the various devices in the NETMF ecosystem, including Netduino, Netduino Go, and .NET Gadgeteer boards like the FEZ Spider. He was even kind enough to do a demo of the Meeblipiator project I put together using his Gadgeteer MIDI module (the demo is at around 45 minutes in). Here's the video…enjoy!

Getting Started With a FEZ Hydra Basic Kit

I’m teaching a NETMF Gadgeteer lab for the Nashville .NET User’s Group this week. Our gracious sponsors have purchased several FEZ Hydra Basic Kits made by GHI Electronics that we’ll be using to introduce everyone to the Gadgeteer world.

The FEZ Hydra Basic Kit consists of:

• FEZ Hydra Mainboard
• USB Client SP Module
• LED7R Module
• Joystick Module
• LightSense Module

So, the challenge is what can we build with just these components?

THE PLAN
We’re going to build a simple project that will use the green LEDs on the outer ring of the LED7R module as a gauge to show the light level that is reported by the LightSense module. In the center of the LED7R module is a bright red LED. I’m going to use the Joystick module to vary the brightness of this LED.

...

Run the program and now when you move the Joystick up & down you should see a slight variation in the brightness of the red LED in the middle of the LED7R module.

Surely, you’re having fun now???

You may be asking yourself why we used Timers instead of just putting a loop in our ProgramStarted() event. This is one area where Gadgeteer is very different than traditional NETMF projects. Read this blog post for an excellent explanation and you’ll save yourself some frustration down the line.

WHAT NEXT?
Well, that’s up to you! I’ve given you a simple example of a couple things you can do with the FEZ Hydra Basic Kit. Now, take this example and improve it or come up with something better (a game maybe?). Be sure to leave a comment on this post and tell me what you did or even better post a link to a video.

Have fun!

Download the complete project source here.

Key Design Decisions

It’s important to keep in mind that this car was built for a TV show with a set schedule. As a result, there are a number of unique design decisions that came into play.

Schedule

Working backwards, the reveal for the car was set for Monday November 28, 2011 at the Microsoft Store in Bellevue, Washington. We started the project in early August, which gave us approximately 12 weeks for research, development, vehicle assembly, and testing. This was by far the #1 design decision as any ideas or features for the car had to be implemented by the reveal date.

Off the Shelf Parts

Another key design decision was to, where possible, use off-the-shelf hardware and software in order to allow  interested developers to build and reuse some of the subsystems for their own car (at least the ones that don’t require welding). For example, instead of buying pricey custom sized displays for the instrument cluster or passenger display, we used stock Samsung Series 7 Slate PCs and had West Coast Customs do the hard work of building a custom dash to hold the PC.

Hardware and Networking

The car is packed with a variety of computers and networking hardware.

• Instrument Cluster Slate – This slate is on the driver's side and manages the instrument cluster application and the On-Board Diagnostic (OBD) connection to read telemetry data from the car.
• Passenger Slate – This slate, which is built into the passenger's side, runs a custom Windows 8 application (see Passenger slate below).
  
• Laptop 1 – This laptop runs the REST service to control different parts of the car, the Kinect socket service for the front Kinect, and the user message service to display messages on the rear glass while driving.
• Laptop 2 – This laptop runs the Heads Up Display (HUD) service, the Kinect socket service for the back Kinect, the OBD-II database, and Azure services.
• Windows Phone – A Nokia Lumia 800 connects via WiFi and a custom Windows Phone 7 application (See Windows Phone application below).
• Xbox 360 – The Xbox 360 displays on either the passenger HUD or the rear glass display.
• Networking – A NETGEAR N600/WNDR3700 wireless router provides wired and wireless access for everything in the car, which is used in conjunction with a Verizon USB network card plugged into a Cradle Point MBR900 to provide an always-on 3G/4G LTE internet connection. The slates, laptops, and Xbox 360 are connected via CAT5e cable, while the Windows Phone 7 connects via WiFi.

Note: One of the limitations of the Kinect SDK is that if you have multiple Kinects plugged into one PC, only one of those Kinects can do skeletal tracking at a time (color/depth data works just fine). Because of this, we decided to have a dedicated laptop plugged into the front Kinect and another laptop plugged into the back Kinect in order to allow front and back skeletal tracking at the same time. If we'd not used simultaneous skeletal tracking, we could have combined all of the systems onto a single laptop.

Architecture

Here is a quick overview of the application architecture.

REST Service Layer

The REST Service Layer allowed different systems talk to one another. More importantly, it allowed different services to control hardware they normally wouldn't be able to access.

1. Thin client approach
   The solution we chose was to have all the services that control different parts of the car reside on the laptops and have client applications like the Windows Phone application send REST commands to execute an action so the service layer would execute the request.
   
   
2. REST-enable hardware
   Controlling hardware should be invisible to the consuming clients. For example, hardware that requires USB communication would be impossible to control with a Windows Phone. The service layer allowed us to control hardware in a way that was invisible to the end user.
   
   
3. Helper Libraries
   To simplify communication with the service layer, we built a set of helper classes to abstract out repetitive tasks like JSON serialization/deserialization, URI building, etc. For example, to get the list of car horn “ringtones”, the client application can call HornClient.List() to get back a list of available ringtone filenames. To set the car horn, the client calls HornClient.Set(filename), and to play the car horn, it then calls HornClient.Play(filename). The Helper libraries were built to work on Windows 7, Windows 8, and Windows Phone 7.

OBD-II

We have already released an article and library on the OBD-II portion of the car.  In short, OBD-II stands for On-Board Diagnostics. Hooking into this port allows one to query for different types of data from the car, which we use to get the current speed, RPMs, fuel level, etc. for display in the Instrument Cluster and other locations. OBD can do far more than this, but it's all we needed for our project. Please see the linked articles for further details on the OBD-II library itself.

For the car, because only one application can open and communicate with a serial port at one time, we created a WCF service that polls the OBD-II data from the car and GPS data from a Microsoft Streets & Trips GPS locator, and returns it to any application that queries the service.

For the OBD library, we used a manual connection to poll different values at different intervals. For values critical to driving the car—like RPM, speed, etc.—we polled for the values as quickly as the car could return them. With other values that weren’t critical to driving the car—like the fuel level, engine coolant temperature, etc.—we polled at a 1-2 second interval. For GPS, we subscribed to the LocationChanged event, which would fire when the GPS values changed.

Rather than creating a new serial port connection for every WCF request for OBD data, we created a singleton service that is instantiated when the service first runs. Accordingly, there is only one object in the WCF service that represents the last OBD and GPS data returned, which is obtained by the continual reading of the latest OBD data using the OBD library as described above. This means that calls to the WCF service ReadMeasurement method didn’t actually compute anything, but instead serialized the last saved data and returned it via the WCF service.

Since WCF supports multiple protocols, we implemented HTTP and TCP and ensured that any WCF service options we chose worked on Windows Phone, which, for example, can only use basic HTTP bindings.

To enable the ability to change the programming model later and to simplify the polling of the service, we built a helper library for Windows and Windows Phone that abstracts all the WCF calls.

The code below creates a new ObdService class and signs up for an event when the measurement has changed. The Start method does a couple of things: it lets you set the interval that you want to poll the ObdService, in this case every second (while the instrument cluster needs fast polling, the database logger can poll once a second). It also determines what IP address the service is hosted at (localhost), the protocol (HTTP or TCP), and whether to send “demo mode” data. Since one of the main ways the car is showcased is when it’s stopped on display, “demo mode” sends fake data, instead of always returning 0's for MPH, RPM, etc., so people can see what the instrument cluster would look like in action.

_service = new ObdService();
_service.ObdMeasurementChanged += service_ObdMeasurementChanged;
_service.Start(new TimeSpan(0, 0, 0, 0, 1000), localhost, Protocol.Http, false);

void service_ObdMeasurementChanged(object sender, ObdMeasurementChangedEventArgs e)
{
  Debug.Writeline("MPH=” + e.Measurement.MilesPerHour);
}

OBD-II Database & Azure Services

To record and capture the car telemetry data like MPH, RPM, engine load, and throttle (accelerator) position, as well as location data (latitude, longitude, altitude, and course), we used a SQL Server Express database with a simple, flat Entity Framework model, shown below. The primary key, the ObdMeasurementID is a GUID that is returned via the ObdService.  Just like above, the database logger subscribes to the ObdMeasurementChanged event and receives a new reading at the time interval set in the Start() method.

The Windows Azure data model uses Azure Table Services instead of SQL Server. The data mapping is essentially the same since both have a flat schema.

For Azure Table Storage, in addition to the schema above, you also need a partition key and a row key. For the partition key, we used a custom TripID (GUID) to represent a Trip. When the car is turned on/off a new TripID is created. That way we could group all measurements for that particular trip and do calculations based on that trip, like the average miles per gallon, distance traveled, fastest speed, etc. For the row key, we used a DateTimeOffset and a custom extension method, ToEndOfDays() that provides a unique numerical string (since Azure's row key is a string type) that subtracts the time from the DateTime.Max value. The result is that the earlier a DateTime value, the larger the number.

Example:

Time=5/11/2012 9:14:09 AM, EndOfDays=2520655479509478223 //larger
Time=5/11/2012 9:14:11 AM, EndOfDays=2520655479482804811 //smaller

Since they are ordered in reverse order, with the most recent date/time being the first row, we can write an efficient query to pull just the first row to get the current latitude/longitude without needing to scan the entire table for the last measurment.

public override string RowKey
{
    get
    {
        return new DateTimeOffset(TimeStamp).ToEndOfDays();
    }
    set
    {
        //do nothing
    }
}

public static class DateTimeExtensions
{
  public static string ToEndOfDays(this DateTimeOffset source)
  {  
    TimeSpan timeUntilTheEnd = DateTimeOffset.MaxValue.Subtract(source);
    return timeUntilTheEnd.Ticks.ToString();
  }

  public static DateTimeOffset FromEndOfDays(this String daysToEnd)
  {
    TimeSpan timeFromTheEnd = newTimeSpan(Int64.Parse(daysToEnd));
    DateTimeOffset source = DateTimeOffset.MaxValue.Date.Subtract(timeFromTheEnd);
    return source;
  }
}

To upload data to Azure, we used a timer-based background uploader that would check to see if there was an internet connection, and then filter and upload all of the local SQL Express rows that had not been submitted to Azure using the Submitted boolean database field. On the Azure side, we used an ASP.NET MVC controller to submit data. The controller deserializes the data into a List<MeasurementForTransfer> type, it adds the data to a blob, and adds the blob to a queue as shown below.

A worker role (or many) will then read items off the queue and the new OBD measurement rows are placed into Azure Table Storage.

public ActionResult PostData()
{
    try
    {
        StreamReader incomingData = new StreamReader(HttpContext.Request.InputStream);

        string data = incomingData.ReadToEnd();
        JavaScriptSerializer oSerializer =
            new JavaScriptSerializer();

        List<MeasurementForTransfer> measurements;
        measurements = oSerializer.Deserialize(data, typeof(List<MeasurementForTransfer>)) as List<MeasurementForTransfer>;

        if (measurements != null)
        {
            CloudBlob blob = _blob.UploadStringToIncoming(data);
            _queue.PushMessageToPostQueue(blob.Uri.ToString());
            return new HttpStatusCodeResult(200);
        }
        ...
    }
}

Instrument Cluster

Much of this is also covered in our previously released OBD-II library where the instrument cluster application is included as a sample. This is a WPF application that runs on a Windows 7 slate. It contains three different skins designed by 352 Media—a 2012 Mustang dashboard, a 1967 Mustang dashboard, and a Metro-style dashboard—each of which can be "swiped" through. This application queries the OBD-II WCF service described above as quickly as it can to retrieve speed, RPM, fuel level, and other data for display to the driver. The gauges are updated in real-time just as a real dashboard instrument cluster would behave.

HUD

The HUD (or Heads Up Display) application runs on one of the two Windows 7 computers in the car. This is a full-screen application that is output via a projector to a series of mirrors and a projection screen. This is then reflected onto the front glass of the windshield of the car. To install these, we altered the physical car's body and created brackets to mount mirrors and the projectors. In the picture on the left, you can see the dashboard's structural member pivoted outward. You can see the 12" section we removed and added in the base plate to allow light to be reflected through to the windshield. Bill Steele helped design and implement the physical HUD aspect into the car.

The HUD application has several different modes.  The mode is selected from the Windows Phone application.

• POI / Mapping – This uses Bing Maps services. The phone or Windows 8 passenger application can choose one of a select group of categories (Eat, Entertain, Shop, Gas). Once selected, the REST service layer is contacted and the current choice is persisted. The HUD is constantly polling the service to know what the current category is, and when it changes, the HUD switches to an overhead map display with the closest locations of that category displayed, along with your always updated current GPS position and direction. The list of closest items in the category is requested every few seconds from the Bing Maps API and the map is updated appropriately.
  
• Car telemetry - In the car telemetry mode, the OBD data from the WCF service described above is queried and displayed on the screen.  This can be though of as an overall car "status" display with the speed, RPMs, real-time MPG, time, and weather information.
• Weather – We use the World Weather Online API to get weather data for display on the HUD. This API allows queries for weather based on a latitude and longitude, which we have at all times. A quick call to the service gives us the current temperature and a general weather forecast, which we display as an icon next to the temperature in the lower-left portion of the screen.
  
• Kinect – Using our Kinect Service, with the standard WPF client code, we can display the rear camera on the HUD to help the driver when backing up. See the Kinect Service project for more information on how this works and to use the service in an application of your own.

Windows Phone Application

One of the main ways to control the vehicle is through the Windows Phone application. 

The first pivot of the app allows the user to lock, unlock, start the car, and set off the alarm.  This is done through the Viper product from Directed Electronics.

The second pivot contains the remaining ways that a user can interact with the car.

• Kinect – This uses the Kinect service much in the way the HUD does. It can display both the front and rear cameras as well as allow the user to listen to an audio clip and send it up to the car while applying a voice changing effect.
  
• Voice Effect – When the Talk button is pressed, the user can record their voice via the microphone. When released, the audio data is packaged in a simple WAV file and uploaded to the REST service. The user can select from several voice effects, such as Chipmunk and Deep. On the service side, that WAV file is modified with the selected effect and then played through the PA system. The code in this section of the app is very similar to the Coding4Fun Skype Voice Changer. We use NAudio and several pre-made effects to process the WAV file for play.
• Lighting – This controls the external lighting for the car. The user can select a zone, an animation, and a color to apply. Once selected, this is communicated through the REST service to the lighting controller.
  
• Messaging – This presents a list of known pictures and videos for the user. The selection is sent to the car through the REST service and displayed on the projector that is pointed at the rear window, allowing following drivers to see the image, video, or message. 
• Point Of Interest – As described earlier, this is the way the user can turn on the Point of Interest map on the HUD.  Selecting one of the four items sends the selection to the REST service where it is persisted. The polling HUD will know when the selection is changed and display the map interface as shown above.
• Telemetry – This is a replica of the instrument cluster that runs on the Windows 7 slate. OBD data is queried via the WCF service, just like the slate, and displayed on the gauges, just like the slate.
   
• Projection Screen – This will raise and lower the projection screen on the rear of the car.
• Horn – This displays a list of known horn sound effects that live on the REST service layer. Selecting any of the items will send a command through the REST to play that sound file on the external sound system of the car. This selected audio file would play when the horn was pressed in the car.
• Settings – Internal settings for setting up hardware and software for the car.

Passenger Application

The passenger interface runs on a Samsung Series 7 slate running the Windows 8 Consumer Preview. This interface has a subset of the functionality provided by the Windows Phone application, but communicates through the same REST service. From this interface, the passenger can set the car horn sound effect, view the front and back Kinect cameras, select a Point of Interest category to be displayed on the HUD, and select the image, video or message that will be displayed on the rear window.

External Car Lighting

The external lighting system was controlled by a web server running on a Netduino Plus using a Sparkfun protoshield board to simplify wiring, and allow for another shield to be used. The actual lights were Digital Addressable RGB LED w/ PWM. We'll also have a more in-depth article on this system on Coding4Fun shortly.

The car is broken down into different zones—grill, wheels, vents, etc. It also has a bunch of pre-defined procedural animation patterns that have a few adjustable parameters that allow for things like a snake effect, a sensor sweep, or even a police pattern. Each zone has its own thread which provides the ability to have multiple animation patterns going at the same time. When a command is received, the color, pattern, zone, and other data is then processed.

Here is a basic animation loop pattern.

private static void RandomAnimationWorker()
{
    var leds = GetLedsToIlluminate();
    var dataCopy = _data;
    var r = new Random();

    while (IsThreadSignaledToBeAlive(dataCopy.LightingZone))
    {
        for (var i = 0; i < leds.Length; i++)
            SetLed(leds[i], r.Next(255), r.Next(255), r.Next(255));

        LedRefresh();
        Thread.Sleep(dataCopy.TickDuration);
    }
}

Rear Projection Window

The rear projection system consists of two 4” linear actuators, a linear actual controller, the NETMF web server from above, a Seeed Studio Relay Shield, the back glass of a 1967 Ford Mustang, some rear projection film, a low profile yet insanely bright projector that accepts serial port commands, and a standard USB to serial adapter.

The REST service layer toggles the input of the projector based on the selected state.  This would allow us to go from the HDMI output of an Xbox 360 to the VGA output of the laptop.  While doing this, the REST layer sends a command to the NETMF web server to either raise or lower the actuators.

Here is the code for the NETMF to control the raising and lowering the glass:

public static class Relay
{
    // code for for Electronic Brick Relay Shield
    static readonly OutputPort RaisePort = new OutputPort(Pins.GPIO_PIN_D5, false);
    static readonly OutputPort LowerPort = new OutputPort(Pins.GPIO_PIN_D4, false);

    const int OpenCloseDelay = 1000;

    public static bool Raise()
    {
        return ExecuteRelay(RaisePort);
    }

    public static bool Lower()
    {
        return ExecuteRelay(LowerPort);
    }

    private static bool ExecuteRelay(OutputPort port)
    {
        port.Write(true);
        Thread.Sleep(OpenCloseDelay);
        port.Write(false);

        return true;
    }
}

Messaging System

This is a WPF application that leveraged the file system on the computer to communicate between the REST service layer and itself.  Visually, it shows the message/image/video in the rear view mirror but it actually does two other tasks, it operates our car horn system and plays the recorded audio output from the phone.

• Displaying Messages
  To display images, we poll the REST service every second for an update.  Depending on the return type, we either display a TextBlock element or a MediaElement.
  
  
• Detecting and Playing Car Horn
  When someone presses the horn in the car, it is detected by a Phidget 8/8/8 wired into a Digital Input. In-between the car horn and the Phidget, there is a relay as well. This isolates the voltage coming from the horn and solves a grounding issue. We then feed back two wires from that relay and put one into the ground and the other into one of the digital inputs. In the application, we listen to the InputChange event on the Phidget and play / stop the audio based on the state.
  
  
• Detecting new recorded audio from the phone and car horn changes
  When someone talks into the phone or selects a new car horn, the REST service layer places that audio file into a predetermined directory. The Messaging service then uses a FileSystemWatcher to detect when this file is added. The difference between the car horn detection and recorded audio is the recorded audio will play once it is done writing to the file system.

External PA System

To interact with people, we installed an external audio PA or Public Address system. This system is hooked into the laptop that is connected to the car horn, and can play audio data from the phone. Having a PA system that is as simple as an audio jack that plugs into a PC enabled us to have different ringtones for the car horn and to talk through the car using Windows Phone.

Conclusion

After months of planning and building, the Project Detroit car was shown to the world on an episode of Inside West Coast Customs. Though it was a ton of work, the end product is something we are all proud of. We hope that this project inspires other developers to think outside the box and realize what can be done with some off-the-shelf hardware, software, and passion.

MayhemVolumeStart.zip file contains the class library project all set up with the NuGet package references and the reaction that increments the volume.

The first step we are going to take is to add the additional references we need to be configurable. The first reference is MayhemWpf from the same NuGet feed MayhemCore is from. (http://makemayhem.com/nuget). Add a reference to the following assemblies:

• PresentationCore
• PresentationFramework
• System.Runtime.Serialization
• System.Xaml
• WindowsBase.

Next, we switch the code from incrementing the volume to setting the volume to 50%.

To do this, replace the line:

device.AudioEndpointVolume.VolumeStepUp();

with

device.AudioEndpointVolume.MasterVolumeLevelScalar = 50 / 100.0f;

The reaction now works successfully to set the volume to half way. The next step is to make that setting configurable, and persist between application runs. Let’s make that 50 a variable. Add a private integer to the class (I’m calling it level) and change the above line to be:

device.AudioEndpointVolume.MasterVolumeLevelScalar = level / 100.0f;

Since that variable will be user configurable, we want to be able to show a string on the main window that says what the reaction is doing. To do this, we have to tell Mayhem that our reaction is configurable by implement the IWpfConfigurable interface from MayhemWpf.

In order to set that configuration string we have to implement the method GetConfigString. The implementation of this method should return some string that provides a look into what the configuration settings are for the reaction.

I will make the GetConfigString method be the following:

public string GetConfigString()
{
   return string.Format("Set the volume to {0}%",level);
}

Now, in order to make the module save the configured state when the application is shut down and restarted, we need to add a little bit of data contract code. We are going to add the [DataContract] attribute to our class. This attribute comes from the System.Runtime.Serialization framework.

Our class header now looks like:

[DataContract]
[MayhemModule("Set Volume", "Sets the master volume level")]
public class SetVolume : ReactionBase, IWpfConfigurable

Note that I also changed the MayhemModule attribute’s name and description to reflect the new functionality.

We also need to mark the configurable variables to be saved as well. To do that, add the [DataMember] attribute to our level variable:

[DataMember]
private int level;

In order to set a default value for this variable, we shouldn’t initialize it using a constructor. Constructors should never be used in modules. Instead, we have a method that should be overridden that you get from ReactionBase. OnLoadDefaults is to load the settings you want to have by default.

protected override void OnLoadDefaults()
{
   level = 50;
}

Now when you use the SetVolume reaction in Mayhem, it will at 50%, but if that value is changed and the application is restarted, it will be set to the value it was at before it was closed!

The next step is to make the configuration window that actually enables that variable to be modified.

Add a new WPF UserControl item to the project named SetVolumeConfig.xaml.

Replace the contents of the XAML file with:

<src:WpfConfiguration x:Class="Volume.SetVolumeConfig"
   xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
   xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"   
   xmlns:src="clr-namespace:MayhemWpf.UserControls;assembly=MayhemWpf"
   Width="300">
   <Grid>
      <Grid.ColumnDefinitions>
         <ColumnDefinition Width="100" />
         <ColumnDefinition Width="200" />
      </Grid.ColumnDefinitions>
      <Grid.RowDefinitions>
         <RowDefinition />
      </Grid.RowDefinitions>
      <TextBlock Style="{StaticResource ConfigLabel}" Grid.Column="0" Grid.Row="0">Volume:</TextBlock>
      <Slider Grid.Column="1" Grid.Row="0" Height="23" HorizontalAlignment="Left"  Name="VolumeSlider" VerticalAlignment="Top" Width="200" Maximum="100" />
   </Grid>
</src:WpfConfiguration>

This configuration window simply has a volume slider allowing values between 0 and 100. WpfConfiguration handles the display of how the module asks for information. As you hopefully notice, it doesn’t contain the save button, or the title of the window. All that is handled for you so that we can maintain consistency between configuration windows. We also provide default styles that further maintain consistency and user expectation.

We now need to create the backing code that makes the configuration window come together. Change SetVolumeConfig.xaml.cs to extend the base class WpfConfiguration instead of the default UserControl base class.

In the constructor, we should ask for all the information we need to populate the config window with the current settings. In this situation, that only means the current volume level setting. Add an int parameter to the constructor called level. Also add a public property on the class with a private setter called Level. In the constructor, set this property with the parameter passed in to the constructor.

Your class should now look like this:

using MayhemWpf.UserControls;
namespace Volume
{
   public partial class SetVolumeConfig : WpfConfiguration
   {
      public int Level
      {
         get;
         private set;
      }
      public SetVolumeConfig(int level)
      {
         this.Level = level;
         InitializeComponent();
      }
   }
}

WpfConfiguration gives us some handy hooks that we can use. We need to set the value property on our slider. This could be done with data bindings, but for simplicity we will just be doing this with code. Also, there is a Boolean flag in the WpfConfiguration class that enables the Save button on the dialog. When handling user input, the save button should only be enabled if the input is valid. Since the slider always has a value in our range, the Save button can always be enabled. We will set this flag in the OnLoad method.

Override the OnLoad method as follows:

public override void OnLoad()
{
   VolumeSlider.Value = this.Level;
   CanSave = true;
}

We then also override the OnSave method to set the property to the value the SetVolume class will expect.

public override void OnSave()
{
   this.Level = (int)VolumeSlider.Value;
}

Lastly, we need to set the title that shows up on the configuration window. To do this, override the Title property and have it return “Set Volume”:

public override string Title
{
   get { return "Set Volume"; }
}

Now we need to go back to the SetVolume class and implement those configuration related methods.

The ConfigurationControl property needs to return an instance of the SetVolumeConfig class. Remember that its constructor expects the current volume setting:

public WpfConfiguration ConfigurationControl
{
   get
   {
      return new SetVolumeConfig(level);
   }
}

Lastly, OnSaved takes the instance of WpfConfiguration control we just returned in the above property. We need to cast this back to SetVolumeConfig to be able to get access to the Level property:

public void OnSaved(WpfConfiguration configurationControl)
{
   var config = (SetVolumeConfig)configurationControl;
   this.level = config.Level;
}

That’s all it takes! This code provides us with a complete reaction that enables us to set the volume level on the computer. As you can see, we provide a very straightforward API enabling you, the developer, to write user configurable modules for Mayhem.  You can click the download link at the top of the screen to get the final code in MayhemVolumeFinal.zip.

We have more documentation available on the available methods and APIs on our CodePlex documentation page.

The OBD-II Manager project is the OBD-II sub-system and instrument cluster application from the Project Detroit Mustang we built with West Coast Customs.  If you're not familiar with this project, you can find out more about it here and here.

OBD stands for On-Board Diagnostics.  In the world of cars, this can be thought of as the computer which has a variety of data points which can be queried, such as speed, fuel level, and even trouble codes that relate to the check engine light.  If you've ever taken your car in for service, your service center has hooked up the car to a computer via the OBD port to get the status of the car and what may not be working properly. 

This project contains a library you can use to integrate OBD data into your application, as well as a modified version of the actual Instrument Cluster application used in the Project Detroit car which shows how the library can be used.

More Information

• Coding4Fun Article
• Project Detroit
• OBD-II to USB cable (ELM323/327-based OBD-II to USB cable)
• More information on OBD-II
• ELM323/327 Datasheet

Instrument Cluster Screenshots

At the SXSW Interactive Opening Party hosted by Frog Design and Microsoft, the Coding4Fun team has unveiled their latest invention, BoxingBots!

With BoxingBots you are the controller except this time, forget the screen. You’re controlling a 6-foot tall metal robot that’s trying to punch out your opponent’s 6-foot tall machine. It’s Real Steel in real time.

Video Demo

Watch Laura Foy's video of the BoxingBots in action!

BoxingBots combines Kinect’s motion sensing technology with pneumatic robots in a 6-foot square ring. Two players face off to try to knock each other’s blocks off.

Each robot has

• 1 On-board laptop PC
• 70 programmatic LEDs in the head and chest controlled by a Phidgets LED controller board
• 2 2.5 gallon tanks of compressed air @ 150 PSI
• 8 12-volt 17amp batteries
• 2 Roboteq motor controllers that power the four omni-directional wheels
• 2 5" stroke cylinders that punches/retracts the arms
• 1 6' stroke cylinder that lifts the head when the robot is knocked out
• 2 button switches on the chest that are triggered by punches and a Phidget 8/8/8 interface board that controls the relays and buttons switches

How gameplay happens

• Each robot's on-board PC receive commands over the network from the Admin PC for movement, punches, score, etc
• Each player stands in front of a Kinect for Windows-enabled PC that interprets their gestures (punch, move in a direction, strafe, rotate) and sends that as a command to the Admin PC
• The Admin PC manages the game, including pausing, reset, starting, sending commands to the robots and receiving damage data from the robots when the robot has been punched

Photos

WPF Sample Screenshots

Windows Phone Sample Screenshots