Project Detroit: An Overview

In this article, we will give an overview of the technical side of Project Detroit, the Microsoft-West Coast Custom Mustang creation. If you're not already familiar with this project, you can find more information here. 

Key Design Decisions

It’s important to keep in mind that this car was built for a TV show with a set schedule. As a result, there are a number of unique design decisions that came into play.

Schedule

Working backwards, the reveal for the car was set for Monday November 28, 2011 at the Microsoft Store in Bellevue, Washington. We started the project in early August, which gave us approximately 12 weeks for research, development, vehicle assembly, and testing. This was by far the #1 design decision as any ideas or features for the car had to be implemented by the reveal date.

Off the Shelf Parts

Another key design decision was to, where possible, use off-the-shelf hardware and software in order to allow  interested developers to build and reuse some of the subsystems for their own car (at least the ones that don’t require welding). For example, instead of buying pricey custom sized displays for the instrument cluster or passenger display, we used stock Samsung Series 7 Slate PCs and had West Coast Customs do the hard work of building a custom dash to hold the PC.

Hardware and Networking

The car is packed with a variety of computers and networking hardware.

• Instrument Cluster Slate – This slate is on the driver's side and manages the instrument cluster application and the On-Board Diagnostic (OBD) connection to read telemetry data from the car.
• Passenger Slate – This slate, which is built into the passenger's side, runs a custom Windows 8 application (see Passenger slate below).
  
• Laptop 1 – This laptop runs the REST service to control different parts of the car, the Kinect socket service for the front Kinect, and the user message service to display messages on the rear glass while driving.
• Laptop 2 – This laptop runs the Heads Up Display (HUD) service, the Kinect socket service for the back Kinect, the OBD-II database, and Azure services.
• Windows Phone – A Nokia Lumia 800 connects via WiFi and a custom Windows Phone 7 application (See Windows Phone application below).
• Xbox 360 – The Xbox 360 displays on either the passenger HUD or the rear glass display.
• Networking – A NETGEAR N600/WNDR3700 wireless router provides wired and wireless access for everything in the car, which is used in conjunction with a Verizon USB network card plugged into a Cradle Point MBR900 to provide an always-on 3G/4G LTE internet connection. The slates, laptops, and Xbox 360 are connected via CAT5e cable, while the Windows Phone 7 connects via WiFi.

Note: One of the limitations of the Kinect SDK is that if you have multiple Kinects plugged into one PC, only one of those Kinects can do skeletal tracking at a time (color/depth data works just fine). Because of this, we decided to have a dedicated laptop plugged into the front Kinect and another laptop plugged into the back Kinect in order to allow front and back skeletal tracking at the same time. If we'd not used simultaneous skeletal tracking, we could have combined all of the systems onto a single laptop.

Architecture

Here is a quick overview of the application architecture.

REST Service Layer

The REST Service Layer allowed different systems talk to one another. More importantly, it allowed different services to control hardware they normally wouldn't be able to access.

1. Thin client approach
   The solution we chose was to have all the services that control different parts of the car reside on the laptops and have client applications like the Windows Phone application send REST commands to execute an action so the service layer would execute the request.
   
   
2. REST-enable hardware
   Controlling hardware should be invisible to the consuming clients. For example, hardware that requires USB communication would be impossible to control with a Windows Phone. The service layer allowed us to control hardware in a way that was invisible to the end user.
   
   
3. Helper Libraries
   To simplify communication with the service layer, we built a set of helper classes to abstract out repetitive tasks like JSON serialization/deserialization, URI building, etc. For example, to get the list of car horn “ringtones”, the client application can call HornClient.List() to get back a list of available ringtone filenames. To set the car horn, the client calls HornClient.Set(filename), and to play the car horn, it then calls HornClient.Play(filename). The Helper libraries were built to work on Windows 7, Windows 8, and Windows Phone 7.

OBD-II

We have already released an article and library on the OBD-II portion of the car.  In short, OBD-II stands for On-Board Diagnostics. Hooking into this port allows one to query for different types of data from the car, which we use to get the current speed, RPMs, fuel level, etc. for display in the Instrument Cluster and other locations. OBD can do far more than this, but it's all we needed for our project. Please see the linked articles for further details on the OBD-II library itself.

For the car, because only one application can open and communicate with a serial port at one time, we created a WCF service that polls the OBD-II data from the car and GPS data from a Microsoft Streets & Trips GPS locator, and returns it to any application that queries the service.

For the OBD library, we used a manual connection to poll different values at different intervals. For values critical to driving the car—like RPM, speed, etc.—we polled for the values as quickly as the car could return them. With other values that weren’t critical to driving the car—like the fuel level, engine coolant temperature, etc.—we polled at a 1-2 second interval. For GPS, we subscribed to the LocationChanged event, which would fire when the GPS values changed.

Rather than creating a new serial port connection for every WCF request for OBD data, we created a singleton service that is instantiated when the service first runs. Accordingly, there is only one object in the WCF service that represents the last OBD and GPS data returned, which is obtained by the continual reading of the latest OBD data using the OBD library as described above. This means that calls to the WCF service ReadMeasurement method didn’t actually compute anything, but instead serialized the last saved data and returned it via the WCF service.

Since WCF supports multiple protocols, we implemented HTTP and TCP and ensured that any WCF service options we chose worked on Windows Phone, which, for example, can only use basic HTTP bindings.

To enable the ability to change the programming model later and to simplify the polling of the service, we built a helper library for Windows and Windows Phone that abstracts all the WCF calls.

The code below creates a new ObdService class and signs up for an event when the measurement has changed. The Start method does a couple of things: it lets you set the interval that you want to poll the ObdService, in this case every second (while the instrument cluster needs fast polling, the database logger can poll once a second). It also determines what IP address the service is hosted at (localhost), the protocol (HTTP or TCP), and whether to send “demo mode” data. Since one of the main ways the car is showcased is when it’s stopped on display, “demo mode” sends fake data, instead of always returning 0's for MPH, RPM, etc., so people can see what the instrument cluster would look like in action.

_service = new ObdService();
_service.ObdMeasurementChanged += service_ObdMeasurementChanged;
_service.Start(new TimeSpan(0, 0, 0, 0, 1000), localhost, Protocol.Http, false);

void service_ObdMeasurementChanged(object sender, ObdMeasurementChangedEventArgs e)
{
  Debug.Writeline("MPH=” + e.Measurement.MilesPerHour);
}

OBD-II Database & Azure Services

To record and capture the car telemetry data like MPH, RPM, engine load, and throttle (accelerator) position, as well as location data (latitude, longitude, altitude, and course), we used a SQL Server Express database with a simple, flat Entity Framework model, shown below. The primary key, the ObdMeasurementID is a GUID that is returned via the ObdService.  Just like above, the database logger subscribes to the ObdMeasurementChanged event and receives a new reading at the time interval set in the Start() method.

The Windows Azure data model uses Azure Table Services instead of SQL Server. The data mapping is essentially the same since both have a flat schema.

For Azure Table Storage, in addition to the schema above, you also need a partition key and a row key. For the partition key, we used a custom TripID (GUID) to represent a Trip. When the car is turned on/off a new TripID is created. That way we could group all measurements for that particular trip and do calculations based on that trip, like the average miles per gallon, distance traveled, fastest speed, etc. For the row key, we used a DateTimeOffset and a custom extension method, ToEndOfDays() that provides a unique numerical string (since Azure's row key is a string type) that subtracts the time from the DateTime.Max value. The result is that the earlier a DateTime value, the larger the number.

Example:

Time=5/11/2012 9:14:09 AM, EndOfDays=2520655479509478223 //larger
Time=5/11/2012 9:14:11 AM, EndOfDays=2520655479482804811 //smaller

Since they are ordered in reverse order, with the most recent date/time being the first row, we can write an efficient query to pull just the first row to get the current latitude/longitude without needing to scan the entire table for the last measurment.

public override string RowKey
{
    get
    {
        return new DateTimeOffset(TimeStamp).ToEndOfDays();
    }
    set
    {
        //do nothing
    }
}

public static class DateTimeExtensions
{
  public static string ToEndOfDays(this DateTimeOffset source)
  {  
    TimeSpan timeUntilTheEnd = DateTimeOffset.MaxValue.Subtract(source);
    return timeUntilTheEnd.Ticks.ToString();
  }

  public static DateTimeOffset FromEndOfDays(this String daysToEnd)
  {
    TimeSpan timeFromTheEnd = newTimeSpan(Int64.Parse(daysToEnd));
    DateTimeOffset source = DateTimeOffset.MaxValue.Date.Subtract(timeFromTheEnd);
    return source;
  }
}

To upload data to Azure, we used a timer-based background uploader that would check to see if there was an internet connection, and then filter and upload all of the local SQL Express rows that had not been submitted to Azure using the Submitted boolean database field. On the Azure side, we used an ASP.NET MVC controller to submit data. The controller deserializes the data into a List<MeasurementForTransfer> type, it adds the data to a blob, and adds the blob to a queue as shown below.

A worker role (or many) will then read items off the queue and the new OBD measurement rows are placed into Azure Table Storage.

public ActionResult PostData()
{
    try
    {
        StreamReader incomingData = new StreamReader(HttpContext.Request.InputStream);

        string data = incomingData.ReadToEnd();
        JavaScriptSerializer oSerializer =
            new JavaScriptSerializer();

        List<MeasurementForTransfer> measurements;
        measurements = oSerializer.Deserialize(data, typeof(List<MeasurementForTransfer>)) as List<MeasurementForTransfer>;

        if (measurements != null)
        {
            CloudBlob blob = _blob.UploadStringToIncoming(data);
            _queue.PushMessageToPostQueue(blob.Uri.ToString());
            return new HttpStatusCodeResult(200);
        }
        ...
    }
}

Instrument Cluster

Much of this is also covered in our previously released OBD-II library where the instrument cluster application is included as a sample. This is a WPF application that runs on a Windows 7 slate. It contains three different skins designed by 352 Media—a 2012 Mustang dashboard, a 1967 Mustang dashboard, and a Metro-style dashboard—each of which can be "swiped" through. This application queries the OBD-II WCF service described above as quickly as it can to retrieve speed, RPM, fuel level, and other data for display to the driver. The gauges are updated in real-time just as a real dashboard instrument cluster would behave.

HUD

The HUD (or Heads Up Display) application runs on one of the two Windows 7 computers in the car. This is a full-screen application that is output via a projector to a series of mirrors and a projection screen. This is then reflected onto the front glass of the windshield of the car. To install these, we altered the physical car's body and created brackets to mount mirrors and the projectors. In the picture on the left, you can see the dashboard's structural member pivoted outward. You can see the 12" section we removed and added in the base plate to allow light to be reflected through to the windshield. Bill Steele helped design and implement the physical HUD aspect into the car.

The HUD application has several different modes.  The mode is selected from the Windows Phone application.

• POI / Mapping – This uses Bing Maps services. The phone or Windows 8 passenger application can choose one of a select group of categories (Eat, Entertain, Shop, Gas). Once selected, the REST service layer is contacted and the current choice is persisted. The HUD is constantly polling the service to know what the current category is, and when it changes, the HUD switches to an overhead map display with the closest locations of that category displayed, along with your always updated current GPS position and direction. The list of closest items in the category is requested every few seconds from the Bing Maps API and the map is updated appropriately.
  
• Car telemetry - In the car telemetry mode, the OBD data from the WCF service described above is queried and displayed on the screen.  This can be though of as an overall car "status" display with the speed, RPMs, real-time MPG, time, and weather information.
• Weather – We use the World Weather Online API to get weather data for display on the HUD. This API allows queries for weather based on a latitude and longitude, which we have at all times. A quick call to the service gives us the current temperature and a general weather forecast, which we display as an icon next to the temperature in the lower-left portion of the screen.
  
• Kinect – Using our Kinect Service, with the standard WPF client code, we can display the rear camera on the HUD to help the driver when backing up. See the Kinect Service project for more information on how this works and to use the service in an application of your own.

Windows Phone Application

One of the main ways to control the vehicle is through the Windows Phone application. 

The first pivot of the app allows the user to lock, unlock, start the car, and set off the alarm.  This is done through the Viper product from Directed Electronics.

The second pivot contains the remaining ways that a user can interact with the car.

• Kinect – This uses the Kinect service much in the way the HUD does. It can display both the front and rear cameras as well as allow the user to listen to an audio clip and send it up to the car while applying a voice changing effect.
  
• Voice Effect – When the Talk button is pressed, the user can record their voice via the microphone. When released, the audio data is packaged in a simple WAV file and uploaded to the REST service. The user can select from several voice effects, such as Chipmunk and Deep. On the service side, that WAV file is modified with the selected effect and then played through the PA system. The code in this section of the app is very similar to the Coding4Fun Skype Voice Changer. We use NAudio and several pre-made effects to process the WAV file for play.
• Lighting – This controls the external lighting for the car. The user can select a zone, an animation, and a color to apply. Once selected, this is communicated through the REST service to the lighting controller.
  
• Messaging – This presents a list of known pictures and videos for the user. The selection is sent to the car through the REST service and displayed on the projector that is pointed at the rear window, allowing following drivers to see the image, video, or message. 
• Point Of Interest – As described earlier, this is the way the user can turn on the Point of Interest map on the HUD.  Selecting one of the four items sends the selection to the REST service where it is persisted. The polling HUD will know when the selection is changed and display the map interface as shown above.
• Telemetry – This is a replica of the instrument cluster that runs on the Windows 7 slate. OBD data is queried via the WCF service, just like the slate, and displayed on the gauges, just like the slate.
   
• Projection Screen – This will raise and lower the projection screen on the rear of the car.
• Horn – This displays a list of known horn sound effects that live on the REST service layer. Selecting any of the items will send a command through the REST to play that sound file on the external sound system of the car. This selected audio file would play when the horn was pressed in the car.
• Settings – Internal settings for setting up hardware and software for the car.

Passenger Application

The passenger interface runs on a Samsung Series 7 slate running the Windows 8 Consumer Preview. This interface has a subset of the functionality provided by the Windows Phone application, but communicates through the same REST service. From this interface, the passenger can set the car horn sound effect, view the front and back Kinect cameras, select a Point of Interest category to be displayed on the HUD, and select the image, video or message that will be displayed on the rear window.

External Car Lighting

The external lighting system was controlled by a web server running on a Netduino Plus using a Sparkfun protoshield board to simplify wiring, and allow for another shield to be used. The actual lights were Digital Addressable RGB LED w/ PWM. We'll also have a more in-depth article on this system on Coding4Fun shortly.

The car is broken down into different zones—grill, wheels, vents, etc. It also has a bunch of pre-defined procedural animation patterns that have a few adjustable parameters that allow for things like a snake effect, a sensor sweep, or even a police pattern. Each zone has its own thread which provides the ability to have multiple animation patterns going at the same time. When a command is received, the color, pattern, zone, and other data is then processed.

Here is a basic animation loop pattern.

private static void RandomAnimationWorker()
{
    var leds = GetLedsToIlluminate();
    var dataCopy = _data;
    var r = new Random();

    while (IsThreadSignaledToBeAlive(dataCopy.LightingZone))
    {
        for (var i = 0; i < leds.Length; i++)
            SetLed(leds[i], r.Next(255), r.Next(255), r.Next(255));

        LedRefresh();
        Thread.Sleep(dataCopy.TickDuration);
    }
}

Rear Projection Window

The rear projection system consists of two 4” linear actuators, a linear actual controller, the NETMF web server from above, a Seeed Studio Relay Shield, the back glass of a 1967 Ford Mustang, some rear projection film, a low profile yet insanely bright projector that accepts serial port commands, and a standard USB to serial adapter.

The REST service layer toggles the input of the projector based on the selected state.  This would allow us to go from the HDMI output of an Xbox 360 to the VGA output of the laptop.  While doing this, the REST layer sends a command to the NETMF web server to either raise or lower the actuators.

Here is the code for the NETMF to control the raising and lowering the glass:

public static class Relay
{
    // code for for Electronic Brick Relay Shield
    static readonly OutputPort RaisePort = new OutputPort(Pins.GPIO_PIN_D5, false);
    static readonly OutputPort LowerPort = new OutputPort(Pins.GPIO_PIN_D4, false);

    const int OpenCloseDelay = 1000;

    public static bool Raise()
    {
        return ExecuteRelay(RaisePort);
    }

    public static bool Lower()
    {
        return ExecuteRelay(LowerPort);
    }

    private static bool ExecuteRelay(OutputPort port)
    {
        port.Write(true);
        Thread.Sleep(OpenCloseDelay);
        port.Write(false);

        return true;
    }
}

Messaging System

This is a WPF application that leveraged the file system on the computer to communicate between the REST service layer and itself.  Visually, it shows the message/image/video in the rear view mirror but it actually does two other tasks, it operates our car horn system and plays the recorded audio output from the phone.

• Displaying Messages
  To display images, we poll the REST service every second for an update.  Depending on the return type, we either display a TextBlock element or a MediaElement.
  
  
• Detecting and Playing Car Horn
  When someone presses the horn in the car, it is detected by a Phidget 8/8/8 wired into a Digital Input. In-between the car horn and the Phidget, there is a relay as well. This isolates the voltage coming from the horn and solves a grounding issue. We then feed back two wires from that relay and put one into the ground and the other into one of the digital inputs. In the application, we listen to the InputChange event on the Phidget and play / stop the audio based on the state.
  
  
• Detecting new recorded audio from the phone and car horn changes
  When someone talks into the phone or selects a new car horn, the REST service layer places that audio file into a predetermined directory. The Messaging service then uses a FileSystemWatcher to detect when this file is added. The difference between the car horn detection and recorded audio is the recorded audio will play once it is done writing to the file system.

External PA System

To interact with people, we installed an external audio PA or Public Address system. This system is hooked into the laptop that is connected to the car horn, and can play audio data from the phone. Having a PA system that is as simple as an audio jack that plugs into a PC enabled us to have different ringtones for the car horn and to talk through the car using Windows Phone.

Conclusion

After months of planning and building, the Project Detroit car was shown to the world on an episode of Inside West Coast Customs. Though it was a ton of work, the end product is something we are all proud of. We hope that this project inspires other developers to think outside the box and realize what can be done with some off-the-shelf hardware, software, and passion.