Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Content Obsolete

This content is no longer current.

Computer-Controlled R/C Car with Camera

 

Control a remote controlled car with your computer using Microsoft Robotics Studio. Then, add a wireless IP camera for stealthy remote operation.
ASPSOFT, Inc.

Difficulty: Advanced
Time Required: 3-6 hours
Cost: $50-$100
Software: Visual Basic or Visual C# Express Editions, Microsoft Robotics Studio, Phidgets .NET and MSRS libraries
Hardware: A remote-controlled car, PhidgetInterfaceKit 0/0/4 OR PhidgetInterfaceKit 0/16/16, Airlink101 AIC-250W wireless camera (optional), Xbox 360 Controller for Windows (optional), batteries, wire, and solder
Download: Download
 
In 2005, we at ASPSOFT built the world's first .NET-powered battlebot known as The Finalizer. Since then, many robotics enthusiasts and hobbyists have become intrigued with the idea of a .NET-controlled robot, but don't have the cash or hardware required to build something as elaborate as The Finalizer. With that in mind, I set out to create a very simple and very cheap computer controlled vehicle that anyone with a couple bucks and a computer could build.
 

What You Will Need

The Car

First, we will modify the remote controlled car so that it can be interfaced with the Phidget controller. The easiest way to accomplish this is to modify the hand-held controlling unit to receive input from the Phidget controller instead of a human driver.

Start by unscrewing all screws on the back of the unit and then open up the controller.

If you have chosen the right type of controller, that is, one with digital inputs, you will see that the joysticks simply push down on contacts on the board, closing the circuit for forward/backward/left/right. We will be wiring these contacts to our Phidget board so when the relays/IO ports open and close, they will be closing and opening the circuits.

To do this, cut 8 equal lengths of wire of about 6 inches. Strip off a bit of insulation from each end.

Next, solder one end of each wire to each contact of the controller. You will need to connect one wire to both the ground and the active points of the contact as shown in the picture.

When all wires are soldered, insert the opposite ends into the screw terminals of the Phidget interface board.

Phidget 0/0/4 Interface Kit

You'll notice on the Phidget board that there are 3 terminals per relay, labeled NO, XC, and NC, where “X” is a number from 0 to 3. These stand for “Normally Open”, “Relay X Common”, and “Normally Closed”. Since our circuits are normally open and closed when you press the joystick, you will want to connect one wire from each contact to the NO and the XC terminals of each relay. Wire the relays as follows:

0 – Forward

1 – Backward

2 – Left

3 - Right

 

Phidget 0/16/16 Interface Kit

This board is divided into two sections and is labeled as such: Inputs and Outputs.  On the Output side, there are 16 outputs, numbered 0 through 15.  There are also 8 common ground terminals labeled "G".  As with the above board, you will want to connect one wire from each contact to a numbered output, and one to a common ground contact.  Wire them using the same numbering scheme as above.


The Camera

If you decide to add the network camera to the project continue reading. If you choose to skip the camera, move on to the next section.

The camera will need to be battery powered. The camera I used in this project is the Airlink101 AIC-250W. This camera requires 5V to operate. Getting 5V out of a series of standard batteries isn't easy without a voltage regulator, so I cheated and decided to try pumping 6V (4 AA batteries at 1.5V each) to the device. I have run the cam for over 2 hours on one set of batteries with no ill effects.

To easily connect the batteries to the camera, I used a 4 AA battery holder and a power connector that matched the size of the connector on the included AC power supply. As stated above, the size required is a 5.5mm outer diameter with a 2.5mm inner diameter.

On the back of the camera, above the power connector, there is a symbol showing the positive and negative connections for the power connector.

The symbol shows that the negative terminal is the outside ring and the positive terminal is the inside. With that in mind, we will solder the black wire of the holder to the outside connector, and the red to the ring connector. The wires from the battery holder can be easily soldered onto the power connector as shown.

When all is said and done, you will have a battery holder with a DC power end that can be plugged directly into the camera. Pop 4 AA batteries into the holder and you should see the Power light on the camera turn on. I chose to mount the battery pack to the top of the camera using some double-sided tape as shown.

Finally, mount the camera to the vehicle with some double-sided tape or velcro.

The Software

Now that the hardware is complete, we need to write the software to control the car. First, the required software must be installed.  Install the software in the following order:

  1. Microsoft C# or Visual Basic 2005 Express Edition
  2. Microsoft Robotics Studio
  3. Phidgets libraries
  4. Phidgets Services for Microsoft Robotics Studio

Once all of the software is installed, the files phidget21.dll and Phidget21.NET.dll must be copied from the \Program Files\Phidgets directory to the bin directory of your MSRS installation.  If this is not done, the Phidget services will all fail to start when requested, causing you as much pain and torment as it caused me.

The Code

Before we get started, it is essential that you have some familiarity with Microsoft Robotics Studio. Please read through the included help file as well as the base tutorials. I have also included several links at the bottom of this article with even more information on the internals of MSRS.  The documentation and tutorials will be your best guidance for understanding what we are about to build.

At its core, the MSRS runtime aids in developing applications where concurrency and performance are key issues. In a robot where there are multiple sensors receiving input that needs to be concurrently handled, MSRS excels.  While our remote controlled car may not fit into that mold exactly, it is a great way to delve into Robotics Studio and learn about what it has to offer.

To start, we must first create a service for our car. This can be done by opening the Microsoft Robotics Studio Command Prompt from the Microsoft Robotics Studio (1.0) program group on the Start menu.

To generate the template for our service, the command dssnewservice.exe is used. Change to the directory where you wish to create your project and type one of the following commands, depending on the language you wish to use, to create a new service named “RCCar”:

C#:

dssnewservice /service:RCCar

VB

dssnewservice /language:VB /service:RCCar

This will generate a folder with several items, including project and solution files.  Note that it will lower-case RCCar to Rccar in various places.  This is normal.

Open the generated RCCar.sln file in Microsoft C# 2005 Express Edition or Microsoft Visual Basic 2005 Express Edition. You will see that several files were generated.

RCCar.cs/.vb – Contains actual implementation of the RCCar service

RCCarTypes.cs/.vb – Contains list of messages and default state object

RCCar.manifest.xml – A description of the services started when the application is run

Take some time to review the base generated code. This will help as we move forward.

Open the RCCar.sln file to start working.  The first thing we will need to do is access the service which controls our Phidget Interface Kit. Robotics Studio includes a built-in service for this device. To access it, first, set a reference to the PhidgetBoards.Y2006.M08.proxy namespace. This can be done by right-clicking on the project name in the Solution Explorer and selecting Add Reference from the context menu. When the Add Reference dialog appears, select PhidgetBoards.Y2006.M08.proxy and click the OK button.

To use the contents of this library in our code, we need to import the library.  This can be done by adding the following line to the Rccar class.

C#

using phidgetinterfacekit = Phidgets.Robotics.Services.PhidgetInterfaceKitBoards.Proxy;
using phidgetcommon = Phidgets.Robotics.Services.Proxy;

VB

Imports phidgetinterfacekit = Phidgets.Robotics.Services.PhidgetInterfaceKitBoards.Proxy
Imports phidgetcommon = Phidgets.Robotics.Services.Proxy

Next, we need to setup a partner relationship with the PhidgetInterfaceKit service and declare a port on which to communicate with it. At the top of the RccarService class implementation, add the following code to the service:

C#

[Partner("PhidgetInterfaceKit", Contract = phidgetinterfacekit.Contract.Identifier,
CreationPolicy = PartnerCreationPolicy.UseExistingOrCreate)]
private phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations _ikPort =
new phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations();
VB
<Partner("PhidgetInterfaceKit", Contract:=phidgetinterfacekit.Contract.Identifier, _
CreationPolicy:=PartnerCreationPolicy.UseExistingOrCreate)> _
Private _ikPort As New phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations()

Next, we will setup a few items that will be required to talk to the Phidget controller.  First, we must subscribe to the PhidgetInterfaceKit service.  We will be subscribing to additional services later on, so create a SubscribeToServices method in the RccarService class and then call this method from the bottom of the Start method:

C#

private void SubscribeToServices()
{
_ikPort.SelectiveSubscribe(null,
new phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations());
}

VB

Private Sub SubscribeToServices()
_ikPort.SelectiveSubscribe(Nothing, New phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations())
End Sub

This will subscribe to the PhidgetInterfaceKit service without requesting any notifications on events, as they will not be needed for this project.

Next, create an enumeration above the definition for the RccarService class for the possible directions:

C#

public enum Direction
{
Forward = 0,
Backward,
Left,
Right,
None
}

VB
Public Enum Direction
Forward = 0
Backward
Left
Right
None
End Enum

The value of each item in the enumeration will correspond to the relay used for that direction as described above.  Next, we will write a method named SetOutput to communicate with the Phidget Interface Kit and enable or disable the output relay as appropriate:

C#

public void SetOutput(Direction dir, bool enabled)
{
// if no direction sent, return
if(dir == Direction.None)
return;

// create a new request to send to the Phidgets service
phidgetcommon.SetOutputRequestType req = new phidgetcommon.SetOutputRequestType();

// assign the index and state (true/false)
req.Index = (int)dir;
req.State = enabled;

// send the request to the port
_ikPort.SetOutput(req);
}

VB

Public Sub SetOutput(ByVal dir As Direction, ByVal enabled As Boolean)
' if no direction sent, return
If dir = Direction.None Then Return

' create a new request to send to the Phidgets service
Dim req As New phidgetcommon.SetOutputRequestType

' assign the index and state (true/false)
req.Index = dir
req.State = enabled

' send the request to the port
_ikPort.SetOutput(req)
End Sub

The SetOutput method creates a new SetOutputRequestType request object and fills in the appropriate Index and State. The Index refers to the index number of the relay to change, and the State property is a boolean of whether the relay should be opened or closed.

Next, we will need a user interface to interact with to control the robot. Right-click on the RCCar project and select Add -> Windows Form from the context menu. Name this form RemoteControlForm.cs/.vb.

<NOTE FOR VB USERS>

Due to an oddity in the dssproxy application which runs during the compilation process, one addition must be made to the form created in the Visual Basic project.  To make this change, first click on the Show all files button on the Solution Explorer toolbar as shown:

Once that is done, you should see a "+" next to the RemoteControlForm.vb file.  First, right-click on the RemoteControlForm.vb file and select View Code.  Add the following namespace definition around the the class definition as shown:

Namespace Robotics.Rccarvb
Public Class RemoteControlForm

End Class
End Namespace

Next, open the RemoteControlForm.Designer.vb file using the same method above and add the same namespace definition around the class implementation.  Once this is done, verify that the project compile and continue on.

</NOTE FOR VB USERS>

For the moment, we will setup the form with four buttons (up, down, left and right) and control the vehicle with those. Drag four buttons onto the form and label them appropriately. You can easily get arrow glyphs on the buttons by using the Marlett font and setting the Text property of each button to one of the following:

3 = Left

4 = Right

5 = Up

6 = Down

Name the buttons btnForward, btnBackward, btnLeft, and btnRight.  In an effort to make things easy for determining which button is pressed, assign the values 0, 1, 2, and 3 to the Tag property of each button to match the enumeration created above.  So, the Tag property of the forward button would be 0, backward would be 1, and so on.  When you are done, you will have a form similar to the following:

This form will need to call the SetOutput method we created in our service above.  Therefore, it will need an instance of the RccarService available to it.  To easily accomplish this, the constructor for the form will be written to accept the current instance of the service and store it in a member variable for later use.  Add the following code to RemoteControlForm.cs/.vb to facilitate this:

C#

RccarService _service = null;

public RemoteControlForm(RccarService service)
{
InitializeComponent();

_service = service;
}

VB

Private _service As RccarService

Public Sub New(ByVal service As RccarService)

' This call is required by the Windows Form Designer.
InitializeComponent()

' Add any initialization after the InitializeComponent() call.
_service = service
End Sub

To properly drive the car, we need to listen for both the button being clicked (close the circuit) and the button being released (open the circuit). This can be done by hooking the MouseDown and MouseUp events on each button. Open RemoteControlForm.cs/.vb in the designer. Then, choose the events list in the properties window by clicking on the “lightning bolt” icon. This will toggle the event list into view.

Highlight all of the buttons on the form. Then, enter btn_MouseDown into the MouseDown event and btn_MouseUp in the MouseUp event. The IDE will automatically generate method stubs for each of those events.

Next, add the code for the MouseDown and MouseUp events.  If you are using VB, you will need to import the System.Windows.Forms namespace into the RemoteControlForm code.  The Tag property created above will be used from each button to determine which was clicked:

C#

private void btn_MouseDown(object sender, MouseEventArgs e)
{
_service.SetOutput((Direction)int.Parse((sender as Button).Tag.ToString()), true);
}

private void btn_MouseUp(object sender, MouseEventArgs e)
{
_service.SetOutput((Direction)int.Parse((sender as Button).Tag.ToString()), false);
}
VB
Imports System.Windows.Forms

...

Private Sub btn_MouseDown(ByVal sender As System.Object, ByVal e As System.Windows.Forms.MouseEventArgs) Handles btnRight.MouseDown, btnLeft.MouseDown, btnForward.MouseDown, btnBackward.MouseDown
_service.SetOutput(CType(Integer.Parse(CType(sender, Button).Tag.ToString()), Direction), True)
End Sub

Private Sub btn_MouseUp(ByVal sender As System.Object, ByVal e As System.Windows.Forms.MouseEventArgs) Handles btnRight.MouseUp, btnLeft.MouseUp, btnForward.MouseUp, btnBackward.MouseUp
_service.SetOutput(CType(Integer.Parse(CType(sender, Button).Tag.ToString()), Direction), False)
End Sub

The events simply look at the button being sent in as the sender parameter, convert the Tag to an entry in the Direction enumeration, and set the associated relay using the SetOutput method.

Now that our base user interface is setup, we need to launch the form from our service. To do this, we need to add reference to the Ccr.Adapters.WinForms namespace. This can be done in the same way as we added the reference to the Phidget library. Once that is done, back in Rccar.cs/.vb, bring the WinForms library into the class by adding the following using statement:

C#

using Microsoft.Ccr.Adapters.WinForms;

VB

Imports Microsoft.Ccr.Adapters.WinForms

To actually launch the form, add the following line to the bottom of the Start method, below the previous call to SubscribeToServices:

C#

WinFormsServicePort.Post(new RunForm(StartForm));

VB

WinFormsServicePort.Post(New RunForm(AddressOf StartForm))

This will launch the method StartForm which actually creates the form. Add a member variable to the class to store the instance of the created form and add the StartMethod form as follows:

C#

RemoteControlForm _remoteForm = null;
public System.Windows.Forms.Form StartForm()
{
_remoteForm = new RemoteControlForm(this);
return _remoteForm;
}

VB

Private _remoteForm As RemoteControlForm

Private Function StartForm() As System.Windows.Forms.Form
_remoteForm = New RemoteControlForm(Me)
Return _remoteForm
End Function

This will create a new instance of the form, assign it to the local member variable, and return the instance back to the calling method.

At this point, everything should be in place to actually run the project. Make sure the Phidget board is connected to the PC via the USB cable and press the F5 key to build and execute the project. If all went to plan, you should see the UI form start up and you should be able to click the arrow buttons and hear the relays click on the 0/0/4 board, or the lights flicker on the 0/16/16 board. If the car is turned on and the remote is turned on, the car will drive around based on the buttons you click.

The Keyboard

We can easily add support to drive the car with the keyboard's arrow keys. To use the arrows, we cannot use the native KeyDown event because the arrow keys are normally used to cycle through the active controls of a form. Therefore, we need to trap the key down events earlier in the message chain. To do this, we must override the ProcessCmdKey method of the Form class. Additionally, the form's KeyPreview property must be set to false.  Add the following code to the RemoteControlForm.cs/.vb file:

C#

const int WM_KEYDOWN = 0x100;

protected override bool ProcessCmdKey(ref Message msg, Keys keyData)
{
if(msg.Msg == WM_KEYDOWN)
{
switch(keyData)
{
case Keys.Up:
_service.SetOutput(Direction.Forward, true);
break;

case Keys.Down:
_service.SetOutput(Direction.Backward, true);
break;

case Keys.Left:
_service.SetOutput(Direction.Left, true);
break;

case Keys.Right:
_service.SetOutput(Direction.Right, true);
break;
}
return true;
}
return false;
}

VB

Dim WM_KEYDOWN As Integer = 256

Protected Overrides Function ProcessCmdKey(ByRef msg As Message, ByVal keyData As Keys) As Boolean
If (msg.Msg = WM_KEYDOWN) Then
Select Case (keyData)
Case Keys.Up
_service.SetOutput(Direction.Forward, true)
Case Keys.Down
_service.SetOutput(Direction.Backward, true)
Case Keys.Left
_service.SetOutput(Direction.Left, true)
Case Keys.Right
_service.SetOutput(Direction.Right, true)
End Select
Return true
End If
Return false
End Function

This looks much like the button processors from above.

Just like the buttons, we also need a KeyUp event. Go back to the designer for the form and double-click on the KeyUp event in the properties pane, just like we did with the mouse button events. This will generate an event stub for the KeyUp event. Fill in the method with the following:

C#

private void RemoteControlForm_KeyUp(object sender, KeyEventArgs e)
{
switch(e.KeyCode)
{
case Keys.Up:
_service.SetOutput(Direction.Forward, false);
break;

case Keys.Down:
_service.SetOutput(Direction.Backward, false);
break;

case Keys.Left:
_service.SetOutput(Direction.Left, false);
break;

case Keys.Right:
_service.SetOutput(Direction.Right, false);
break;
}
}

VB

Private Sub RemoteControlForm_KeyUp(ByVal sender As System.Object, ByVal e As System.Windows.Forms.KeyEventArgs) Handles MyBase.KeyUp
Select Case (e.KeyCode)
Case Keys.Up
_service.SetOutput(Direction.Forward, False)
Case Keys.Down
_service.SetOutput(Direction.Backward, False)
Case Keys.Left
_service.SetOutput(Direction.Left, False)
Case Keys.Right
_service.SetOutput(Direction.Right, False)
End Select
End Sub

Now, if you execute the application again, you will be able to drive the car around just by pressing the arrow keys on the keyboard.

Gamepad Support

If you own an Xbox 360 Controller for Windows, or a similar gamepad device, we can use it to drive our vehicle around. Ensure that the Xbox 360 Controller for Windows drivers are installed and the controller is working normally.

To use the gamepad, we first need to partner with the gamepad service, much like we did with the PhidgetInterfaceKit service.

First, set a reference to the XInputGamepad.Y2006.M09.proxy namespace. Bring the namespace into the RCCar.cs/.vb file by adding the following using statement, and then partner with the XInputGamepad service like we did with the Phidget service above:

C#

[Partner("XInputGamepad", Contract = gamepad.Contract.Identifier,
CreationPolicy = PartnerCreationPolicy.UseExistingOrCreate)]
private gamepad.XInputGamepadOperations _gamepadPort = new gamepad.XInputGamepadOperations();

VB

<Partner("XInputGamepad", Contract:=gamepad.Contract.Identifier, _
CreationPolicy:=PartnerCreationPolicy.UseExistingOrCreate)> _
Private _gamepadPort As New gamepad.XInputGamepadOperations()

With that in place, we now need to subscribe to the XInputGamepad service to receive notifications when the state of the controller changes. This code will be added to our previous SubscribeToServices method: 

C#

private void SubscribeToServices()
{
_ikPort.SelectiveSubscribe(null, new phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations());

gamepad.XInputGamepadOperations _gamepadNotify = new gamepad.XInputGamepadOperations();

Activate(Arbiter.Receive<gamepad.DPadChanged>(true, _gamepadNotify, GamepadDpadHandler));

_gamepadPort.Subscribe(_gamepadNotify);
}

VB

Private Sub SubscribeToServices()
_ikPort.SelectiveSubscribe(Nothing, New phidgetinterfacekit.PhidgetInterfaceKitBoardsOperations())

Dim _gamepadNotify As gamepad.XInputGamepadOperations = New gamepad.XInputGamepadOperations

Activate(Arbiter.Receive(Of gamepad.DPadChanged)(True, _gamepadNotify, AddressOf GamepadDpadHandler))

_gamepadPort.Subscribe(_gamepadNotify)
End Sub

This method creates a port named _gamepadNotify which will receive the notifications from the Gamepad service. The Activate method sets up relationships between communication ports and arbiters. An arbiter is simply an object which automatically manages concurrency. So, the second line sets up a relationship between the _gamepadNotify port and a method we will implement, GamepadDpadHandler. This is setup on the DPadChanged message.

The last line in the method sets up a subscription. A subscription is required to receive the messages on the port specified.  You may wish to wrap the call to Subscribe a call to Arbiter.Choice.  This will allow you to determine if the subscription was successful or not, and display a message either way.  The source package for this article contains the code on how this is done.  For simplicity, it has been removed for the listing above.

Next, we must actually implement the GamepadDpadHandler method. It looks like the following snippet:

C#

public void GamepadDpadHandler(gamepad.DPadChanged msg)
{
gamepad.DPad dpad = msg.Body;

SetOutput(Direction.Forward, dpad.Up);
SetOutput(Direction.Backward, dpad.Down);
SetOutput(Direction.Left, dpad.Left);
SetOutput(Direction.Right, dpad.Right);
}

VB

Private Sub GamepadDpadHandler(ByVal msg As gamepad.DPadChanged)
Dim dpad As gamepad.DPad = msg.Body

SetOutput(Direction.Forward, dpad.Up)
SetOutput(Direction.Backward, dpad.Down)
SetOutput(Direction.Left, dpad.Left)
SetOutput(Direction.Right, dpad.Right)
End Sub

This code simply maps the current state of the d-pad to the outputs on the Phidget interface kit.  We can also use the thumbsticks of the Xbox 360 controller to drive the vehicle.  You will find the code for this in the source package linked above.

With this code in place, we can once again press F5 to build and execute the project. Ensure the Phidget board and the controller are connected. When the application runs, you will be able to drive the car with the left thumbstick.

Note: This XInputGamepad service specifically works with devices that are defined as gamepads. If you have a true analog joystick, like a flight stick, you will need to replace the code above with a partnership with the GameController service instead of the XInputGamepad service and propagate accordingly.

The Camera

Now that the car is up and running, we can add the code required to drive the wireless camera. 

<NOTE FOR VB USERS>

Unfortunately, I have been unsuccessful in getting the following code to work in Visual Basic due to its lack of custom iterators (i.e. the lack of the yield keyword).  I am currently pursuing a workaround for this, but until one is found, the following section will consist of C# code only.  I will update this portion of the article with VB code if/when a solution is found.

</NOTE FOR VB USERS>

I decided to create a brand new service to operate the wireless camera so it could be later used in other projects very easily. In this section, we will learn how to create a brand new service and use it to post messages to other services that are listening.

As with the RCCar service, you will use the dssnewservice.exe command to generate the template for the new service. Open the Robotics Studio Command Prompt as before and run the following:

C#

dssnewservice /service:NetCam

This will, as before, generate a directory and code for a new service named NetCam.

Back in Microsoft C# Express 2005, right-click on the solution in the Solution Explorer and select Add -> Existing Project and navigate to the generated NetCam.csproj file. This will bring the project into our existing solution for easy editing and debugging.

The first thing we need to do is add a reference to the System.Drawing assembly.  This can be done as described above.

Next, switch to the NetCamTypes.cs file. We will be using the NetCamState to store the most recently captured image from the camera. To do this, we will need to bring the System.Drawing reference into the class and modify the NetCamState object to look like the following:

C#

using System.Drawing;

...

[DataContract()]
public class NetCamState
{
private Bitmap _image;
private int _size;
private DateTime _timeStamp;

public Bitmap Image
{
get { return _image; }
set { _image = value; }
}

[DataMember]
public Size Size
{
get
{
if (_image != null)
{
return _image.Size;
}
return Size.Empty;
}
set { return; }
}

[DataMember]
public DateTime TimeStamp
{
get { return _timeStamp; }
set { _timeStamp = value; }
}
}

The DataContract() attribute states that the NetCamState object is serializable via XML. The DataMember attribute must be added to every property and field that you would like to be serialized. A Bitmap object cannot be directly serialized, so we will be working around that shortly.

The network camera I chose for this project streams its live video feed via MotionJPEG, or MJPEG. This is a quirky little non-standard that basically sends a never-ending stream of JPG images down the pipe. When displayed fast enough, it looks like smooth animation. Think of an electronic flipbook and you've got the right idea.

The net cam expects a web request to its internal web server and then responds with a multipart response which is a stream of JPEG images, each prefaced with some boundary tags and information about the image ahead. A standard GET request to the camera's IP address will result in something that looks like the following:

HTTP/1.0 200 OK 
Server: Camera Web Server/1.0
Auther: Steven Wu
MIME-version: 1.0
Cache-Control: no-cache
Content-Type: multipart/x-mixed-replace;boundary=--video boundary--
--video boundary--Content-length: 9052
Date: 2006-10-01 22:21:47 IO_00000000_PT_000_114
Content-type: image/jpeg
<binary data of length 9052 bytes>
--video boundary--Content-length: 9375
Date: 2006-10-01 22:21:48 IO_00000000_PT_000_114
Content-type: image/jpeg
<binary data of length 9375 bytes>

Our NetCam service is going to connect to the network camera's MJPEG source URL and loop through the response data, pulling out each frame as it is downloaded. When a partnered service wishes to request the current frame, it will do so by sending the QueryFrame message that we are about to define.

Open the NetCamType.cs file and you will see class definitions for NetCamOperations, Get, and Replace. Here we will add our new message.

Add a class named QueryFrame to the file with the rest of the request types.  This will inherit from the base Query object and define its request object, QueryFrameRequest, and it's response, one of two types: a QueryFrameResponse object, or a Fault in case of an error.  Next, we need to add this QueryFrame message to the list of valid messages that NetCamOperations will handle. This is done by changing the class definition as shown.  Finally, we need to create the definitions for the QueryFrameRequest and QueryFrameResponse objects.

C#

public class NetCamOperations : PortSet<DsspDefaultLookup, DsspDefaultDrop, Get, QueryFrame>
{
}

public class QueryFrame : Query<QueryFrameRequest, PortSet<QueryFrameResponse, Fault>>
{
}

[DataContract]
public class QueryFrameRequest
{
}

[DataContract]
public class QueryFrameResponse
{
[DataMember]
public Size Size;

[DataMember]
public byte[] FrameBuffer;

[DataMember]
public DateTime TimeStamp;
}

As you can see, for right now, the QueryFrameRequest object remains empty, while the QueryFrameResponse contains a handful of properties which define the size of the image, a byte buffer for the image data itself, and a timestamp for the time the image was grabbed.

Now we need to handle the camera data itself and provide the current frame data to a service requesting it.  This will be accomplished by creating a processing thread which will parse each image from the webcam into a JPEG image, store it in memory to pass as a response to a request for it, and repeat.  Internally, we will parse our JPEG, post a message to an internal port stating that the current frame is ready, store the frame, and start over.  Since many requests can be received concurrently, we will be using the built-in Arbiter class to manage incoming requests while at the same time, parsing the data from the webcam to ensure the service does not get into a situation where frame data is being manipulated while a request is simultaneously being processed.

First, the internal message port will be created to handle notifications that a complete frame is available.  The port will be named FramePort and designed to handle messages of type Frame.  The implementation these types can be added directly to the NetCam.cs file either above or below the implementation of the NetCamService class.  The Frame class will be using types from the System.Drawing library, so ensure that the library is both referenced and imported.

C#

using System.Drawing;
using System.Drawing.Imaging;

...

internal class Frame
{
private Bitmap _image;
private Size _size;
private DateTime _timeStamp;

public Bitmap Image
{
get { return _image; }
set { _image = value; }
}

public Size Size
{
get
{
if (_image != null)
{
return _image.Size;
}
return Size.Empty;
}
set { return; }
}

public DateTime TimeStamp
{
get { return _timeStamp; }
set { _timeStamp = value; }
}
}

internal class FramePort : Port<Frame> { }

Now we need to hook up the QueryFrameRequest message so that it can be responded to from incoming requests, and we need to grab the data from the camera. To do this, first, add the following member variable definition to the top of the NetCamService class along with a declaration of a boolean variable to maintain the status of the connection to the camera, and then modify the Start method of the NetCam service to look like the snippet below:

C#

FramePort _framePort = new FramePort();
private bool _connected = false;

protected override void Start()
{
base.Start();

Activate(
Arbiter.Interleave(
new TeardownReceiverGroup(),
new ExclusiveReceiverGroup(
Arbiter.Receive(true, _framePort, FrameHandler)
),
new ConcurrentReceiverGroup
(
Arbiter.Receive<QueryFrame>(true, _mainPort, QueryFrameHandler)
)
)
);

DirectoryInsert();

Thread t = new Thread(new ThreadStart(NetCamProcessor));
t.IsBackground = true;
t.Start();

LogInfo(LogGroups.Console, "Service uri: ");
}

You will notice a new Arbiter method. The Interleave allows you to setup several handlers at once, defining the concurrency of each. Handlers that require Exclusive access are registered in the ExclusiveReceiverGroup constructor, and those that can be called simultaneously are registered in the ConcurrentReceiverGroup constructor.

In the code above, we setup a handler on the internal _framePort port, calling the method FrameHandler. Additionally, a handler is created on the main service port, _mainPort for the QueryFrame message, which will be handled by the QueryFrameHandler method.

Finally, a thread is created which will be responsible for connecting to the network camera and grabbing each JPEG image as it is sent. I am not going to list every single line here, but I will highlight the important points.  The full code, of course, can be found in the source package.  First, let's implement the FrameHandler method.  As described above, a message will be posted from the processing thread when a new frame is ready to be requested.  The FrameHandler method is the method that will respond to that notification.  Simply, it will expect a Frame object and store it away inside of the NetCamState object we created earlier.

C#

private void FrameHandler(Frame frame)
{
_state.Image = frame.Image;
_state.TimeStamp = frame.TimeStamp;
}

Next, we need to implement the NetCamProcessor thread handler.  The code here is a bit complex, and I have taken a number of shortcuts to make it as simple as possible.  I am the first to admit that the parsing code is not very robust, however it will properly parse the output of the Airlink AIC-250W.  The thread processor will use the HttpWebRequest and HttpWebResponse methods of the System.Net class, several IO methods from System.IO and the methods from the System.Text class.  As always, ensure this is referenced and imported into the class.

The processor will sit in an infinite loop which connects to the IP address of the camera, requests a specific page, and parses the MJPEG data as described earlier.  When a full frame has been parsed, it posts that Frame object to our internal _framePort to store away until a QueryFrameRequest comes in.

C#

using System.Net;
using System.IO;
using System.Text;

...

private void CameraProcessor()
{
byte[] buff = new byte[1024*1024];
byte[] lenBuff = new byte[10];
HttpWebRequest req = null;
HttpWebResponse resp = null;
Stream s = null;
int len = 0;

while(true)
{
try
{
// NOTE: change the IP address for the camera on your network
req = (HttpWebRequest)HttpWebRequest.Create("http://192.168.1.100/mjpeg.cgi");
resp = (HttpWebResponse)req.GetResponse();

_connected = true;

s = resp.GetResponseStream();
BinaryReader br = new BinaryReader(s);

while(true)
{
Array.Clear(buff, 0, buff.Length);
Array.Clear(lenBuff, 0, lenBuff.Length);

// --video boundary--
// Content-length:
buff = br.ReadBytes(34);

// content length
byte by;
int i = 0;
while((by = br.ReadByte()) != 0x0d)
lenBuff[i++] = by;
len = int.Parse(Encoding.ASCII.GetString(lenBuff));

// Date: 2000-01-01 01:23:45 IO_00000000_PT_000_114
// Content-type: image/jpeg
buff = br.ReadBytes(79);

// image data
buff = br.ReadBytes(len);

// create a new Frame object and post it to our internal port
Frame frame = new Frame();
frame.Image = (Bitmap)Bitmap.FromStream(new MemoryStream(buff, 0, len));
frame.TimeStamp = DateTime.Now;

_framePort.Post(frame);

// new lines before the next --video boundary-- segment
buff = br.ReadBytes(6);
}
}
catch(Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.Message);
_connected = false;
}
finally
{
if(req != null)
req.Abort();

if(s != null)
s.Close();

if(resp != null)
resp.Close();
}
}
}

The NetCamProcessor method connects to the mjpeg.cgi file on the network camera's IP address. Once connected, it loops forever reading data from the response stream, parsing out the Content-length information and reading precisely that much information into a byte buffer. With this in hand, it fills in an instance of the Frame object and posts that to the internal _framePort communications port. As we set it up earlier, the _framePort handles messages sent to it with the FrameHandler method. This was setup in the exclusive group for a reason. Because images are streaming in as quickly as possible, before all processing can be completed and the next image arrives, by posting the message to the _framePort port, the thread will block until the FrameHandler method returns, as this method was registered in the ExclusiveReceiverGroup above.  This will allow us to copy the current Image and TimeStamp into the internal state object before moving on to the next image. Therefore, when a partnered service requests the current frame, it will return the current frame in the state object without worry of being overwritten mid-process.

The final thing to setup in our NetCamService is the method to respond to the QueryFrame message. Recall that this handler was setup in the constructor above.  This code uses the Fault object in the W3C.Soap namespace, so import accordingly.

C#

using W3C.Soap;

...

private void QueryFrameHandler(QueryFrame query)
{
QueryFrameResponse response = new QueryFrameResponse();

if(!_connected)
{
response.Size = Size.Empty;
response.FrameBuffer = null;
response.TimeStamp = DateTime.MinValue;
query.ResponsePort.Post(new Fault());
}
else
{
if(_state.Image != null)
{
ImageFormat format = ImageFormat.Bmp;

using (MemoryStream stream = new MemoryStream())
{
Size size = _state.Image.Size;
_state.Image.Save(stream, format);

response.TimeStamp = _state.TimeStamp;
response.FrameBuffer = new byte[(int)stream.Length];
response.Size = size;

stream.Position = 0;
stream.Read(response.FrameBuffer, 0, response.FrameBuffer.Length);
}
query.ResponsePort.Post(response);
}
}
}

This method takes the current image stored in the internal state object and converts it into a byte stream to be returned in our previously defined QueryFrameResponse object. The response object is then sent back to the caller via the ResponsePort object contained within the passed in QueryFrame object.

Now that we have the camera setup to grab images and allow requests for the currently available frame, we can partner with the service in our RCCar service and display the live stream from the camera.

As with the other services, you will need to add a reference to the NetCam service. You can do this by adding a reference to the NetCam.Y200X.MYY where X is the current year, and Y is the current month. Do not add add a reference to the project-level NetCam assembly.

Now, switch back to the RCCar.cs file. At the top with the other partner definitions, add the following two lines to partner with the NetCam service:

C#

[Partner("NetCam", Contract = netcam.Contract.Identifier,
CreationPolicy = PartnerCreationPolicy.UseExistingOrCreate)]
private netcam.NetCamOperations _netcamPort = new netcam.NetCamOperations();

Next, we need to add a method which will continually send the QueryFrame message to the NetCam service and handle the resulting image data. This can be accomplished by adding the following call to the Start method:

C#

SpawnIterator(NetCamHandler);

The SpawnIterator method will allow you to call a method of type IEnumerator<ITask> asynchronously.  Let's take a look at the NetCamHandler method that is spawned.  This will use code from the System.Threading library, so import as usual.

C#

using System.Threading;

...

private IEnumerator<ITask> NetCamHandler()
{
bool pollCamera = true;

while(true)
{
// wait a few ticks for the cam to get connected
Thread.Sleep(4000);
pollCamera = true;

while(pollCamera)
{
// send the QueryFrame message and send the result to our form to be displayed
yield return Arbiter.Choice(
_netcamPort.QueryFrame(new netcam.QueryFrameRequest()),
delegate(netcam.QueryFrameResponse response)
{
WinFormsServicePort.FormInvoke(
delegate()
{
_remoteForm.ImageUpdate(response);
}
);
},
delegate(Fault fault)
{
LogError("Error querying frame from camera");
pollCamera = false;
}
);
}
}
}

This will wait for 4 seconds for the camera to become available and then continually poll the NetCam service for the latest available frame. The Choice method from the Arbiter object will allow you to send a message to a specified port and handle the result in both a success and a failure. In this case, we are creating a QueryFrameRequest object, posting it to the _netCam port, and handing the response with two delegates, the first being the successful return, and the second being a fault from the NetCam service.

When a valid QueryFrameResponse object is returned, the delegate we've defined calls the ImageUpdate method on our user interface via WinFormsServicePort.FormInvoke to ensure that we do not attempt to update the UI thread from a thread other than it was created on.  ImageUpdate passes the QueryFrameResponse response object which contains the image data to be displayed.

The yield keyword seen above is new to C# and .NET 2.0 .  Also note that the NetCamHandler method is defined as type IEnumerator<ITask>.  When yield return is used with an enumerator, this tells the CCR that more data will be available.  When yield break is used, this signals the end of the enumeration.  In this case, since we will constantly be polling images, this signals to the CCR that more data is forthcoming.

On the RemoteControlForm, a PictureBox control must be added to display the images from the camera.  Do this in the designer and set the box dimensions to 320x240, the default size returned by the network camera. Name the control pbImage.

And finally, following code is added to the RemoteControlForm.cs class.  Note that the following method, ImageUpdate, uses methods from the System.IO and our NetCam namespaces, so import them at the top.

C#

using System.IO;
using netcam = Robotics.NetCam.Proxy;

...

public void ImageUpdate(netcam.QueryFrameResponse response)
{
pbImage.Image = Bitmap.FromStream(new MemoryStream(response.FrameBuffer));
}

This simply takes the response from our above QueryFrame request, turns it back into a Bitmap object, and then assigns it to the PictureBox.

We're Done!

Pressing F5 to build and run the solution should now allow you to:

  • Drive the car with the keyboard
  • Drive the car with the onscreen buttons
  • Drive the car with a gamepad controller
  • View the output from the network camera attached to the roof of the vehicle.

So where do we go from here?  Well, there are a variety of features that can be added to the project, such as:

  • Implement the remote control form using the MSRS-included DirectionDialog form.  Note that you will likely have to split out the camera display to a separate form.
  • Implement the above with the VPL language.
  • Try building a new service for another platform (Lego NXT, BoeBot, etc.) and use the NetCam service and IP camera with that.
  • Add configuration and state persistence to allow the IP address for the camera to be specified in an XML store instead of hard-coding it into the application.
  • Add logic to replay a series of commands to drive the vehicle in a set path.
  • With the above, add motion detection code to have the car surprise an unsuspecting victim.
  • If you are using a Phidget 0/16/16 or Phidget 8/8/8 board, find a remote control car with additional features, like the Tyco N.S.E.C.T, assuming it has a digital vs. analog control unit.

Conclusion

As you can see, Microsoft Robotics Studio is a very functional, very sophisticated and very complex library which tackles many of the issues involved with creating software for use in robotics.  With the issues of concurrency and decentralization handled by the CCR and DSS runtimes, you are free to focus your efforts at building robust software for controlling the machine at hand.

More Information

Here are a few links regarding MSRS, the CCR and various other useful tidbits to help you along:

Bio

Though Brian is a recognized .NET expert with over 6 years experience developing .NET solutions, and over 9 years of professional experience architecting and developing solutions using Microsoft technologies and platforms, he has been "coding for fun" for as long as he can remember.  Outside the world of .NET and business applications, Brian enjoys developing both hardware and software projects in the areas of gaming, robotics, and whatever else strikes his fancy for the next ten minutes. He rarely passes up an opportunity to dive into a C/C++ or assembly language project.  You can reach Brian via his blog at http://www.brianpeek.com/.

Tags:

Follow the Discussion

  • DutchDutch

    if you make it with one remote control will it control just the car it was made from or will 1 comp controller control multiple cars

  • David BDavid B

    So, if you are using an 8/8/8 board versus a 0/0/4 board you have digital outputs versus physical relays.  Does this make wiring the RC transmitter different for the 8/8/8 compared to the 0/0/4?  In one, the power is coming from the USB port, and in the other, the power is coming from the RC transmitter itself.

    Thanks!

  • Clint RutkasClint I'm a "developer"

    @David B:   yes, 8/8/8 is digital, 0/0/4 are analog.

    As for wiring, click the contact us link at the top, I'll put you in contact with Brian Peek.

  • sewayehu ezgebusewayehu ezgebu

    A nice project.i want to do a similar project with some modification like go and stop buttons.

  • JimJim

    Great project but Please help! I have installed all needed software but had to use Microsoft Robotics 1.5. Everything works so far but the keyup event. I hit the keyboard direction keys and the relay click but when I take my finger off it does not set the relay to false. What am I doing wrong? Thanks

  • Mrunmoy SamalMrunmoy Samal

    This is a great work done by you Brian.

    But, what I would like to comment here is, we really do not need a complex hardware interface. A simpler and cheaper solution would be to use a stacked buffer chip.

    if you'd like to know more on how to do it, I can share the circuit diagram and concept with you. I had done this as my final semester project during my bachelors degree.

    mail me at mrunmoyAThotmailDOTcom or mrunmoyATyahooDOTcom (please replace DOT with . and AT with @, I've put them just to avoid spams from webbots)

    otherwise, great work!

    Best Regards,

    Mrunmoy

  • mackdunbarmackdunbar

    is there a way to do this with perportioal steering and throttle control

  • Aakash KambadAakash Kambad

    Hi,

    Thanks for this details. this is realy a good thing

  • ChrisChris

    How do you tell if your remote control is digital or not?

  • Scott KScott K

    This is great!  I am working with 0/16/16 phidget board.  I have everything installed but I just can't seem to get the wiring right.  Is there an article or other project that would go into more detail on wiring, relays, circuits etc...  Also, do I need the battery on the remote control or is the USB sufficiant for the controller's power?  

    thanks again.

  • FAVIFAVI

    So How the CAmera is Linked With The PC IS it wifi CAmera Or Blutooth CAmera.,.??

    Plz Tel Me.,.,

  • Clint RutkasClint I'm a "developer"

    @FAVI It is a wifi camera.  http://www.airlink101.com/products/aic250w.html

  • Clint RutkasClint I'm a "developer"

    @srikant the source code is listed at the top of the page with a link called "Download"

    You can save this page as a webpage to your desktop as well.

  • srikantsrikant

    i request u to mail the whole coding & working of this step by step as early as possible .plz

    email id

    pnd.srikant@gmail.com

    thank u!!!!!!!!

  • rahulrahul

    can you please suggest the car model that can be used for the project.the car should cost anything between $10-15..thank u...

  • Clint RutkasClint I'm a "developer"

    @BeefMasterFlex this is a pretty old project, so I don't know if we still have the hardware laying around.  

    Imagine controlling a remote control car with the computer as a "man in the middle".  This lets you automate movements.  I'll see if Brian still has some of this but the video would basically be him using an xbox controller or a wiimote to drive it around.  It will look as if he is using the remote control.

  • BeefMasterFlexBeefMaster​Flex

    Would it be possible to get a video of this running on YouTube or something.  I'm aware this is relatively old, but I would like to see it running before I spend the amount of money required to make this.  Thanks.

  • NazirNazir

    Wonderful project, guy! Thanks! Its really helpful to understand some complex stuff.

  • ollieollie

    nice, im making a VTOL UAV for a bit of fun, and this is exactly what i needed. i have four high speed motors which i connected to four rc car controls, then i used your article and created a program which runs all four of them, two automatically and the other two on the left and right arrow keys (direction)

  • timtim

    i love it

Remove this comment

Remove this thread

close

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.