If all these things have already been mentioned before why revisit them? Because today's post is an hour long session that gently introduces you to the world of augmented reality, doing AR on Windows Phone 7.1(5), and two sample code projects too.

Recording of “Programming Augmented Reality for Windows Phone”

As promised here’s the recording of the webcast session I did for AT&T on Augmented Reality with Windows Phone.

It was great going through this stuff and hopefully inspiring more people to think beyond with their apps and bring in the real world to augment users experiences!

Items that were covered in the session were:

• Recap on what it means to have Augmented Reality
• Review of the different types of AR solutions
• A High level description of what sensors and inputs you can use to augment your solution
• A Run down of the top 3 AR toolkits for Windows Phone in both Silverlight and XNA
• A code walkthrough of a Silverlight solution using SLARToolkit
• A Code walkthrough of a 3D SilverXNA project also using the SLARToolkit
• Q&A

from Vimeo.

The source for the two mentioned projects are available here, Programming Augmented Reality for WP Source. If you grab these sample projects and run into an error where the "SLARToolKitWinPhone" project cannot be found, there's a couple fixes.

What I did was just grab the latest source from the SLAR Project (http://slartoolkit.codeplex.com/SourceControl/list/changesets), unziped it and replaced that project with the one in \trunk\SLARToolKit\Source\SLARToolKit, SLARToolKitWinPhone.csproj.

Or you can download the SLAR binaries (http://slartoolkit.codeplex.com/releases/) and use them.

The important thing is that Simon spends an hour with us, talking about AR, challenges and just the fun things you can do with it on Windows Phone.

Here’s a few more links you might find interesting:

• SLARToolkit
• How about a little physics in your augmented reality?
• Augmented Reality Comes to Silverlight
• (YASL53DP) Supercharging Silverlight 5 3D projects with the Babylon Toolkit
• Augmented Reality, 3D, Web cam, Silverlight 5 beta and you.

Kintext: be what's next [12:43]

Illusionist or Augmented Reality?  [02:22]

Farewell XP [08:13]

Get your Lumia!  [16:10]

Key questions: What makes HTML5 the "right" programming model for driving personal computing forward into the future? What's the relationship between augmented reality, ambient computing, and HTML5? Allen and Rob answer these questions.

What do you think?

Tune in. Enjoy.

Allen's YOW! speaker page

Rob's YOW! speaker page

The YOW! Developer Conference offers outstanding opportunities to learn more about the latest practices, technologies, and methodologies for building innovative software solutions as well as the chance to meet and network with international software experts and other talented developers in Australia. Thanks to Dave Thomas and the event's excellent staff—Mary Catherine (MC), Lisa, Aino, Melissa, and others—for inviting me to this excellent pure developer event and thanks to all of the speakers for letting me take some of their time to record conversations for Channel 9. If you live in Australia, or aren't too far away, or just like to travel (who doesn't?), then you need to go to this yearly event. It's outstanding. There are many great developers down under. That's for sure. The speakers are exceptional—Dave and team set a high bar!

Inter touch media creates multi-touch immersive media experiences. They create widgets in WPF for advertisement and other media scenarios. Intertouch Media has recently moved into augmented reality using Silverlight and Microsoft platform.

 Learn more @ http://www.intertouchmedia.com/

Introduction

This is an advanced tutorial, and as such we expect that you are already familiar with XNA and scene graph based 3D programming. The game is a single player game in which the player shoots virtual balls into a real-world environment to knock down all of the virtual dominos overlaid on top of a board. The player holds a webcam in one hand and shoots the virtual ball by clicking a mouse using the other hand. Before the game starts, the player is allowed to add virtual dominos or modify the positions and orientations of the existing ones overlaid on the board as desired.

How To Compile and Run The Game

After you download the Goblin XNA distribution from http://goblinxna.codeplex.com, as well as all other necessary libraries indicated in the installation guide, compile the Goblin XNA project under the /src directory. If you're not familiar with Goblin XNA framework, it is strongly recommended that you first go through the tutorials in the /tutorials directory. Then, find the ARDominos directory under /projects directory and open up the solution file in order to build and run the game.

How To Play

Before you run the application, you need to print out the marker array. The marker array can be found in the DominoGroud.ppt file. The 1^st slide is for the ARTag library, and the 2^nd one is for the ALVAR library. The size of the PowerPoint slide is set to 25.6 x 17.52 inches, so if you want the entire marker array to fit in A4 size paper, you should choose "Scale to fit" option when you print. Once you have the print out, you're ready to run the game.

The virtual dominos are overlaid around the center of the marker array (referred to as ‘game board' from now on), so you need to aim your webcam around the center of the game board. If you move your mouse cursor around on top of the game window, you will see a yellow-green cross-hair cursor, and this cursor position will be used to control the game play. All of the mouse-based interactions are handled by the left mouse button.

The initial game mode is ‘Add' (as indicated on the upper-right corner of the game window), so if you click on any part on the game board, a new domino will be added. If you press-and-hold, then you can move the newly added domino around, and once you release the mouse the domino will be added at the released position. While the domino is rendered in transparent, you are allowed to modify its orientation along the normal vector of the game board by using the ‘Left' and ‘Right' arrow keys. You are not, however, allowed to add a domino outside of the gameboard, and you are initially limited to maximum of 40 dominos on the game board (you can change this limit in the code if you want).

If you want to edit the current position or orientation of an existing domino on the game board, change the game mode to ‘Edit' by clicking the ‘E' key. To modify the position, press-and-hold the mouse on the desired domino, and drag it around (again, it won't allow you to place the domino outside of the game board). The selected domino is rendered transparent, and it remains selected until you select another domino or change to a different game mode. If a domino is selected, you can change its orientation by using the ‘Left' and ‘Right' arrow keys. If you want to delete a domino, select the domino you want to delete and press the ‘D' or ‘Delete' key.

Once you are satisfied with the current domino configuration, start the game by pressing the ‘P' key. If you want to reset the configuration to the original one, then press the ‘R' key. Once the game starts, the elapsed time will display in the upper-right corner and the number of balls shot will display in the lower-right corner. The goal of the game is to knock all of the virtual dominos off the game board by shooting as many virtual balls as necessary, and you will get different trophies based on the elapsed time.

The virtual ball is shot from the cursor location when you click the mouse. You can move the webcam around to move closer to certain dominos and improve your chances of hitting your target. However, if the game board becomes invisible to the camera, the game doesn't allow you to shoot any balls.

Once you win, you need to press the ‘P' key to restart the game.

List of command keys used in the game:

· 'H' -- Display short-cut key help menu

· 'A' -- Switch to "Add" mode

· 'E' -- Switch to "Edit" mode

· 'P' -- Switch to "Play" mode. If pressed during "Play" mode, it restarts the game.

· 'R' -- Reset the game to the initial state

· 'S' -- Toggles the shadow mapping (NOTE: On non-NVidia graphics cards, the shadow mapping may have some aliasing problem)

· 'G' -- Toggles the GUI

· 'D' -- Delete selected dominos during "Edit" mode

· 'C' -- Toggle center cursor mode during "Play" mode

Step 1. Setup Scene Basics

First, you need to setup the basics for the scene, such as lighting and ground plane where the virtual dominos will be placed. Usually, you also need to setup a camera for the scene, but since you will setup the marker tracking system later on, the camera will be created and added to the scene automatically.

NOTE: Before going through this part, we strongly recommend that you go through Tutorial 1 and Tutorial 5 that comes with the Goblin XNA distribution.

Add Lighting Sources

We will add two directional lights to the scene, each coming from different directions. We also set the ambient light color to a dark gray color so that none of the objects in the scene will appear totally black.

C#

// Create a directional light source
LightSource lightSource = new LightSource();
lightSource.Direction = new Vector3(-1, -1, -1);
lightSource.Diffuse = Color.White.ToVector4();
lightSource.Specular = new Vector4(0.6f, 0.6f, 0.6f, 1);

LightSource lightSource2 = new LightSource();
lightSource2.Direction = new Vector3(1, -1, -1);
lightSource2.Diffuse = Color.White.ToVector4();
lightSource2.Specular = new Vector4(0.6f, 0.6f, 0.6f, 1);

// Create a light node to hold the light source
LightNode lightNode = new LightNode();
lightNode.AmbientLightColor = new Vector4(0.3f, 0.3f, 0.3f, 1);
lightNode.LightSources.Add(lightSource);
lightNode.LightSources.Add(lightSource2);

// Add this light node to the root node
scene.RootNode.AddChild(lightNode);

Create a Ground Plane

Since the game will have physics simulation, we will need a ground plane where the dominos will be placed. We will create a GeometryNode to hold the cubic geometry that represents a flat ground plane, and assign physics properties to this ground. Since we want to show the game board rather than this virtual ground plane, we set IsOcculder property to true. If this property is set to true, this geometry won't be rendered in the frame buffer; rather, it will be used for z-buffer testing. This means that you won't see this ground plane, but if any virtual objects go behind this ground plane, those virtual objects won't be rendered. Since this ground plane won't be visible, we won't assign any materials to it. Finally, we add this ground to the marker node that tracks the game board.

C#

GeometryNode groundNode = new GeometryNode("Ground");
groundNode.Model = new Box(129.5f, 99, 0.2f);

groundNode.Physics.Collidable = true;
groundNode.Physics.Shape = ShapeType.Box;
groundNode.Physics.MaterialName = "Ground";
groundNode.Physics.Pickable = true;
groundNode.AddToPhysicsEngine = true;

groundNode.IsOccluder = true;

groundNode.Model.ReceiveShadows = true;

markerNode.AddChild(groundNode);

Step 2. Setup Capture Device and Marker Tracker

We need to have a video capture device (e.g., a webcam) to capture the real-world environment, and an optical marker tracking system to track the marker array printed on the game board.

NOTE: Before going through this part, we strongly recommend that you go through Tutorial 8, which comes with the Goblin XNA distribution.

Setup Capture Device

Since we're using a webcam to capture the live video, we'll use the DirectShowCapture class. We then initialize this capture device with deviceID = 0, 30Hz framerate, and 640-by-480 resolution, and attach it to our scene. Since we want to show the live video stream in the background, we set ShowCameraImage to true.

If your computer has only one webcam attached (including the embedded cameras on laptop computers), then using deviceID = 0 will work fine. However, if you have more than one webcam, then you need to change this ID to indicate which webcam you want to use.

C#

DirectShowCapture captureDevice = new DirectShowCapture();
captureDevice.InitVideoCapture(0, FrameRate._30Hz, 
    Resolution._640x480, ImageFormat.R8G8B8_24, false);

scene.AddVideoCaptureDevice(captureDevice);

scene.ShowCameraImage = true;

Setup Optical Marker Tracking System

We will use the ALVAR optical marker tracking library from VTT (http://virtual.vtt.fi/virtual/proj2/multimedia/alvar.html). We initialize the tracker with the size of the image we'll be passing for the tracker to analyze, the camera calibration file generated by SampleCamCalib project that comes with ALVAR (you should perform calibration for each of your webcams in order to achieve optimal tracking performance), and the pixel size of the square marker (9.0 means that if you overlay a cube with size of 9x9x9 pixels, then the cube will overlay exactly on top of the marker square). Finally, we assign this tracking system to our scene.

C#

ALVARMarkerTracker tracker = new ALVARMarkerTracker();
tracker.MaxMarkerError = 0.02f;
tracker.InitTracker(captureDevice.Width, captureDevice.Height, "calib.xml", 9.0);
scene.MarkerTracker = tracker;

Create a MarkerNode To Track Gameboard

Since we want to overlay the dominos on top of a game board with marker arrays printed on it, we need to create a MarkerNode that holds the information of the marker configuration we want to track. Then, any geometry attached to this node will be overlaid on top the game board. We create a MarkerNode by denoting which marker tracker will be used to track this marker, the marker configuration file, and a list of marker IDs in this marker configuration. Finally, we add this node to our scene's root node.

C#

// Create a marker node to track the ground plane
int[] ids = new int[54];
for (int i = 0; i < ids.Length; i++)
    ids[i] = i;

markerNode = new MarkerNode(
scene.MarkerTracker, "ARDominoALVAR.txt", ids);

scene.RootNode.AddChild(markerNode);

Step 3. Create 3D Dominos and Balls

Now, we are ready to add 3D objects on top of the game board.

Create 3D Dominos

We'll add textured mapped cubic geometries that represent dominos, and lay them out in a circle. We also need to assign appropriate physics properties to each domino so that they will behave correctly in the physics simulation. Finally, we add all dominos to the marker node that tracks the game board.

C#

dominoModel = new DominoBox(
    new Vector3(dominoSize.X, dominoSize.Z, dominoSize.Y),
    new Vector2(0.663f, 0.707f));

dominoModel.CastShadows = true;
dominoModel.ReceiveShadows = true;

float radius = 18;
for (int x = 0; x < 360; x += 30)
{
    GeometryNode dominoNode = 
        new GeometryNode("Domino " + dominos.Count);
    dominoNode.Model = dominoModel;

    dominoNode.Physics.Mass = 20;
    dominoNode.Physics.Shape = ShapeType.Box;
    dominoNode.Physics.MaterialName = "Domino";
    dominoNode.Physics.Pickable = true;
    dominoNode.AddToPhysicsEngine = true;

    Material dominoMaterial = new Material();
    dominoMaterial.Diffuse = Color.White.ToVector4();
    dominoMaterial.Specular = Color.White.ToVector4();
    dominoMaterial.SpecularPower = 10;
    dominoMaterial.Texture = cguiLogoTexture;

    dominoNode.Material = dominoMaterial;

    TransformNode dominoTransNode = new TransformNode();
    dominoTransNode.Translation = new Vector3(
        (float)(radius * Math.Cos(MathHelper.ToRadians(x))),
        radius * (float)(Math.Sin(MathHelper.ToRadians(x))), 
        dominoSize.Y / 2);
    dominoTransNode.Rotation = 
        Quaternion.CreateFromAxisAngle(Vector3.UnitZ, 
            MathHelper.ToRadians(x + 90));

    markerNode.AddChild(dominoTransNode);

    dominoTransNode.AddChild(dominoNode);

    dominos.Add(dominoNode);
}

Create 3D Balls

We will create a list of 3D balls and add them to the scene even before the player shoots the ball in order to have best performance. Since we don't want to show these balls yet, we'll add them outside of the view frustum. When the player shoots a ball, we'll simply pick a ball from this list and give it a new position and velocity. Since the code is quite similar to creating 3D dominos, we will omit the codes here (see CreateBalls() function from line 1459)

Step 4. Setup Physics Simulation and Sound Effect

It is very easy to setup the physics simulation using Goblin XNA as well as use a sound effect when two objects collide into each other. The physics properties set to the GeometryNode in the previous steps are used by the physics engine. There are many properties associated with GeometryNode.Physics, but there are only few properties that you really have to define to make the simulation work.

C#

dominoNode.Physics.Mass = 20;
dominoNode.Physics.Shape = ShapeType.Box;
dominoNode.Physics.MaterialName = "Domino";
dominoNode.Physics.Pickable = true;
dominoNode.AddToPhysicsEngine = true;

Taking the domino geometry as an example, we need to define its mass, the shape primitive used in the physics simulation, the material name is necessary only if we're going to modify its physics material later on, which is the case for this game), and whether this geometry can be picked (we want to pick and manipulate the dominos in certain game modes, so we set this to true). Actually, we also need to set Collidable and Interactable properties to true for the geometry to collide and have physical response when other geometry hits this geometry. However, the game mode starts with ‘Add' mode, which will be explained later, and we don't want the geometry to collide and react on the collision when we move the selected domino around on the game board using mouse. So we leave them as false when we create them, and then set those two properties to true once the game starts.

NOTE: Before going through this part, we strongly recommend that you go through Tutorial 5 and Tutorial 9, which come with the Goblin XNA distribution.

Setup Physics Simulation

Since Goblin XNA supports Newton dynamics library, we will use NewtonPhysics implementation for our physics simulation. The default simulation space defined by Newton library is 200x200x200 centered at the origin, and if any objects go outside of this bound, those objects won't be simulated for optimization purposes. Since our scene is larger than this default space size, we'll set it to 500x500x500 centered at the center. The default gravity is 9.8, but this is quite slow for a realistic simulation, so we'll increase the gravity to 30. We will also increase the maximum simulation sub-steps to 5 so that even if the frame rate gets below 60 FPS, the physics engine will simulate as many as 5 sub-steps to simulate at least 60 FPS.

C#

scene.PhysicsEngine = new NewtonPhysics();
// Make the physics simulation space larger 
// to 500x500x500 centered at the origin
((NewtonPhysics)scene.PhysicsEngine).WorldSize =
    new BoundingBox(Vector3.One * -250, Vector3.One * 250);
// Increase the gravity
scene.PhysicsEngine.Gravity = 30.0f;

((NewtonPhysics)scene.PhysicsEngine).MaxSimulationSubSteps = 5;

Setup Physics Material and Sound Effect

We will setup some material properties for each physics entity that define how each entity will behave when they collide. Each entity is defined by a material name (for example, “Domino” for domino geometries), and in the example code below, this material definition is applied between “Ground” and “Ball” entities. For detailed explanation of each material properties (e.g., elasticity, friction, etc), please see the user manual that comes with the Goblin XNA distribution. When we set the material properties between two physics entities, we can also define a callback function, which will be invoked when the two entities collide. For our game, we want to add a sound effect to indicate the collision. For better performance, we limit the number of concurrent sound effects, and we only play a sound effect if the contact speed between the two entities is greater than certain speed (e.g., in the code example below, it will only play sound effect if the contact speed is above 3 pixels/sec). We also change the volume of the sound effect based on its contact speed.

C#

NewtonMaterial physMat2 = new NewtonMaterial();
// Gound to ball material interaction
physMat2.MaterialName1 = "Ground";
physMat2.MaterialName2 = "Ball";
physMat2.Elasticity = 0.7f;
physMat2.ContactProcessCallback = delegate(Vector3 contactPosition, 
    Vector3 contactNormal, float contactSpeed, 
    float colObj1ContactTangentSpeed, float colObj2ContactTangentSpeed,
    Vector3 colObj1ContactTangentDirection, 
    Vector3 colObj2ContactTangentDirection)
{
    if (contactSpeed > 3f)
    {
        if (soundsPlaying.Count >= SOUND_LIMIT)
            return;

        try
        {
            Sound.SetVolume("Default", contactSpeed * volume);
        }
        catch (Exception exp) { }
        try
        {
            Sound.Play("rubber_ball_01");
            soundsPlaying.Add(DateTime.Now.TimeOfDay.TotalMilliseconds);
        }
        catch (Exception exp) { }
    }

};
((NewtonPhysics)scene.PhysicsEngine).AddPhysicsMaterial(physMat2);

The above code only defines the material properties between “Ground” and “Ball” entities. In the actual game code, however, it defines the material properties for each pair of existing physics entities.

Step 5. Add User Interaction System

Now, we have all of the components necessary to start playing the AR domino knock-down game except for a way of interacting with the game for the player. Ideally, we want to play the game using a touch-sensitive mobile device with an embedded camera, such as an iPhone or an Ultra-Mobile-PC. Due to platform restriction and the high cost of those devices, we will use a webcam with mouse and keyboard based interaction for this demo. But first, we will add several callback functions for mouse and keyboard events.

C#

MouseInput.Instance.MousePressEvent += 
    new HandleMousePress(MousePressHandler);
MouseInput.Instance.MouseDragEvent += new HandleMouseDrag(MouseDragHandler);
MouseInput.Instance.MouseReleaseEvent += 
    new HandleMouseRelease(MouseReleaseHandler);
MouseInput.Instance.MouseMoveEvent += new HandleMouseMove(MouseMoveHandler);

KeyboardInput.Instance.KeyPressEvent += new HandleKeyPress(KeyPressHandler);
KeyboardInput.Instance.KeyReleaseEvent += 
    new HandleKeyRelease(KeyReleaseHandler); 

For this game, we have implemented three different modes, “Add”, “Edit”, and “Play”.

Add Mode

In “Add” mode, we want the players to be able to add dominoes anywhere on the game board by clicking the mouse and dragging the newly added domino around to finalize its position. We also want the players to be able to rotate the added domino along the normal vector of the game board by pressing “Left” and “Right” arrow keys.

NOTE: The following codes are in the MousePressHandler function.

C#

Vector3 nearSource = 
    new Vector3(mouseLocation.X, mouseLocation.Y, 0);
Vector3 farSource = 
    new Vector3(mouseLocation.X, mouseLocation.Y, 1);

Matrix viewMatrix = 
    markerNode.WorldTransformation * State.ViewMatrix;

Vector3 nearPoint = 
    graphics.GraphicsDevice.Viewport.Unproject(
        nearSource, State.ProjectionMatrix, 
        viewMatrix, Matrix.Identity);
Vector3 farPoint = 
    graphics.GraphicsDevice.Viewport.Unproject(farSource, 
        State.ProjectionMatrix, viewMatrix, Matrix.Identity);
// Cast a ray to the scene to pick a point on the ground where 
// the domino will be added
List<PickedObject> pickedObjects = 
    ((NewtonPhysics)scene.PhysicsEngine).PickRayCast(nearPoint, farPoint);

for (int i = 0; i < pickedObjects.Count; i++)
{
    if (((GeometryNode)pickedObjects[i].PickedPhysicsObject.Container).
        Name.Equals("Ground"))
    {
        Vector3 intersectPoint = nearPoint * (1 – 
            pickedObjects[i].IntersectParam) +
            pickedObjects[i].IntersectParam * farPoint;

        dominoTransNode.Translation = intersectPoint + Vector3.UnitZ * 
            dominoSize.Y / 2;
        break;
    }
}

Before we add a domino, we need to figure out the 3D point on the gameboard that corresponds to the 2D mouse click point on the screen. To do this, we need to create a 3D ray originating from the 3D point on the near clipping plane to the 3D point on the far clipping plane, both of which correspond to the clicked 2D point on the screen. XNA's Viewport class supports the Unproject function that performs this 2D to 3D mapping, which requires the 3D plane to map to (in this case, it's the nearSource and farSource), the projection matrix, the view matrix, and the world matrix. The tricky part here is the concept of the computation of the view matrix. In Goblin XNA, to make things simple, we decided not to have the marker transformation affect the physics simulation by default, so even if the marker transformation changes, nothing changes in the physics simulation world. However, if we want to figure out the actual view matrix in the physics simulation world, we need to multiply the marker transformation with the current view matrix, which is the State.ViewMatrix.

Once we get the 3D ray, we simply pass it to Newton library to perform the picking and then it will return a list of intersecting objects. We then check to see if there is an intersecting geometry named “Ground,” which is the gameboard. If the game board is intersected, then we calculate the 3D intersection point on the game board and put the newly added domino in that intersection point. In the MouseDragHandler function, you will see similar areas of code that let the player drag the newly added domino around on the game board.

There are also codes that let the player rotate the new domino along the normal vector of the game board in KeyPressHandler and Update functions, but we will omit the explanation here since the codes are self-explanatory.

Edit Mode

In “Edit” mode, we want the players to be able to pick an existing domino on the gameboard and drag it around. We also want the players to be able to rotate the selected domino along the normal vector of the game board by pressing “Left” or “Right” arrow keys.

NOTE: The following codes are in the MousePressHandler function.

C#

// Cast a ray to the scene to select the closest 
// object (to the eye location) that hits the ray
List<PickedObject> pickedObjects =   
    ((NewtonPhysics)scene.PhysicsEngine).PickRayCast(nearPoint, farPoint);

// If one or more objects intersect with our ray vector
if (pickedObjects.Count > 0)
{
    // Sort the picked objects in ascending order of intersection parameter
    // (the closest to the eye is the first one in the sorted list)
    pickedObjects.Sort();

    // If the closest object is the ground, then don't do anything
    if (((GeometryNode)pickedObjects[0].PickedPhysicsObject.Container).
        Name.Equals("Ground"))
        return;

    selectedDominos.Add(
        (GeometryNode)pickedObjects[0].PickedPhysicsObject.Container);
    
    GeometryNode domino = selectedDominos[0];
    Vector4 mod = domino.Material.Diffuse;
    domino.Material.Diffuse = 
        new Vector4(mod.X, mod.Y, mod.Z, 0.5f);
}

The picking codes are the same as before, but now we want to pick an existing domino instead of the game board. Since there are multiple dominoes on the game board, it is possible that the picking ray intersects with more than one domino (and most likely the game board as well). Thus, we need to sort the intersected geometries based on their distances from the camera, and then we want to select the one closest to the camera. If the closest geometry is not the game board, then we add this domino to the list of selected dominoes and set its transparency to half-transparent to indicate the selection. Then, in the MouseDragHandler function, we will see similar codes with “Add” mode that let the player drag the selected domino around.

Play Mode

In “Play” mode, we want the players to be able to shoot balls from the clicked location. We allow the player to shoot either heavy or normal balls depending on whether the click was on the right or left mouse button.

NOTE: The following code snippets are in the MouseReleaseHandler function since we allow the player to inject additional speed when pushing the camera forward while press-and-holding the mouse button, and then releasing the button (this interaction makes more sense when the game is played on an UMPC).

C#

bool heavy = (button == MouseInput.RightButton);

float additionalSpeed = 0;

// Add extra speed to the ball being shot if the camera point is moved from 
// far point to closer point (to the ground)
float ballReleaseDistance = 
    markerNode.WorldTransformation.Translation.Length();

if (ballReleaseDistance < ballPressDistance)
    additionalSpeed = ballPressDistance - ballReleaseDistance;

Vector3 nearSource = 
    new Vector3(uiManager.CrossHairPoint.X, 
        uiManager.CrossHairPoint.Y, 0);
Vector3 farSource = 
    new Vector3(uiManager.CrossHairPoint.X, 
        uiManager.CrossHairPoint.Y, 1);

Matrix viewMatrix = markerNode.WorldTransformation * State.ViewMatrix;

Vector3 nearPoint = 
    graphics.GraphicsDevice.Viewport.Unproject(nearSource,
        State.ProjectionMatrix, viewMatrix, Matrix.Identity);
Vector3 farPoint = 
    graphics.GraphicsDevice.Viewport.Unproject(farSource,
        State.ProjectionMatrix, viewMatrix, Matrix.Identity);

ballCount++;
uiManager.BallCount++;
// Create a ball and shoot it
CreateBallCharactor(nearPoint, farPoint, additionalSpeed, heavy);

C#

Vector3 linVel = far - near;
linVel.Normalize();

GeometryNode ballNode = (heavy) ? heavyBalls[curHeavyBall] : balls[curBall];
ballNode.Model.CastShadows = true;
ballNode.Model.ReceiveShadows = true;
Vector4 orig = ballNode.Material.Diffuse;
ballNode.Material.Diffuse = 
    new Vector4(orig.X, orig.Y, orig.Z, 1);

Vector3 v = near + linVel * 10;

// Forces the physics engine to 'transport' 
// this ball to the location 'v' in the simulation world
((NewtonPhysics)scene.PhysicsEngine).SetTransform(
    ballNode.Physics, Matrix.CreateTranslation(v));

// Apply a linear velocity to this ball
((NewtonPhysics)scene.PhysicsEngine).ApplyLinearVelocity(
    ballNode.Physics, linVel * (50f + additionalSpeed));

if (heavy)
{
    curHeavyBall++;
    // Circles through if we run out of heavy balls
    if (curHeavyBall >= BALL_NUM / 2)
        curHeavyBall = 0;
}
else
{
    curBall++;
    // Circles through if we run out of normal balls
    if (curBall >= BALL_NUM / 2)
        curBall = 0;
}

Step 6. Add Heads-Up-Display (HUD)

Finally, we add a HUD for displaying information such as elapsed game time, number of balls used, and a victory message when the game is over. We also add a cross hair to indicate the position of the mouse on the screen.

NOTE: The above codes are in the Draw function of UIManager class.

C#

// Count the time elapsed in the play mode if game is not over yet
if (gameState.CurrentGameMode == GameState.GameMode.Play &&  
    !gameState.GameOver)
{
    gameState.ElapsedSecond += gameTime.ElapsedRealTime.TotalSeconds;
    if (gameState.ElapsedSecond >= 60)
    {
        gameState.ElapsedMinute++;
        gameState.ElapsedSecond = 0;
    }
    // Update the time elapsed label with the new elapsed time
    modeStatusLabel = "Play: " + (int)gameState.ElapsedMinute + ":" +  
        String.Format("{0:D2}", (int)gameState.ElapsedSecond) + "  ";
}

// Draws the cross hair mark to indicate the mouse position which is used for
// ball shooting, domino addition, domino edition, and GUI interaction (if 
// GUI is shown)
UI2DRenderer.FillRectangle(new Rectangle(crossHairPoint.X - 10, 
    crossHairPoint.Y, 23, 3), null, new Color(50, 205, 50, 150));
UI2DRenderer.FillRectangle(new Rectangle(crossHairPoint.X, crossHairPoint.Y -  
    10, 3, 23), null, new Color(50, 205, 50, 150));

UI2DRenderer.WriteText(Vector2.Zero, modeStatusLabel, 
    Color.White, labelFont,
    GoblinEnums.HorizontalAlignment.Right, 
    GoblinEnums.VerticalAlignment.Top);

// Displays how many balls are used so far
if (gameState.CurrentGameMode == GameState.GameMode.Play)
{
    UI2DRenderer.WriteText(Vector2.Zero, 
        "Balls Used: " + ballCount, Color.Red, 
        labelFont, GoblinEnums.HorizontalAlignment.Right,       
        GoblinEnums.VerticalAlignment.Bottom);
}

// Shows the victory texts as well as play the victory sound if won
if (gameState.GameOver)
{
    UI2DRenderer.WriteText(new Vector2(0, 130), "Victory!! You Won!!", 
        Color.Red, victoryFont, GoblinEnums.HorizontalAlignment.Center, 
        GoblinEnums.VerticalAlignment.None);

    UI2DRenderer.WriteText(new Vector2(0, 180), 
        "Time: " + (int)gameState.ElapsedMinute + ":" + 
        String.Format("{0:D2}", (int)gameState.ElapsedSecond), 
        Color.Red,victoryFont, 
        GoblinEnums.HorizontalAlignment.Center, 
        GoblinEnums.VerticalAlignment.None);

    UI2DRenderer.WriteText(new Vector2(0, 220), 
        "Balls Used: " + ballCount, Color.Red, 
        victoryFont, GoblinEnums.HorizontalAlignment.Center, 
        GoblinEnums.VerticalAlignment.None);

    Texture2D trophy = null;
    if (gameState.ElapsedMinute < 1 && 
        gameState.ElapsedSecond <= 14)
        trophy = trophyGold;
    else if (gameState.ElapsedMinute < 1 && 
        gameState.ElapsedSecond <= 25)
        trophy = trophySilver;
    else
        trophy = trophyBronze;

    UI2DRenderer.FillRectangle(new Rectangle(
        (State.Width - trophy.Width) / 2, 280, 
        trophy.Width, trophy.Height), 
        trophy, Color.White);
}

Conclusion

This article demonstrated how to implement a simple yet interesting interactive AR game. Since this article is rather advanced, and most of the codes are specific to Goblin XNA framework, you will have a better understanding of the source codes if you go through some of the suggested tutorials first. If you have an UMPC (Ultra-Mobile-PC) with you, then you can also run the same code, and you can shoot the balls by tapping on the screen rather than using a mouse.

If you want to try this out, download the source code via the link at the top of the article!

About The Author

Ohan Oda is a Ph.D student at Columbia University studying Computer Science. He specializes in designing systems and user interaction techniques for entertainment in Augmented Reality. He is the lead developer of Goblin XNA framework, and has worked on several Augmented Reality projects using this framework. Please contact ohan@cs.columbia.edu for any questions.

While the above feature list may mean something to application developers, what it means for consumers is that we’ll soon start seeing augmented reality applications developed using the Silverlight platform. These apps will be able to take advantage of your computer’s webcam to launch “augmented reality” applications – that is, those that use technologies like computer vision and object recognition, for example.

There are already two sample apps available on Codeplex, one being a 3D game called Balder and the other a sample that uses Silverlight's 3D projection capabilities. To run these samples, you’ll need a webcam and at least the Silverlight 4 runtime. For more information on this toolkit, check out the documentation.

(via ReadWriteWeb and Channel 9)

I’m seeing this company everywhere lately, it seems. And they’re doing all kinds of cool stuff. Before happening upon the Photosynth video, I had also favorited a video of theirs on Vimeo. That one, called “Playing in an Augmented World,” was a demo of some “YDreamers” interacting with virtual elements in real physical setting. As it turns out, the apps they used in that video were developed in C# 3.0 / .NET 3.5. 

This is definitely a company to watch, I’m totally subscribed to their channel now.