Silverlight Dynamic Video Puzzle

Sign in to queue


My blog title says it all, Monkey see, Monkey Build.  I saw Microsoft Surface's video puzzle and I needed to build it.  I took this opportunity to play with Microsoft's new technology, Silverlight and build my own puzzle game.  I couldn't find anything out on the Internet regarding dynamic video creation with Silverlight so I took it upon myself to do this.  The screen shots in this demo will be in c#, however, everything should hold true for VB.

Clint Rutkas - Academic Developer Evangelist - Microsoft
Monkey see, Monkey Build

Difficulty: Intermediate
Time Required: 6-10 hours
Cost: Free
Software: Visual Studio Express, Silverlight SDK and Runtime for 1.1 2.0 beta2
Download: Download c# - Download VB Updated to Silverlight 2.0 Beta - Download c#


  • Fixed the Silverlight.Js, used the new version that is included with the 1.1 Refresh SDK so the proper installer will prompt now.  Downloads are corrected also too.
  • 6/20/2008 - Verified the solution works with Silverlight 2.0beta2.

Spec's, Requirements and headaches

So I had a few mental requirements for the application to "entertain" myself. 

  1. Be able to rotate the video blocks
  2. Be able to translate
  3. Be able to increase the difficulty of the puzzle on the fly.
  4. Zero interaction with the keyboard

Simple, right?  You want to see a demo too see some awesomeness?  No problem, head over to my site and see it in person.

Microsoft Surface Video Puzzle


Clint's Silverlight Video Puzzle


To the Internet Batman!

To help save time building this, I first when out and attempting to see if anything has been done anything close to this before.  I found

Also I need 2 small controls which I found out are in the Silverlight SDK.  I needed horizontal slider controls for this app.  While I know a text box control would have worked just as well, see requirement 4.

On the Silverlight community site, there is also a very nice demo showing the photo application in Surface with source code.  This is where I learned the majority of what I needed.  This application was very close to where I needed to go with mine.  However, while it was close, it wasn't perfect.  Parts of the photo application code are pretty much copied.  This isn't the best way of doing this, but it is one way of doing it.

Toolbelt: check, Keyboard: check, Band-aids: check

So on to building it.  I first started off with a new project and some XAML.


After clicking "OK", I get a very nice blank project.


We'll want to add an additional XAML object for the video blocks.  Go to "Project -> Add New Item", select Silverlight Page, name it "VideoBlock.xaml" and click "OK".

So now we have everything we need for the most part, grab any video you have, if you don't have one, go to a site like TeamXbox and grab one from there.  I used Windows Movie Maker and grabbed a snippet from The Office (best show ever).

In addition to all these files, we need to add in a reference to the Silverlight SDK to get a hold of the Slider controls. Go to "Project -> Add Reference", and add in the Silverlight.Samples.Controls.dll.

While we have a reference in the application, we still need the XAML to know about this also.  So we'll add a line to link this in.  It is the xmlns line that I bolded, XML name space, real clever, eh?  We'll also  change the background color to a gray so we can see it more easy.

<Canvas x:Name="parentCanvas"
        Background="#CCCCCC" Width="640" Height="480">

Silverlight 101

So in Silverlight, you have brushes which can "painted" onto objects.  One of these brushes is called the VideoBrush which I'll talk about later on.  For playing videos, one would use the MediaElement.  Lets add this to our canvas.

<MediaElement x:Name="media" Source="theOffice.wmv" />

Now if we run the application at this point, we'll get the video playing.  Lets sit back with some popcorn and enjoy this victory for a second or two.  Now that we have this, lets hide it since this isn't what we want.  We do that by adding in an XML attribute Opacity and setting it to 0.

On top of that element, lets add on some TextBlock elements and the sliders.

<Canvas x:Name="parentCanvas"
        Background="#CCCCCC" >
    <MediaElement x:Name="media" Source="theOffice.wmv" Opacity="0" />
    <TextBlock x:Name="sliderX" Canvas.Top="5" Canvas.Left="5" Text="X (01):" />
    <TextBlock x:Name="sliderY"  Canvas.Top="21" Canvas.Left="5" Text="Y (01):" />
    <uicontrol:Slider x:Name="hSliderX" Canvas.Top="8" Canvas.Left="55" />
    <uicontrol:Slider x:Name="hSliderY" Canvas.Top="25" Canvas.Left="55" />

So now when we run the application, we get a blank screen but we can hear the audio from the video.


Super exciting I know.  Now if you resize the browser, you'll notice also the canvas isn't resizing automatically.  Lets fix that.

So in the code behind lets add in a new event on the BrowserHost object with the Resize event.  We'll attach the Browser_Resize event to it.


private void BrowserHost_Resize(object sender, EventArgs e)
    // Set size to host size
    Width = BrowserHost.ActualWidth;
    Height = BrowserHost.ActualHeight;


Private Sub BrowserHost_Resize(ByVal sender As Object, ByVal e As EventArgs)
    ' Set size to host size
    Width = BrowserHost.ActualWidth
    Height = BrowserHost.ActualHeight
End Sub

Video as Paint?

So now that we have that small part done, lets get into more of the nitty gritty with video.  As I said earlier, you'll use a VideoBrush to paint on the video.

We'll use the brush as the Fill for the rectangle we'll be using to show off the clip of the video we want.

Still not moving?  Try explosives?

While doing this project, I discovered an interesting thing when using the VideoBrush and dynamically appending items on.  Having items in the XAML rather than programmatically creating everything caused it not to work properly.  Nothing would appear.

Throwback to how UI's use to be done

The XAML for this is a simplified version of the Surface demo Photo.XAML.  You have a 3 Rectangles, 1 for the video, 1 for translation and 1 for rotation.  Simple, no?  Doing all this by hand gets a bit annoying since you need to nest everything properly.

Here is the more interesting VideoBrush code.  You need the name from the MediaElement along with the how much of an offset you want the video.  Since your moving the "camera" in, this value needs to be negative.



public delegate void SetActiveVideo(VideoBlock videoBlock, ActionType actionType, Point photoCenter, Point lastPosition);
public VideoBlock(string mediaName, int offsetX, int offsetY, int rectWidth, int rectHeight, SetActiveVideo functionPointer)
    // additional code here
    Rectangle video = new Rectangle();
    VideoBrush vb = new VideoBrush();
    TranslateTransform videoTransform = new TranslateTransform();
    videoTransform.X = -1 * offsetX;
    videoTransform.Y = -1 * offsetY;

    vb.SourceName = mediaName;
    vb.Transform = videoTransform;
    vb.Stretch = Stretch.None;
    vb.AlignmentX = AlignmentX.Left;
    vb.AlignmentY = AlignmentY.Top;

    video.Width = Width;
    video.Height = Height;
    video.Fill = vb;
    // additional code here


Public Delegate Sub SetActiveVideo(ByVal videoBlock As VideoBlock, ByVal actionType As ActionType, ByVal photoCenter As Point, ByVal lastPosition As Point)
Public Sub VideoBlock(ByVal mediaName As String, ByVal offsetX As Integer, ByVal offsetY As Integer, ByVal rectWidth As Integer, ByVal rectHeight As Integer, ByVal functionPointer As SetActiveVideo)
      ' additional code here
      Dim video As Rectangle = New Rectangle
      Dim vb As VideoBrush = New VideoBrush
      Dim videoTransform As TranslateTransform = New TranslateTransform
      videoTransform.X = ((1 * offsetX)  * -1)
      videoTransform.Y = ((1 * offsetY)  * -1)
      vb.SourceName = mediaName
      vb.Transform = videoTransform
      vb.Stretch = Stretch.None
      vb.AlignmentX = AlignmentX.Left
      vb.AlignmentY = AlignmentY.Top
      video.Width = Width
      video.Height = Height
      video.Fill = vb
      ' additional code here
End Sub

There is a mouse on my object!

So now we have our video block element prepped and ready.  There are additional events added in, but they are fairly boiler plate.  When looking at this code, you see I pass in a delegate, a function pointer if you will.  Why do I do this?  This is how the mouse position is calculated.  I bet this could be redone better, but this is how I did it.  The parent element, the canvas, has the mouse position I need to calculate proper translation and rotation, however, mouse events on the VideoBlock object will give me the mouse position only on that element.  So I may be at point 150, 150 on the screen, on the VideoBlock object, I'm actually at 10,15.

Additionally, this approach allows me to move the VideoBlock to the top of the objects.  Once again, this could be done other ways, this is how I chose to do it.

So on all my mouse related all boils down to doing the exact same function.  Since events are bubbled, a mouse movement on a VideoBlock is also a mouse movement on the root canvas.


private void root_MouseLeftButtonDown(object sender, MouseEventArgs e)
    HandleMouseLeftButtonDown(ActionType.Selecting, e);

private void translateControls_MouseLeftButtonDown(object sender, MouseEventArgs e)
    HandleMouseLeftButtonDown(ActionType.Moving, e);

private void rotateScaleControls_MouseLeftButtonDown(object sender, MouseEventArgs e)
    HandleMouseLeftButtonDown(ActionType.RotatingScaling, e);

private void HandleMouseLeftButtonDown(ActionType actionType, MouseEventArgs e)
    if (functionPointer != null)
        functionPointer(this, actionType,
            new Point(translateTransform.X + rotateTransform.CenterX,
                  translateTransform.Y + rotateTransform.CenterY), e.GetPosition(null));


    Private Sub root_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseEventArgs)
        HandleMouseLeftButtonDown(ActionType.Selecting, e)
    End Sub
    Private Sub translateControls_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseEventArgs)
        HandleMouseLeftButtonDown(ActionType.Moving, e)
    End Sub
    Private Sub rotateScaleControls_MouseLeftButtonDown(ByVal sender As Object, ByVal e As MouseEventArgs)
        HandleMouseLeftButtonDown(ActionType.RotatingScaling, e)
    End Sub
    Private Sub HandleMouseLeftButtonDown(ByVal actionType As ActionType, ByVal e As MouseEventArgs)
        If (Not (functionPointer) Is Nothing) Then
            functionPointer(Me, actionType, New Point((translateTransform.X + rotateTransform.CenterX), 
              (translateTransform.Y + rotateTransform.CenterY)), e.GetPosition(Nothing)) End If End Sub

You built a car out of all these parts?

So we move back from VideoBlock.xaml.cs to Page.xaml.cs

We've created our primary object for painting video, lets populate it.  First, how do we know how big the media file is?  With the MediaOpened event!  And we'll restart the video with the MediaEnded event.

So here is the full Page_Load event from Page.xaml.cs


public void Page_Loaded(object o, EventArgs e)
    // Required to initialize variables

    BrowserHost.Resize +=new EventHandler(BrowserHost_Resize);
    media.MediaOpened += new EventHandler(media_MediaOpened);
    media.MediaEnded += new EventHandler(media_MediaEnded);
    MouseLeftButtonUp += Page_MouseLeftButtonUpOrLeave;
    MouseLeave += Page_MouseLeftButtonUpOrLeave;
    MouseMove += Page_MouseMove;

    hSliderX.Range = new ValueRange(1, 20);
    hSliderY.Range = new ValueRange(1, 20);
    hSliderX.ValueChanged += new EventHandler(slider_ValueChanged);
    hSliderY.ValueChanged += new EventHandler(slider_ValueChanged);

    hSliderX.SetValue(ZIndexProperty, 500);
    hSliderY.SetValue(ZIndexProperty, 500);
    sliderX.SetValue(ZIndexProperty, 500);
    sliderY.SetValue(ZIndexProperty, 500);


    Public Sub Page_Loaded(ByVal o As Object, ByVal e As EventArgs)
        ' Required to initialize variables
        AddHandler BrowserHost.Resize, AddressOf Me.BrowserHost_Resize
        AddHandler media.MediaOpened, AddressOf Me.media_MediaOpened
        AddHandler media.MediaEnded, AddressOf Me.media_MediaEnded
        MouseLeftButtonUp = (MouseLeftButtonUp + Page_MouseLeftButtonUpOrLeave)
        MouseLeave = (MouseLeave + Page_MouseLeftButtonUpOrLeave)
        MouseMove = (MouseMove + Page_MouseMove)
        hSliderX.Range = New ValueRange(1, 20)
        hSliderY.Range = New ValueRange(1, 20)
        AddHandler hSliderX.ValueChanged, AddressOf Me.slider_ValueChanged
        AddHandler hSliderY.ValueChanged, AddressOf Me.slider_ValueChanged
        hSliderX.SetValue(ZIndexProperty, 500)
        hSliderY.SetValue(ZIndexProperty, 500)
        sliderX.SetValue(ZIndexProperty, 500)
        sliderY.SetValue(ZIndexProperty, 500)
    End Sub

So now that we have the media loaded, we can use the media.NaturalVideoWidth and media.NaturalVideoHeight properties to find out exactly how big the video is!  I have a function that is fired off when the page is first loaded or a slider value has changed.  This is how we chop it up.



private void prepVideoObjects()
    int totalPhotosX = (int)hSliderX.Value;
    int totalPhotosY = (int)hSliderY.Value;

    if (lastX != totalPhotosX || lastY != totalPhotosY)
        lastX = totalPhotosX;
        lastY = totalPhotosY;

        sliderX.Text = string.Format("X ({0}):", totalPhotosX.ToString("D2"));
        sliderY.Text = string.Format("Y ({0}):", totalPhotosY.ToString("D2"));

        int rectWidth = (int)Math.Floor(media.NaturalVideoWidth / totalPhotosX);
        int rectHeight = (int)Math.Floor(media.NaturalVideoHeight / totalPhotosY);
        Random random = new Random();

        // remove all old videoblocks
        for (int i = Children.Count - 1; i > 0; i--)
            if (Children[i].GetType() == typeof(VideoBlock))

        for (int x = 0; x < totalPhotosX; x++)
            for (int y = 0; y < totalPhotosY; y++)
                VideoBlock vb = new VideoBlock(media.Name, x * rectWidth, y * rectHeight, rectWidth, rectHeight, SetActivePhoto);
                shuffle(vb, random);


Private Sub prepVideoObjects()
        Dim totalPhotosX As Integer = CType(hSliderX.Value,Integer)
        Dim totalPhotosY As Integer = CType(hSliderY.Value,Integer)
        If ((lastX <> totalPhotosX)  _
                    OrElse (lastY <> totalPhotosY)) Then
            lastX = totalPhotosX
            lastY = totalPhotosY
            sliderX.Text = String.Format("X ({0}):", totalPhotosX.ToString("D2"))
            sliderY.Text = String.Format("Y ({0}):", totalPhotosY.ToString("D2"))
            Dim rectWidth As Integer = CType(Math.Floor((media.NaturalVideoWidth / totalPhotosX)),Integer)
            Dim rectHeight As Integer = CType(Math.Floor((media.NaturalVideoHeight / totalPhotosY)),Integer)
            Dim random As Random = New Random
            ' remove all old videoblocks
            Dim i As Integer = (Children.Count - 1)
            Do While (i > 0)
                If (Children(i).GetType = GetType(VideoBlock)) Then
                End If
                i = (i - 1)
            Dim x As Integer = 0
            Do While (x < totalPhotosX)
                Dim y As Integer = 0
                Do While (y < totalPhotosY)
                    Dim vb As VideoBlock = New VideoBlock(media.Name, (x * rectWidth), (y * rectHeight), rectWidth, rectHeight, SetActivePhoto)
                    shuffle(vb, random)
                    y = (y + 1)
                x = (x + 1)
        End If
    End Sub

Last but not least, lets see how the mouse movements are handled on the page.


private void Page_MouseMove(object sender, MouseEventArgs e)
    if (null != activeVideoBlock)
        // Perform the appropriate transform on the active photo
        Point position = e.GetPosition(null);
        switch (currentActionType)
            case VideoBlock.ActionType.Moving:
                // Move it by the amount of the mouse move
                activeVideoBlock.Translate(position.X - currentLastPosition.X, position.Y - currentLastPosition.Y);
            case VideoBlock.ActionType.RotatingScaling:
                // Rotate it according to the angle the mouse moved around the photo's center
                double radiansToDegrees = 360 / (2 * Math.PI);
                double lastAngle = Math.Atan2(currentLastPosition.Y - currentVideoCenter.Y, currentLastPosition.X - currentVideoCenter.X) * radiansToDegrees;
                double currentAngle = Math.Atan2(position.Y - currentVideoCenter.Y, position.X - currentVideoCenter.X) * radiansToDegrees;
                activeVideoBlock.Rotate(currentAngle - lastAngle);

        currentLastPosition = position;


    Private Sub Page_MouseMove(ByVal sender As Object, ByVal e As MouseEventArgs)
        If (Not (activeVideoBlock) Is Nothing) Then
            ' Perform the appropriate transform on the active photo
            Dim position As Point = e.GetPosition(Nothing)
            Select Case (currentActionType)
                Case VideoBlock.ActionType.Moving
                    ' Move it by the amount of the mouse move
                    activeVideoBlock.Translate((position.X - currentLastPosition.X), (position.Y - currentLastPosition.Y))
                Case VideoBlock.ActionType.RotatingScaling
                    ' Rotate it according to the angle the mouse moved around the photo's center
                    Dim radiansToDegrees As Double = (360 / (2 * Math.PI))
                    Dim lastAngle As Double = (Math.Atan2((currentLastPosition.Y - currentVideoCenter.Y), (currentLastPosition.X - currentVideoCenter.X)) * radiansToDegrees)
                    Dim currentAngle As Double = (Math.Atan2((position.Y - currentVideoCenter.Y), (position.X - currentVideoCenter.X)) * radiansToDegrees)
                    activeVideoBlock.Rotate((currentAngle - lastAngle))
            End Select
            currentLastPosition = position
        End If
    End Sub

Wrapping it up

So as you can see, Silverlight is pretty powerful once you figure out how to tame it.  I used example base code and modified it to suit my own needs to dynamically alter a video in real time.  There are a few code tweaks that could happen to improve this code and I'm betting with some additional effort, one could remove the delegate.  In addition, you could add in edge detection code and a timer to make this into a true video puzzle game.  This was a pet project I worked on while I was at the airport.  This demo can be downgraded to Silverlight 1.0 also.  I don't believe anything I did was really Silverlight 1.1 (now 2.0) only.

Clint's Bio:

Clint is an academic developer evangelist for Microsoft.  His two primary development languages are C# and JavaScript. He has built a Disco Dance Floor too! In his off time, he whips up other random weird projects and does twenty something activities with his friends.  His next two big projects are an automated bartender and a skateboard segway.  Clint's blog is and can be emailed at if you have any question.

The Discussion

Add Your 2 Cents