FaceLight – Silverlight 4 Real-Time Face Detection

The webcam and microphone API was the first thing I played with after the Silverlight 4 beta was released at last year's PDC. In my opinion, it's one of the coolest features. You can do a lot of fun Silverlight applications with it too.

When the SLARToolkit was released, I finally had time to implement real time face detection using the Silverlight 4 webcam API.

This article will describe the simple facial recognition method that searches for a certain sized skin color region in a webcam snapshot. This technique is not as perfect as a professional computer vision library like OpenCV and the Haar-like features they use, but it runs in real time and works for most webcam scenarios.

Demo Application

You need a webcam and at least the Silverlight 4 runtime installed to run the sample. At the moment the release candidate is available for Windows and Mac. The facial region should be illuminated well and the background should be in skin color contrast to get the best results.

Open the sample

How To Use

You can start and stop the webcam with the Button, or you can load an image from disk with the Button. Use the ComboBox to change the demo mode from “Highlight” to “Image.” “Highlight” simply draws a red ellipse around the detected face and “Image” overlays the facial region with an image. The ape's head is the default image (Figure 11), but it's possible to apply a different picture by entering its URI in the TextBox. You can use the Slider controls to alter the skin color thresholds in the YCbCr color space (see Step 2: Filtering the Skin Color). Save the result of the face detection (including the overlaid image) to disk with the Button.

When you click the Button for the first time, you'll need to give permission for the capturing. This application uses the default Silverlight capture devices. You can specify default video and audio devices with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.

How It Works

The idea is to take snapshots from the webcam, filter the skin color out using color thresholds, apply filters to reduce the noise, find the facial region—and then do fun things with this information. In the following sections I'll show you how to do this using a simple skin color segmentation approach. For clarity, I've reduced code listings to only what's relevant here. The complete source code is available at CodePlex and licensed under the Ms-PL.

Figure 1: The six steps

Step 1: Capturing the Webcam

The Silverlight 4 webcam API is easy to use. The CaptureSource class provides the webcam stream and you can use it as the source of a VideoBrush, which in turn fills a rectangle to show the video feed from the webcam. We'll use the default video capture device, but you can also iterate CaptureDeviceConfiguration class to get all the capture devices on the system. The user can specify the default video and audio devices with the Silverlight configuration; he or she only has to press the right mouse button over the Silverlight application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.

The initialization of the webcam:

C#

// Create capturesource and use the default video capture device 
captureSource = new CaptureSource();
captureSource.VideoCaptureDevice = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();
captureSource.CaptureImageCompleted += new EventHandler<CaptureImageCompletedEventArgs>(captureSource_CaptureImageCompleted);

// Start capturing
if (captureSource.State != CaptureState.Started)
{
   // Create video brush and fill the WebcamVideo rectangle with it
   var vidBrush = new VideoBrush();
   vidBrush.Stretch = Stretch.Uniform;
   vidBrush.SetSource(captureSource);
   WebcamVideo.Fill = vidBrush;

   // Ask user for permission and start the capturing
   if (CaptureDeviceConfiguration.RequestDeviceAccess())
   {
      captureSource.Start();
   }
}

Now that we have the webcam up and running, we need to take snapshots from the video stream to feed the face detection. You can use a custom VideoSink implementation for this task, but the CaptureSource also provides the built-in CaptureImageAsync method, which is easier to use. The performance is not much worse than a custom VideoSink.

C#

// Part of the RunUpdate method of the MainPage class
var dispatcherTimer = new DispatcherTimer();
dispatcherTimer.Interval = new TimeSpan(0, 0, 0, 0, 40); // 25 fps
dispatcherTimer.Tick += (s, e) =>
{
   // Process camera snapshot if started
   if (captureSource.State == CaptureState.Started)
   {
      // CaptureImageAsync fires the captureSource_CaptureImageCompleted event
      captureSource.CaptureImageAsync();
   }
};
dispatcherTimer.Start();

// captureSource_CaptureImageCompleted event handler of the MainPage class
private void captureSource_CaptureImageCompleted(object sender, CaptureImageCompletedEventArgs e)
{
   // Process camera snapshot
   Process(e.Result);
}

We use a DispatcherTimer here to call the CaptureImageAsync method. Every time the capturing is done, the CaptureImageAsync method fires a CaptureImageCompleted event, and the EventArgs provides a WriteableBitmap that contains a completed snapshot. The DispatcherTimer is initialized to perform this capturing task every 40 ms.

Figure 2: The result of the first step is a webcam stream

Step 2: Filtering the Skin Color

After grabbing an image from the webcam, you need to filter the skin color; this lets you find the face in a following step. The WriteableBitmap provided through the CaptureImageCompleted event uses the RGB color space to represent the pixels and is actually just a 32 bit integer array that stores the alpha, red, green and blue (ARGB) byte components for all the pixels.

The RGB color space has some major drawbacks when it comes to filtering certain color ranges. For example: we'd like to filter the skin color with a robust method that's not too much affected by the brightness of the image, but doing this in the RGB color space would imply a larger color space volume—so you can't use simple value thresholding.

Fortunately, there are other color spaces available that don't suffer from such problems. The HSV color space, for example, defines the color in its three components as Hue, Saturation and Value (Brightness), where the actual color (Hue) is represented as a circle from 0° to 360° and the brightness is the height of a cylinder.

Since the relevant skin color Hue ranges from 0° - 60° and 300° - 360° (which involves extra calculations) and the RGB to HSV conversion is more computationally expensive than other color space conversions, we use the YCbCr color space for the skin color filtering. YCbCr stores the brightness in the Y component and the chroma (color) information in the Cb component as blue-difference and in Cr component as red-difference. The RGB-YCbCr conversion can be done with simple addition and multiplication operations. The Y component ranges from 0 to 1, Cb and Cr from -0.5 to 0.5.

I tested several photos of different colored persons, and found that the following values cover most skin color ranges (except Martian or Avatar's Navis, perhaps).

Y = [0, 1] Cb = [-0.15, 0.05] Cr = [0.05, 0.20]

The sample application also provides Sliders to change these thresholds dynamically.

Figure 3 illustrates the YCbCr color range that's used for thresholding. The Y is constant at 0.5; the Cb lower threshold is left and the upper threshold is right; the Cr lower threshold is at the top and the upper threshold is at the bottom.

Figure 3: YCbCr color range. Y=0.5 Cb=[-0.15, 0.05] Cr=[0.05, 0.20]

This the source code used to generate the bitmap shown in Figure 3:

C#

// Visualize method of the HistogramVisualizer class
public void Visualize(WriteableBitmap surface)
{
   var w = surface.PixelWidth;
   var h = surface.PixelHeight;
   var pixels = surface.Pixels;
   var min = this.Min;
   var max = this.Max;
   int i;
   float xf, yf, cb, cr;

   // Use constant Y
   float v = min.Y + (max.Y - min.Y) * YFactor;

   // Interpolate between min and max and set pixel
   for (int y = 0; y < h; y++)
   {
      for (int x = 0; x < w; x++)
      {
         i = y * w + x;
         xf = (float)x / w;
         yf = (float)y / h;
         cb = min.Cb + (max.Cb - min.Cb) * xf;
         cr = min.Cr + (max.Cr - min.Cr) * yf;
         pixels[y * w + x] = new YCbCrColor(v, cb, cr).ToArgbColori();
      }
   }
}

Since the WriteableBitmap camera snapshot uses the RGB color space, we have to convert from RGB to YCbCr before we can apply the thresholds:

C#

// FromArgbColori method of the YCbCrColor class
public static YCbCrColor FromArgbColori(int color)
{
   // Extract the RGB components from the int color and convert to the range [0, 1]
   const float f = 1f / 255f;
   var r = (byte)(color >> 16) * f;
   var g = (byte)(color >> 8) * f;
   var b = (byte)(color) * f;

   // Create new YCbCr color from RGB color
   var y = 0.299f * r + 0.587f * g + 0.114f * b;
   var cb = -0.168736f * r + -0.331264f * g + 0.5f * b;
   var cr = 0.5f * r + -0.418688f * g + -0.081312f * b;

   return new YCbCrColor(y, cb, cr);
}

During the thresholding process, each pixel of the WriteableBitmap is converted from RGB to YCbCr and tested against the defined upper and lower threshold:

C#

// Process method of the ColorRangeFilter class
public WriteableBitmap Process(WriteableBitmap snapshot)
{
   var p = snapshot.Pixels;
   var result = new WriteableBitmap(snapshot.PixelWidth, snapshot.PixelHeight);
   var rp = result.Pixels;

   // Threshold every pixel
   for (int i = 0; i < p.Length; i++)
   {
      var ycbcr = YCbCrColor.FromArgbColori(p[i]);
      if (ycbcr.Y >= LowerThreshold.Y && ycbcr.Y <= UpperThreshold.Y
       && ycbcr.Cb >= LowerThreshold.Cb && ycbcr.Cb <= UpperThreshold.Cb
       && ycbcr.Cr >= LowerThreshold.Cr && ycbcr.Cr <= UpperThreshold.Cr)
      {
         rp[i] = 0xFFFFFF;
      }
   }

   return result;
}

After the thresholding is performed and the pixel falls in the skin color range, a white pixel is written to the resulting WriteableBitmap. This results in a binary black & white image that masks skin color and is then used in the next step.

Figure 4: The result of the second step is a skin color filtered binary image

Step 3: Reducing Noise with Erosion

The skin color-filtered image contains tiny pixel areas from the background that are mostly caused by image noise. Image noise can prevent a clear segmentation in later steps, so we have to remove it.

Erosion is a common image filter you can use for this task. The morphological Erosion operator reduces the boundaries of the colored regions so that very small regions are removed and only larger regions remain. It works by iterating over all pixels of the WriteableBitmap and testing to see if the neighboring pixels of the current pixel c are empty (zero) or not. If one of the neighboring pixels is empty, the current pixel c has to be a boundary pixel and thus must be removed (set to black). How many neighboring pixels are tested depends on the concrete implementation. This set of test coordinate points is usually called a kernel in image processing.

Since the previous step produced a black & white image, we can use a simple binary Erosion operator here. It turns out that a 5 x 5 kernel is optimal for this use case. The for-loop that would usually be used to implement a generic Erosion operator is unrolled into 5 * 5 = 25 neighbor pixel tests for better performance.

C#

// Process method of the Erode5x5Filter class
public WriteableBitmap Process(WriteableBitmap input)
{
   var p = input.Pixels;
   var w = input.PixelWidth;
   var h = input.PixelHeight;
   var result = new WriteableBitmap(w, h);
   var rp = result.Pixels;
   var empty = CompareEmptyColor; // = 0
   int c, cm;
   int i = 0;

   // Erode every pixel
   for (int y = 0; y < h; y++)
   {
      for (int x = 0; x < w; x++, i++)
      {
         // Middle pixel
         cm = p[y * w + x];
         if (cm == empty) { continue; }

         // Row 0
         // Left pixel
         if (x - 2 > 0 && y - 2 > 0)
         {
            c = p[(y - 2) * w + (x - 2)];
            if (c == empty) { continue; }
         }
         // Middle left pixel
         if (x - 1 > 0 && y - 2 > 0)
         {
            c = p[(y - 2) * w + (x - 1)];
            if (c == empty) { continue; }
         }
         if (y - 2 > 0)
         {
            c = p[(y - 2) * w + x];
            if (c == empty) { continue; }
         }
         if (x + 1 < w && y - 2 > 0)
         {
            c = p[(y - 2) * w + (x + 1)];
            if (c == empty) { continue; }
         }
         if (x + 2 < w && y - 2 > 0)
         {
            c = p[(y - 2) * w + (x + 2)];
            if (c == empty) { continue; }
         }

         // Row 1
         // Left pixel
         if (x - 2 > 0 && y - 1 > 0)
         {
            c = p[(y - 1) * w + (x - 2)];
            if (c == empty) { continue; }
         }


         // ... 
         // ... Process the rest of the 24 neighboring pixels
         // ...

         
         // If all neighboring pixels are processed 
         // it's clear that the current pixel is not a boundary pixel.
         rp[i] = cm;
      }
   }
   
   return result;
}

Figure 5: The result of the third step is a reduced-noise image

Step 4: Expanding with Dilation

In addition to removing noise, Erosion shrinks the face region. Unfortunately this causes some holes— especially in the area around the eyes—to arise or enlarge, which can lead to faulty color segmentation. That's where the other fundamental morphology operator, Dilation, comes into play

The Dilation enlarges the boundaries and expands an area by iterating over all pixels of the WriteableBitmap. This time, though, it's checked if one of the neighboring pixels of the current pixel c is not empty (white). If only one of the neighboring pixels is not empty, the current pixel c will be set to white.

The best results are achieved when the Dilation with a 5 x 5 kernel is applied three times.

C#

// Process method of the Dilate5x5Filter class which is applied 3 times
public WriteableBitmap Process(WriteableBitmap input)
{
   var p = input.Pixels;
   var w = input.PixelWidth;
   var h = input.PixelHeight;
   var result = new WriteableBitmap(w, h);
   var rp = result.Pixels;
   var r = this.ResultColor;
   var empty = CompareEmptyColor; // = 0
   int c, cm;
   int i = 0;

   // Dilate every pixel
   for (int y = 0; y < h; y++)
   {
      for (int x = 0; x < w; x++, i++)
      {
         // Middle pixel
         cm = p[y * w + x];
         // Is the pixel empty? 
         // If not we set the result and continue with the next pixel
         if (cm != empty) { rp[i] = r; continue; }

         // Row 0
         // Left pixel
         if (x - 2 > 0 && y - 2 > 0)
         {
            c = p[(y - 2) * w + (x - 2)];
            // If only one of the neighboring pixels is not empty,
            // we set the result and continue with the next pixel.
            if (c != empty) { rp[i] = r; continue; }
         }
         // Middle left pixel
         if (x - 1 > 0 && y - 2 > 0)
         {
            c = p[(y - 2) * w + (x - 1)];
            if (c != empty) { rp[i] = r; continue; }
         }
         if (y - 2 > 0)
         {
            c = p[(y - 2) * w + x];
            if (c != empty) { rp[i] = r; continue; }
         }
         if (x + 1 < w && y - 2 > 0)
         {
            c = p[(y - 2) * w + (x + 1)];
            if (c != empty) { rp[i] = r; continue; }
         }
         if (x + 2 < w && y - 2 > 0)
         {
            c = p[(y - 2) * w + (x + 2)];
            if (c != empty) { rp[i] = r; continue; }
         }

         // Row 1
         // Left pixel
         if (x - 2 > 0 && y - 1 > 0)
         {
            c = p[(y - 1) * w + (x - 2)];
            if (c != empty) { rp[i] = r; continue; }
         }


         // ... 
         // ... Process the rest of the 24 neighboring pixels
         // ...         
      }
   }
   
   return result;
}

Figure 6: The result of the fourth step after applying the Dilation three times

Step 5: Finding the face with histogram segmentation

The skin color-filtered, eroded and dilated image is a good starting point for image segmentation, the process of portioning an image into multiple sets of pixels (segments). Image segmentation is typically used to find the location of certain objects.

There are many different techniques available; a fast and simple method is histogram-based segmentation. An image histogram is a statistical representation of all the pixels present in the image. Most image editing tools have a color histogram functionality, which is commonly implemented as a graph that visualizes the amount of certain colors in an image. Figure 7 shows the color histogram of the sample image where the x-axis of the graph represents the brightness of each color component from 0 to 255, and the y-axis represents the number of pixels for each intensity.

Figure 7: Color histogram of the sample image

Since the skin color filtering produced a binary image, the histogram contains only two values for the number of black and white images and doesn‘t help us to find the skin segment. To fix this, we need the location of the white pixels maximum for the y-axis (rows) and x-axis (columns). Counting the white pixels separately for the rows and columns accomplishes this.

Figure 8 shows the row and column histogram for the webcam image where the blue color stands for row and the green for the column distribution of white pixels. The yellow lines highlight the maximum value for each.

Figure 8: Row and column histogram of the webcam image

C#

// FromWriteableBitmap method of the Histogram class
public static Histogram FromWriteabelBitmap(WriteableBitmap input)
{
   var p = input.Pixels;
   var w = input.PixelWidth;
   var h = input.PixelHeight;
   var histX = new int[w];
   var histY = new int[h];
   var empty = CompareEmptyColor; // = 0

   // Create row and column statistics / histogram
   for (int y = 0; y < h; y++)
   {
      for (int x = 0; x < w; x++)
      {
         if (p[y * w + x] != empty)
         {
            histX[x]++;
            histY[y]++;
         }
      }
   }

   return new Histogram(histX, histY);
}


// The constructor of the Histogram class that is used in the FromWriteableBitmap method
public Histogram(int[] histX, int[] histY)
{
   X = histX;
   Y = histY;

   // Find maximum value and index (coordinate) for x
   int ix = 0, iy = 0, mx = 0, my = 01;
   for (int i = 0; i < histX.Length; i++)
   {
      if (histX[i] > mx)
      {
         mx = histX[i];
         ix = i;
      }
   }

   // Find maximum value and index (coordinate) for y
   for (int i = 0; i < histY.Length; i++)
   {
      if (histY[i] > my)
      {
         my = histY[i];
         iy = i;
      }
   }

   // Keep results in member variables
   Max = new Vector(mx, my);
   MaxIndex = new Vector(ix, iy);
}

The code that draws the row and column histogram into the Dilation image shown in Figure 8 uses the WriteableBitmapEx library's DrawLine method:

C#

// Visualize method of the HistogramVisualizer class
public void Visualize(WriteableBitmap surface)
{
   var w = surface.PixelWidth;
   var h = surface.PixelHeight;
   var scale = this.Scale;
   var histogram = this.Histogram;
   var histX = histogram.X;
   var histY = histogram.Y;

   // Histogram X
   for (int x = 0; x < w; x++)
   {
      var hx = histX[x];
      if (hx != 0)
      {
         var norm = (int)(((float)hx / histogram.Max.X) * scale);
         surface.DrawLine(x, h - 1, x, h - norm, Colors.Green);
      }
   }
   // Draw max
   surface.DrawLine(histogram.MaxIndex.X, h - 1, histogram.MaxIndex.X, 0, Colors.Yellow);


   // Histogram Y
   for (int y = 0; y < h; y++)
   {
      var hy = histY[y];
      if (hy != 0)
      {
         var norm = (int)(((float)hy / histogram.Max.Y) * scale);
         surface.DrawLine(w - 1, y, w - norm, y, Colors.Blue);
      }
   }
   // Draw max
   surface.DrawLine(w - 1, histogram.MaxIndex.Y, 0, histogram.MaxIndex.Y, Colors.Yellow);
}

Figure 9: The result of the row and column histogram determination

Now that we know the x- and y-coordinates of the white pixels' start / end and where the maximum peak is, we can use this information to perform the actual histogram-based segmentation. The simple approach we'll use here finds initial seeds where the histogram value is above half of the histogram's maximum. In the next sub step, the algorithm starts at the found seeds and tries to detect the segment's boundaries where the histogram value is below a certain threshold. Usually the face region will be the largest segment in the result list, which is why the segments are sorted by size.

This approach is not perfect, but it's fast enough for real time and works for common webcam scenarios.

C#

// Process method of the HistogramMinMaxSegmentator class
public IEnumerable<Segment> Process(WriteableBitmap input)
{
   var hx = Histogram.X;
   var hy = Histogram.Y;
   var histUpperThreshold = Histogram.Max * 0.5f;

   // Find seeds for the segmentation:
   // All the x and y histogram indices where the value is above 
   // the half maximum and have a min distance 
const int step = 10;
   // x
   var ix = GetIndicesAboveThreshold(Histogram.MaxIndex.X, -step, hx, histUpperThreshold.X);
   ix.AddRange(GetIndicesAboveThreshold(Histogram.MaxIndex.X + step, step, hx, histUpperThreshold.X));
   // y
   var iy = GetIndicesAboveThreshold(Histogram.MaxIndex.Y, -step, hy, histUpperThreshold.Y);
   iy.AddRange(GetIndicesAboveThreshold(Histogram.MaxIndex.Y + step, step, hy, histUpperThreshold.Y));


   // Find the boundaries for the segments defined by the seeds
   var segments = new List<Segment>();
   foreach (var y0 in iy)
   {
      foreach (var x0 in ix)
      {
         var segment = new Segment(0, 0, 0, 0);
         segment.Min.X = GetIndexBelowThreshold(x0, -1, hx, ThresholdLuminance.X);
         segment.Max.X = GetIndexBelowThreshold(x0, 1, hx, ThresholdLuminance.X);
         segment.Min.Y = GetIndexBelowThreshold(y0, -1, hy, ThresholdLuminance.Y);
         segment.Max.Y = GetIndexBelowThreshold(y0, 1, hy, ThresholdLuminance.Y);
         segments.Add(segment);
      }
   }

   // Order by the largest segment
   return segments.OrderByDescending(s => s.DiagonalSq);
}

// GetIndicesAboveThreshold method of the HistogramMinMaxSegmentator class
private List<int> GetIndicesAboveThreshold(int start, int step, int[] hist, int threshold)
{
   var result = new List<int>();
   int hi;
   for (int i = start; i < hist.Length && i > 0; i += step)
   {
      hi = hist[i];
      if (hi > threshold)
      {
         result.Add(i);
      }
   }
   return result;
}

// GetIndexBelowThreshold method of the HistogramMinMaxSegmentator class
private int GetIndexBelowThreshold(int start, int step, int[] hist, int threshold)
{
   int result = start, hi;
   for (int i = start; i < hist.Length && i > 0; i += step)
   {
      hi = hist[i];
      result = i;
      if (hi < threshold)
      {
         break;
      }
   }
   return result;
}

Step 6: Overlaying the detected face

After we get information about the position of the face, we can finally do something useful and fun with this data.

Highlight facial region

The segment information calculated in the previous step contains the x- / y-coordinates of the segment's center and the width / height. It's easy to highlight the facial region with this data (Figure 10): We use an empty image that's laid over the webcam output and draw a red ellipse into the image's bitmap using the WriteableBitmapEx library's DrawEllipse method.

Figure 10: Highlighted facial region

C#

// Overlay method of the MainPage class
private void Overlay(IEnumerable<Segment> foundSegments, int w, int h)
{
   // Highlight the found segments with a red ellipse
   var result = new WriteableBitmap(w, h);
   foreach (var s in foundSegments)
   {
      // Uses the segment's center and half width, height
      var c = s.Center;
      result.DrawEllipseCentered(c.X, c.Y, s.Width >> 1, s.Height >> 1, Colors.Red);
   }
   ImgResult.Source = result;
}

Overlay facial region

Highlighting the face is nice (although not much fun), but we can also use the segment information to put an image over the face and move / scale it to the appropriate position / size. In the first version of the application I've used a photo of Chuck Norris, but I changed it (due to legal issues). Fortunately I found a funny snapshot of an orangutan that Ethan Hein took and released under the Creative Commons license. The ape's head is the default overlay image (Figure 11), but you could apply a different picture by entering its URI in a TextBox.

C#

// TransformOverlaidImage method of the MainPage class
private void TransformOverlaidImage(IEnumerable<Segment> foundSegments, int w, int h)
{
   // Set width and height of image by using the first segment's information
   var s = foundSegments.First();
   var iw = s.Width;
   var ih = s.Height;
   ImgOverlay.Width = iw;
   ImgOverlay.Height = ih;

   // Create the transform to keep the image's size and position in sync with the facial region
   var transform = new TransformGroup();
   
   // Scale image and move it to the segment's position
   transform.Children.Add(new ScaleTransform { 
      ScaleX = 1.5, 
      ScaleY = 1.5, 
      CenterX = iw >> 1, 
      CenterY = ih >> 1 });
   transform.Children.Add(new TranslateTransform { X = s.Min.X, Y = s.Min.Y + 10 });

   // Calcualte scale from Bitmap to actual size and create the transformation
   var sx = (GrdContent.ActualWidth / w);
   var sy = (GrdContent.ActualHeight / h);
   transform.Children.Add(new ScaleTransform { ScaleX = sx, ScaleY = sy });
   
   // Apply transformation
   ImgOverlay.RenderTransform = transform;
}

Voilà the iApe:

Figure 11: Result of the sixth step after overlaying the orangutan head

The Extras

Beside the algorithms we discussed above, the application also has some extra functionality. You don't need these for detection itself, but they're worth explanation.

Take result snapshot

You can take a snapshot of the final result (including the overlay) with the Button and send it to your friends, or use it as a profile picture or for your next passport. The code behind this functionality uses the WriteableBitmapEx library's WriteTga method to save the result into a TGA image file.

C#

// BtnSnapshot_Click method of the MainPage class
private void BtnSnapshot_Click(object sender, RoutedEventArgs e)
{
   // Render
   var wb = new WriteableBitmap(GrdContent, null);

   // Init SaveFileDialog
   var saveFileDlg = new SaveFileDialog
   {
      DefaultExt = ".tga",
      Filter = "TGA Image (*tga)|*.tga",
   };

   // SaveFileDialog.ShowDialog() can only be called from user-initiated code 
   // like an event handler, otherwise a SecurityException is thrown.
   if (saveFileDlg.ShowDialog().Value)
   {
      using (var dstStream = saveFileDlg.OpenFile())
      {
         wb.WriteTga(dstStream);
      }
   }
}

Load detection image

If you don't have a webcam or just want to try a photo with the skin color face detection, you can use the Button to load an image from disk. The code behind the Button opens a file dialog and fills a WriteableBitmap with the image's data. If it's larger than the size of the element that will show it, the picture will be resized with WriteableBitmapEx library's Resize method.

C#

// BtnOpenPic_Click method of the MainPage class
private void BtnOpenPic_Click(object sender, RoutedEventArgs e)
{
   // OpenFileDialog.ShowDialog() can only be called from user-initiated code 
   // like an event handler, otherwise a SecurityException is thrown.
   var openFileDialog = new OpenFileDialog();
   if (openFileDialog.ShowDialog().Value)
   {
      // Open the stream and load the image
      using (var stream = openFileDialog.File.OpenRead())
      {
         // Fill WriteableBitmap from stream
         var bmpImg = new BitmapImage();
         bmpImg.SetSource(stream);
         loadedPicture = new WriteableBitmap(bmpImg);
         
         // Resize the image if it's too large
         var w = (int)GrdContent.Width;
         var h = (int)GrdContent.Height;
         if (loadedPicture.PixelWidth > w || loadedPicture.PixelHeight > h)
         {
            loadedPicture = loadedPicture.Resize(w, h, WriteableBitmapExtensions.Interpolation.Bilinear);
         }
      }
   }
}

Conclusion

This article demonstrated how to implement a simple facial recognition system with Silverlight's new webcam feature and basic histogram-based skin color segmentation. This technique is not perfect, but it works in most webcam scenarios and in real time. There's definitely room for improvement in the segmentation method and in performance.

If you like this article, I'll write a second part that covers a better segmentation approach, overall performance optimization and an extended demo.

And if you want to try this out and learn more, the links to the live demo app and source code are at the top of the article!

About The Author

René Schulte is a .Net and Silverlight software developer from Dresden, Germany. He has a passion for computer graphics, physical simulations, AI and algorithms and loves C#, Shaders, Augmented Reality and the WriteableBitmap. He started the SLARToolkit, the WriteableBitmapEx and the Matrix3DEx Silverlight open source projects and has a Silverlight website powered by real time soft body physics. Contact information can be found on his Silverlight website or blog or via Twitter.