You can see a quick demonstration of the program in action in the following video:

In the next section, we'll learn how to use the program to create a movie. Then we'll look at how the program works and dive into the software's implementation.

How to Use the Program

This section will give an overview of how to use the program to make a video. You will need three basic pieces:

• A pen light or laser pointer to draw sketches with. You can buy cheap laser pointers and LED penlights at almost any convenience or office supply store.
• A web camera to capture the light pen.
• Software to get the video from the web camera and encode the video file. We'll discuss this program below.

You can create the program in simple six steps:

1. Select a video camera to use (plug one in, if necessary)
2. Configure the camera
3. Set the threshold for the pen
4. Select the background image or video, if you'll be using one
5. Start recording
6. Draw!

Let's look at the application's controls:

 Figure 1: A screenshot of the application

There are nine controls for the video camera, audio input and recording:

• A combo box for selecting the camera
• An option to flip the camera's video
• An option to tweak the camera's video control settings
• A slider to control the discrimination between the pen light and normal scene
• A combo box for selecting the microphone to use when recording
• An option to choose the background
• An option to choose the desired size of the output video
• Buttons to start, stop or pause recording
• A button to clear the current drawing

Some recommended settings for the camera:

• Reduce exposure setting until there is no glare
• Lower the brightness until there is no glare
• Disable the white balance “auto” setting and manually adjust it
• Disable the auto-exposure setting (if applicable)

Adjust these while watching the video preview area.

The Pen Threshold Slider

The next step is to move the pen light and adjust the threshold until the response to the light feels right. Anything brighter than the threshold will be considered a pen; everything else is just regular pixels. If the setting is too low, everything in the video will appear to be a pen; too high and it won't pick up on the pen. You may wish to go back and adjust the camera's parameters (see the previous section).

• If you are using a laser pointer, I recommend putting Scotch tape over the pointer. This will diffuse the light and keep it from overwhelming the CCD in the webcam and ruining the image.

The clear button erases the current pen drawing. You can use this periodically as you adjust the settings.

Figure 2: The pen light detector threshold

Other Tips for Making a Video

Making a movie takes some getting used to. Here are a few other tips I learned:

• Try to use a dark background behind you.
• Use a light low, and pointed toward the front of your face. This will make you look better and ensure the light isn't too bright on your skin.
• Try not to wear anything shiny, such as shirts or buttons. Depending on your lighting, the shiny part can look like a pen.
• Don't have any glass or other items behind you. Light might reflect off glass or a forehead, and look like a pen.

• Don't vary the lighting. Some webcams change their parameters automatically based on light level; others change their frame rate when it's darker (to allow longer exposure).

DirectShow

To understand how the software is implemented, we first need a quick overview of DirectShow. Software built using DirectShow employs component (object) graphs to do its work and create overall behavior. Some components—such as video camera or video encoder—are necessary. Some add features, and a few are needed to connect it all. When the graph is built and run, DirectShow makes sure each node agrees on the exact media formats that will be provided. I enjoy object graphs as a design structuring technique, but I found DirectShow a challenge to master.

In a sense, you lay out the basic schematic of the system—or at least the DirectShow portion—and then create code to complete the fine details and add functionality. You can prototype a graph (schematic) with GraphEdit or Monograph EditStudio. These tools have a large catalog of pieces, making it easy to try out different ideas. Then you can lay out the pieces to see if they go together (some pieces just don't) and test it.

This project uses three different graphs:

1. A graph to show the camera preview, which is used when not recording or when sketching on a movie
2. A graph to record from web camera, with or without a still image in the background
3. A graph to paint on a video from a file

The Preview Graph

Let's start with the simple camera preview:

Figure 3: Second DirectShow graph to get the pen

Here are the components and what they do:

• The Camera: your webcam (or other video input device)
• A Sample Grabber to get an image frame and pass it to a delegate. This graph uses two: The first, Flip Video, flips the video to make the preview more like a mirror. The second, Light Paint, scans for pen light and adds points in for the previous pen. I'll go into a bit more detail below.
• The Video Renderer is a helper object that displays the camera image on the window. It requires a window region to display on.

Finding the Camera

Let's look at how the program code does this. The following enumeration is used by the display pull down to list all the video sources:

C#

static public IEnumerable<VideoSource> VideoDevices()
{
  IEnumMoniker em = DeviceEnum(ref DirectShowNode.CLSID_VideoInputDeviceCategory);
  if (null == em)
    yield break;

  foreach (IMoniker Moniker in COM.Enumerator(em))
  {
     VideoSource S = new VideoSource(Moniker), T;
     string Key = S.DevicePath;
     if (null == Key)
       Key = S.DisplayName;
     if (DevicePath2Source.TryGetValue(Key, out T))
       {
          S.Dispose();
          S = T;
       }
      else
       DevicePath2Source[Key] = S;

     yield return S;
  }

  Marshal.ReleaseComObject(em);
}

Whenever the video source selection changes, the _VideoSource instance variable is updated accordingly.

Creating the Preview Graph

The following code builds the video graph (see Figure 3):

C#

DirectShowGraph CamVideo=null;
public void BuildPreviewGraph(Control CamPreview)
{
  // Disable any face tracking
  _VideoSource.FaceTracking  = PluralMode.None;

  // Add the camera source
  CamVideo = new DirectShowGraph();
  CamVideo.Add(_VideoSource, "source", null);

  // Add the flip video item, as a delegate of a sample grabber
  SampleGrabber CamFrameGrabber1 = new SampleGrabber();
  Flip = new FlipVideo();
  CamFrameGrabber1.Callback(Flip);
  Flip.FlipHorizontal = FlipHorizontal;
  AMMediaType  Media = CamVideo.BestMediaType(RankMediaType);
  CamVideo.Add(CamFrameGrabber1, "flipgrabber",  Media);
  CamFrameGrabber1.MediaType = Media;

  // Add the paint-with light item, as a delegate of a sample grabber
  SampleGrabber CamFrameGrabber = new SampleGrabber();
  PaintedArea = new LightPaint();
  CamFrameGrabber.Callback(PaintedArea);
  Media = CamVideo.BestMediaType(RankMediaType);
  CamVideo.Add(CamFrameGrabber, "grabber",  Media);
  CamFrameGrabber.MediaType = Media;

  DirectShowNode Preview = new DirectShowNode(DirectShowNode.CLSID_VideoRenderer);
  CamVideo.Add(Preview, "render1", null);

  Preview.RenderOnto(CamPreview);

  // Add a null renderer to consume any extra pins from the camera source
  DirectShowNode N = new DirectShowNode(DirectShowNode.CLSID_NULLRenderer);
  CamVideo.Add(N, "null",null);

  // The size isn't known until we've built a sample grabber graph
  CamFrameGrabber1.UpdateFrameSize();
  Flip.Size = CamFrameGrabber1.FrameSize;
  CamFrameGrabber.UpdateFrameSize();
  PaintedArea.Size = CamFrameGrabber.FrameSize;

  // Start the camera graph
  CamVideo.Start();
}

If it has the option, the first thing it does is tell the camera to disable face tracking. Otherwise, the camera might move its view on us, causing all sorts of confusion.

Then it builds the graph using a helper class called DirectShowGraph to add each of the nodes. The helper automatically connects the pins between the passed node and the most recent node with available output pins.

Next, the frame sizes are updated and the graph is executed. DirectShow takes over, moving video from the camera through the graph and onto the display.

The FlipVideo Sample Grabber Delegate

The SampleGrabber class is a proxy to the DirectShow COM ISampleGrabber objects. It has a procedure called Callback() that registers a callback to a delegate implementing the ISampleGrabberCB interface.

This project includes a class called FlipVideo whose instances are used here. If enabled, each instance is responsible for flipping the video. Some cameras don't have a built-in setting to do this, so we provide a way to do it in code.

Here is a portion of the code that does this. Because it is byte manipulation, it looks a like C:

C#

public int BufferCB(double SampleTime, IntPtr Buffer, int BufferLen)
{
    unsafe
    {
        byte* Buf = (byte*) Buffer;
        byte* End = Buf + BufferLen-(BufferLen%3);

        // The width of our buffer
        int Width  = Size . Width;

        if (!FlipHorizontal)
        return 0;
        // This takes about 8 ms (640x480)
        int Width3 = Width*3;
        byte* BufEnd = Buf + Width3 * Size.Height;
        for (byte* BPtr= Buf; BPtr != BufEnd; BPtr+= Width3)
            for (byte* B = BPtr, BEnd = B+Width3-3; B < BEnd;)
            {
                byte Tmp =*BEnd;
                *BEnd++ = *B;
                *B++    = Tmp;
                Tmp =*BEnd;
                *BEnd++ = *B;
                *B++    = Tmp;
                Tmp     = *BEnd;
                *BEnd   = *B;
                *B++    = Tmp;
                BEnd -= 5;
            }
    }
    return 0;
}

The LightPaint Sample Grabber Delegate

This project includes a class called LightPaint, whose instances are used as here. Each LightPaint instance is responsible for three things:

1. Finding the pen points
2. Replacing the video with the background image (optional). Although this isn't necessary for the simple preview case, it is necessary in the more complex configurations. We'll look at those later.
3. Painting all the pen strokes

Below is a portion of the code that does this. I've excluded a nearly identical chunk of code that removes the step where the background image pixels are copied in. (We allow this duplication for performance reasons—checking whether to include a background image with every pixel gets expensive!)

C#

public int BufferCB(double SampleTime, IntPtr Buffer, int BufferLen)
{
    unsafe
    {
        // This scary construct speeds up the processing of the buffer a lot
        // by 10ms or more.  This is critical in speeding up acsess
        fixed (byte* _CurrentPoints = CurrentPoints)
        fixed (byte* _IsPenPoint   = IsPoint)
        fixed (byte* Bknd         = Bkgnd)
        {
            byte* Buf = (byte*) Buffer;
            int Width3 = _Size.Width*3, Width=_Size.Width;

            BufferLen -= BufferLen % 3;
            byte* End = Buf + BufferLen;
            byte* CurrentPoint = _CurrentPoints;
            byte* IsPenPoint = _IsPenPoint;

            // Scan the image for the points brighter than threshold
            for (int PI=0,I=0; Buf != End;  PI++, I += 3)
            {
                byte B1= Buf[0];
                byte G1= Buf[1];
                byte R1= Buf[2];
                byte B2 = _CurrentPoints[I+0];
                byte G2 = _CurrentPoints[I+1];
                byte R2 = _CurrentPoints[I+2];

                // This is the key spot that detects the pen light.  
                // This must be very fast
                // Tweak this in different ways to see what works
                if (R1 * RedScale + G1 * GreenScale + 
                    B1*BlueScale >= Threshold)
                {
                    if (B1>B2 || G1>G2 || R1>R2)
                    {
                        _IsPenPoint[PI] = 1;
                        _CurrentPoints[I+0] = B1;
                        _CurrentPoints[I+1] = G1;
                        _CurrentPoints[I+2] = R1;
                    }
                    Buf+=3;
                    continue;
                }

                if (0 == _IsPenPoint[PI])
                {
                    // Add the current points
                    B2 = Bknd[I+0];
                    G2 = Bknd[I+1];
                    R2 = Bknd[I+2];
                }

                *Buf++ = B2;
                *Buf++ = G2;
                *Buf++ = R2;
            }
        }
    }

    return 0;
}

The Scariest Keyword in .NET…

… is the “fixed” keyword. It turns an array into a pointer and keeps the garbage collector from touching the array while you use it. In short, “fixed” is everything your mother warned you about in C. But it gives a big performance improvement.

A frame needs to complete processing, compression and be written to the file in less than 30ms. (Usually it must do so in a lot less time, to provide adequate safety margin.) If not, the processing buffers will fill, the video quality could deteriorate, and the video preview will lag so much it will make the program unusable.

I found that “fixed” saves 10ms (about 30%) of the processing time spent in the Sample Grabber delegates. That's big.

The Graph to Painting on the Video Stream or Still Image

Now that we've learned how the pieces go together for painting on a preview, let's look at how we can record the stream.

You can use the next graph to record yourself painting on a still image or the video stream. It's a lot more complicated than the previous one:

Figure 4: DirectShow when painting on the Webcam video or a still picture

This adds a lot more components. Here are the components and what they do:

• A WM Asf Writer that encodes the movie and writes it to a file. It needs both an audio and a video input, as well as a special profile to support encoding at different sizes.
• Three Sample Grabbers. The first two delegates are the FlipVideo and LightPaint objects discussed earlier. The third has an Overlay delegate that puts the pen stroke onto the video preview so you can see where your hand and pen are while you paint.
• A Smart Tee splits an image stream into two copies. There are two tees used. The first splits the camera video into streams with and without the still background image. The second splits the video into two screens: one that is recorded, and another that is seen on the preview. As a bonus, the Smart Tee gives priority to the movie encoder and can drop frames to the preview.
• Microphone. The WM Asf Writer requires an audio source. The microphone gives you a chance to narrate something fun onto the movie.
• Two Video Renderers. One provides previews of the pen over a still background image; the other goes over the webcam video to show the user is painting. I found that I needed to see where the pen is in the camera's view, and where it is on the resulting movie, in order to paint effectively.

The Overlay Delegate

The overlay delegate is a much simpler version of the LightPaint class. Like the LightPaint class, it flips the video (if appropriate) and puts the pen stroke onto the video preview so you can see where your hand and pen are while painting. It's synchronized with the LightPaint object in order to grab the pen strokes from it.

C#

public int BufferCB(double SampleTime, 
        IntPtr Buffer, int BufferLen)
{
    unsafe
    {
        // This scary construct speeds up the 
        // processing of the buffer a lot by 10ms or more.  
        // This is critical in speeding up access: each frame
        // has to make it thru the whole DS graph 
        // in far less than 30ms.
        fixed (byte* CurrentPoint = SrcPoints.CurrentPoints)
        fixed (byte* IsPenPoint   = SrcPoints.IsPoint)
        {
            byte* Buf = (byte*) Buffer;
            byte* End = Buf + BufferLen-(BufferLen%3);

            // The width of the LightPaint delegate and it's size
            int Width2 = SrcPoints.Size.Width;

            // The width of our buffer
            int Width  = Size.Width;

            if (Size == SrcPoints.Size)
            {
                // This is for the common, but special, 
                // case where the LightPaint
                // delegate and use have the size and 
                // we don't need to resize
                for (int I=0; Buf != End; Buf+=3, I++)
                    if (0 != IsPenPoint[I])
                    {
                       int J = I*3;
                       Buf[0] = CurrentPoint[J++];
                       Buf[1] = CurrentPoint[J++];
                       Buf[2] = CurrentPoint[J++];
                    }
            }
            else
            {
                // The loop used to scan over the points
                // Note: This is designed to allow different 
                // sizes for the LightPaint delegate and 
                // the buffer we are painting on
                for (int Y = 0, Y2=0; Buf != End; Y2+= dY2, Y=(Y2>>10))
                    for (int X=0,J=Y*Width2,I=J*3,I2=Y*Width2*1024;
                        Buf != End && X < Width;
                        X++, Buf+=3, I2+= dX2, J=(I2>>10),I=3*J)
                    {
                        if (0 != IsPenPoint[J])
                        {
                            Buf[0] = CurrentPoint[I];
                            Buf[1] = CurrentPoint[I+1];
                            Buf[2] = CurrentPoint[I+2];
                        }
                    }
            }
        }
    }
    return 0;
}

The Graph to Paint on a Movie

We can take this a step further by painting onto a movie. This means we need two video sources: one for the camera (which can see the pen light), and one for the movie.

Figure 5: DirectShow when painting on a movie, using your own microphone or the movie's audio

There are two separate filter graphs here. The top graph captures the pen strokes from the camera. The lower graph captures video from a video file, overlays the pen strokes, and previews and encodes the video. The code selects one of the dashed lines at run-time. If the user chooses to use the movie's original audio track, the WM Asf Writer's audio input is connected to the movie's output. Otherwise, the input is connected to the microphone.

We've also introduce two more components:

• The WM Asf Reader, which reads from a video file
• The Color Convert, which converts the video to RGB format. The video cameras allow a variety of output formats, and the delegate selects the RGB format. However, movie files tend to have only one output encoding, and it's seldom RGB.

Conclusion

In this article, I described how to create a paint-with-light effect on pictures, movies, etc. using a webcam. I reviewed the key concepts of DirectShow and how they can be used to create the video. Watching a movie of a sketch being drawn is what makes this different from a simple doodle on a picture.

If you want to try this out, check the download link for the source code at the top of the article!

About The Author

Randall Maas writes firmware for medical devices, and consults in embedded firmware. Before that, he did a lot of other things… like everyone else in the software industry. You can contact him at randym@acm.org.

Introduction

This article describes “Gremlin”, a quick and easy application that lets you play Halloween tricks. When your victim's computer is idle, Gremlin moves windows around on the screen, changes the focus window, moves your mouse, scrolls windows, and types nonsensical stuff. When there is background noise – like someone talking – it will shake the screen, even as your victim is typing away.

Deployment

Run it right away. You can download, copy theGremlin.exe executable and NAudio.dll to the victim's computer (say in c:\ directory), and then double click on the executable to run it.

Run it later. The other option is to copy the executable and dll files (or a shortcut to it) to the Startup folder on the victim's machine and watch the fun begin when they start up their machine in the morning!

If the computer you're using runs on Windows XP the path is:

C:\Documents and Settings\All Users\Start Menu\Programs\Startup

With Vista and Windows 7 the path looks like:

C:\Users\USERNAME\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup

Just remember to change USERNAME to the name of the user on your machine.

Commandline

Gremlin is compiled as a windows application, so it won't popup a terminal window if you double click on it, etc. But you can run it from the command line with flags:

-aggressive will make the gremlin's actions more obvious

-help will display the command line options

-name NAME is useful for testing. Gremlin will only use windows with this specific title or from this specific application. (Remember to drop the ".exe" from the application filename)

To stop the program, press CTRL+F2.

And now … the dodgy bits

The overall structure of the program is broken down into three kinds of functionality – actions, triggers and some interstitial glue:

Actions

• The screen-shaker is a window that makes the monitor look like it is shaking. Sort of like a loose cable on the back of the monitor.
• A random event might kick the windows around—either friction will slow them down, or...
• Gremlin will type nonsensical messages from the keyboard. These messages are then sent from a virtual keyboard, or they:
• Move the mouse around, or
• Press the mouse buttons, or
• Randomly switch the focus to another window

Triggers

• An audio module listens for sounds that trigger the module that shakes your screen.
• Otherwise, the software uses a p/invoke to GetLastInputInfo() and waits for the user to be idle for a minute or two. When it detects that the user is inactive, it randomly selects and carries out actions.

Other

• A hidden window receives key press events. I tend to reuse this module a lot, as a way to stop experimental programs that may have made my machine unusable.

For fun, I'll describe a couple of these modules below.

Screen shaker

My favorite bit is the screen shaker. It grabs a picture of the screens, creates a window that spreads across them, and then moves that image back and forth a few pixels every 30ms or so, with a little bit of random rotation. (Faster computers, of course, get a better effect).

The screen shaker is special in four ways:

• It is a top-level window – no other windows are allowed to go over it.
• This top-level window never becomes the focus window (the window that gets the keyboard events).
• Mouse button events (e.g. clicks and double-clicks) are passed through to the windows and desktop underneath. This gives the illusion that this is the "real" desktop shaking, by allowing a person to click on a button or text that passes the click thru to the real button or text.
• It shakes each of the monitor's screens independently

Some background: to create a window that looks like the shaking screen, I used two classes. The first is ScrShake (in ScreenShake.cs), which derives from the second class UnfocusableForm.  I'll describe ScrShake first.

The screen shaking process requires five instance variables:

• animTimer is a System.Windows.Forms.Timer used to force painting of a new frame on the screen.
• bx, and by are how far the screen has shaken up or down, left or right.
• angle is how much the screen has twisted during shaking.
• screenBitmap is an array of bitmaps, one for each of the monitors.

This creates a window without a border or other trappings, then makes it the topmost window and sets a flag in the method variable ExStyle to ignore mouse clicks. More on this flag in a little while.

C#

public partial class ScrShake : UnfocusableForm
{
   public ScrShake(double ShakeCoef, double AngleCoef) : base(true)
   {
      this.SuspendLayout();
      … other setup code …
      // This is needed since we're over the whole display, and we don't
      // want the other areas to be blurry
      TransparencyKey =  BackColor = ForeColor = System.Drawing.Color.Fuchsia;
      DoubleBuffered = true;
      TopMost = true;
      FormBorderStyle = System.Windows.Forms.FormBorderStyle.None;

      ClientSize = new System.Drawing.Size(300, 300);
      Name = "topimage";
      ResumeLayout(false);

      // This is need to pass mouse clicks thru to lower layers
      ExStyle |= (int) WS . EX_TRANSPARENT;

      … More code that will be described later…
   }
}

When the timer has done a given number of animations, it will call the Stop() method. This will stop the animation, clean it up, and hide the window:

C#

void Stop()
{
   animTimer . Stop();
   Hide();
   screenBitmap = null;
}

When the main loops that start the animated screen shaking process, it calls the Screens() method. This grabs an image of the screens, determines the shape of each monitor, and starts the animation for 10 frames.

C#

internal void Screens()
{
   if (Visible)
   {
      Stop();
      return ;
   }
   // Grab the screens
   screenBitmap = Program.GrabScreens();
   CountDown = 10;

   animTimer.Start();
   angle = 0.0f;
   … code to display the window and get shape of monitors (see below)…
}

The screen capture portion is in ScreenCapture.cs. The method GrabScreens() creates an individual bitmap for each monitor and returns them as an array.

C#

internal void Screens()
{
   if (Visible)
   {
      Stop();
      return ;
   }
   // Grab the screens
   screenBitmap = Program.GrabScreens();
   CountDown = 10;

   animTimer.Start();
   angle = 0.0f;
   … code to display the window and get shape of monitors (see below)…
}

The code below determines the bounds of each screen and builds up a size of all of the screens put together. The window will be set to be this size.

C#

Screen[] AllScreens = Screen.AllScreens;

// full width/height of all monitors combined
Rectangle fullSize = AllScreens[0].Bounds;

// find a rectangle that will encompass all monitors on the system
// (assuming the primary monitor is on the left/top!)
for (int i = 1; i < AllScreens.Length && i < screenBitmap.Length; i++)
{
    Rectangle Bounds = AllScreens[i].Bounds;

    if (Bounds.Left < fullSize.Left)
        fullSize.X = Bounds.X;

    if (Bounds.Right > fullSize.Right)
        fullSize.Width = Bounds.Right - fullSize.X;

    if (Bounds.Top < fullSize.Top)
        fullSize.Y = Bounds.Y;

    if (Bounds.Bottom > fullSize.Bottom)
        fullSize.Height = Bounds.Bottom - fullSize.Y;
}

The code to show the window, fill the whole screen, and move it to the top looks like:

C#

Show();
WindowState = FormWindowState.Normal;
// cover all monitors with one gigantic window
Location = new Point(fullSize.Left, fullSize.Top);
Size = new Size(fullSize.Width, fullSize.Height);

// bring it to the top
BringToFront();

The constructor also created a timer that drives the screen shaking effect. The timer has a delegate that randomly selects the angle of rotation and the offset of the image, and triggers a repaint of the window. (The amount of shaking is controlled by two external variables called AngleCoef and ShakeCoef).

C#

animTimer = new System.Windows.Forms.Timer();
animTimer.Tick += delegate(object A, EventArgs E)
{
    if (--CountDown < 1)
        Stop();

    angle += (float)((Program.Rnd.NextDouble() - 0.5) * AngleCoef);
    bx = (float)((Program.Rnd.NextDouble() - 0.5) * ShakeCoef);
    by = (float)((Program.Rnd.NextDouble() - 0.5) * ShakeCoef);
    Invalidate();
};

// Sets the timer interval to 30 milliseconds.
animTimer.Interval = 30;

Although each monitor shifts up & down, left & right by the same amount, and rotates by the same angle, they are painted independently. This gives the illusion that each monitor has a shaky image. Paint the window is using the following code.

C#

protected override void OnPaint(PaintEventArgs e)
{
    base.OnPaint(e);
    Graphics g = e.Graphics;
    g.CompositingQuality = CompositingQuality.HighQuality;

    // for each monitor, draw the effect
    Screen[] AllScreens = Screen.AllScreens;
    for(int i = 0; i < screenBitmap.Length; i++)
    {
        if (null == screenBitmap[i])   continue;
        g.SmoothingMode = SmoothingMode.HighQuality;

        // grab the size of the current monitor
        Rectangle region = AllScreens[i].Bounds;

        double ImWidth = screenBitmap[i].Width  * 
           e.Graphics.DpiX / screenBitmap[i].HorizontalResolution;
        double ImHeight= screenBitmap[i].Height * 
           e.Graphics.DpiY / screenBitmap[i].VerticalResolution;
        Matrix m = new Matrix();
        m.Translate( (float)(- ImWidth  /2), 
                  (float)(- ImHeight /2), MatrixOrder.Append);

        // rotate the bitmap about the center
        m.RotateAt(angle, new Point(0,0), MatrixOrder.Append);

        // center the image no matter what its size is
        m.Translate( region.Width /2 - bx, 
                  region.Height/2 - by, MatrixOrder.Append);

        // assign our transformation matrix
        g.Transform = m;

        // draw it
        g.DrawImage(screenBitmap[i], region.Left, region.Top);
    }

The Form that does nothing!

The ScrShaker class uses a help class called UnfocusableForm (in UnfocusableForm.cs) to create a window that never becomes the focus window – it never receives key presses, etc.

This window has no trappings – no border, no resize gripper, no icon, no minimize / maximize buttons, no title bar.

C#

public partial class UnfocusableForm : Form
{
   public UnfocusableForm(bool X) : base()
   {
      if (X)
        {
           ControlBox = false;
           FormBorderStyle = System.Windows.Forms.FormBorderStyle.None;
           ShowInTaskbar = false;
           ShowIcon = false;
           MinimizeBox = false;
           SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide;
           StartPosition = System.Windows.Forms.FormStartPosition.Manual;
        }
   }

Then it overrides the ShowWithoutActivation method so that the window will not become the active when it is shown.

C#

protected override bool ShowWithoutActivation
{ get {  return true; } }

The UnfocusableForm class also overrides the CreateParams() method to set extra styles when creating the window. I haven't found a way to set these styles flexibly. Instead, the UnfocusableForm uses a method variable called ExStyle that allows derived classes to specify what flags they desire. In this case, The ScrShake class used the method variable to set a flag that ignore mouse clicks:

C#

protected override CreateParams CreateParams
{
  get
  {
     CreateParams cp=base.CreateParams;
     cp . ExStyle |= ExStyle;
     return cp;
  }
}

Finally, it captures a couple of events that ask a window if it would like to become the active window. (These usually happen when the user clicks on an inactive window):

C#

internal const int MA_NOACTIVATE = 0x0003;
protected override void WndProc(ref Message m)
{
  if (m.Msg == (int) WM.MOUSEACTIVATE)
  {
     m.Result = (IntPtr) MA_NOACTIVATE;
     return;
  }
  if (m.Msg == (int) WM.FOCUS)
  {
     m.Result = (IntPtr)1;
     return;
  }
  base.WndProc(ref m);
}

Sending Mouse Events

Sending Mouse Events is simple, but not trivial. The wiki at Pinvoke.net provides the signature to the procedures and structures we need to pass to it. (Note: these structures are a bit different, based on a blog posting by Raymond Chen)

C#

[DllImport("user32.dll", EntryPoint = "SendInput", SetLastError = true)]
static extern uint SendInput(uint nInputs, INPUT[] Inputs, int cbSize);
[DllImport("user32.dll", EntryPoint = "GetMessageExtraInfo", SetLastError = true)]
static extern IntPtr GetMessageExtraInfo();

[StructLayout(LayoutKind.Sequential)]
struct INPUT
{
   internal INPUTTYPE   type;
   internal INPUT_UNION i;
}

 // This generates the anonymous union
[StructLayout(LayoutKind.Explicit)]
struct INPUT_UNION
{
   [FieldOffset(0)]
   public MOUSEINPUT mi;
   [FieldOffset(0)]
   public KEYBDINPUT ki;
   [FieldOffset(0)]
   public HARDWAREINPUT hi;
};

[StructLayout(LayoutKind.Sequential)]
struct MOUSEINPUT
{
   public int    dx;
   public int    dy;
   public int    mouseData;
   public MOUSEEVENTF dwFlags;
   public int    time;
   public IntPtr dwExtraInfo;
}

[Flags]
enum MOUSEEVENTF : int
{
   MOVE       = 0x01,
   LEFTDOWN   = 0x02,
   LEFTUP     = 0x04,
   RIGHTDOWN  = 0x08,
   RIGHTUP    = 0x10,
   MIDDLEDOWN = 0x20,
   MIDDLEUP   = 0x40,
   ABSOLUTE   = 0x8000
}

Each of these pieces go together in a specific way. To do a mouse movement, a mouse event structure needs to be created. This involves setting dwFlags to the kind of mouse event (a relative movement in this case), dx and dy to the number of pixels moved. It is also important to set dwExtraInfo to whatever GetMessageExtraInfo() is set to.

C#

MOUSEINPUT MEvent = new MOUSEINPUT();
MEvent.dwFlags = MOUSEEVENTF.MOVE;
MEvent.dx = Rnd . Next(-8, 8) ;
MEvent.dy = Rnd . Next(-8, 8);
MEvent.dwExtraInfo = GetMessageExtraInfo();
Send(MEvent);

Sending the event takes a few more steps that are all wrapped in a helper methdo called Send(). Send these human interface device events via the SendInput() procedure (earlier). The procedure takes an array of events, for different kinds of devices. We have to create the array, set the event type, copy the event data, and then make the call:

C#

public virtual bool Send(MOUSEINPUT Event)
{
  INPUT[] Events = new INPUT[1];
  Events[0] . type = INPUTTYPE . MOUSE;
  Events[0] . i.mi = MEvent;
  return SendInput((uint)Events.Length, Events, Marshal.SizeOf(Events[0])) > 0; 
}

From here, sending a mouse button click is pretty straight forward. A mouse click is really a mouse button press event, followed by a mouse button release event. To make things interesting the mouse button is choosen at random. The “ugly” part is that each of the mouse buttons is assigned a bit, so a binary shift is used to convert a button number to its bit. And the button release is yet another bit, so the second message shifts the flags, to move the bit from the “button pressed” state to the “button released state”.

C#

MOUSEINPUT MEvent = new MOUSEINPUT();
MEvent.dwFlags = (MOUSEEVENTF) (1 << (2*Rnd.Next(0, 3)+1)) ;
MEvent.dwExtraInfo = GetMessageExtraInfo ();
Send(MEvent);

MEvent.dwFlags = (MOUSEEVENTF)((int) MEvent.dwFlags << 1);
MEvent.dwExtraInfo = GetMessageExtraInfo ();
Send(MEvent);

Generating deranged text

The gremlin can also type deranged sentences that will appear in editors, if the user left one open. The SendWait() method in the SendKeys class does the typing itself:

C# 

SendKeys . SendWait(GenerateSentence());
SendKeys . SendWait("{ENTER}");

Generating the sentences is not difficult; I used a simplistic Markov generator. This generator starts by randomly selecting a word that can start a sentence. The Starts variable is array which holds just these words. Then, in the Transition method, this word is used to find the next word that can be included in the sentence. It does this by using a dictionary, called NonStart, which given the word as a key, will return a list of all the words that can come after it. This process then repeats, appending the words onto the end of a string. It stops if it finds a word that can end a sentence.

C#

static Dictionary<string,string[]> NonStart ;
static string[] Starts;
static Dictionary<string,string> Terminal ;

static string GenerateSentence()
{
  StringBuilder SB = new StringBuilder();
  string Word = Starts[ Rnd.Next(0, Starts.Length) ];

  Transition(SB, Word);
  return SB.ToString();
}

static void Transition(StringBuilder SB, string Word)
{
  while (true)
  {
     SB.Append(Word);
     SB.Append(' ');

     // Look up word after this
     string[] Nexts;
     if (!NonStart.TryGetValue(Word, out Nexts))
       break;
     int Idx = Rnd.Next(Terminal . ContainsKey(Word) ? -1 :0, Nexts.Length);
     if (Idx < 0)
       break;
     Word = Nexts[Idx];
  }
}

Building these two dictionaries and array is a pretty simple process. The BuildSuffixTree() method takes a string and splits it up into sentences. Then it splits each sentence into (lower case) words. The first word of the sentence goes onto the end of the Starts array. Otherwise, the previous word is used to look up a list in a dictionary, and the current word is appended on to the list. This dictionary will become the NonStarts dictionary. The last word of the sentence is also placed into the Terminal dictionary, to indicate that this might be the end of a sentence.

C#

static void BuildSuffixTree()
{
  // First, build up a list of words that start sentence, and transitions
  List<string> Starts1 = new List<string>();
  Dictionary<string,List<string>> Trans = new Dictionary<string,List<string>>();
  foreach(string S1 in S.Split(new char[]{'.','?','!'}))
  {
     string Prev=null;
     foreach(string S2 in S1.Split(new char[]{' ','\n','\r','\t',',',';'}))
     {
        if (S2 . Length < 1)
          continue;
        if (null == Prev)
          {
             Starts1.Add(string.Intern(S2.ToLower()));
             Prev = string.Intern(S2.ToLower());
             continue;
          }
        List<string> Nextsa;
        if (! Trans.TryGetValue(Prev, out Nextsa))
          Trans[Prev] = Nextsa = new List<string>();
        Prev = string.Intern(S2.ToLower());
        Nextsa.Add(Prev);
     }
     if (null != Prev)
       Terminal[Prev] = Prev;
 }

 // Next, flatten the list of words that start a sentence
 Starts = Starts1.ToArray();

 // Flatten the transition table
 foreach (string S3 in Trans.Keys)
  NonStart[S3] = Trans [S3].ToArray();
}

The things that trigger action

The audio trigger

The audio module (in file AudioTrigger.cs) listens in on all of the microphones. It uses NAudio, with a basic setup and delegate structure swiped from Mark Heath's article “.NET Audio Recording”. The first difference is that it registers all audio input devices:

C#

static WaveIn[] StartMics()
{
  int NumDevices = WaveIn.DeviceCount;
  WaveIn[] AudIns = new WaveIn[NumDevices];
  for (int waveInDevice = 0; waveInDevice < NumDevices ; waveInDevice++)
  {
     AudIns[waveInDevice] = new WaveIn();
     AudIns[waveInDevice].DeviceNumber = waveInDevice;
     AudIns[waveInDevice].DataAvailable += waveIn_DataAvailable;
     AudIns[waveInDevice].WaveFormat = new WaveFormat(8000, 1);
     AudIns[waveInDevice].StartRecording();
  }
  return AudIns;
}

The real magic in the trigger is a modified version of the waveIn_DataAvailable delegate. Instead of saving the audio, it checks for any sound activity. It starts by checking to see if any of the sound amplitudes are greater than a pre-defined, very high threshold. Then the samples are squared and summed up, and the result is compared with a threshold. If it exceeds the threshold, the trigger is set. These two are good at catching the kind of sound made when a person is talking, futzing on the desk, or typing.

C#

static double AudioThresh  = 0.8;
static double AudioThresh2 = 0.09;

static void waveIn_DataAvailable(object sender, WaveInEventArgs e)
{
  bool Tr = false;
  double Sum2  = 0;
  int Count = e.BytesRecorded / 2;
  for (int index = 0; index < e.BytesRecorded; index += 2)
  {
     double Tmp = (short)((e.Buffer[index + 1] << 8) | e.Buffer[index + 0]);
     Tmp /= 32768.0;
     Sum2 += Tmp*Tmp;
     if (Tmp > AudioThresh)
       Tr = true;
  }
  Sum2 /= Count;

  // If the Mean-Square is greater than a threshold, set a flag to indicate that noise has happened
  if (Tr || Sum2 > AudioThresh2)
    Interlocked.Exchange(ref AudioTrigger, 1);
}

This sets a flag – AudioTrigger – which will be seen (and reset) in the main loop, with a bit of code that looks like:

C#

while (0 == Shutdown)
{
    // Do some lovely events
    Application . DoEvents();
    Thread . Sleep (20);

    //… do some other stuff ..

    // Check for a kick from the sound system
    if (0 != AudioTrigger)
    {
        Interlocked . Exchange(ref AudioTrigger, 0);
        …
        ScreenShaker . Screens();
        continue;
    }
    //… more checks for the user idle trigger…
}

Waiting for the user to be idle

The check to see if the user is not doing anything, the main loop uses a helper method called LastInputTime(), which returns the number of milliseconds the user did any input activity. The loop ensures that the user has been idle for long enough, and then calls the method that selects random gremlin actions:

C#

uint LastTime = Win32.LastInputTime();
if ((uint) Environment.TickCount < IdleTimeTrigger + LastTime)
    continue;

The LastInputTime() helper method uses a p/invoke call to the GetLastInputInfo() API call. The wiki at Pinvoke.net provides the signature to the procedure, and a suitable structure that we need to pass to it:

C#

[DllImport("User32.dll")]
static extern bool GetLastInputInfo(ref LASTINPUTINFO LastInfo);

[StructLayout(LayoutKind.Sequential)]
struct LASTINPUTINFO
{
  public uint cbSize;

  /// <summary>
  /// Number of system tickes
  /// </summary>
  public uint dwTime;
}

Then, these two structures are wrapped up into the LastInputTime() helper procedure, which does the dirty work of allocating and initializing a structure.

C#

internal static uint LastInputTime()
{
  LASTINPUTINFO lastInput=new LASTINPUTINFO();
  lastInput.cbSize = (uint)Marshal.SizeOf(lastInput);
  GetLastInputInfo(ref lastInput);
  return lastInput . dwTime;
}

Conclusion

The tricks gremlin plays are intended to be somewhat subtle. There is an aggressive option to make it react more, and move windows around more visibly.

If you want to try this out, the download link for the executable and source code are at the top of the article!

Resources and References

This article is a blatant grab bag of experimental and reused code from earlier experiments and other demo programs. Below are some of the projects I lifted code examples from:

• Pinvoke.net – a wiki-style site with many useful bits of example code related to calling the Windows API from within .NET.
• “.NET Audio Recorder” by Mark Heath, from which I stole code to make the audio trigger.
• “Possessed PC Pranks for Halloween” by Brian Peek, from which I stole the basic code for sending keystrokes.
• “April Fools' Day Application” by Brian Peek, from which I stole code to grab the screen and rotate it.

About The Author

Randall Maas writes firmware for medical devices, and consults in embedded software. Before that he did a lot of other things… like everyone else in the software industry. You can contact him at randym@acm.org.

Skype: I find your lack of feedback disturbing.

This article will discuss how to build a software tool that makes it easier to talk into Skype with a headset.

Recently my family began experimenting with Skype, but we found that talking over Skype can be exhausting with headphones. After doing a little research, I learned that the problem was that we only heard the other party. You'd think that this is a good thing, but we have a social brain that works hard to gauge our behavior and adjust.

The answer is to feed a little bit of the microphone back into the earpiece, so that our brain knows how loud we're talking. This is called side tone in the telecommunication industry. Without it, we talk louder and louder until we're sure that we'll be heard.

I couldn't find a Skype plug-in to do this… but I had just read a Coding4Fun article, by Mark Heath, about adding audio effects to Skype. I decided I was going to write a tool to add this feedback.

How to use the tool

First, let's take a look at how to use the application. Once it is running, there will be a microphone icon in the lower right hand corner of the screen:

Figure 1: Icon in the system notification tray

Clicking on it, will get the application control window:

Figure 2: The application's controls

Let's look at the side tone controls:

• The “Side Tones” check box enables or disables the side tone. If you have headphones, you'll want this on; if you have loud speakers, you'll want it off.
• The slider controls the volume of the microphone in the headset. You can change it while talking. The volume will vary with microphone and headset.

You can do a “sound check” to see if the feedback is working by clicking on the Sound Check button, and adjusting the volume. I found that the volume setting that works best in a conversation is much, much lower than what works in a sound check.

Next, let's look at the AGC (“Automatic Gain Control”) section. When making a call, the software can automatically adjust how loud you sound to the other party:

• When “Low Pass” is checked, the headphone feedback is at 44100 samples/sec. The microphone sound is filtered to keep only the sounds below 8 Khz, converted to 16000/samples per second and sent to Skype. When it is not checked, headphone feedback is at 16000 samples/sec, and sent to Skype without the low pass filter.
• When "Skype AutoGain" is checked, it signals to Skype that Skype can use its own algorithm. When clear, Skype is told not to apply any adjustments.
• When "AutoGain" is checked, the custom Automatic Gain Control algorithm is used.

The Automatic Gain Control has three sliders:

• The “Cutoff” slider controls the distinction between background noise and conversation. Sound below this level is cutoff and silence is sent to Skype. This will vary with microphone – more sensitive (expensive) microphones will pick more background noise and be better with a higher setting.
• The “Normal” slider controls the volume when you are talking normally. The Gain Control tries to raise the volume to this level.
• The “Loud” slider controls the volume when you talk exceptionally loud. This rarely happens, but when you do talk louder than the Normal level, the Gain Control tries to adjust the volume to this level.

How the Software Works

I originally started this project by creating an effect for the Skype Voice Changer. My plan was to open a WaveStream, begin playing it on the headphones and copy the microphone stream to it.

I quickly found that this was not the way to add feedback. The underlying “WaveOut” system had a huge latency: Everything I heard in the headphones was at least a second or more behind what I was saying. This made it even harder to talk than before, and I had to abandon it.

While researching the problem I found a DirectSound code sample that I could modify into doing what I wanted. (This initial prototype didn't coordinate with Skype – it attached to the microphone and copied the sounds to the output, at a lower volume. But it was on always on.) The sound in the headphones is still slightly behind (the microphone) but is barely perceptible, and we'll muffle it a bit more to make it less distinguishable.

From here on I shall describe the major – or technically interesting – components of the program. We'll look at:

1. DirectSound and Circular buffers
2. Sample Window and Sizing the buffers
3. Using WaitHandle's to Synchronize with DirectSound
4. IIR Filters
5. Automatic Gain Control
6. Estimating loudness

Note: I won't be describing how to connect to Skype. Mark Heath's description is very good.

DirectSound and Circular Buffers

The DirectSound code is in AudioLoop.cs. The module sets DirectSound to capture sound from the default recording device at 16bits / sample at either 16000 or 44100 samples per second. The capture and playback buffers are configured in “looping” mode to act as circular buffers. The capture buffer eventually overwrites samples, and we'll lose them if we don't act fast enough; if we don't update the playback buffer, it will repeat the same sound over and over.

The StartMicrophone() procedure sets up the capture and playback buffers. Then it creates a thread to do the work. The thread is at a high priority so that if the OS has a choice between (say) email or processing sound, it does the sound.

The StopMicrophone() procedure stops the worker thread and cleans up the resources.

The software processes a fixed number of samples at a time, called the “sample window.” The buffers are several times the size of sample window, so that the system can keep capturing and playing while the software is processing them. The sound processing loop is the heart of the application:

1. Wait for the next sample window to be ready
2. Copy the samples from the capture buffer
3. Process the samples and send the results to the playback buffer
4. Process the samples and sends the results to Skype

C#

while (Runnable)
{
    for (int I = 0, offset = 0; Runnable && I < _bufferPositions; I++, offset += bufferSize)
    {
        // Wait for the sample areas to be ready
        notificationEvent[I].WaitOne(Timeout.Infinite, true);

        // Get the sound samples
        byte[] buffer = (byte[]) captureBuffer.Read(offset, typeof (byte), LockFlag.None, bufferSize);

        // Convert samples to 16bit PCM 
        for (int L = buffer.Length, J = 10, K = 0; K < L; K += 2)
            PCM16Buffer[J++] = (Int16) ((UInt16) buffer[K] | (((UInt16) buffer[K + 1]) << 8));

        // Play them out to the ear, if applicable
        if (null != playbackBuffer)
        {
            // Perform a low pass filter to "muffle" the sound
            Butterworth(PCM16Buffer, 10, LPSample, Coefs);

            // put the muffled sample into the output buffer
            // -- The lock flag seems to work, but others may work too
            playbackBuffer.Write(Idx, LPSample, LockFlag.None);
            Idx += buffer.Length;
            if (Idx >= 4*bufferSize)
                Idx -= 4*bufferSize;
            if (!playing)
            {
                playbackBuffer.Volume = _Volume;
                playbackBuffer.Play(0, BufferPlayFlags.Looping);
                playing = true;
            }
        }

        // Process the sound and deliver it to Skype
        if (null != outStream)
        {
            int L = AGC.Process(PCM16Buffer, 10, buffer);
            if (0 != L)
                outStream.BeginSend(buffer, 0, L, SocketFlags.None, SendCallback, null);
            // Note: could send out pink noise if L == 0
        }

        // Move the sliding window of the previous 10 samples into the start
        // of the PCM16Buffer
        for (int K = 0, J = PCM16Buffer.Length - 10; K < 10; K++, J++)
            PCM16Buffer[K] = PCM16Buffer[J];
    }
}

Note the “for” loop at the bottom of the code. This preserves the last 10 incoming samples at the start of the buffer. This is needed to make the sound processing smooth, and will be discussed a bit later.

Sample Window and Sizing the Buffers

How big should the sample window be? This is bit of a trade off in responsiveness and design complexity.

I chose a window big enough to hold 10 milliseconds of sound. Since the ear is sensitive sound to delays of even 30 milliseconds, I cut this done so that a delay wouldn't be perceptible. (When I tried a 50 millisecond window, my voice came out the headphone sounding like an echo... and I found myself talking slower and slower.) The sample window could be made smaller, but I am sure that there is a point where the OS won't schedule the audio loop to wake-n-run more frequently. And, as the sample window gets smaller, the processing may drop in quality, because it doesn't have enough to work with.

The capture buffer is 8 times the size of the sample window. This ratio is arbitrary, but I wanted the buffer to be about an order of magnitude larger. My rationale is that if the processing falls behind, the sound – for the Skype call – won't be dropped. I feel that it is more important to preserve sound quality for the other party than to preserve the quality of feedback.

The playback buffer is four times the sample window. I wanted it small, so that if the processing fell behind, the replaying of a sound will seem to be a continuation of a current sound.

When writing the sound to the playback buffer, we have to track where in the buffer to put the samples. I tried to use GetCurrentPosition() to find where to write to next into the playback buffer; this created terrible sound. Instead, the software uses a local variable to track where to write next.

DirectSound and Notifications

How do we keep in sync with the sound capture – how do we know when a sample buffer is ready?

The application gives a table of buffer indices and WaitHandle's to DirectSound. When the capture buffer's write index reaches one of those indices, it signals the corresponding WaitHandle. The worker thread cycles performs a WaitOne() one each of the WaitHandle's, one at a time. As a convenience, we use a specific kind of WaitHandle called AutoResetEvent. This type of WaitHandle sets itself back to a “wait” state once WaitOne() returns.

If the thread has gotten behind, the WaitOne will return immediately, the loop processes the sample, and begins to catch up with the work.

We must use a separate AutoResetEvent for each of the 8 capture windows. The AutoResetEvent doesn't tell us if it was signaled multiple times. If only one AutoResetEvent handle were used, it wouldn't know that two (or more) sample windows were ready. Instead, it would process just one, falling further behind, adding latency. This would happen randomly overtime, and be hard to test consistently.

IIR: Infinite Impulse Response Filters

This project came together so quickly, so easy – once I found the right approach – that I couldn't resist getting fancy. I added a low-pass filter to muffle the feedback a little. And I added automatic gain control, as an experimental option.

For both of these I used a filtering algorithm called “IIR” (this stands for Infinite Impulse Response – but that term is a confusing mouthful, so let's just call it IIR). IIR is a special purpose virtual machine. Low-pass filters, high-pass filters, combinations of those filters, and even equalizers, can be specified, and use very specific techniques (like a compiler) to convert them into an IIR implementation.

(You could, instead, “compile” the filters to be the resistor values to use in a hardware circuit. That's programming in solder!)

The machine code for these IIR virtual machines is just two list of coefficients, called A & B. The software emulator is code that looks like the following bit of code:

C#

Out[0] = Sample[0];
Out[1] = Sample[1];
for (int Idx = 2; Idx < N; Idx++)
{
    Out[Idx] =
    B0 * Sample[Idx]
 + B1 * Sample[Idx - 1]
 + B2 * Sample[Idx - 2]
        // … more like this …
        // Next, the feed back
 - A1 * Out[Idx - 1]
 - A2 * Out[Idx - 2]
        // … more like this …
 ;
}

IIRs are easy to implement - and take less CPU power than other methods. But sometimes they sound poor; if they sound too bad, you'll want to use a different technique. I found that the low-pass filters in this project work will for some microphones, and add a slight crackle to others.

Example Low Pass Filter

For the low pass filter to create the muffling, I used a Butterworth filter, using the code below. It takes a buffer of signed, 16-bit samples, and then converts the 16-bit values into a byte array suitable for the sound buffer.

The filter code is a bit different than the example code in the previous section. Most of the differences are for speed.

• This code doesn't use a buffer for the old values, instead uses separate variables for the elements of the buffer.  It uses I_0,I_1,I_2 instead of Sample[Idx], Sample[Idx-1], and Sample[Idx-2]. It also uses O_1, and O_2 instead of Out[Idx-1] and Out[Idx-2].
• The A and B coefficients are put into a single array. It also adds two of the sample input values, and is missing a coefficient; this is because the B0 and B2 coefficients are always the same for this kind of filter.

There is one difference that is not for speed. These are tricks done to make the filter smooth, and needed because the sample window is so small. They preserve the state of filter. If we didn't preserve them, the filter would be starting and stopping so frequently that it would add distracting clicks to the output sound. The filters performance would be weakened, because the sample window isn't big enough to hold sounds lower than (about) 200 Hz. Preserving these values, the filter isn't starting and stopping, and doesn't really know about the sample window. All of the IIR filters in this program use similar techniques.

• O_1, O_2 are explicitly preserved across calls by being stored in class variables
• I_1 and I_2 are preserved by the audio loop (remember the warning about preserving 10 samples at the bottom of the loop?) The audio loop preserves the last 10 samples at the start of InBuffer. When this procedure is called, it retrieves the last two samples.

C#

static double O_1 = 0.0, O_2=0.0;

static void Butterworth(Int16[] InBuffer, int Ofs, byte[] OutBuffer, double[] Coefs)
{
  double C0=Coefs[0], C1=Coefs[1], C2 = Coefs[2], C3=Coefs[3];
  double I_1=InBuffer[Ofs-1], I_2= InBuffer[Ofs-2];
  for (int L = InBuffer . Length, J=0, I = Ofs; I < L; I++)
  {
     double I_0 = InBuffer[I];

     // Filter the samples
     double A = (I_0 + I_2) * C0 + I_1 * C1;
     I_2 = I_1;
     I_1 = I_0;

     A = A - O_1 * C2 - O_2 * C3;
     O_2 = O_1;
     O_1 = A;

     // Convert it back to 16 bit
     Int16 S;
     if (A < -32767) S = -32767;
     if (A > 32767)  S = 32767;
     else S = (Int16) A;

     // Store it
     OutBuffer[J++] = (byte)(S & 0xFF);
     OutBuffer[J++] = (byte)(((UInt16)S >> 8) & 0xFF);
  }
}

Automatic Gain Control

I decided next to tackle a problem where my wife's voice did not carry well on calls. This happens a lot to her with cell phones – and answering machines. I was pretty sure that the problem was poor automatic-gain-control (AGC). The typical amplifier in a headset (and in Skype) estimates how loud our voice is, then increases – or decreases – the volume to a reasonable level. It was deciding that my wife's voice was background noise, and cutting her off.

I chose to write an alternate gain control that amplified the sound and passed it to Skype. That way we'd have four to choose from: The one built into the Microphone, the Soundcard's, Mine, and Skype's. (To be fair, these automatic gain controls work well in most cases).

The main portion of the gain control is implemented in the file GainControl.cs. The control algorithm is:

1. Calculate (or estimate) how loud we are currently talking (using the Analyze() procedure)
2. If the loudness is very low, no one is talking… so set the output to zero. (Without this step, the volume of noise and hum would be cranked up)
3. Otherwise, compute the guess-gain by dividing how loud the sound of our voice should be by how loud it currently is
4. Compute the gain (called MaxGain) at which the sound will start clipping. If the guess-gain is louder than this, reduce it to MaxGain.
5. The software adjusts the gain for a gentle transition – especially in the case when we go from absolute quiet, to the start of talking. It does this by tracking the gain (called PrevGain) used in previous sample and the current one.
6. Multiple all the samples by this gain value.
7. If the sample rate is greater than 16000 samples/sec
   1. Do a low pass filter at 8 khz (again in IIR form). This helps prevent artifacts from down sampling
   2. Resample the sound to 16000

The portion of code that calculates the gain looks like (CutOff_dB, LowGain_dB, and TgtGain_dB are the three slider values):

C#

if (!AutoGain)
    Gain = 1.0;
else
{
    double MaxGain;
    double dB = Analyze(InBuffer, Ofs, out MaxGain);

    if (dB < CutOff_dB)
        Gain = 0.0;
    else if (dB < LowGain_dB + 4.0)
    {
        Gain = Math.Exp((LowGain_dB - dB) * db2log);
    }
    else
    {
        Gain = Math.Exp((TgtGain_dB - dB) * db2log);
    }
    Gain = (0.4 * Gain + 0.6 * PrevGain);
    if (Gain > MaxGain)
        Gain = MaxGain;
    PrevGain = Gain;
}

// Skip further process if there is silence
if (0.0 == Gain)
{
    return 0;
}

If you look at the code, you'll see that we don't compare directly with LowGain_db; rather we compare the estimate volume with LowGain_db+4. This gives a little “hysteresis” – if we raise our voice momentarily, the software won't suddenly make it the highest possible volume. Instead, the software lowers the volume a little bit.

When the software changes the sample rate, it basically needs to know how many input samples to skip. At the start of a call, the software computes this, calling it InInc:

C#

// Calculate how we resample to 16Khz
InInc = (int)(1024.0 * SampleRate / 16000.0);

The process of applying the gain adjustment, performing a low pass filter and re-sampling is below:

C#

int NumSamples = InBuffer.Length;
int End = OutBuffer.Length;
int NextIdxForOut = -InInc;
int OutIdx = 0;
for (int I = Ofs; OutIdx < End && I < NumSamples; I++)
{
    // Retrieve the sample
    Int16 S = InBuffer[I];
    double I_0 = S;

    // Apply Gain
    I_0 *= Gain;

    // 8khz low pass filter 
    if (DoLP)
    {
        // Simple Butterworth 8KHz low-pass filter
        double A = (I_0 + DS_I_2) * LP[0] + DS_I_1 * LP[1];
        DS_I_2 = DS_I_1;
        DS_I_1 = I_0;
        I_0 = A - DS_O_1 * LP[2] - DS_O_2 * LP[3];
        DS_O_2 = DS_O_1;
        DS_O_1 = I_0;

        // Change sample rate
        int Tmp = NextIdxForOut / 1024;
        if (I < Tmp)
            continue;
        NextIdxForOut += InInc;
    }

    // Convert it back to 16 bit
    S = (Int16)I_0;

    // Store it
    OutBuffer[OutIdx++] = (byte)(S & 0xFF);
    OutBuffer[OutIdx++] = (byte)(((UInt16)S >> 8) & 0xFF);
}

Estimating Volume

How loud “it should be” is controlled by a slider on the screen. The software estimates how loud the sound is by using an algorithm devised by David Robinson that takes into account how it sounds to a person. This way we can increase the gain on hard to hear sounds, and reduce the gain on sounds that a person is very sensitive to.

The loudness estimator, implemented in the Analysis procedure in GainAnalysis.cs, uses the following algorithm:

1. Uses a combination of two (IIR) filters to make the sound to reflect how our ears hear it.
2. Compute the Mean-Square value of the filtered sound (called MS)
3. Track the last 750 ms of these values.
4. Make a sorted copy of these.
5. Retrieve the first non-zero value at least 95% of the way into the buffer. This is so that we don't take the loudest sample and assume that is how the person is talking.
6. If MS (the value computed in step 2) is much quieter, use that one instead
7. Convert this value into dB scale by performing a logarithm on it.

The code to “normalize” the sound into how a person hears it is below. Along the way it computes the square of the samples (used in step 2). Like the earlier IIR filters, these preserves their variables across the calls. The first IIR is a yulewalk filter, but it preserves it old intermediary values in an array. Like the trick in AudioLoop, where we copy the last 10 samples into the start of the current buffer, the analysis procedure copies the last 10 immediate values into the start of YuleTmp array.

The output of the yulewalk filter is feed into a 150Hz high pass filter. It is essentially the same as the low-pass filter described earlier.

C#

for (int L = Samples.Length, N = Ofs; N < L; N++)
{
    int _V = Samples[N];
    double V = _V;
    if (_V > MaxSample) MaxSample = _V;
    if (-_V > MaxSample) MaxSample = -_V;

    // Perform yulewalk filter
    double S = V * YuleCoefs[0];
    for (int J = N - 1, I = 1; I < 11; I++, J--)
        S += Samples[J] * YuleCoefs[I];
    for (int J = N - 1, I = 11; I < 21; I++, J--)
        S -= YuleTmp[J] * YuleCoefs[I];

    // Store for the feedback into the next stage of the yule walk
    YuleTmp[N] = S;


    // Perform butterworth high-pass filter stage, using S as an input
    double Accum =
       (S + GA_I_2) * HPCoefs[0]
       + GA_I_1 * HPCoefs[1];
    GA_I_2 = GA_I_1;
    GA_I_1 = S;
    Accum = Accum - GA_O_1 * HPCoefs[2] - GA_O_2 * HPCoefs[3];

    GA_O_2 = GA_O_1;
    GA_O_1 = Accum;

    // The square of the filtered results
    Sum += Accum * Accum;
}

// Copy the intermediate yulewalk state for the next
// (this is needed since we are looking at a fairly small time window)
for (int I = 0, J = YuleTmp.Length - 10; I < 10; I++)
    YuleTmp[I] = YuleTmp[J];

The mean-squared is computed:

C#

// The mean square of the filtered results
double MS = Sum / NumSamples;

Tracking the last 750ms of samples is a simple matter of putting it into a circular queue:

C#

MSQueue[QIdx++] = MS;
if (QIdx >= MSQueue . Length)
   QIdx = 0;

Next, is the code to finding the first non-zero value 95% of the way into the buffer. It is a straightforward copy-the-array, sort it, and fetch:

C#

Array.Copy(MSQueue, SortedQ, MSQueue.Length);
Array.Sort(SortedQ);

// Return the 95% 
double X = SortedQ[Q95Idx]; 
for (int I = Q95Idx +1; X < 400.0 && I < SortedQ . Length; I++)
      X = SortedQ[I];

Next, override this value if the current sample window is very, very quiet – that is, the user stopped talking. (If we don't do this, we'll amplify background noise between words)

C#

if (MS < X * 0.40 && MS < 12800.0)
   X = MS;

Finally, convert the result into decibels (or a reasonable approximation of a decibel)

C#

return 10.0 * Math . Log10 (X * 0.5 + double . Epsilon);

Note: The logarithm function takes a positive, non-zero floating point number. However, the value we pass to it can be zero; if we pass a zero, though, the Logarithm function would return a bad value. The simplest thing to do is check to see if the value we are passing is “zero” and not call Logarithm. However, I learned a long time ago to just add “epsilon” to the value to whatever we pass. This can really improve performance on number crunching.

Conclusion

This concludes how to add a little bit feedback and fancy amplification to you Skype phone calls.

If you want to try this out, the download link for the source code is at the top of the article!

If you'd like to experiment further, here are ideas of what can be done:

1. DirectSound has echo cancellation and noise suppression. These seem desirable, esp. if you wanted to try making your own speakerphone. I was not able to get them to work and I would love to learn how.
2. I'm sure it is possible to trim even more latency off of the side-tone playback, and I would be interested in learning better techniques to do so.
3. Another would be to create the ideal equalizer from Robinson's Equal Loudness model, and use that filter and amplify the sound.
4. It might be useful for the other party to control the settings, so that they decide when your voice has the right volume.

Resources and References

• David Robinson (while doing his PhD at University of Essex on human hearing). His algorithm is intended to find the overall loudness of an album. He includes MATLAB scripts to compute coefficients at custom sample rates, as well as some pre-computed coefficients.
• Mark Heath's Skype Voice Changer
• Derek Pierson's DirectX tutorial

About The Author

Randall Maas writes firmware for medical devices, and consults in embedded software. Before that he did a lot of other things… like everyone else in the software industry. You can contact him at randym@acm.org.