So what takes care of a bad-pitch day? Autotune—an effect that corrects the pitch of your voice so you'll never again sing out of tune. And now, with the power of modern microprocessors, autotune is possible in real-time, allowing singers to benefit from its almost magical powers during live concerts.

The company most famous for its autotune effect is Antares. Antares Auto-Tune currently retails for $249, and a stripped down version is available for $100. In addition to simply improving the pitch of a dodgy singer, autotune can be used to create unique robotic sounding vocal effects, a technique massively popular in recent years thanks to its use by artists such as T-Pain and the group behind the “Auto-Tune the News” YouTube videos. In 1998, when the effect was first used on Cher's “Believe” single, the producer used such extreme settings that instead of subtly adjusting the pitch, autotune “snapped” instantaneously to the nearest “correct” note.

Here is a nerdy example of what Autotune can do.

How does Autotune work?

An autotune effect has two parts. The first is pitch detection, which calculates the dominant frequency of the incoming signal, and is the reason autotune is normally used on monophonic audio sources (i.e. playing one note at a time, not whole chords). So, if your guitar is out of tune, you're out of luck (Celemony's Melodyne product, however, features some incredible capabilities for pitch-shifting polyphonic audio).

The second stage is pitch shifting, or “correcting” a given note. However, the bigger the pitch shift required, the more artificial the end result will be, and it is worth noting that absolutely perfect pitch is not always desirable. Sometimes the blended notes resulting from vibrato, for example, are an important part of the performance, and eliminating them would be detrimental.

Creating a .NET Autotune Algorithm

For this project, we will be creating an autotune effect for .NET. Serious recording enthusiasts should just go out and buy a decent autotune effect, but to have a little fun, we'll see if we can make an autotune that can give us a poor man's Cher effect (or T-Pain effect, if you prefer).

To get started, I searched to see if there were some pre-existing open source autotune implementations, which brought me to awesomebox, a project created by Ravi Parikh and Keegan Poppen while they were students at Stanford University. They kindly gave me permission to make use of their code, which uses an auto-correlator for pitch detection and an open source pitch-shifting algorithm from audio DSP expert Stephan M. Bernsee.

Porting C++ to C#

Although porting from C/C++ to C# is not exactly fun, there is enough similarity in the syntax that it is possible to complete without too many changes. You need to remember, however, that a long in C is an int in C# (i.e. 32 bits long not 64).

Additionally, the C# compiler is fussier than C/C++ when it comes to casting between floats, doubles, and ints. Putting the “f” suffix on numeric literals sorts out most of these compiler errors.

Pointers can be a pain. I tend to replace them with integer variables used to index into an array. You can of course use unsafe code, but that limits your options if you plan to port to Silverlight or Windows Phone 7 at a later date, neither of which allow unsafe code or interop into unmanaged code. The necessary mathematical functions are available in the System.Math class.

To see an example, compare this pitch shifting C++ source file with my C# conversion of it.

Capturing Audio with NAudio

Interop wrappers for the NAudio Windows WaveIn APIs capture the audio. Here is the code used to start recording:

c#:

waveIn = new WaveIn();
waveIn.DeviceNumber = recordingDevice;
waveIn.DataAvailable += waveIn_DataAvailable;
waveIn.RecordingStopped += new EventHandler(waveIn_RecordingStopped);
waveIn.WaveFormat = recordingFormat;
waveIn.StartRecording();

VB.Net:

waveIn = New WaveIn
waveIn.DeviceNumber = recordingDevice
AddHandler waveIn.DataAvailable, AddressOf waveIn_DataAvailable
AddHandler waveIn.RecordingStopped, AddressOf waveIn_RecordingStopped
waveIn.WaveFormat = _recordingFormat
waveIn.StartRecording()

The steps are:

1. Create a new WaveIn device
2. [Optional] set up the device number (0 for default recording device)
3. Add a handler to the DataAvailable event—this is where we will receive the raw audio data
4. Add a handler for the RecordingStopped event. This allows us to close the temporary WAV file we created
5. Set up the recording format. For this project we are going to record in mono (i.e. one channel), 16 bit, 44.1kHz audio—the default setting for most microphones
6. Call the StartRecording method

Whenever the soundcard reports a new buffer of recorded audio, we receive it in the DataAvailable event handler:

c#:

void waveIn_DataAvailable(object sender, WaveInEventArgs e)
{
    byte[] buffer = e.Buffer;
    int bytesRecorded = e.BytesRecorded;
    WriteToFile(buffer, bytesRecorded);

    for (int index = 0; index < e.BytesRecorded; index += 2)
    {
        short sample = (short)((buffer[index + 1] << 8) |
                                buffer[index + 0]);
        float sample32 = sample / 32768f;
        sampleAggregator.Add(sample32);
    }
}

VB.Net

Private Sub waveIn_DataAvailable(ByVal sender As Object, ByVal e As WaveInEventArgs)
    Dim buffer() = e.Buffer
    Dim bytesRecorded = e.BytesRecorded
    WriteToFile(buffer, bytesRecorded)

    For index = 0 To e.BytesRecorded - 1 Step 2
        Dim sample = CShort(buffer(index + 1)) << 8 Or CShort(buffer(index + 0))
        Dim sample32 = sample / 32768.0F
        _sampleAggregator.Add(sample32)
    Next index
End Sub

The WaveInEventArgs contains the number of bytes recorded (e.BytesRecorded) and a pointer to the buffer containing those bytes (e.Buffer). The handler does two things with the recorded data. First, it calls WriteToFile, which uses the WaveFileWriter class from NAudio to write the data to disk:

c#:

// before we start recording, set up a WaveFileWriter...
writer = new WaveFileWriter(waveFileName, recordingFormat);

// ... every block we receive we write it to the WaveFileWriter:
writer.WriteData(buffer, 0, bytesRecorded);

// ... and when recording stops we must call Dispose to finalize the
// .WAV file properly
writer.Dispose()

VB.Net:

writer = New WaveFileWriter(waveFileName, _recordingFormat)
writer.WriteData(buffer, 0, bytesRecorded)
writer.Dispose()

Converting Audio to Floating Point and Back Again

Once recording has completed, we have a WAV file on which to perform our autotune effect. However, our WAV file consists of 16 bit samples (i.e. System.Int16 aka short). In other words, we have a sequence of byte pairs, each of which represent a number in the range -32768 to 32767. For the digital signal processing we will be performing, it's best to have a sequence of floating point numbers (System.Single or float) in the range -1.0f to 1.0f. This is a common requirement, so NAudio provides a utility class to convert audio from short to float called Wave16ToFloatProvider. Here's the code that takes a WAV file and implements the autotune algorithm on it:

c#:

public static void ApplyAutoTune(string fileToProcess, string tempFile, AutoTuneSettings autotuneSettings)
{
    using (WaveFileReader reader = new WaveFileReader(fileToProcess))
    {
        IWaveProvider stream32 = new Wave16toFloatProvider(reader);
        IWaveProvider streamEffect = new AutoTuneWaveProvider(stream32, autotuneSettings);
        IWaveProvider stream16 = new WaveFloatTo16Provider(streamEffect);
        using (WaveFileWriter converted = new WaveFileWriter(tempFile, stream16.WaveFormat))
        {
            // buffer length needs to be a power of 2 for FFT to work nicely
            // however, make the buffer too long and pitches aren't detected fast enough
            // successful buffer sizes: 8192, 4096, 2048, 1024
            // (some pitch detection algorithms need at least 2048)
            byte[] buffer = new byte[8192]; 
            int bytesRead;
            do
            {
                bytesRead = stream16.Read(buffer, 0, buffer.Length);
                converted.WriteData(buffer, 0, bytesRead);
            } while (bytesRead != 0 && converted.Length < reader.Length);
        }
    }
}

VB.Net

Public Shared Sub ApplyAutoTune(ByVal fileToProcess As String,
    ByVal tempFile As String,
    ByVal autotuneSettings As AutoTuneSettings)
    Using reader As New WaveFileReader(fileToProcess)
        Dim stream32 As IWaveProvider = New Wave16ToFloatProvider(reader)
        Dim streamEffect As IWaveProvider = New AutoTuneWaveProvider(stream32, autotuneSettings)
        Dim stream16 As IWaveProvider = New WaveFloatTo16Provider(streamEffect)
        Using converted As New WaveFileWriter(tempFile, stream16.WaveFormat)
            ' buffer length needs to be a power of 2 for FFT to work nicely
            ' however, make the buffer too long and pitches aren't detected fast enough
            ' successful buffer sizes: 8192, 4096, 2048, 1024
            ' (some pitch detection algorithms need at least 2048)
            Dim buffer(8191) As Byte
            Dim bytesRead As Integer
            Do
                bytesRead = stream16.Read(buffer, 0, buffer.Length)
                converted.WriteData(buffer, 0, bytesRead)
            Loop While bytesRead <> 0 AndAlso converted.Length < reader.Length
        End Using
    End Using
End Sub

Here's how it works:

1. First we use a WaveFileReader to open the file that we've just created, which contains 16 bit audio samples
2. Then we use the Wave16ToFloatProvider to perform the conversion to floating point samples
3. Next we pass it through our autotune effect (the AutotuneWaveProvider). We'll explain how this works this later
4. Then we use the WaveFloatTo16Provider to convert back to 16 bit samples ready for saving to WAV (we could save a 32 bit WAV, but it would be rather wasteful of disk space)
5. Having set up the audio pipeline, we can read from the WaveFloatTo16Provider and pull audio right through from the WAV file. We need to read in block sizes that are a power of 2, since we are passing the data through FFTs. If we want to read arbitrary block sizes, we need to introduce another element into our pipeline to buffer up enough data to pass through FFTs
6. Finally, we use the WaveFileWriter to write the data we have read into a WAV file

The AutoTuneWaveProvider

As we saw in the last code snippet, the AutoTuneWaveProvider is the piece in our audio pipeline that actually performs the autotune effect. It implements the NAudio IWaveProvider interface, which allows it to be used in the pipeline for real-time playback if necessary, even though our example code is not doing this (see the section on performance later). Here's the AutoTuneWaveProvider constructor:

c#:

public AutoTuneWaveProvider(IWaveProvider source, AutoTuneSettings autoTuneSettings)
{
    this.autoTuneSettings = autoTuneSettings;
    if (source.WaveFormat.SampleRate != 44100)
        throw new ArgumentException("AutoTune only works at 44.1kHz");
    if (source.WaveFormat.Encoding != WaveFormatEncoding.IeeeFloat)
        throw new ArgumentException("AutoTune only works on IEEE floating point audio data");
    if (source.WaveFormat.Channels != 1)
        throw new ArgumentException("AutoTune only works on mono input sources");

    this.source = source;
    this.pitchDetector = new AutoCorrelator(source.WaveFormat.SampleRate);
    this.pitchShifter = new SmbPitchShifter(Settings);
    this.waveBuffer = new WaveBuffer(8192);
}

VB.Net

Public Sub New(ByVal source As IWaveProvider, ByVal autoTuneSettings As AutoTuneSettings)
    Me.autoTuneSettings = autoTuneSettings
    If source.WaveFormat.SampleRate <> 44100 Then
        Throw New ArgumentException("AutoTune only works at 44.1kHz")
    End If
    If source.WaveFormat.Encoding <> WaveFormatEncoding.IeeeFloat Then
        Throw New ArgumentException("AutoTune only works on IEEE floating point audio data")
    End If
    If source.WaveFormat.Channels <> 1 Then
        Throw New ArgumentException("AutoTune only works on mono input sources")
    End If

    Me.source = source
    Me.pitchDetector = New AutoCorrelator(source.WaveFormat.SampleRate)
    ' alternative pitch detector:
    ' Me.pitchDetector = New FftPitchDetector(source.WaveFormat.SampleRate)
    Me.pitchShifter = New SmbPitchShifter(Settings, source.WaveFormat.SampleRate)
    Me.waveBuffer = New WaveBuffer(8192)
End Sub

Some points to notice:

1. We pass in a source IWaveProvider—this is where the data will be coming from
2. We check that the source is in the right format—floating point mono input.
3. We also pass in an AutoTuneSettings object. This not only encapsulates the settings for autotune, it is important if you want to adjust the settings in real-time while the effect is running
4. We then create the two key components of our autotune effect: a pitch detector (which uses an autocorrelator), and a pitch shifter
5. Finally we create a buffer to use for audio processing. This can be a byte[] array, but we use WaveBuffer from NAudio because it uses a clever trick that allows us to cast a byte[] into a float[] without using unsafe code or having to copy all of the data

The key method on any implementation of IWaveProvider is its Read method. This is where the audio consumer, usually the sound card or a WaveFileWriter, asks for data. The data must be supplied as a byte array, and if at all possible you should return exactly the number of bytes you were asked for (if you can't, an extra layer of buffering is usually required, or audio playback will be choppy). Here's our implementation of the Read method:

c#:

public int Read(byte[] buffer, int offset, int count)
{
    if (waveBuffer == null || waveBuffer.MaxSize < count)
    {
        waveBuffer = new WaveBuffer(count);
    }

    int bytesRead = source.Read(waveBuffer, 0, count);

    // the last bit sometimes needs to be rounded up:
    if (bytesRead > 0) bytesRead = count;

    int frames = bytesRead / sizeof(float); 
    float pitch = pitchDetector.DetectPitch(waveBuffer.FloatBuffer, frames);
        
    // an attempt to make it less "warbly" by holding onto the pitch 
    // for at least one more buffer
    if (pitch == 0f && release < maxHold)
    {
        pitch = previousPitch;
        release++;
    }
    else
    {
        this.previousPitch = pitch;
        release = 0;
    }
    
    WaveBuffer outBuffer = new WaveBuffer(buffer);

    pitchShifter.ShiftPitch(waveBuffer.FloatBuffer, pitch, 0.0f, outBuffer.FloatBuffer, frames);

    return frames * 4;
}

VB.Net:

Public Function Read(ByVal buffer() As Byte, ByVal offset As Integer,
                        ByVal count As Integer) As Integer Implements NAudio.Wave.IWaveProvider.Read
    If waveBuffer Is Nothing OrElse waveBuffer.MaxSize < count Then
        waveBuffer = New WaveBuffer(count)
    End If

    Dim bytesRead = source.Read(waveBuffer, 0, count)
    'Debug.Assert(bytesRead = count)

    ' the last bit sometimes needs to be rounded up:
    If bytesRead > 0 Then
        bytesRead = count
    End If

    'pitchsource->getPitches()
    Dim frames = bytesRead \ Len(New Single) ' MRH: was count
    Dim pitch = pitchDetector.DetectPitch(waveBuffer.FloatBuffer, frames)

    ' MRH: an attempt to make it less "warbly" by holding onto the pitch for at least one more buffer
    If pitch = 0.0F AndAlso release < maxHold Then
        pitch = previousPitch
        release += 1
    Else
        Me.previousPitch = pitch
        release = 0
    End If

    Dim midiNoteNumber = 40
    Dim targetPitch = CSng(8.175 * Math.Pow(1.05946309, midiNoteNumber))

    Dim outBuffer As New WaveBuffer(buffer)

    pitchShifter.ShiftPitch(waveBuffer.FloatBuffer, pitch, targetPitch, outBuffer.FloatBuffer, frames)

    Return frames * 4
End Function

Here's what's going on

1. First we need to read from our source (in our case, a WAV file converted to floating point samples)
2. If we get less data than we were expecting, we know that means we're at the end of the file, so we'll just pretend we got a full buffer
3. We then work out how many audio ‘frames' are present (which is the same as the number of samples since this is mono audio). It's floating point audio, so frames equal bytes divided by four
4. We then pass the data through our pitch detector algorithm (see below)
5. Next we use some experimental code to stabilize pitch detection by reporting the previous frequency when no pitch is picked up
6. Finally we pass the data into our pitch shifter, including details of the detected pitch

Now that we've seen the big picture of the AutotuneWaveProvider, let's drill down into its two main components—the pitch detector and pitch shifter.

Pitch Detection with Autocorrelation

The pitch detection part of autotune is vital to getting good results. If it can't accurately detect the input pitch, it will incorrectly calculate how much the pitch needs to be adjusted. However, high quality pitch detection is quite difficult to get right. First of all, the microphone may well pick up background noise. Second, when you sing a into a microphone, the signal consists not only of a single frequency, but also “harmonics” at different frequencies.

The good news is that we need to detect only the primary pitch.

The awesomebox algorithm makes use of “autocorrelation” for its pitch detection, but I made a few small tweaks to how the algorithm is implemented in an attempt to improve its accuracy. Autocorrelation has the advantage of being a relatively quick process. The basic principle is that if a signal is periodic, it will “correlate” well with itself when shifted forward (or backwards) one cycle.

Let's say we are looking to see if the note “Middle C” is being sung. The frequency of Middle C is around 262Hz. If we are sampling at 44.1kHz (which is standard for CD quality audio), then we will expect the signal to repeat at approximately every 168 samples (44100/262). Accordingly, for every sample in the buffer, we calculate the sum of squares of that sample and the sample 168 samples previous. We do this for every possible offset that measures a frequency in the range we want to detect (I am using 85Hz to 300Hz, which is adequate for pitch detecting vocals). The offset with the highest score is the most likely frequency.

Let's have a look at the code for an autocorrelation algorithm, starting with the constructor for the AutoCorrelator class:

c#:

public AutoCorrelator(int sampleRate)
{
    this.sampleRate = (float)sampleRate;
    int minFreq = 85;
    int maxFreq = 255;

    this.maxOffset = sampleRate / minFreq;
    this.minOffset = sampleRate / maxFreq;
}

VB.Net

Public Sub New(ByVal sampleRate As Integer)
    Me.sampleRate = CSng(sampleRate)
    Dim minFreq = 85
    Dim maxFreq = 255

    Me.maxOffset = sampleRate \ minFreq
    Me.minOffset = sampleRate \ maxFreq
End Sub

First of all, we pre-calculate some values based on the minimum and maximum frequencies we are looking for. Remember that lower frequencies are harder to detect than higher frequencies, so don't set minFreq too low. MaxOffset and MinOffset are the maximum and minimum backwards distances we will be seeking while looking for a match.

c#:

public float DetectPitch(float[] buffer, int frames)
{
    if (prevBuffer == null)
    {
        prevBuffer = new float[frames];
    }

    float maxCorr = 0;
    int maxLag = 0;

    // starting with low frequencies, working to higher
    for (int lag = maxOffset; lag >= minOffset; lag--)
    {
        float corr = 0; //  sum of squares
        for (int i = 0; i < frames; i++)
        {
            int oldIndex = i - lag;
            float sample = ((oldIndex < 0) ? prevBuffer[frames + 
            corr += (sample * buffer[i]);
        }
        if (corr > maxCorr)
        {
            maxCorr = corr;
            maxLag = lag;
        }

    }
    for (int n = 0; n < frames; n++)
    { 
        prevBuffer[n] = buffer[n]; 
    }
    float noiseThreshold = frames / 1000f;

    if (maxCorr < noiseThreshold || maxLag == 0) return 0.0f;
    return this.sampleRate / maxLag;
}

VB.Net

Public Function DetectPitch(ByVal buffer() As Single, ByVal frames As Integer) As Single Implements IPitchDetector.DetectPitch
    If prevBuffer Is Nothing Then
        prevBuffer = New Single(frames - 1){}
    End If
    Dim secCor As Single = 0
    Dim secLag = 0

    Dim maxCorr As Single = 0
    Dim maxLag = 0

    ' starting with low frequencies, working to higher
    For lag = maxOffset To minOffset Step -1
        Dim corr As Single = 0 ' this is calculated as the sum of squares
        For i = 0 To frames - 1
            Dim oldIndex = i - lag
            Dim sample = (If(oldIndex < 0, prevBuffer(frames + oldIndex), buffer(oldIndex)))
            corr += (sample * buffer(i))
        Next i
        If corr > maxCorr Then
            maxCorr = corr
            maxLag = lag
        End If
        If corr >= 0.9 * maxCorr Then
            secCor = corr
            secLag = lag
        End If
    Next lag
    For n = 0 To frames - 1
        prevBuffer(n) = buffer(n)
    Next n
    Dim noiseThreshold = frames / 1000.0F
    'Debug.WriteLine(String.Format("Max Corr: {0} ({1}), Sec Corr: {2} ({3})", Me.sampleRate / maxLag, maxCorr, Me.sampleRate / secLag, secCor))
    If maxCorr < noiseThreshold OrElse maxLag = 0 Then
        Return 0.0F
    End If
    'Return 44100.0f / secLag '--works better for singing
    Return Me.sampleRate / maxLag
End Function

A few things to notice:

1. Notice that the audio comes in as an array of floating point numbers. NAudio performs this conversion from 16 bit audio for us by using Wave16ToFloatProvider
2. We store the previous buffer. This allows us to look backwards for correlation
3. We then work through each and every possible integer offset within our range, and calculate a correlation value
4. The correlation is calculated as the sum of squares
5. If it is the largest so far, we store the “lag” value (i.e. number of samples back that we correlated with)
6. Notice that we return 0 (i.e. no frequency detected) if we don't find a strong frequency. This noise threshold may need to be tweaked depending on your input
7. Finally, we convert into a frequency with the formula sampleRate / maxLag

I wrote some unit tests to measure the accuracy of detection with sine waves (which admittedly are the easiest to detect). Here are the results for audio sampled at 44.1kHz:

Test Frequency	Detected Pitch
109.99Hz	108.35Hz
116.53Hz	118.23Hz
123.46Hz	123.18Hz
130.80Hz	129.71Hz
138.58Hz	140.00Hz
146.82Hz	148.48Hz
155.55Hz	154.74Hz
164.80Hz	163.33Hz
174.60Hz	172.94Hz
184.98Hz	183.75Hz
195.98Hz	194.27Hz
207.63Hz	206.07Hz
219.98Hz	219.40Hz
233.06Hz	234.57Hz
246.92Hz	247.75Hz
261.60Hz	256.40Hz
277.16Hz	139.56Hz
293.64Hz	146.03Hz

Notice that the detected frequencies from the final two tests are actually half the correct amount. This doesn't actually matter for our purposes, since this just means the frequency has been detected as one octave below the correct note.

To improve on the accuracy of the autocorrelator's results, there are a couple of things you can do:

• Run a band-pass filter before-hand, to remove any frequencies outside of the desired range
• Combine the results with those obtained from a different technique, such as counting zero-crossings

Pitch Detection with the Fast Fourier Transform

I decided to implement an alternative pitch detection algorithm to see if I could get better results. A different approach is to use the Fast Fourier Transform, which converts signals from the “time domain” into the “frequency domain.”

The basic approach is to take a block of samples (which must be a power of 2 – e.g. 1024), and run the FFT on them. The FFT takes complex numbers as inputs, which for audio signals are entirely real. The implementation I am using expects real and complex parts interleaved for the input buffer. Here's our code setting up fftBuffer with interleaved samples:

c#:

private float[] fftBuffer;
private float[] prevBuffer;

public float DetectPitch(float[] buffer, int inFrames)
{
    Func<int, int, float> window = HammingWindow;
    if (prevBuffer == null)
    {
        prevBuffer = new float[inFrames];
    }
 
    // double frames since we are combining present and previous buffers
    int frames = inFrames * 2;
    if (fftBuffer == null)
    {
        fftBuffer = new float[frames * 2]; // times 2 because it is complex input
    }
 
    for (int n = 0; n < frames; n++)
    {
        if (n < inFrames)
        {
            fftBuffer[n * 2] = prevBuffer[n] * window(n, frames);
            fftBuffer[n * 2 + 1] = 0; // need to clear out as fft modifies buffer
        }
        else
        {
            fftBuffer[n * 2] = buffer[n-inFrames] * window(n, frames);
            fftBuffer[n * 2 + 1] = 0; // need to clear out as fft modifies buffer
        }
    }

VB.Net

Private fftBuffer() As Single
Private prevBuffer() As Single

Public Function DetectPitch(ByVal buffer() As Single,
                            ByVal inFrames As Integer) As Single Implements IPitchDetector.DetectPitch
    Dim window As Func(Of Integer, Integer, Single) = AddressOf HammingWindow
    If prevBuffer Is Nothing Then
        prevBuffer = New Single(inFrames - 1) {}
    End If

    ' double frames since we are combining present and previous buffers
    Dim frames = inFrames * 2
    If fftBuffer Is Nothing Then
        fftBuffer = New Single(frames * 2 - 1) {} ' times 2 because it is complex input
    End If

    For n = 0 To frames - 1
        If n < inFrames Then
            fftBuffer(n * 2) = prevBuffer(n) * window(n, frames)
            fftBuffer(n * 2 + 1) = 0 ' need to clear out as fft modifies buffer
        Else
            fftBuffer(n * 2) = buffer(n - inFrames) * window(n, frames)
            fftBuffer(n * 2 + 1) = 0 ' need to clear out as fft modifies buffer
        End If
    Next n

Notice that we prepend the previous buffer we were passed. This is a common way of increasing the accuracy and resolution of an FFT by using overlapping windows, and can be further extended to store three previous buffers, allowing us to have 75% overlapping windows instead of just the 50% that we have in this example.

For better peak frequency detection, the signal that is passed into the FFT is best pre-processed with a “windowing” function. There are several to choose from, each with its own strengths and weaknesses. I used the Hamming window, which is a fairly common choice:

c#:

private float HammingWindow(int n, int N) 
{
    return 0.54f - 0.46f * (float)Math.Cos((2 * Math.PI * n) / (N - 1));
}

VB.Net

Private Function HammingWindow(ByVal n As Integer, ByVal _N As Integer) As Single
    Return 0.54F - 0.46F * CSng(Math.Cos((2 * Math.PI * n) / (_N - 1)))
End Function

The next step is to pass on our interleaved buffer to the FFT algorithm. I am using Stephan Bernsee's here, though there is an alternative implementation in NAudio that I could have used. Since the same function can be used for an inverse FFT, the -1 parameter means (rather counter-intuitively), do a forwards FFT. It processes the data in place, which is fine since we don't need to keep the contents of the input buffer:

c#:

// assuming frames is a power of 2
SmbPitchShift.smbFft(fftBuffer, frames, -1);

VB.Net

' assuming frames is a power of 2
SmbPitchShift.smbFft(fftBuffer, frames, -1)

Once we have completed the FFT, we are ready to interpret its output. The output of the FFT consists of complex numbers (again real followed by imaginary in our buffer), which represent frequency “bins.”

We start off by calculating the bin size and working out which bins correspond to the range of frequencies we are interested in detecting:

c#:

float binSize = sampleRate / frames;
int minBin = (int)(85 / binSize);
int maxBin = (int)(300 / binSize);

VB.Net

Dim binSize = sampleRate / frames
Dim minBin = CInt(Fix(85 / binSize))
Dim maxBin = CInt(Fix(300 / binSize))

For example, if our sample rate is 44.1kHz and we analyse a block of 1024 samples, then each bin represents 43Hz, which is hardly the granularity we are looking for. To increase resolution, our options are to either sample at a higher rate or analyse a bigger chunk. Our approach is to use overlapping blocks of 8192 samples, as we read 4096 samples each time. This means we have a resolution of around 5Hz, which is much more acceptable.

Now we can calculate the magnitude or “intensity” for each frequency by calculating the sum of squares (strictly we should then take the square root, but we don't need to since we are just looking for the largest value):

c#:

float maxIntensity = 0f;
int maxBinIndex = 0;

for (int bin = minBin; bin <= maxBin; bin++)
{
    float real = fftBuffer[bin * 2];
    float imaginary = fftBuffer[bin * 2 + 1];
    float intensity = real * real + imaginary * imaginary;
    if (intensity > maxIntensity)
    {
        maxIntensity = intensity;
        maxBinIndex = bin;
    }
}

VB.Net

Dim maxIntensity = 0.0F
Dim maxBinIndex = 0
For bin = minBin To maxBin
    Dim real = fftBuffer(bin * 2)
    Dim imaginary = fftBuffer(bin * 2 + 1)
    Dim intensity = real * real + imaginary * imaginary
    If intensity > maxIntensity Then
        maxIntensity = intensity
        maxBinIndex = bin
    End If
Next bin

Since we have identified the bin with the maximum intensity, we can calculate the detected frequency:

c#:

return binSize * maxBinIndex;

VB.Net

Return binSize * maxBinIndex

I don't currently specify a minimum threshold for maxIntensity, but perhaps if it were very low, the FFT pitch detector would return zero to indicate no pitch detected instead of returning an answer that is probably not accurate.

Let's have a look at how the FFT pitch detector does:

Test Frequency	Detected Pitch
109.99Hz	107.67Hz
116.53Hz	118.43Hz
123.46Hz	123.82Hz
130.80Hz	129.20Hz
138.58Hz	139.97Hz
146.82Hz	145.35Hz
155.55Hz	156.12Hz
164.80Hz	166.88Hz
174.60Hz	172.27Hz
184.98Hz	183.03Hz
195.98Hz	193.80Hz
207.63Hz	209.95Hz
219.98Hz	220.72Hz
233.06Hz	231.48Hz
246.92Hz	247.63Hz
261.60Hz	263.78Hz
277.16Hz	274.55Hz
293.64Hz	296.08Hz

As can be seen, it correctly picks out the primary frequencies of the higher notes, but overall it doesn't get that much closer than the autocorrelator, so I've left that as the default algorithm. You, however, can swap in the FFT detector in the code if it works better for the material you are auto-tuning.

There are ways of using the phase information from the FFT output to increase the accuracy of pitch detection even further, but I have left that as an exercise for the reader!

Pitch Shifting

The next step is to determine how much we will shift the pitch. The simplest way to do this is to look for the musical pitch that is closest to the detected pitch. Then, the amount of the shift by is simply the ratio of those two notes.

There are, however, some additional considerations. First, we may want to select a subset of musical notes that are acceptable. For example, only notes in the key of C#, or maybe F# minor pentatonic. This may require a slightly more radical adjustment.

Second, depending on the effect we are after, we may not want to instantaneously jump to the new frequency. The code I am using utilizes a fairly rudimentary “attack” time parameter, allowing you to gradually move to the new frequency.

The actual DSP for the pitch-shifting effect is more or less untouched from Stephan Bernsee's code, and this is because it works really well. The Bernsee's code makes use of the Fast Fourier Transform, plus a bunch of clever mathematics, which I almost understand, but not quite well enough to try and explain here! You're better off reading an article in which the man himself explains how it works.

The class that manages the pitch-shifting algorithm is called SmbPitchShifer and inherits from a PitchShifter base class. This does the bulk of its work in the ShiftPitch function:

c#:

public void ShiftPitch(float[] inputBuff, float inputPitch,
                       float targetPitch, float[] outputBuff, int nFrames)
{
     UpdateSettings();
     detectedPitch = inputPitch;

VB.Net

Public Sub ShiftPitch(ByVal inputBuff() As Single, ByVal inputPitch As Single,
                    ByVal targetPitch As Single, ByVal outputBuff() As Single, ByVal nFrames As Integer)
    UpdateSettings()
    detectedPitch = inputPitch

The inputPitch parameter is set to the frequency detected by the PitchDetector. The targetPitch parameter is currently unused, but will be used to specify the target pitch in real-time when accepting input from, say, a MIDI keyboard. In any case, we call UpdateSettings in order to see if any of the autotune algorithm settings have changed since last time.

Next we calculate the amount we need to shift the pitch shift. A shift factor of 1 means no change. We don't allow the shift factor to go above 2 or below 0.5, since these figures represent a whole octave change:

c#:

float shiftFactor = 1.0f;

if (inputPitch > 0)
{
    shiftFactor = snapFactor(inputPitch);
}

if (shiftFactor > 2.0) shiftFactor = 2.0f;
if (shiftFactor < 0.5) shiftFactor = 0.5f;

VB.Net

Dim shiftFactor = 1.0F

If inputPitch > 0 Then
   shiftFactor = snapFactor(inputPitch)
   shiftFactor += addVibrato(nFrames)
End If

If shiftFactor > 2.0 Then
   shiftFactor = 2.0F
End If
If shiftFactor < 0.5 Then
   shiftFactor = 0.5F
End If

The decision of what the target note is takes place in the snapFactor function:

c#:

protected float snapFactor(float freq)
{
    float previousFrequency = 0.0f;
    float correctedFrequency = 0.0f;
    int previousNote = 0;
    int correctedNote = 0;
    for (int i = 1; i < 120; i++)
    {
        bool endLoop = false;
        foreach (int note in this.settings.AutoPitches)
        {
            if (i % 12 == note)
            {
                previousFrequency = correctedFrequency;
                previousNote = correctedNote;
                correctedFrequency = (float)(8.175 * Math.Pow(1.05946309, (float)i));
                correctedNote = i;
                if (correctedFrequency > freq) { endLoop = true; }
                break;
            }
        }
        if (endLoop)
        {
            break;
        }
    }
    if (correctedFrequency == 0.0) { return 1.0f; }
    int destinationNote = 0;
    double destinationFrequency = 0.0;
    // decide whether we are shifting up or down
    if (correctedFrequency - freq > freq - previousFrequency)
    {
        destinationNote = previousNote;
        destinationFrequency = previousFrequency;
    }
    else
    {
        destinationNote = correctedNote;
        destinationFrequency = correctedFrequency;
    }
    if (destinationNote != currPitch)
    {
        numElapsed = 0;
        currPitch = destinationNote;
    }
    if (attack > numElapsed)
    {
        double n = (destinationFrequency - freq) / attack * numElapsed;
        destinationFrequency = freq + n;
    }
    numElapsed++;
    return (float)(destinationFrequency / freq);
}

VB.Net:

Protected Function snapFactor(ByVal freq As Single) As Single
    Dim previousFrequency = 0.0F
    Dim correctedFrequency = 0.0F
    Dim previousNote = 0
    Dim correctedNote = 0
    For i = 1 To 119
        Dim endLoop = False
        For Each note As Integer In Me.settings.AutoPitches
            If i Mod 12 = note Then
                previousFrequency = correctedFrequency
                previousNote = correctedNote
                correctedFrequency = CSng(8.175 * Math.Pow(1.05946309, CSng(i)))
                correctedNote = i
                If correctedFrequency > freq Then
                    endLoop = True
                End If
                Exit For
            End If
        Next note
        If endLoop Then
            Exit For
        End If
    Next i
    If correctedFrequency = 0.0 Then
        Return 1.0f
    End If
    Dim destinationNote = 0
    Dim destinationFrequency = 0.0
    ' decide whether we are shifting up or down
    If correctedFrequency - freq > freq - previousFrequency Then
        destinationNote = previousNote
        destinationFrequency = previousFrequency
    Else
        destinationNote = correctedNote
        destinationFrequency = correctedFrequency
    End If
    If destinationNote <> currPitch Then
        numElapsed = 0
        currPitch = destinationNote
    End If
    If attack > numElapsed Then
        Dim n = (destinationFrequency - freq) / attack * numElapsed
        destinationFrequency = freq + n
    End If
    numElapsed += 1
    Return CSng(destinationFrequency / freq)
End Function

The way this function works is that it runs through the MIDI notes 0-120 and, if that note is selected as one of the valid pitches we support, we remember the “corrected frequency,” which can be calculated from the MIDI note number with the following formula:

c#:

correctedFrequency = (float)(8.175 * Math.Pow(1.05946309, (float)midiNoteNumber));

VB.Net

correctedFrequency = CSng(8.175 * Math.Pow(1.05946309, CSng(i)))

Obviously, a pitch is likely to fall somewhere in between two valid notes, so we choose the which pitch to correct by determining which one is closest to the detected frequency.

The snapFactor function is also responsible for implementing the attack time parameter. This allows the destinationFrequency to be slowly moved to the target note over the duration of the attack period. Having calculated our shift factor, we are now ready to pass our data on to the actual pitch-shifting algorithm:

c#:

int fftFrameSize = 2048;
int osamp = 8; // 32 is best quality
SmbPitchShift.smbPitchShift(shiftFactor, nFrames, fftFrameSize, osamp, this.sampleRate, inputBuff, outputBuff);

VB.Net

Dim fftFrameSize = 2048
Dim osamp = 8 ' 32 is best quality
SmbPitchShift.smbPitchShift(shiftFactor, nFrames, fftFrameSize, osamp, Me.sampleRate, inputBuff, outputBuff)

The final thing we do in the ShiftPitch function is keep a record of the pitch shifts we have made. These are stored in a queue (maximum of 5000 entries) and are very useful for diagnosing what is going on if you are not getting the results you wanted from the algorithm:

c#:

shiftedPitch = inputPitch * shiftFactor;
updateShifts(detectedPitch, shiftedPitch, this.currPitch);

VB.Net

shiftedPitch = inputPitch * shiftFactor
updateShifts(detectedPitch, shiftedPitch, Me.currPitch)

Performance

Performance, as you might expect in a managed application that has not been extensively optimized, was not good. Using my laptop, I could autotune one minute of audio in about 90 seconds. Obviously, that rules out real-time autotuning. I decided to profile the application to see if there were any quick ways I could improve things.

The profiling tools in Visual Studio revealed that 20% of the time was spent on pitch detection and 80% pitch shifting. Unfortunately, there were not too many options available for optimisation, since further investigation pointed to calls to Math.Sin taking the bulk of the time. Possibly creating lookup tables could save a bit more time.

Fortunately, we have another option for speeding things up. The pitch-shifting algorithm takes an “oversampling” parameter, which by default is set to 32, the highest value. However, we can trade off speed for quality. Setting it to 16 meant that I could autotune a minute of audio in 55ms (on my 2.4GHz Core2Duo laptop) – realtime but only just. Setting it to 8 reduced that down to 36s. The results still sounded reasonable, so I have left it set at 8 in the code.

An alternative way of speeding it up though would be to swap in a different pitch-shifting algorithm. You could start by trying out one I created as part of the Skype Voice Changer project previously featured on Coding4Fun, which is also able to operate in real-time (although I haven't done any quality comparisons).

Creating a Test GUI

Rather than starting from scratch, I decided to build upon .NET Voice Recorder, a WPF application I created for a previous Coding4Fun article. This takes advantage of the NAudio .NET audio library for audio recording and playback capabilities. The GUI has three screens. On the first is the input device used for recording. The second records a short voice clip. And the third allows you to edit a small portion of saved audio.

Here's a screenshot of the second screen showing a recording in progress:

And here's the screen that allows you to trim the recording, preview it, and save it as WAV:

As you can see, I have added a new button allowing access to the autotune effect settings. On this screen, you can select which notes are valid, and you can also adjust the “attack time” if you prefer to not go for the robotic effect. I've included a drop-down menu that automatically selects the appropriate notes from various keys.

When you click “Apply,” the autotune effect is applied (while you wait on a background thread) and then you are returned to the screen, allowing you to play back your recording and see how it sounds. If you'd like, you can then go back and change the autotune settings (or turn it off).

MVVM Light

The original VoiceRecorder application used a MVVM (model-view-viewmodel) architecture for binding data to each view. I have updated it to make use of Laurent Bugnion's excellent MVVM Light library. This removes the need for my own RelayCommand and ViewModelBase classes, and also enables me to replace my ViewManager with a more extensible framework using the event aggregator (“Messenger”) that is included with MVVM light. This allows me to quickly navigate from one view to another by sending out a message on the event aggregator:

c#:

private void NavigateToSaveView()
{
   Messenger.Default.Send(new NavigateMessage(SaveViewModel.ViewName, this.voiceRecorderState));
}

VB.Net

Private Sub NavigateToSaveView()
    Messenger.Default.Send(New NavigateMessage(SaveViewModel.ViewName, Me.voiceRecorderState))
End Sub

Getting the Best out of Autotune

Unfortunately, Autotune is an effect that doesn't always produce the desired result . Obviously, if you want great autotune, you're best off buying a commercial implementation, but here are a few tips for getting the most out of an autotune algorithm:

• Get a good quality recording. Avoid background noise, hum, and too quiet or too loud (distorted) recordings. If you need to sing against a backing track, play it in headphones so the microphone doesn't pick it up.
• Know what key you are singing in. This is where the test application won't help you out, since if you don't know what key you are singing in, you can hardly expect to be able to select the appropriate key from the list. You might even be singing in between two keys! If you have an in-tune musical instrument at hand, play a note to give yourself a starting pitch.
• Choose your scale. The easiest option is to just go with a chromatic scale, which means all 12 notes are valid. However, you can try to make the autotune force you into a monotone. Pentatonic scales are useful for instant gratification. They have five notes in them, and so long as you stick to those five, almost anything you sing will fit in with a backing track in your chosen key.
• Adjust the attack time. An attack time of zero is great for the robot effect. A longer attack time will smooth out transitions.
• Why is it “warbling”? With this autotune algorithm, it is quite common to get a “warbling” effect. This is because it is either changing its mind about what note to pitch shift to (because the pitch detector is not providing stable pitch detection), or because the pitch detector couldn't determine a pitch at all, so your voice isn't being pitch shifted. You can play around with the release slider (and may need to modify the release algorithm) if you want to eliminate warbling.

Taking it further

.NET Voice Recorder is open source and hosted on CodePlex in a Mercurial repository. So what you waiting for? Make a fork and have a go at improving it:

• Improve the pitch detector algorithm. I have already given some suggestions for how this can be done. One idea that might be worth experimenting with is making it “hold” the detected frequency for a short period until a strong, new frequency is detected.
• Display the detected pitches. The autotune effect stores data of the detected pitches as well as the pitches it attempts to convert to. You might display this information to the user, perhaps underneath the waveform, so they can see what it is detecting. (It currently outputs this information with Debug.WriteLine, which is useful for debugging purposes).
• Suggest an appropriate scale? Instead of leaving it to the user to select what key to snap the notes to, how about auto-selecting the detected notes?
• Allow direct input of desired pitch. Instead of letting the autotune effect try to work out what note it should be targeting at any given time, it is far more effective to let the user input what note they would like to shift to. This could be done by entering the note using a MIDI keyboard in real-time, or by drawing the notes in, perhaps on a “piano-roll” control. Or, users could implement a simple, domain-specific language to specify the desired note. For example:
  • 0:00.0 C#
  • 0:01.5 E
  • 0:02.7 G#
• Port it to Windows Phone 7. This would be quite a cool effect to have on your phone. Of course, you might need to optimize performance a bit more to save battery life. And you'll want to put your graphic designer hat on to give it a more beautiful look than it currently has.

About the Author

Mark Heath is the author of several open source .NET applications and libraries, including NAudio and the Skype Voice Changer. He works for NICE Systems, developing applications that search, display, and play back vast amounts of multimedia data. He has a blog, Sound Code, and you can follow him on his sporadically updated Twitter account.

I really should pick up some portable speakers, but the ones I’ve checked out previously are either way too big for true portability, or a small, tinny and gobble up batteries like there’s no tomorrow.

The X Mini Max II may lift my downer on portable speakers. They’re small, and are held together in an egg shape by magnets for portability. There’s a simple 3.5” connector jack for plugging into your MP3 player or laptop, and forget the need for batteries – the X Mini Max II has a rechargable battery that charges via USB, so that’s one less power cable to bring along.

The speakers have a couple of additional tricks that set them apart from the rest. Firstly, you can connect multiple sets of speakers together in daisy-chain formation to spread the sound around the room. Finally, unscrew the speaker and it springs open accordion style to effectively mimic a sub-woofer.

Small speaker, big sound, large bass, no hassle. Love it. There’s a range of speakers on offer in a choice of colours, plus one model with an integrated audio player and SD card slot. That’s another item on my Christmas gift list filled. 

Optional Reading

I cover the basics of this article in a multi-part blog series, which you should check out if you have trouble:

Part 1 - How Audio Data is Represented
Part 2 - Demystifying the WAV Format
Part 3 - Synthesizing Simple WAV Audio Using C#
Part 4 - Algorithms for Different Sound Waves in C#

What's An Oscillator?

An oscillator is a device or application that generates a waveform. In electrical engineering terms, it's a device that outputs an electrical current with varying voltage. If you plot the voltage over time, you get a regular wave in a particular form, such as a sine, square, triangle or sawtooth.

An oscillator is the most basic type of synthesizer. Analog synths use electrical circuits to output a sound wave. Digital synthesizers do the same thing, but with software.

You can create a pretty neat sounding instrument by combining the outputs of multiple oscillators. For example, if you have three oscillators oscillating at a frequency of 440Hz (concert A pitch), but each of them has a different waveform (saw, square, sine) you get a very interesting, layered sound.

But before we get too deep into this subject, let's briefly explore the physics of sound.

The Physics of Sound

Sound happens when air pressure changes on your ear drum. When you clap in an empty room, pressure waves bounce all over the place and dance on your eardrum. The changes in pressure are detected continuously by your ear.

Digitally, “pressure” is referred to by a scalar value called amplitude. The amplitude (loudness) of the wave is measured thousands of times per second (44,100 times per second on CDs). Every measurement of pressure (aka amplitude) is called a sample—CDs are recorded with 44,100 samples per second, each with a value between the minimum and maximum amplitude for the bit depth.

Think about 44,100 samples per second. That's a lot of stuff for your ear to detect. That's how we're able to hear so much stuff going on in the mix of a song, especially in stereo tracks where you have 44,100 samples per second, per ear.

It turns out that there is a horribly intense mathematical theorem which basically tells us that 44,100 samples per second is enough to accurately represent a pitch as high as 22 KHz. The human ear can really hear only up to 20KHz, so a 44.1KHz sampling rate is a more than high-enough sampling rate.

This whole section is expanded in detail on my blog:

Part 1 - How Audio Data is Represented

Terminology

So now you have a rather glancing overview of how sound works, and perhaps some clues as to how we should go about representing it in computers. Let's go over all this new terminology (plus some even newer terms) in delicious, bulleted format:

· Sample: A measurement of a sound wave at a very small point in time. 44,100 of these measurements in a row form a single channel of CD-quality audio.

· Amplitude: The value of a sample. Max and min values are dependent upon the bit depth.

· Bit depth: The number of bits used to represent a sample. 16-bit, 32-bit, etc. Max amplitude is (2^depth) / 2 – 1.

· Sample rate (aka sampling rate, aka bit rate): The number of samples per second of audio. 44,100 is standard for CD-quality audio.

How sound is represented

By now, you've probably surmised that a second of audio data is somehow represented by an array of some integer data type, which has a length of 44,100. You would be correct in that assumption. However, if you want sound to play from a computer's sound card, that data has to be accompanied with a bunch of format information. WAV is probably the easiest format to deal with.

See more in the following article:

Part 2 - Demystifying the WAV Format

You can also see how to build out a WAV file, old school and binary style, in the 3^rd part of that series:

Part 3 - Synthesizing Simple WAV Audio Using C#

However, we are taking a slightly easier route, by using DirectSound. DirectSound gives us a lot of nice classes for all the format information, abstracting all that stuff away and allowing us to pump a stream of data into a DirectSound object and play it. Perfect for a synthesizer app!

So, let's get started!

Building the app

I learned some Blend while working with this app, since it's built on WPF. The image buttons are just radio buttons. I had to differentiate the group number per instance of the user control at runtime (in the constructor of the Oscillator class).

I'm a terrible UI designer for the most part, so this is about as sexy as I'm willing to make this application. But feel free to make it look and act better!

Designing the UI

There's a dirty little secret in this application. It says it can oscillate 3 waves, but in truth, there's a constant (set to 3) that you can modify. You could have six if you wanted. How did I accomplish this? Each synth that you see is an instance of a WPF user control called Oscillator.xaml:

I have a StackPanel called Oscs in the main window. In the Window_Loaded event handler of the main window, I use this bit of code to add instances of the usercontrol:

C#

// Add 3 oscillators
Oscillator tmp;
for (int i = 0; i < NUM_GENERATORS; i++)
{
    tmp = new Oscillator();
    Oscs.Children.Add(tmp);
    mixer.Oscillators.Add(tmp);
}

VB

' Add 3 oscillators
Dim tmp As Oscillator
Dim i As Integer = 0
While i < NUM_GENERATORS
    tmp = New Oscillator()
    Oscs.Children.Add(tmp)
    mixer.Oscillators.Add(tmp)
    System.Math.Max(System.Threading.Interlocked.Increment(i),i - 1)
End While

The long rectangular canvas is used to plot the values of the generated wave, so you can visualize the wave as it's played. It is scaled along the X axis so you can see the general shape of the wave, which would be impossible without scaling it with 44,100 samples per second.

Earlier in the article, I noted that a sound file is basically a really, really long array of 16- or 32-bit floating point numbers between -1 and 1. We use this data to plot the graph as well. More on that later.

Now that we have the UI figured out (dynamic addition of oscillators), let's take a look at exactly how the sound is produced.

Bzzzzt! Making Sounds and the Mixer

One of the many cool things about DirectSound is that it basically wraps the WAV format for you. You set the buffering/format options and then shove a bunch of data into it, and it will play. Magic.

The way I've architected the solution is a little more modular. None of the oscillators has the ability to play itself—rather, uses its UI to control some values such as frequency, amplitude and wave type. These values are tied to public properties. The Oscillator component does virtually no audio work at all.

The generation of audio data is handled by the custom Mixer class, which takes a collection of Oscillators and, based on their properties, creates a composite of all the generators. This is done by averaging the samples in every oscillator and putting them into a new array of data.

The Mixer class looks like this:

One of the workhorses of the Mixer class is the method GenerateOscillatorSampleData. This takes an Oscillator as an argument to give access to the public properties set in the UI. From there, the algorithm generates 1 second of sample data (specified by the member bufferDurationSeconds) based on the wave type that has been selected in the UI. This is where the mathy stuff comes in to play. Check out this method and the different cases in the switch statement that determine what kind of wave to create below.

C#

public short[] GenerateOscillatorSampleData(Oscillator osc)
{
    // Creates a looping buffer based on the params given
    // Fill the buffer with whatever waveform at the specified frequency            
    int numSamples = Convert.ToInt32(bufferDurationSeconds * 
        waveFormat.SamplesPerSecond);
    short[] sampleData = new short[numSamples];
    double frequency = osc.Frequency;
    int amplitude = osc.Amplitude;
    double angle = (Math.PI * 2 * frequency) / 
        (waveFormat.SamplesPerSecond * waveFormat.Channels);            

    switch (osc.WaveType)
    {
        case WaveType.Sine:
            {
                for (int i = 0; i < numSamples; i++)
                    // Generate a sine wave in both channels.
                    sampleData[i] = Convert.ToInt16(amplitude * 
        Math.Sin(angle * i));
            }
            break;
        case WaveType.Square:
            {
                for (int i = 0; i < numSamples; i++)
                {
                    // Generate a square wave in both channels.
                    if (Math.Sin(angle * i) > 0)
                        sampleData[i] = Convert.ToInt16(amplitude);
                    else
                        sampleData[i] = Convert.ToInt16(-amplitude);
                }
            }
            break;
        case WaveType.Sawtooth:
            {
                int samplesPerPeriod = Convert.ToInt32(
        waveFormat.SamplesPerSecond / 
        (frequency / waveFormat.Channels));
                short sampleStep = Convert.ToInt16(
        (amplitude * 2) / samplesPerPeriod);
                short tempSample = 0;

                int i = 0;
                int totalSamplesWritten = 0;
                while (totalSamplesWritten < numSamples)
                {
                    tempSample = (short)-amplitude;
                    for (i = 0; i < samplesPerPeriod && 
        totalSamplesWritten < numSamples; i++)
                    {
                        tempSample += sampleStep;
                        sampleData[totalSamplesWritten] = tempSample;

                        totalSamplesWritten++;
                    }
                }
            }
            break;
        case WaveType.Noise:
            {
                Random rnd = new Random();
                for (int i = 0; i < numSamples; i++)
                {
                    sampleData[i] = Convert.ToInt16(
        rnd.Next(-amplitude, amplitude));
                }
            }
            break;
    }
    return sampleData;
}

VB.Net

Public Function GenerateOscillatorSampleData(ByVal osc As Oscillator) As Short()
    ' Creates a looping buffer based on the params given
    ' Fill the buffer with whatever waveform at the specified frequency 
    Dim numSamples As Integer = Convert.ToInt32(
     bufferDurationSeconds * waveFormat.SamplesPerSecond)
    Dim sampleData As Short() = New Short(numSamples - 1) {}
    Dim frequency As Double = osc.Frequency
    Dim amplitude As Integer = osc.Amplitude
    Dim angle As Double = (Math.PI * 2 * frequency) /
     (waveFormat.SamplesPerSecond * waveFormat.Channels)

    Select Case osc.WaveType
        Case WaveType.Sine
            If True Then
                For i As Integer = 0 To numSamples - 1
                    ' Generate a sine wave in both channels.
                    sampleData(i) =
                     Convert.ToInt16(amplitude * Math.Sin(angle * i))
                Next
            End If
            Exit Select
        Case WaveType.Square
            If True Then
                For i As Integer = 0 To numSamples - 1
                    ' Generate a square wave in both channels.
                    If Math.Sin(angle * i) > 0 Then
                        sampleData(i) = Convert.ToInt16(amplitude)
                    Else
                        sampleData(i) = Convert.ToInt16(-amplitude)
                    End If
                Next
            End If
            Exit Select
        Case WaveType.Sawtooth
            If True Then
                Dim samplesPerPeriod As Integer =
                 Convert.ToInt32(waveFormat.SamplesPerSecond /
                      (frequency / waveFormat.Channels))
                Dim sampleStep As Short =
                 Convert.ToInt16((amplitude * 2) / samplesPerPeriod)
                Dim tempSample As Short = 0

                Dim i As Integer = 0
                Dim totalSamplesWritten As Integer = 0
                While totalSamplesWritten < numSamples
                    tempSample = CShort(-amplitude)
                    i = 0
                    While i < samplesPerPeriod AndAlso totalSamplesWritten < numSamples
                        tempSample += sampleStep
                        sampleData(totalSamplesWritten) = tempSample

                        totalSamplesWritten += 1
                        i += 1
                    End While
                End While
            End If
            Exit Select
        Case WaveType.Noise
            If True Then
                Dim rnd As New Random()
                For i As Integer = 0 To numSamples - 1
                    sampleData(i) = Convert.ToInt16(
                     rnd.[Next](-amplitude, amplitude))
                Next
            End If
            Exit Select
    End Select
    Return sampleData
End Function

The Mixer is the heart of the app, and it's a beautiful example of object orientation and cohesion. Give it three things (oscillators) and it spits out a new thing you can use (an array of sample data).

Now that we have the sample data, all we have to do is play it back using DirectSound.

Sound Playback with DirectSound

As I mentioned, DirectSound provides a wrapper over the WAV format. You set up your buffer and format information and then feed it a bunch of data in the form of an array of shorts (arrays of trousers are known to cause errors).

First, we initialize the format information and buffer in the Window_Loaded event handler of the main form. The values below are not really arbitrary; there is an explanation of them in the Optional Reading section above (see Demystifying the WAV Format). This code also contains the code to add the oscillators, as shown earlier in the article.

C#

private void Window_Loaded(object sender, System.Windows.RoutedEventArgs e)
{
    WindowInteropHelper helper = 
        new WindowInteropHelper(Application.Current.MainWindow);
    device.SetCooperativeLevel(helper.Handle, CooperativeLevel.Normal);

    waveFormat = new Microsoft.DirectX.DirectSound.WaveFormat();
    waveFormat.SamplesPerSecond = 44100;
    waveFormat.Channels = 2;
    waveFormat.FormatTag = WaveFormatTag.Pcm;
    waveFormat.BitsPerSample = 16;
    waveFormat.BlockAlign = 4;
    waveFormat.AverageBytesPerSecond = 176400;

    bufferDesc = new BufferDescription(waveFormat);
    bufferDesc.DeferLocation = true;
    bufferDesc.BufferBytes = Convert.ToInt32(
        bufferDurationSeconds * waveFormat.AverageBytesPerSecond / waveFormat.Channels);

    // Add 3 oscillators
    Oscillator tmp;
    for (int i = 0; i < NUM_GENERATORS; i++)
    {
        tmp = new Oscillator();
        Oscs.Children.Add(tmp);
        mixer.Oscillators.Add(tmp);
    }            
}

VB

Private Sub Window_Loaded(ByVal sender As Object, ByVal e As System.Windows.RoutedEventArgs)
    Dim helper As New WindowInteropHelper(Application.Current.MainWindow)
    device.SetCooperativeLevel(helper.Handle, CooperativeLevel.Normal)

    waveFormat = New Microsoft.DirectX.DirectSound.WaveFormat()
    waveFormat.SamplesPerSecond = 44100
    waveFormat.Channels = 2
    waveFormat.FormatTag = WaveFormatTag.Pcm
    waveFormat.BitsPerSample = 16
    waveFormat.BlockAlign = 4
    waveFormat.AverageBytesPerSecond = 176400

    bufferDesc = New BufferDescription(waveFormat)
    bufferDesc.DeferLocation = True
    bufferDesc.BufferBytes = Convert.ToInt32(
     bufferDurationSeconds * waveFormat.AverageBytesPerSecond / waveFormat.Channels)

    ' Add 3 oscillators
    Dim tmp As Oscillator
    For i As Integer = 0 To NUM_GENERATORS - 1
        tmp = New Oscillator()
        Oscs.Children.Add(tmp)
        mixer.Oscillators.Add(tmp)
    Next
End Sub

When you click the Play button, the application takes its collection of oscillators and passes the values of the UI controls to the Mixer (which is initialized on each click with a reference to the main form window, so it can grab the Oscillator user controls).

The mixer outputs an array of shorts, which we write to a DirectSound buffer.

Here is the code for the Play button's click event handler:

C#

private void btnPlay_Click(object sender, System.Windows.RoutedEventArgs e)
{                                    
    mixer.Initialize(Application.Current.MainWindow);

    short[] sampleData = mixer.MixToStream();
    buffer = new SecondaryBuffer(bufferDesc, device);            
    buffer.Write(0, sampleData, LockFlag.EntireBuffer);
    buffer.Play(0, BufferPlayFlags.Default);            

    GraphWaveform(sampleData);
}

VB

Private Sub btnPlay_Click(sender As Object, e As System.Windows.RoutedEventArgs)
    mixer.Initialize(Application.Current.MainWindow)

    Dim sampleData As Short() = mixer.MixToStream()
    buffer = New SecondaryBuffer(bufferDesc, device)
    buffer.Write(0, sampleData, LockFlag.EntireBuffer)
    buffer.Play(0, BufferPlayFlags.[Default])

    GraphWaveform(sampleData)
End Sub

Drawing Pretty Graphs

All that's left is to draw the graph of the waveform on the canvas. Below is the GraphWaveform method. This method could graph anything it wanted to, as long as it was an array of shorts (not trousers). It's reminiscent of trying to graph things using Flash back in the day, when you had to actually figure out points and lines (most likely on paper), but WPF's Polyline object makes this rather trivial.

C#

private void GraphWaveform(short[] data)
{
    cvDrawingArea.Children.Clear();

    double canvasHeight = cvDrawingArea.Height;
    double canvasWidth = cvDrawingArea.Width;

    int observablePoints = 1800;
            
    double xScale = canvasWidth / observablePoints;
    double yScale = (canvasHeight / 
        (double)(amplitude * 2)) * ((double)amplitude / MAX_AMPLITUDE);            

    Polyline graphLine = new Polyline();
    graphLine.Stroke = Brushes.Black;
    graphLine.StrokeThickness = 1;

    for (int i = 0; i < observablePoints; i++)
    {
        graphLine.Points.Add(
            new Point(i * xScale, (canvasHeight / 2) - (data[i] * yScale) ));
    }

    cvDrawingArea.Children.Add(graphLine);            
}

VB

Private Sub GraphWaveform(ByVal data As Short())
    cvDrawingArea.Children.Clear()

    Dim canvasHeight As Double = cvDrawingArea.Height
    Dim canvasWidth As Double = cvDrawingArea.Width

    Dim observablePoints As Integer = 1800

    Dim xScale As Double = canvasWidth / observablePoints
    Dim yScale As Double = (canvasHeight / CDbl((amplitude * 2))) * (CDbl(amplitude) / MAX_AMPLITUDE)

    Dim graphLine As New Polyline()
    graphLine.Stroke = Brushes.Black
    graphLine.StrokeThickness = 1

    For i As Integer = 0 To observablePoints - 1
        graphLine.Points.Add(
         New Point(i * xScale, (canvasHeight / 2) - (data(i) * yScale)))
    Next

    cvDrawingArea.Children.Add(graphLine)
End Sub

Conclusion

This was a really fun little project that took way less time to code than it does to explain. It's a great exercise because it requires you to think about an ancillary field of science before you can sit down and code, which is really what coding for fun's all about, anyway!

If you want to try this out, the download link for the source code is at the top of the article.

About The Author

Dan Waters is an Academic Evangelist at Microsoft, covering schools in the Pacific Northwest, Alaska, and Hawaii. He is based in Bellevue, WA. Dan has way too many guitars at home and tries to entice both of his young daughters to learn how to play them. Music, technology, and music+technology are among his favorite hobbies, along with snowboarding and trying to maintain cool dad status. You can find his blog at www.danwaters.com or follow him on Twitter at www.twitter.com/danwaters.

Introduction

This article describes “Gremlin”, a quick and easy application that lets you play Halloween tricks. When your victim's computer is idle, Gremlin moves windows around on the screen, changes the focus window, moves your mouse, scrolls windows, and types nonsensical stuff. When there is background noise – like someone talking – it will shake the screen, even as your victim is typing away.

Deployment

Run it right away. You can download, copy theGremlin.exe executable and NAudio.dll to the victim's computer (say in c:\ directory), and then double click on the executable to run it.

Run it later. The other option is to copy the executable and dll files (or a shortcut to it) to the Startup folder on the victim's machine and watch the fun begin when they start up their machine in the morning!

If the computer you're using runs on Windows XP the path is:

C:\Documents and Settings\All Users\Start Menu\Programs\Startup

With Vista and Windows 7 the path looks like:

C:\Users\USERNAME\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup

Just remember to change USERNAME to the name of the user on your machine.

Commandline

Gremlin is compiled as a windows application, so it won't popup a terminal window if you double click on it, etc. But you can run it from the command line with flags:

-aggressive will make the gremlin's actions more obvious

-help will display the command line options

-name NAME is useful for testing. Gremlin will only use windows with this specific title or from this specific application. (Remember to drop the ".exe" from the application filename)

To stop the program, press CTRL+F2.

And now … the dodgy bits

The overall structure of the program is broken down into three kinds of functionality – actions, triggers and some interstitial glue:

Actions

• The screen-shaker is a window that makes the monitor look like it is shaking. Sort of like a loose cable on the back of the monitor.
• A random event might kick the windows around—either friction will slow them down, or...
• Gremlin will type nonsensical messages from the keyboard. These messages are then sent from a virtual keyboard, or they:
• Move the mouse around, or
• Press the mouse buttons, or
• Randomly switch the focus to another window

Triggers

• An audio module listens for sounds that trigger the module that shakes your screen.
• Otherwise, the software uses a p/invoke to GetLastInputInfo() and waits for the user to be idle for a minute or two. When it detects that the user is inactive, it randomly selects and carries out actions.

Other

• A hidden window receives key press events. I tend to reuse this module a lot, as a way to stop experimental programs that may have made my machine unusable.

For fun, I'll describe a couple of these modules below.

Screen shaker

My favorite bit is the screen shaker. It grabs a picture of the screens, creates a window that spreads across them, and then moves that image back and forth a few pixels every 30ms or so, with a little bit of random rotation. (Faster computers, of course, get a better effect).

The screen shaker is special in four ways:

• It is a top-level window – no other windows are allowed to go over it.
• This top-level window never becomes the focus window (the window that gets the keyboard events).
• Mouse button events (e.g. clicks and double-clicks) are passed through to the windows and desktop underneath. This gives the illusion that this is the "real" desktop shaking, by allowing a person to click on a button or text that passes the click thru to the real button or text.
• It shakes each of the monitor's screens independently

Some background: to create a window that looks like the shaking screen, I used two classes. The first is ScrShake (in ScreenShake.cs), which derives from the second class UnfocusableForm.  I'll describe ScrShake first.

The screen shaking process requires five instance variables:

• animTimer is a System.Windows.Forms.Timer used to force painting of a new frame on the screen.
• bx, and by are how far the screen has shaken up or down, left or right.
• angle is how much the screen has twisted during shaking.
• screenBitmap is an array of bitmaps, one for each of the monitors.

This creates a window without a border or other trappings, then makes it the topmost window and sets a flag in the method variable ExStyle to ignore mouse clicks. More on this flag in a little while.

C#

public partial class ScrShake : UnfocusableForm
{
   public ScrShake(double ShakeCoef, double AngleCoef) : base(true)
   {
      this.SuspendLayout();
      … other setup code …
      // This is needed since we're over the whole display, and we don't
      // want the other areas to be blurry
      TransparencyKey =  BackColor = ForeColor = System.Drawing.Color.Fuchsia;
      DoubleBuffered = true;
      TopMost = true;
      FormBorderStyle = System.Windows.Forms.FormBorderStyle.None;

      ClientSize = new System.Drawing.Size(300, 300);
      Name = "topimage";
      ResumeLayout(false);

      // This is need to pass mouse clicks thru to lower layers
      ExStyle |= (int) WS . EX_TRANSPARENT;

      … More code that will be described later…
   }
}

When the timer has done a given number of animations, it will call the Stop() method. This will stop the animation, clean it up, and hide the window:

C#

void Stop()
{
   animTimer . Stop();
   Hide();
   screenBitmap = null;
}

When the main loops that start the animated screen shaking process, it calls the Screens() method. This grabs an image of the screens, determines the shape of each monitor, and starts the animation for 10 frames.

C#

internal void Screens()
{
   if (Visible)
   {
      Stop();
      return ;
   }
   // Grab the screens
   screenBitmap = Program.GrabScreens();
   CountDown = 10;

   animTimer.Start();
   angle = 0.0f;
   … code to display the window and get shape of monitors (see below)…
}

The screen capture portion is in ScreenCapture.cs. The method GrabScreens() creates an individual bitmap for each monitor and returns them as an array.

C#

internal void Screens()
{
   if (Visible)
   {
      Stop();
      return ;
   }
   // Grab the screens
   screenBitmap = Program.GrabScreens();
   CountDown = 10;

   animTimer.Start();
   angle = 0.0f;
   … code to display the window and get shape of monitors (see below)…
}

The code below determines the bounds of each screen and builds up a size of all of the screens put together. The window will be set to be this size.

C#

Screen[] AllScreens = Screen.AllScreens;

// full width/height of all monitors combined
Rectangle fullSize = AllScreens[0].Bounds;

// find a rectangle that will encompass all monitors on the system
// (assuming the primary monitor is on the left/top!)
for (int i = 1; i < AllScreens.Length && i < screenBitmap.Length; i++)
{
    Rectangle Bounds = AllScreens[i].Bounds;

    if (Bounds.Left < fullSize.Left)
        fullSize.X = Bounds.X;

    if (Bounds.Right > fullSize.Right)
        fullSize.Width = Bounds.Right - fullSize.X;

    if (Bounds.Top < fullSize.Top)
        fullSize.Y = Bounds.Y;

    if (Bounds.Bottom > fullSize.Bottom)
        fullSize.Height = Bounds.Bottom - fullSize.Y;
}

The code to show the window, fill the whole screen, and move it to the top looks like:

C#

Show();
WindowState = FormWindowState.Normal;
// cover all monitors with one gigantic window
Location = new Point(fullSize.Left, fullSize.Top);
Size = new Size(fullSize.Width, fullSize.Height);

// bring it to the top
BringToFront();

The constructor also created a timer that drives the screen shaking effect. The timer has a delegate that randomly selects the angle of rotation and the offset of the image, and triggers a repaint of the window. (The amount of shaking is controlled by two external variables called AngleCoef and ShakeCoef).

C#

animTimer = new System.Windows.Forms.Timer();
animTimer.Tick += delegate(object A, EventArgs E)
{
    if (--CountDown < 1)
        Stop();

    angle += (float)((Program.Rnd.NextDouble() - 0.5) * AngleCoef);
    bx = (float)((Program.Rnd.NextDouble() - 0.5) * ShakeCoef);
    by = (float)((Program.Rnd.NextDouble() - 0.5) * ShakeCoef);
    Invalidate();
};

// Sets the timer interval to 30 milliseconds.
animTimer.Interval = 30;

Although each monitor shifts up & down, left & right by the same amount, and rotates by the same angle, they are painted independently. This gives the illusion that each monitor has a shaky image. Paint the window is using the following code.

C#

protected override void OnPaint(PaintEventArgs e)
{
    base.OnPaint(e);
    Graphics g = e.Graphics;
    g.CompositingQuality = CompositingQuality.HighQuality;

    // for each monitor, draw the effect
    Screen[] AllScreens = Screen.AllScreens;
    for(int i = 0; i < screenBitmap.Length; i++)
    {
        if (null == screenBitmap[i])   continue;
        g.SmoothingMode = SmoothingMode.HighQuality;

        // grab the size of the current monitor
        Rectangle region = AllScreens[i].Bounds;

        double ImWidth = screenBitmap[i].Width  * 
           e.Graphics.DpiX / screenBitmap[i].HorizontalResolution;
        double ImHeight= screenBitmap[i].Height * 
           e.Graphics.DpiY / screenBitmap[i].VerticalResolution;
        Matrix m = new Matrix();
        m.Translate( (float)(- ImWidth  /2), 
                  (float)(- ImHeight /2), MatrixOrder.Append);

        // rotate the bitmap about the center
        m.RotateAt(angle, new Point(0,0), MatrixOrder.Append);

        // center the image no matter what its size is
        m.Translate( region.Width /2 - bx, 
                  region.Height/2 - by, MatrixOrder.Append);

        // assign our transformation matrix
        g.Transform = m;

        // draw it
        g.DrawImage(screenBitmap[i], region.Left, region.Top);
    }

The Form that does nothing!

The ScrShaker class uses a help class called UnfocusableForm (in UnfocusableForm.cs) to create a window that never becomes the focus window – it never receives key presses, etc.

This window has no trappings – no border, no resize gripper, no icon, no minimize / maximize buttons, no title bar.

C#

public partial class UnfocusableForm : Form
{
   public UnfocusableForm(bool X) : base()
   {
      if (X)
        {
           ControlBox = false;
           FormBorderStyle = System.Windows.Forms.FormBorderStyle.None;
           ShowInTaskbar = false;
           ShowIcon = false;
           MinimizeBox = false;
           SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide;
           StartPosition = System.Windows.Forms.FormStartPosition.Manual;
        }
   }

Then it overrides the ShowWithoutActivation method so that the window will not become the active when it is shown.

C#

protected override bool ShowWithoutActivation
{ get {  return true; } }

The UnfocusableForm class also overrides the CreateParams() method to set extra styles when creating the window. I haven't found a way to set these styles flexibly. Instead, the UnfocusableForm uses a method variable called ExStyle that allows derived classes to specify what flags they desire. In this case, The ScrShake class used the method variable to set a flag that ignore mouse clicks:

C#

protected override CreateParams CreateParams
{
  get
  {
     CreateParams cp=base.CreateParams;
     cp . ExStyle |= ExStyle;
     return cp;
  }
}

Finally, it captures a couple of events that ask a window if it would like to become the active window. (These usually happen when the user clicks on an inactive window):

C#

internal const int MA_NOACTIVATE = 0x0003;
protected override void WndProc(ref Message m)
{
  if (m.Msg == (int) WM.MOUSEACTIVATE)
  {
     m.Result = (IntPtr) MA_NOACTIVATE;
     return;
  }
  if (m.Msg == (int) WM.FOCUS)
  {
     m.Result = (IntPtr)1;
     return;
  }
  base.WndProc(ref m);
}

Sending Mouse Events

Sending Mouse Events is simple, but not trivial. The wiki at Pinvoke.net provides the signature to the procedures and structures we need to pass to it. (Note: these structures are a bit different, based on a blog posting by Raymond Chen)

C#

[DllImport("user32.dll", EntryPoint = "SendInput", SetLastError = true)]
static extern uint SendInput(uint nInputs, INPUT[] Inputs, int cbSize);
[DllImport("user32.dll", EntryPoint = "GetMessageExtraInfo", SetLastError = true)]
static extern IntPtr GetMessageExtraInfo();

[StructLayout(LayoutKind.Sequential)]
struct INPUT
{
   internal INPUTTYPE   type;
   internal INPUT_UNION i;
}

 // This generates the anonymous union
[StructLayout(LayoutKind.Explicit)]
struct INPUT_UNION
{
   [FieldOffset(0)]
   public MOUSEINPUT mi;
   [FieldOffset(0)]
   public KEYBDINPUT ki;
   [FieldOffset(0)]
   public HARDWAREINPUT hi;
};

[StructLayout(LayoutKind.Sequential)]
struct MOUSEINPUT
{
   public int    dx;
   public int    dy;
   public int    mouseData;
   public MOUSEEVENTF dwFlags;
   public int    time;
   public IntPtr dwExtraInfo;
}

[Flags]
enum MOUSEEVENTF : int
{
   MOVE       = 0x01,
   LEFTDOWN   = 0x02,
   LEFTUP     = 0x04,
   RIGHTDOWN  = 0x08,
   RIGHTUP    = 0x10,
   MIDDLEDOWN = 0x20,
   MIDDLEUP   = 0x40,
   ABSOLUTE   = 0x8000
}

Each of these pieces go together in a specific way. To do a mouse movement, a mouse event structure needs to be created. This involves setting dwFlags to the kind of mouse event (a relative movement in this case), dx and dy to the number of pixels moved. It is also important to set dwExtraInfo to whatever GetMessageExtraInfo() is set to.

C#

MOUSEINPUT MEvent = new MOUSEINPUT();
MEvent.dwFlags = MOUSEEVENTF.MOVE;
MEvent.dx = Rnd . Next(-8, 8) ;
MEvent.dy = Rnd . Next(-8, 8);
MEvent.dwExtraInfo = GetMessageExtraInfo();
Send(MEvent);

Sending the event takes a few more steps that are all wrapped in a helper methdo called Send(). Send these human interface device events via the SendInput() procedure (earlier). The procedure takes an array of events, for different kinds of devices. We have to create the array, set the event type, copy the event data, and then make the call:

C#

public virtual bool Send(MOUSEINPUT Event)
{
  INPUT[] Events = new INPUT[1];
  Events[0] . type = INPUTTYPE . MOUSE;
  Events[0] . i.mi = MEvent;
  return SendInput((uint)Events.Length, Events, Marshal.SizeOf(Events[0])) > 0; 
}

From here, sending a mouse button click is pretty straight forward. A mouse click is really a mouse button press event, followed by a mouse button release event. To make things interesting the mouse button is choosen at random. The “ugly” part is that each of the mouse buttons is assigned a bit, so a binary shift is used to convert a button number to its bit. And the button release is yet another bit, so the second message shifts the flags, to move the bit from the “button pressed” state to the “button released state”.

C#

MOUSEINPUT MEvent = new MOUSEINPUT();
MEvent.dwFlags = (MOUSEEVENTF) (1 << (2*Rnd.Next(0, 3)+1)) ;
MEvent.dwExtraInfo = GetMessageExtraInfo ();
Send(MEvent);

MEvent.dwFlags = (MOUSEEVENTF)((int) MEvent.dwFlags << 1);
MEvent.dwExtraInfo = GetMessageExtraInfo ();
Send(MEvent);

Generating deranged text

The gremlin can also type deranged sentences that will appear in editors, if the user left one open. The SendWait() method in the SendKeys class does the typing itself:

C# 

SendKeys . SendWait(GenerateSentence());
SendKeys . SendWait("{ENTER}");

Generating the sentences is not difficult; I used a simplistic Markov generator. This generator starts by randomly selecting a word that can start a sentence. The Starts variable is array which holds just these words. Then, in the Transition method, this word is used to find the next word that can be included in the sentence. It does this by using a dictionary, called NonStart, which given the word as a key, will return a list of all the words that can come after it. This process then repeats, appending the words onto the end of a string. It stops if it finds a word that can end a sentence.

C#

static Dictionary<string,string[]> NonStart ;
static string[] Starts;
static Dictionary<string,string> Terminal ;

static string GenerateSentence()
{
  StringBuilder SB = new StringBuilder();
  string Word = Starts[ Rnd.Next(0, Starts.Length) ];

  Transition(SB, Word);
  return SB.ToString();
}

static void Transition(StringBuilder SB, string Word)
{
  while (true)
  {
     SB.Append(Word);
     SB.Append(' ');

     // Look up word after this
     string[] Nexts;
     if (!NonStart.TryGetValue(Word, out Nexts))
       break;
     int Idx = Rnd.Next(Terminal . ContainsKey(Word) ? -1 :0, Nexts.Length);
     if (Idx < 0)
       break;
     Word = Nexts[Idx];
  }
}

Building these two dictionaries and array is a pretty simple process. The BuildSuffixTree() method takes a string and splits it up into sentences. Then it splits each sentence into (lower case) words. The first word of the sentence goes onto the end of the Starts array. Otherwise, the previous word is used to look up a list in a dictionary, and the current word is appended on to the list. This dictionary will become the NonStarts dictionary. The last word of the sentence is also placed into the Terminal dictionary, to indicate that this might be the end of a sentence.

C#

static void BuildSuffixTree()
{
  // First, build up a list of words that start sentence, and transitions
  List<string> Starts1 = new List<string>();
  Dictionary<string,List<string>> Trans = new Dictionary<string,List<string>>();
  foreach(string S1 in S.Split(new char[]{'.','?','!'}))
  {
     string Prev=null;
     foreach(string S2 in S1.Split(new char[]{' ','\n','\r','\t',',',';'}))
     {
        if (S2 . Length < 1)
          continue;
        if (null == Prev)
          {
             Starts1.Add(string.Intern(S2.ToLower()));
             Prev = string.Intern(S2.ToLower());
             continue;
          }
        List<string> Nextsa;
        if (! Trans.TryGetValue(Prev, out Nextsa))
          Trans[Prev] = Nextsa = new List<string>();
        Prev = string.Intern(S2.ToLower());
        Nextsa.Add(Prev);
     }
     if (null != Prev)
       Terminal[Prev] = Prev;
 }

 // Next, flatten the list of words that start a sentence
 Starts = Starts1.ToArray();

 // Flatten the transition table
 foreach (string S3 in Trans.Keys)
  NonStart[S3] = Trans [S3].ToArray();
}

The things that trigger action

The audio trigger

The audio module (in file AudioTrigger.cs) listens in on all of the microphones. It uses NAudio, with a basic setup and delegate structure swiped from Mark Heath's article “.NET Audio Recording”. The first difference is that it registers all audio input devices:

C#

static WaveIn[] StartMics()
{
  int NumDevices = WaveIn.DeviceCount;
  WaveIn[] AudIns = new WaveIn[NumDevices];
  for (int waveInDevice = 0; waveInDevice < NumDevices ; waveInDevice++)
  {
     AudIns[waveInDevice] = new WaveIn();
     AudIns[waveInDevice].DeviceNumber = waveInDevice;
     AudIns[waveInDevice].DataAvailable += waveIn_DataAvailable;
     AudIns[waveInDevice].WaveFormat = new WaveFormat(8000, 1);
     AudIns[waveInDevice].StartRecording();
  }
  return AudIns;
}

The real magic in the trigger is a modified version of the waveIn_DataAvailable delegate. Instead of saving the audio, it checks for any sound activity. It starts by checking to see if any of the sound amplitudes are greater than a pre-defined, very high threshold. Then the samples are squared and summed up, and the result is compared with a threshold. If it exceeds the threshold, the trigger is set. These two are good at catching the kind of sound made when a person is talking, futzing on the desk, or typing.

C#

static double AudioThresh  = 0.8;
static double AudioThresh2 = 0.09;

static void waveIn_DataAvailable(object sender, WaveInEventArgs e)
{
  bool Tr = false;
  double Sum2  = 0;
  int Count = e.BytesRecorded / 2;
  for (int index = 0; index < e.BytesRecorded; index += 2)
  {
     double Tmp = (short)((e.Buffer[index + 1] << 8) | e.Buffer[index + 0]);
     Tmp /= 32768.0;
     Sum2 += Tmp*Tmp;
     if (Tmp > AudioThresh)
       Tr = true;
  }
  Sum2 /= Count;

  // If the Mean-Square is greater than a threshold, set a flag to indicate that noise has happened
  if (Tr || Sum2 > AudioThresh2)
    Interlocked.Exchange(ref AudioTrigger, 1);
}

This sets a flag – AudioTrigger – which will be seen (and reset) in the main loop, with a bit of code that looks like:

C#

while (0 == Shutdown)
{
    // Do some lovely events
    Application . DoEvents();
    Thread . Sleep (20);

    //… do some other stuff ..

    // Check for a kick from the sound system
    if (0 != AudioTrigger)
    {
        Interlocked . Exchange(ref AudioTrigger, 0);
        …
        ScreenShaker . Screens();
        continue;
    }
    //… more checks for the user idle trigger…
}

Waiting for the user to be idle

The check to see if the user is not doing anything, the main loop uses a helper method called LastInputTime(), which returns the number of milliseconds the user did any input activity. The loop ensures that the user has been idle for long enough, and then calls the method that selects random gremlin actions:

C#

uint LastTime = Win32.LastInputTime();
if ((uint) Environment.TickCount < IdleTimeTrigger + LastTime)
    continue;

The LastInputTime() helper method uses a p/invoke call to the GetLastInputInfo() API call. The wiki at Pinvoke.net provides the signature to the procedure, and a suitable structure that we need to pass to it:

C#

[DllImport("User32.dll")]
static extern bool GetLastInputInfo(ref LASTINPUTINFO LastInfo);

[StructLayout(LayoutKind.Sequential)]
struct LASTINPUTINFO
{
  public uint cbSize;

  /// <summary>
  /// Number of system tickes
  /// </summary>
  public uint dwTime;
}

Then, these two structures are wrapped up into the LastInputTime() helper procedure, which does the dirty work of allocating and initializing a structure.

C#

internal static uint LastInputTime()
{
  LASTINPUTINFO lastInput=new LASTINPUTINFO();
  lastInput.cbSize = (uint)Marshal.SizeOf(lastInput);
  GetLastInputInfo(ref lastInput);
  return lastInput . dwTime;
}

Conclusion

The tricks gremlin plays are intended to be somewhat subtle. There is an aggressive option to make it react more, and move windows around more visibly.

If you want to try this out, the download link for the executable and source code are at the top of the article!

Resources and References

This article is a blatant grab bag of experimental and reused code from earlier experiments and other demo programs. Below are some of the projects I lifted code examples from:

• Pinvoke.net – a wiki-style site with many useful bits of example code related to calling the Windows API from within .NET.
• “.NET Audio Recorder” by Mark Heath, from which I stole code to make the audio trigger.
• “Possessed PC Pranks for Halloween” by Brian Peek, from which I stole the basic code for sending keystrokes.
• “April Fools' Day Application” by Brian Peek, from which I stole code to grab the screen and rotate it.

About The Author

Randall Maas writes firmware for medical devices, and consults in embedded software. Before that he did a lot of other things… like everyone else in the software industry. You can contact him at randym@acm.org.

Audio Recording in .NET

The .NET framework does not provide any direct support for recording audio, so I will make use of the open source NAudio project, which includes wrappers for a number of Windows audio recording APIs.

Note: It is important to point out that .NET is not an appropriate choice for high sample rate and low latency audio recording, such as that found in Digital Audio Workstation software used in recording studios. This is because the .NET garbage collector can interrupt the process at any point. However, for purposes of recording speech from the microphone, the .NET framework is more than capable. By default, NAudio asks the soundcard to give us data every 100ms, which gives plenty of time for the garbage collector to run as well as our own code.

We will make use of the wrappers for the waveIn API's, as these are the most universally supported, and allow us freedom to choose the sample rate. We will record in mono, 16 bit at 8kHz, which is more than good enough audio quality for speech, and will not overly tax the processor, which is important as we want to visualize the waveform as well.

Choosing a Capture Device

Normally, you will be able to use the default audio capture device without any difficulties, but should you need to offer the user a choice, NAudio will allow you to do so. You can use the WaveIn.DeviceCount and WaveIn.GetDeviceCapabilities to find out how many recording devices are present, and query for their name and number of supported channels.

On my computer, I have a single waveIn device (Microphone Array) until I plug my headset in, at which point, a new device appears and becomes the default (device 0 is always the default).

int waveInDevices = WaveIn.DeviceCount;
for (int waveInDevice = 0; waveInDevice < waveInDevices; waveInDevice++)
{
    WaveInCapabilities deviceInfo = WaveIn.GetCapabilities(waveInDevice);
    Console.WriteLine("Device {0}: {1}, {2} channels", 
        waveInDevice, deviceInfo.ProductName, deviceInfo.Channels);
}

This produces the following output on my computer:

Device 0: Microphone / Line In (SigmaTel , 2 channels
Device 1: Microphone Array (SigmaTel High, 2 channels

Unfortunately these device names are truncated because the WAVEINCAPS structure only supports 31 characters. There is a way of getting the full device name, but it is rather convoluted.

Normally, you will choose Device 0 (the default), but if you wish to select a different input device, simply set the DeviceNumber property on your WaveIn object to the desired number.

Checking the Recording Level

The first step in recording is usually to help the user determine if their microphone is working or not. This is especially important if the user has more than one input on their soundcard. The way we achieve this is simply by starting recording and displaying the level of audio detected to the user with a volume meter. The waveIn APIs do not write anything to disk, so no audio is actually being ‘recorded' at this point, we are simply examining the input level and then throwing the captured audio samples away.

To begin capturing audio from the soundcard, we use the WaveIn class in NAudio. We configure it with the WaveFormat in which we would like to record (in our case 8kHz mono), before calling StartRecording, to start capturing audio from the device.

waveIn = new WaveIn();
waveIn.DeviceNumber = selectedDevice;
waveIn.DataAvailable += waveIn_DataAvailable;
int sampleRate = 8000; // 8 kHz
int channels = 1; // mono
waveIn.WaveFormat = new WaveFormat(sampleRate, channels);
waveIn.StartRecording();

The DataAvailable event handler will notify us whenever a buffer of audio has been returned to us from the sound card. The data comes back as an array of bytes, representing PCM sample data. This is fine if we are planning to write the audio directly to disk, but what if we wish to have a look at the audio data itself? Each audio sample is 16 bits, i.e. two bytes, meaning that we will need to convert pairs of bytes into shorts to be able to make sense of the data.

Note: if we were recording in stereo, the 16 bit samples would themselves come in pairs, first the left sample, then the right sample.

The following code shows how we might process the raw bytes in the DataAvailable event, and read the individual audio samples out. Notice that we use the BytesRecorded field, not the buffer's Length property. Also, I have chosen to convert the samples to 32 bit floating point format and scaled them so the maximum volume is 1.0f. This makes processing them through effects and visualizing them much easier.

void waveIn_DataAvailable(object sender, WaveInEventArgs e)
{
    for (int index = 0; index < e.BytesRecorded; index += 2)
    {
        short sample = (short)((e.Buffer[index + 1] << 8) | 
                                e.Buffer[index + 0]);
        float sample32 = sample / 32768f;
        ProcessSample(sample32);
    }
}

Note: One complication of using the waveIn and waveOut APIs is deciding on a callback mechanism. NAudio offers three options. First is function callbacks. This means that the waveIn API is given a (pinned) function pointer which it calls back onto. This means that your DataAvailable callback will come in on a background thread. In some ways this is the cleanest approach, but you need to beware of rogue soundcard drivers that can hang in calls to waveOutReset when using function callbacks (the SoundMAX chipset found on a lot of laptops is particularly prone to this problem).

The second is to supply a window handle. The waveIn APIs will post a message back to be handled on the message queue of that window handle. This method tends to be the most reliable and most commonly used. One gotcha to watch out for is that if you stop recording and immediately restart, a message from the old recording session could get handled in the new session resulting in a nasty exception.

The third is to let NAudio create its own new window and post messages to that. This gets round any danger of messages from one recording session getting muddled up with another. This is the callback method that NAudio will use by default if you call the default WaveIn constructor. But don't use this from a background thread or from a console application, or the new window that NAudio creates won't actually get round to processing its message queue.

Visualizing the Recording Level

We have seen how we can begin to capture audio from the soundcard for the purposes of checking the recording level. Now we need to give the user some visual feedback. We will use WPF for our sample recording application. The simplest control we have available to display a single numeric value graphically is the ProgressBar. And because it is WPF, we can fully customize the graphical appearance of the progress bar to look a little more like a volume meter. I have used a gradient going from green to red to show the current volume level. You can read more about how I created this ProgressBar template here.

Figure 1 - A Progress Bar Showing the Current Microphone Volume Level

To help provide the volume level to display, I have created a SampleAggregator class. This is passed every audio sample value we receive and keeps track of the maximum and minimum values. Then, after a specified number of samples, it raises an event allowing the GUI components to respond. We need to be careful not to raise too many of these events or performance will be badly affected. I am raising one every 800 samples, meaning we get 10 updates per second to the screen.

Because I am using data binding, when one of these updates fires, I must raise a PropertyChangedEvent on my DataContext object (also known as the “ViewModel” in the MVVM pattern). Here's the XAML syntax for binding to my CurrentInputLevel property:

<ProgressBar Orientation="Horizontal" 
    Value="{Binding CurrentInputLevel, Mode=OneWay}" 
    Height="20" />

And here's the code in the ViewModel that ensures that the GUI updates whenever we calculate a new maximum input level:

private float lastPeak;

void recorder_MaximumCalculated(object sender, MaxSampleEventArgs e) 
{
    lastPeak = Math.Max(e.MaxSample, Math.Abs(e.MinSample));
    RaisePropertyChangedEvent("CurrentInputLevel");
}

// multiply by 100 because the Progress bar's default maximum value is 100 
public float CurrentInputLevel { get { return lastPeak * 100; } }

Note: Model View ViewModel (MVVM) is a pattern that is growing in popularity amongst WPF and Silverlight developers. The basic idea is that you have no code behind whatsoever on your View (i.e. your xaml markup file), and simply specify all communications with your business logic by means of data binding. The ViewModel serves as an adapter to ease the process of data binding. This approach gives very good separation of appearance and behavior. For the most part, this pattern works very well, but there are a few tricky areas, for which you will need to either write a few lines of code behind, or make use of some cunning tricks such as attached dependency properties or custom triggers. There are several excellent open source helper libraries that can take some of the work out of getting an MVVM application up and running. Have a look here for a comprehensive list.

Adjusting the Recording Level

Suppose the current input level is too high or too soft. We would like to be able to support modifying the recording level. Again, we would like to use data binding to do so, so we will add a volume slider to our XAML:

<Slider Orientation="Horizontal" 
    Value="{Binding MicrophoneLevel, Mode=TwoWay}" 
    Maximum="100" 
    Margin="5" />

Now we have to get hold of the MixerLine that will allow us to access the input volume control for our waveIn device. This requires us to make use of the Windows mixer APIs, which also have wrappers in NAudio. Getting hold of this volume control is not always as straightforward as you might hope (and can require different approaches for XP and Vista), but the following is code that seems to work on most systems:

private void TryGetVolumeControl()
{
    int waveInDeviceNumber = 0;
    var mixerLine = new MixerLine((IntPtr)waveInDeviceNumber, 
                                   0, MixerFlags.WaveIn);
    foreach (var control in mixerLine.Controls)
    {
        if (control.ControlType == MixerControlType.Volume)
        {
            volumeControl = control as UnsignedMixerControl;        
            break;
        }
    }
}

Now we can use the Percent property on the UnsignedMixerControl to set volume to a value anywhere between 0 and 100.

Starting Recording

Now we have got our recording levels set up correctly, we are ready to actually start recording. But since we have already opened our waveIn device, all we need to do is start writing the data we have received into a file.

NAudio has a class called WaveFileWriter which will allow us to write our recorded data to a file. For now, we will write it to a temporary file in PCM format, and convert it later into a better compressed format such as MP3. The following code creates a new WAV file:

writer = new WaveFileWriter(waveFileName, recordingFormat);

Now we can write to the file as we receive notifications from the waveIn device:

void waveIn_DataAvailable(object sender, WaveInEventArgs e)
{
    if (recordingState == RecordingState.Recording)
        writer.WriteData(e.Buffer, 0, e.BytesRecorded);            

   // ...
}

Note: There are three main options for how to store audio while it is being recorded. First, you can write it to a MemoryStream. This saves the inconvenience of dealing with a temporary file, but you need to be careful not to run out of memory. Also, if your recording program crashes half way through, you have lost everything. At the sample rate we are using for this demo, one minute of audio takes just under 1 MB of memory, but if you were recording at 44.1kHz stereo (the standard for music), you would need about 10 MB per minute.

Second, you can write to a temporary WAV file to be converted to another format later, as we are doing here. While this is not a disk space efficient format, it is very easy to work with, and particularly useful if you are planning to apply any effects or edit the audio in any way after recording.

Third, you can pass the audio directly to an encoder (such as WMA or MP3) as it is being recorded. This might be the best option if you are making a longer recording, and have no need to edit it after recording.

Stopping Recording

Obviously we will stop when the user clicks the stop recording button, but we might also want to set a maximum recording duration to stop the user inadvertently filling up their hard disk. For this example, we will allow one minute of recording.

long maxFileLength = this.recordingFormat.AverageBytesPerSecond * 60;
 
int toWrite = (int)Math.Min(maxFileLength - writer.Length, bytesRecorded);
if (toWrite > 0)
    writer.WriteData(buffer, 0, bytesRecorded);
else
    Stop();

Note: Something that can be slightly confusing for users is that when using window callbacks with WaveIn, the last bit of audio you recorded comes in after you have asked recording to stop, so make sure you don't close the file you are saving to until you have got all the audio back. The FinishedRecording event on the WaveIn object will help you determine when it is safe to close the WaveFileWriter and clean up your resources.

Visualizing the Wave Form

It is often desirable to display the audio waveform to the user. Displaying the waveform while you are recording is sometimes called “confidence recording”, because it allows you to see that audio is being recorded as expected and the levels are still right.

There are a variety of possible approaches for drawing audio waveforms. The simplest is to draw a vertical line showing the minimum and maximum values every time our sample aggregator fires:

Figure 2 - Audio Waveform using vertical lines

At first glance it may seem that this would be trivial to implement in WPF, but there is a real danger of consuming too many resources. For example, simply adding a new line to a Canvas every time a new maximum sample is calculated performs very badly, so it is better to have a fixed number of vertical lines and resize them dynamically.

Another approach is to create a polygon. This requires us to add two points to a Polygon's Points collection every time we receive a new sample. The trick is to add these points in the middle of the Points collection, rather than at the end, so that the end result is a single shape. This means our waveform can have a different outline color and fill color. To stop the edges from appearing too jagged, we plot points two units apart along on the X axis.

Figure 3 - Audio Waveform rendered using a Polygon

Like the microphone volume meter, the waveform drawing control needs to receive several notifications a second of the maximum and minimum sample values received by the SampleAggregator. When each sample value is received, we either insert new points into our polygon, or, if the whole screen is full, we go back to the left-hand edge and continue drawing from there.

For the confidence recording display I have used the Polygon method, which is in a class called PolygonWaveFormControl. Here's the code which calculates the new points or updated point locations as we receive a new maximum sample:

public void AddValue(float maxValue, float minValue)
{
    int visiblePixels = (int)(ActualWidth / xScale);
    if (visiblePixels > 0)
    {
        CreatePoint(maxValue, minValue);

        if (renderPosition > visiblePixels)
        {
            renderPosition = 0;
        }
        int erasePosition = (renderPosition + blankZone) % visiblePixels;
        if (erasePosition < Points)
        {
            double yPos = SampleToYPosition(0);
            waveForm.Points[erasePosition] = 
               new Point(erasePosition * xScale, yPos);
            waveForm.Points[BottomPointIndex(erasePosition)] = 
               new Point(erasePosition * xScale, yPos);
        }
    }
}

private void CreatePoint(float topValue, float bottomValue)
{
    double topYPos = SampleToYPosition(topValue);
    double bottomYPos = SampleToYPosition(bottomValue);
    double xPos = renderPosition * xScale;
    if (renderPosition >= Points)
    {
        int insertPos = Points;
        waveForm.Points.Insert(insertPos, new Point(xPos, topYPos));
        waveForm.Points.Insert(insertPos + 1, new Point(xPos, bottomYPos));
    }
    else
    {
        waveForm.Points[renderPosition] = new Point(xPos, topYPos);
        waveForm.Points[BottomPointIndex(renderPosition)] = 
              new Point(xPos, bottomYPos);
    }
    renderPosition++;
}

The erase position calculation is to blank out some previous sample values to make it obvious where the new data is appearing after we have wrapped around once:

Figure 4 PolygonWaveForm control's “blank zone”

Note: There are faster ways to perform rendering in WPF. One option is to use the WriteableBitmap class and draw directly onto it. This could be a good approach if you were using the vertical lines method of rendering. The second is to use DrawingVisual objects, which are lightweight drawing objects offering better performance than using classes derived from Shape. The down-side is the loss of features such as DataBinding and the ability to fully describe the picture in XAML, but for WaveForm drawing this is not really a drawback. I use the DrawingVisual method in the Save Audio part of this application.

Another challenge was how the waveform drawing control could receive notifications since I am using MVVM so I have no direct access to the SampleAggregator. A simple way around this was to create a Dependency Property on PolygonWaveFormControl:

public static readonly DependencyProperty SampleAggregatorProperty = 
       DependencyProperty.Register(
          "SampleAggregator", 
          typeof(SampleAggregator), 
          typeof(PolygonWaveFormControl), 
          new PropertyMetadata(null, OnSampleAggregatorChanged));

public SampleAggregator SampleAggregator
{
    get { return (SampleAggregator)this.GetValue(SampleAggregatorProperty); }
    set { this.SetValue(SampleAggregatorProperty, value); }
}
        
private static void OnSampleAggregatorChanged(object sender, DependencyPropertyChangedEventArgs e)
{
    PolygonWaveFormControl control = (PolygonWaveFormControl)sender;
    control.Subscribe();
}

This allows us to bind the PolygonWaveFormControl to the SampleAggregator made public on our DataContext:

<my:PolygonWaveFormControl 
    Height="40" 
    SampleAggregator="{Binding SampleAggregator}" />

Trimming the Audio

We have created a temporary WAV file, but before the user saves it to a file of their choosing, we want to allow them to trim off any unwanted parts from the start and end of the recording. To do this I would like to display the entire recorded waveform, with a selection rectangle superimposed on top to allow a sub-range to be selected.

Figure 5 - GUI to allow selection of a portion of the recorded audio

To accomplish this kind of interface we need three components. The first is a ScrollViewer. The ScrollViewer allows us to scroll left and right through the WaveForm if it is too big to fit onto a screen, which is likely if you record more than a few seconds of audio.

The second is a new type of WaveForm renderer that will render an entire file, rather than my PolygonWaveFormControl which started again at the left when the screen filled up. For this I created WaveFormVisual which uses DrawingVisual objects to draw the entire WaveForm. Obviously if we wanted to record for a long period, this approach would need to be optimised as the polygon it creates would have thousands of points, but for short recordings, it works fine.

The third piece was the hardest to get right – the selection rectangle to support mouse dragging selection of the waveform. For this I created the RangeSelectionControl.

The RangeSelectionControl is simply a blue rectangle with a solid outline and semi-transparent fill sitting on a Canvas. The magic occurs in the mouse handler. We need to detect when the user hovers over the left or right edge of the rectangle, and set the cursor to show a horizontal resizing icon. This can be done in the MouseMove event, checking the X coordinate and then setting the Cursor property:

Cursor = Cursors.SizeWE;

When the user clicks the left-button while over the edge, we begin to drag. Key to this is calling Canvas.CaptureMouse. If we don't do this, as soon as you try to drag the rectangle bigger, the mouse move events are lost to other controls underneath.

void RangeSelectionControl_MouseDown(object sender, MouseButtonEventArgs e)
{
    if (e.LeftButton == MouseButtonState.Pressed)
    {
        Point position = e.GetPosition(this);
        Edge edge = EdgeAtPosition(position.X);
        DragEdge = edge;
        if (DragEdge != Edge.None)
        {
            mainCanvas.CaptureMouse();
        }
    }
}

Now in the MouseMove methods, we can change the Canvas.Left and Width properties of the rectangle to resize it.

The ScrollViewer is quite straightforward to use, but you must remember to set CanContentScroll property to true, and also to set the size of the items within the ScrollViewer correctly.

<ScrollViewer CanContentScroll="True" 
         HorizontalScrollBarVisibility="Visible" 
         VerticalScrollBarVisibility="Hidden">
   <Grid>
       <my:WaveFormVisual Height="100" 
           HorizontalAlignment="Left" 
           x:Name="waveFormRenderer"/>
       <my:RangeSelectionControl 
           HorizontalAlignment="Left" 
           x:Name="rangeSelection" />
   </Grid>
</ScrollViewer>

We set the appropriate Width of the WaveFormVisual and RangeSelectionControl based on the total number of points we have drawn in the waveform.

Saving the Audio

So we are finally ready to save the audio. We will offer the user two choices of format to save in. The first is simply to save as a WAV file. If the user has selected the entire recording, we only need to copy the audio across to their desired location. If, however, the user has selected a sub-range, then we need to trim the WAV file. This can be quickly accomplished using a TrimWavFile utility function that copies from a WAV file reader to a WAV file writer, skipping over a certain number of bytes from the beginning and end.

public static void TrimWavFile(string inPath, string outPath, 
                TimeSpan cutFromStart, TimeSpan cutFromEnd)
{
    using (WaveFileReader reader = new WaveFileReader(inPath))
    {
        using (WaveFileWriter writer = 
               new WaveFileWriter(outPath, reader.WaveFormat))
        {
            int bytesPerMillisecond = 
                reader.WaveFormat.AverageBytesPerSecond / 1000;

            int startPos = (int)cutFromStart.TotalMilliseconds * 
                           bytesPerMillisecond;
            startPos = startPos - startPos % reader.WaveFormat.BlockAlign;

            int endBytes = (int)cutFromEnd.TotalMilliseconds * 
                           bytesPerMillisecond;
            endBytes = endBytes - endBytes % reader.WaveFormat.BlockAlign;
            int endPos = (int)reader.Length - endBytes; 

            TrimWavFile(reader, writer, startPos, endPos);
        }
    }
}

private static void TrimWavFile(WaveFileReader reader, 
                    WaveFileWriter writer, int startPos, int endPos)
{
    reader.Position = startPos;
    byte[] buffer = new byte[1024];
    while (reader.Position < endPos)
    {
        int bytesRequired = (int)(endPos - reader.Position);
        if (bytesRequired > 0)
        {
            int bytesToRead = Math.Min(bytesRequired, buffer.Length);
            int bytesRead = reader.Read(buffer, 0, bytesToRead);
            if (bytesRead > 0)
            {
                writer.WriteData(buffer, 0, bytesRead);
            }
        }
    }
}

We also want to offer the ability to save as MP3. The easiest way to create MP3 files is to use the open source LAME MP3 encoder (do a web search for lame.exe to get hold of this application if you haven't already got it). Our application will look in the current directory, and prompt the user to find lame.exe if it is not present, as we do not include it in the application download. Assuming you do provide a valid path, we can then convert our (trimmed) WAV file to MP3 by simply calling lame.exe with the appropriate parameters.

public static void ConvertToMp3(string lameExePath, 
     string waveFile, string mp3File)
{
   Process converter = Process.Start(lameExePath, "-V2 \"" + waveFile 
                            + "\" \"" + mp3File + "\"");
   converter.WaitForExit();
}

We end up with a nice compact MP3 file containing the selected portion of our microphone recording.

Exploring the Sample Code Solution

The main WPF sample application is found in the VoiceRecorder project. This contains the main window along with the three views and their associated ViewModels. VoiceRecorder.Core contains some WPF helper classes and user controls to help with the plumbing and GUI of the application, while VoiceRecorder.Audio contains the classes that actually perform the recording, editing and converting of audio.

About the Author

Mark Heath is a software developer currently working for NICE CTI Systems in Southampton, UK. He specializes in .NET development with a particular focus on client side technologies and audio playback. He blogs about audio, WPF, Silverlight and software engineering best practices at http://mark-dot-net.blogspot.com. He is the author of several open source projects hosted at CodePlex, including NAudio, a low-level .NET audio toolkit (http://www.codeplex.com/naudio).

Inspiration

There are a number of examples and tutorials out there to help get started creating games. You will find the basics of this game to be very similar to a Visual C# Coding4Fun article here. This article will show you how to create a shooter style game in Silverlight. My first Silverlight game experience was playing with a Asteroids style game by Bill Reiss. My usage of vectors and how to do a game loop came from great examples like his and others. There are a number of sites and blogs where you can get good info on game development with Silverlight. Here are just a few.

• BlueRoseGames.com
• Farseer Games
• Cameron Albert
• Mike Snow

Basic Layout

To begin, Open Visual Studio and create a new Silverlight Application in either C# or VB.net. First, we will start with the layout of the screen. First, I added a canvas to the grid inside the Page.xaml that is created for you. I set it's background to black, and named it 'gameRoot'.

<UserControl x:Class="SimpleShooter.Page"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" 
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
    Width="400" Height="300">
   <Grid x:Name="LayoutRoot">
        <Canvas x:Name="gameRoot" Width="500" Height="400" Background="Black" >
        </Canvas>
    </Grid>
</UserControl>

Next we will start adding controls to our project for things like game entities, and information displays. In this example. To complete layout of the game, add user controls to the progect called: Info, LivesRemaining, Score, and WaveInfo. In those controls, To begin I added basic TextBlocks to display information on the state of the game. Please note there are better ways to arrange things in a user control, beyond what I show here. I am keeping to a simple canvas and direct positioning to introduce the concept of Canvas.Top and Canvas.Left. I have now placed these controls on Page.xaml and assigned an x:Name to each. At this point, the controls have only a TextBlock in them.

<UserControl x:Class="SimpleShooter.Page"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" 
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
    xmlns:SimpleShooter="clr-namespace:SimpleShooter"
    Width="500" Height="400">
    <Grid x:Name="LayoutRoot">
        <Canvas x:Name="gameRoot" Width="500" Height="400" Background="Black">
            <SimpleShooter:RemainingLives x:Name="ctlLives" Canvas.Top="380" Canvas.Left="10" />
            <SimpleShooter:Score x:Name="ctlScore" Canvas.Top="10" Canvas.Left="10" />
            <SimpleShooter:WaveInfo x:Name="ctlWaveInfo" Canvas.Left="440" Canvas.Top="10" />
            <SimpleShooter:Info x:Name="ctlInfo" Canvas.Top="10"/>
        </Canvas>
    </Grid>
</UserControl>

To finish our layout lets add a star field. To do this, we will write function to generate random numbers, and another that will randomly distribute ellipses on our base canvas. We will need to inherit from System.Security.Cryptography for this:

C#

public partial class Page : UserControl
{
    public Page()
    {
        InitializeComponent();

        GenerateStarField(350);
    }

    void GenerateStarField(int numberOfStars)
    {

        for (int i = 0; i < numberOfStars; i++)
        {

            Ellipse star = new Ellipse();
            double size = GetRandInt(10, 800) * .01;
            star.Width = size;
            star.Height = size;
            star.Opacity = GetRandInt(1, 5) * .1;
            star.Fill = new SolidColorBrush(Colors.White);
            int x = GetRandInt(0, (int)Math.Round(gameRoot.Height, 0));
            int y = GetRandInt(0, (int)Math.Round(gameRoot.Width, 0));
            star.SetValue(Canvas.TopProperty, (double)x);
            star.SetValue(Canvas.LeftProperty, (double)y);
            gameRoot.Children.Add(star);
        }
    }

    public int GetRandInt(int min, int max)
    {
        Byte[] rndBytes = new Byte[10];
        RNGCryptoServiceProvider rndC = new RNGCryptoServiceProvider();
        rndC.GetBytes(rndBytes);
        int seed = BitConverter.ToInt32(rndBytes, 0);
        Random rand = new Random(seed);
        return rand.Next(min, max);
    }
}

VB

Partial Public Class Page
    Inherits UserControl

    Public Sub New()
        InitializeComponent()
        GenerateStarField(350)
    End Sub

    Private Sub GenerateStarField(ByVal numberOfStars As Integer)

        For i As Integer = 0 To numberOfStars - 1

            Dim star As New Ellipse()
            Dim size As Double = GetRandInt(10, 800) * 0.01
            star.Width = size
            star.Height = size
            star.Opacity = GetRandInt(1, 5) * 0.1
            star.Fill = New SolidColorBrush(Colors.White)
            Dim x As Integer = GetRandInt(0, CInt(Math.Round(gameRoot.Height, 0)))
            Dim y As Integer = GetRandInt(0, CInt(Math.Round(gameRoot.Width, 0)))
            star.SetValue(Canvas.TopProperty, CDbl(x))
            star.SetValue(Canvas.LeftProperty, CDbl(y))
            gameRoot.Children.Add(star)
        Next
    End Sub

    Public Function GetRandInt(ByVal min As Integer, ByVal max As Integer) As Integer
        Dim rndBytes As [Byte]() = New [Byte](9) {}
        Dim rndC As New RNGCryptoServiceProvider()
        rndC.GetBytes(rndBytes)
        Dim seed As Integer = BitConverter.ToInt32(rndBytes, 0)
        Dim rand As New Random(seed)
        Return rand.[Next](min, max)
    End Function
End Class

We now have a function called GenerateStarField. Note that it is adding each ellipse to the 'Children' of our base gameRoot canvas, and how the Top and Left properties determine the positions of those ellipses. Now we have a basic layout of our game and a background.

Sprites and Vectors

I am not going to go in to great detail on sprites or vectors. There are a number of great resources (see other articles linked to in this post), as well as another good Coding4Fun article on this topic. We are however, going to add a class to represent a sprite and a vector to our code:

C#

public abstract class Sprite
{
    public double Width { get; set; }
    public double Height { get; set; }
    public Vector Velocity { get; set; }
    public Canvas SpriteCanvas { get; set; }
    private Point _position;
    public Point Position
    {
        get
        {
            return _position;
        }
        set
        {
            _position = value;
            SpriteCanvas.SetValue(Canvas.TopProperty, _position.Y - (Height / 2));
            SpriteCanvas.SetValue(Canvas.LeftProperty, _position.X - (Width / 2));
        }
    }

    public Sprite(Double width, Double height, Point position)
    {
        Width = width;
        Height = height;

        SpriteCanvas = RenderSpriteCanvas();

        SpriteCanvas.Width = width;
        SpriteCanvas.Height = height;
        //NOTE: because the setter for Position uses both Height and Width, it is important this comes after they are set.
        Position = position;
    }

    public abstract Canvas RenderSpriteCanvas();

    public Canvas LoadSpriteCanvas(string xamlPath)
    {
        System.IO.Stream s = this.GetType().Assembly.GetManifestResourceStream(xamlPath);
        return (Canvas)XamlReader.Load(new System.IO.StreamReader(s).ReadToEnd());
    }

    public virtual void Update(TimeSpan elapsedTime)
    {
        Position = (Position + Velocity * elapsedTime.TotalSeconds);
    }
}

VB

Public MustInherit Class Sprite
    Private _Width As Double
    Public Property Width() As Double
        Get
            Return _Width
        End Get
        Set(ByVal value As Double)
            _Width = value
        End Set
    End Property
    Private _Height As Double
    Public Property Height() As Double
        Get
            Return _Height
        End Get
        Set(ByVal value As Double)
            _Height = value
        End Set
    End Property
    Private _Velocity As Vector
    Public Property Velocity() As Vector
        Get
            Return _Velocity
        End Get
        Set(ByVal value As Vector)
            _Velocity = value
        End Set
    End Property
    Private _SpriteCanvas As Canvas
    Public Property SpriteCanvas() As Canvas
        Get
            Return _SpriteCanvas
        End Get
        Set(ByVal value As Canvas)
            _SpriteCanvas = value
        End Set
    End Property
    Private _position As Point
    Public Property Position() As Point
        Get
            Return _position
        End Get
        Set(ByVal value As Point)
            _position = value
            SpriteCanvas.SetValue(Canvas.TopProperty, _position.Y - (Height / 2))
            SpriteCanvas.SetValue(Canvas.LeftProperty, _position.X - (Width / 2))
        End Set
    End Property

    Public Sub New(ByVal initialWidth As [Double], ByVal initialHeight As [Double], ByVal initialPosition As Point)
        Width = initialWidth
        Height = initialHeight

        SpriteCanvas = RenderSpriteCanvas()

        SpriteCanvas.Width = Width
        SpriteCanvas.Height = Height
        'NOTE: because the setter for Position uses both Height and Width, it is important this comes after they are set. 
        Position = initialPosition
    End Sub

    Public MustOverride Function RenderSpriteCanvas() As Canvas

    Public Function LoadSpriteCanvas(ByVal xamlPath As String) As Canvas
        Dim s As System.IO.Stream = Me.[GetType]().Assembly.GetManifestResourceStream(xamlPath)
        Return DirectCast(XamlReader.Load(New System.IO.StreamReader(s).ReadToEnd()), Canvas)
    End Function

    Public Overridable Sub Update(ByVal elapsedTime As TimeSpan)
        Position = (Position + Velocity * elapsedTime.TotalSeconds)
    End Sub
End Class

This Sprite class will give us the basis for entities in the game such as a ship, aliens, and projectiles. For all these items, we need to know the location of the item, the site, and what it looks like. We will take advantage of Point to track our position, and a property of type Canvas to display the XAML for each of these items. The constructor sets these initial parameters, and calls a RenderSpriteCanvas method to be implemented by each class that inherits from it. This method allows the inheriting class to set the contents of the canvas and therefore as control of the way the sprite looks.

Next we have our vector class that will help us control movement of our sprite. Again, I will not go into detail on vectors as this code is borrowed and adapted from many other readily available sources:

C#

public struct Vector
{
    public double X;
    public double Y;

    public Vector(double x, double y)
    {
        X = x;
        Y = y;
    }

    public double Length
    {
        get
        {
            return Math.Sqrt(LengthSquared);
        }
    }

    public double LengthSquared
    {
        get
        {
            return X * X + Y * Y;
        }
    }

    public void Normalize()
    {
        double length = Length;
        X /= length;
        Y /= length;
    }

    public static Vector operator -(Vector vector)
    {
        return new Vector(-vector.X, -vector.Y);
    }

    public static Vector operator *(Vector vector, double scalar)
    {
        return new Vector(scalar * vector.X, scalar * vector.Y);
    }

    public static Point operator +(Point point, Vector vector)
    {
        return new Point(point.X + vector.X, point.Y + vector.Y);
    }

    static public Vector CreateVectorFromAngle(double angleInDegrees, double length)
    {
        double x = Math.Sin(DegreesToRadians(180 - angleInDegrees)) * length;
        double y = Math.Cos(DegreesToRadians(180 - angleInDegrees)) * length;
        return new Vector(x, y);
    }

    static public double DegreesToRadians(double degrees)
    {
        double radians = ((degrees / 360) * 2 * Math.PI);
        return radians;
    }
}

VB

Public Structure Vector
    Public X As Double
    Public Y As Double

    Public Sub New(ByVal x__1 As Double, ByVal y__2 As Double)
        X = x__1
        Y = y__2
    End Sub

    Public ReadOnly Property Length() As Double
        Get
            Return Math.Sqrt(LengthSquared)
        End Get
    End Property

    Public ReadOnly Property LengthSquared() As Double
        Get
            Return X * X + Y * Y
        End Get
    End Property

    Public Sub Normalize()
        Dim length__1 As Double = Length
        X /= length__1
        Y /= length__1
    End Sub

    Public Shared Operator -(ByVal vector As Vector) As Vector
        Return New Vector(-vector.X, -vector.Y)
    End Operator

    Public Shared Operator *(ByVal vector As Vector, ByVal scalar As Double) As Vector
        Return New Vector(scalar * vector.X, scalar * vector.Y)
    End Operator

    Public Shared Operator +(ByVal point As Point, ByVal vector As Vector) As Point
        Return New Point(point.X + vector.X, point.Y + vector.Y)
    End Operator


    Public Shared Function CreateVectorFromAngle(ByVal angleInDegrees As Double, ByVal length As Double) As Vector
        Dim x As Double = Math.Sin(DegreesToRadians(180 - angleInDegrees)) * length
        Dim y As Double = Math.Cos(DegreesToRadians(180 - angleInDegrees)) * length
        Return New Vector(x, y)
    End Function

    Public Shared Function DegreesToRadians(ByVal degrees As Double) As Double
        Dim radians As Double = ((degrees / 360) * 2 * Math.PI)
        Return radians
    End Function
End Structure

Now we can implement Sprite with a new Ship class. Add a class called Ship, and a file called Ship.xaml to your project. Be sure to set the properties of Ship.xaml to 'Embedded Resource'. We now need to inherit from our Sprite class:

C#

public class Ship : Sprite
{
    public Ship(double width, double height, Point firstPosition)
        : base(width, height, firstPosition)
    {

    }

    public override Canvas RenderSpriteCanvas()
    {
        return LoadSpriteCanvas("SimpleShooter.Sprites.Ship.xaml");
    }
}

VB

Public Class Ship
    Inherits Sprite
    Public Sub New(ByVal width As Double, ByVal height As Double, ByVal firstPosition As Point)
        MyBase.New(width, height, firstPosition)

    End Sub

    Public Overloads Overrides Function RenderSpriteCanvas() As Canvas
        Return LoadSpriteCanvas("SimpleShooter.Ship.xaml")
    End Function
End Class

When Ship is instantiated, it calls the constructor of it's base, Sprite. It also implements the RenderSpriteCanvas method and specifies the XAML (a simple white square) to load into the sprite's canvas. Now we are ready to add a sprite to our main page. In this simple game, we will have only one ship (the others will be aliens), so lets add a property to our page, and a function that will instantiate our ship:

<Canvas x:Name="LayoutRoot" Width="30" Height="30" 
    xmlns="http://schemas.microsoft.com/client/2007"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" >
    <Rectangle Height="30" Width="30" Fill="White" />
</Canvas>

C#

void InitializeGame()
{
    PlayerShip = new Ship(10, 10, new Point(100, 300));
    gameRoot.Children.Add(PlayerShip.SpriteCanvas);
}

VB

Private Sub InitializeGame()
    PlayerShip = New Ship(10, 10, New Point(100, 300))
    gameRoot.Children.Add(PlayerShip.SpriteCanvas)
End Sub

We can now call this method from our page constructor and when we run the project, we get a white square (our ship) at the bottom left of our Page. Taking a closer look, we set our page size to 500 x 400, and the InitializeGame method places our Ship at a point 100 pixels from the left of the gameRoot canvas, and 300pixels from the top.

Keyboard Input and the Game Loop

Now we are ready to make some things move. To start, we need choose keys on the keyboard to allow movement, and act on those events. We then need to capture those key presses and act on them if they are relevant to our game loop. Once again, may examples of key handlers and game loops are readily available, so I won't go into detail. The keyboard handler captures all key up and down events. We can therefore ask our instance of the handler if a key is pressed at any given time. The game loop is just that, a constant loop. It consists of a storyboard that has the start called on it and it immediately ends. The class raises an event, and starts the storyboard again. Subscribers to Update event are provided a value that reports the number of milliseconds since the last update. That value can be applied to vectors to apply smooth movement to sprites. To take advantage of these classes, we need to add an instance of both a KeyHandler and GameLoop to our Page. To do this, we update the InitializeGame and Page constructor, and add a handler for the GameLoop:

C#

public Page()
{
    InitializeComponent();
    keyHandler = new KeyHandler(this);
    GenerateStarField(350);
    InitializeGame();
}

void InitializeGame()
{
    gameLoop = new GameLoop(this);
    gameLoop.Update += new GameLoop.UpdateHandler(gameLoop_Update);

    PlayerShip = new Ship(10, 10, new Point(100, 360));
    gameRoot.Children.Add(PlayerShip.SpriteCanvas);

    gameLoop.Start();
}

void gameLoop_Update(TimeSpan elapsed)
{
    //clear the current vector so the sprite is not moving unless a keys is pressed
    PlayerShip.Velocity = new Vector(0, 0);
    if (keyHandler.IsKeyPressed(Key.Left))
    {
        PlayerShip.Velocity = Vector.CreateVectorFromAngle(270, 125);
    }
    if (keyHandler.IsKeyPressed(Key.Right))
    {
        PlayerShip.Velocity = Vector.CreateVectorFromAngle(90, 125);
    }
    PlayerShip.Update(elapsed);
}

VB

Public Sub New()
    InitializeComponent()
    keyHandler = New KeyHandler(Me)
    GenerateStarField(350)
    InitializeGame()
End Sub

Private Sub InitializeGame()
    gameLoop = New GameLoop(Me)
    AddHandler gameLoop.Update, AddressOf gameLoop_Update

    PlayerShip = New Ship(10, 10, New Point(100, 360))
    gameRoot.Children.Add(PlayerShip.SpriteCanvas)

    gameLoop.Start()
End Sub

Private Sub gameLoop_Update(ByVal elapsed As TimeSpan)
    'clear the current vector so the sprite is not moving unless a keys is pressed 
    PlayerShip.Velocity = New Vector(0, 0)
    If keyHandler.IsKeyPressed(Key.Left) Then
        PlayerShip.Velocity = Vector.CreateVectorFromAngle(270, 125)
    End If
    If keyHandler.IsKeyPressed(Key.Right) Then
        PlayerShip.Velocity = Vector.CreateVectorFromAngle(90, 125)
    End If

    PlayerShip.Update(elapsed)
End Sub

Now we have a functioning game loop. Launch the app, click on the Silverlight control to give it focus, and you can now use the arrow keys to move our ship right and left. One problem however is that you are able to take the ship completely off the screen. To prevent this, we can add MinX and MaxX properties to our ship class and override the Update method it inherits from Sprite. Be sure to also add those min and max values after the Ship is instantiated in InitializeGame of our Page:

C#

public class Ship : Sprite
{
    public double MaxX { get; set; }
    public double MinX { get; set; }

    public Ship(double width, double height, Point firstPosition)
        : base(width, height, firstPosition)
    {

    }

    public override Canvas RenderSpriteCanvas()
    {
        return LoadSpriteCanvas("SimpleShooter.Sprites.Ship.xaml");
    }

    public override void Update(System.TimeSpan elapsedTime)
    {
        //verify that this is a position we can move to
        if (Position.X > MaxX)
        {
            Position = new Point(MaxX, Position.Y);
            Velocity = new Vector(0, 0);
        }
        if (Position.X < MinX)
        {
            Position = new Point(MinX, Position.Y);
            Velocity = new Vector(0, 0);
        }
        base.Update(elapsedTime);
    }
}

VB

Public Class Ship
    Inherits Sprite
    Private _MaxX As Double
    Public Property MaxX() As Double
        Get
            Return _MaxX
        End Get
        Set(ByVal value As Double)
            _MaxX = value
        End Set
    End Property
    Private _MinX As Double
    Public Property MinX() As Double
        Get
            Return _MinX
        End Get
        Set(ByVal value As Double)
            _MinX = value
        End Set
    End Property

    Public Sub New(ByVal width As Double, ByVal height As Double, ByVal firstPosition As Point)
        MyBase.New(width, height, firstPosition)

    End Sub

    Public Overloads Overrides Function RenderSpriteCanvas() As Canvas
        Return LoadSpriteCanvas("SimpleShooter.Ship.xaml")
    End Function

    Public Overloads Overrides Sub Update(ByVal elapsedTime As System.TimeSpan)
        'verify that this is a position we can move to 
        If Position.X > MaxX Then
            Position = New Point(MaxX, Position.Y)
            Velocity = New Vector(0, 0)
        End If
        If Position.X < MinX Then
            Position = New Point(MinX, Position.Y)
            Velocity = New Vector(0, 0)
        End If
        MyBase.Update(elapsedTime)
    End Sub
End Class

Prepare to Fire!

We now have all the basic plumbing to add additional sprites such as aliens or projectiles. To begin, we will add additional classes inheriting from Sprite: Alien, Missle, and Bomb, along with Alien.xaml, Missle.xaml, and Bomb.xaml (these new .xaml files need to be set as Embedded Resources). These xaml files are just like our Ship.xaml with different sizes and colors. Make the Aliens and their bombs red, and shrink the bomb and missile height and width to 5. The xaml is very similar, but the classes themselves are going to have a few differing capabilities. Bomb and Missile are very similar. They only need to load their corresponding xaml. Here is our Bomb class for example:

C#

public class Bomb : Sprite
{
    public double MaxX { get; set; }
    public double MinX { get; set; }

    public Bomb(double width, double height, Point firstPosition)
        : base(width, height, firstPosition)
    {

    }

    public override Canvas RenderSpriteCanvas()
    {
        return LoadSpriteCanvas("SimpleShooter.Sprites.Bomb.xaml");
    }

    public override void Update(System.TimeSpan elapsedTime)
    {
        base.Update(elapsedTime);
    }
}

VB

Public Class Bomb
    Inherits Sprite
    Private _MaxX As Double
    Public Property MaxX() As Double
        Get
            Return _MaxX
        End Get
        Set(ByVal value As Double)
            _MaxX = value
        End Set
    End Property
    Private _MinX As Double
    Public Property MinX() As Double
        Get
            Return _MinX
        End Get
        Set(ByVal value As Double)
            _MinX = value
        End Set
    End Property

    Public Sub New(ByVal width As Double, ByVal height As Double, ByVal firstPosition As Point)
        MyBase.New(width, height, firstPosition)

    End Sub

    Public Overloads Overrides Function RenderSpriteCanvas() As Canvas
        Return LoadSpriteCanvas("SimpleShooter.Sprites.Bomb.xaml")
    End Function

    Public Overloads Overrides Sub Update(ByVal elapsedTime As System.TimeSpan)
        MyBase.Update(elapsedTime)
    End Sub
End Class

We now need to add an Alien class similar to our Ship class. These will both need a bit more capability. For one, we are going to have them both fire at each other. In the case of the Alien, it will be firing down, and it's monition of choice will be a bomb:

C#

public class Alien : Sprite
{
    public double fireRateMilliseconds = 2000;
    public double fireVelocity = 250;
    public double wayPointMin;
    public double wayPointMax;
    public double speed = 100;
    public bool spawnWait;
    public DateTime spawnComplete;
    public double MaxX { get; set; }
    public double MinX { get; set; }

    public Alien(double width, double height, Point firstPosition)
        : base(width, height, firstPosition)
    {

    }

    public void CheckDirection()
    {
        if (Position.X > wayPointMax)
        {
            Velocity = Vector.CreateVectorFromAngle(270, speed);
        }
        if (Position.X < wayPointMin)
        {
            Velocity = Vector.CreateVectorFromAngle(90, speed);
        }
    }

    public override Canvas RenderSpriteCanvas()
    {
        return LoadSpriteCanvas("SimpleShooter.Sprites.Alien.xaml");
    }

    public override void Update(TimeSpan elapsedTime)
    {
        CheckDirection();
        base.Update(elapsedTime);
    }

    public Bomb Fire()
    {
        Bomb bomb = new Bomb(5, 5, Position);
        bomb.Velocity = Vector.CreateVectorFromAngle(180, fireVelocity);
        return bomb;
    }
}

VB

Public Class Alien
    Inherits Sprite
    Public fireRateMilliseconds As Double = 2000
    Public fireVelocity As Double = 250
    Public wayPointMin As Double
    Public wayPointMax As Double
    Public speed As Double = 100
    Public spawnWait As Boolean
    Public spawnComplete As DateTime
    Private _MaxX As Double
    Public Property MaxX() As Double
        Get
            Return _MaxX
        End Get
        Set(ByVal value As Double)
            _MaxX = value
        End Set
    End Property
    Private _MinX As Double
    Public Property MinX() As Double
        Get
            Return _MinX
        End Get
        Set(ByVal value As Double)
            _MinX = value
        End Set
    End Property

    Public Sub New(ByVal width As Double, ByVal height As Double, ByVal firstPosition As Point)
        MyBase.New(width, height, firstPosition)

    End Sub

    Public Sub CheckDirection()
        If Position.X > wayPointMax Then
            Velocity = Vector.CreateVectorFromAngle(270, speed)
        End If
        If Position.X < wayPointMin Then
            Velocity = Vector.CreateVectorFromAngle(90, speed)
        End If
    End Sub

    Public Overloads Overrides Function RenderSpriteCanvas() As Canvas
        Return LoadSpriteCanvas("SimpleShooter.Alien.xaml")
    End Function

    Public Overloads Overrides Sub Update(ByVal elapsedTime As TimeSpan)
        CheckDirection()
        MyBase.Update(elapsedTime)
    End Sub

    Public Function Fire() As Bomb
        Dim bomb As New Bomb(5, 5, Position)
        bomb.Velocity = Vector.CreateVectorFromAngle(180, fireVelocity)
        Return bomb
    End Function
End Class

Now that we are going to add munitions to the game, we need to be able to know when these munitions collide with other sprites. To do this, we are going to add a collision detection method to our sprite class. As a form of simple collision detection, we will give each sprite a CollisionRadius from its center point. To keep it simple we will make the radius one half the width of our sprites. A vector created from these two points can then detect if the sum of those two radius is greater than the length of our vector. If it is, they have collided:

C#

public static bool Collides(Sprite s1, Sprite s2)
{
    Vector v = new Vector((s1.Position.X) - (s2.Position.X), (s1.Position.Y) - (s2.Position.Y));
    if (s1.CollisionRadius + s2.CollisionRadius > v.Length)
    {
        return true;
    }
    else
    {
        return false;
    }
}

VB

Public Shared Function Collides(ByVal s1 As Sprite, ByVal s2 As Sprite) As Boolean
    Dim v As New Vector((s1.Position.X) - (s2.Position.X), (s1.Position.Y) - (s2.Position.Y))
    If s1.CollisionRadius + s2.CollisionRadius > v.Length Then
        Return True
    Else
        Return False
    End If
End Function

We are now ready to extend our game loop of our Page to do more than track the movement of our Ship. We begin by taking action on more than just the arrow keys. We will wire up the space bar for shooting. Lets add an enumeration to track the state of our game. We will also add a supporting class that will handle the firing of munitions and adding those to our game. We will keep this simple for now, but it is an obvious place to being extending this code to take advantage of things like partial classes that could contain functionality like our 'Fire' method:

C#

public enum GameState
{
    Ready = 0,
    Running = 1,
    Paused = 2,
    BetweenWaves = 3,
    GameOver = 4
}

if (keyHandler.IsKeyPressed(Key.Space))
{
    switch (status)
    {
        case GameState.Ready:
            break;
        case GameState.Running:
            EntityFired(PlayerShip);
            break;
        case GameState.Paused:
            break;
        case GameState.BetweenWaves:
            status = GameState.Running;
            ctlInfo.GameInfo = "";
            StartWave();
            break;
        case GameState.GameOver:
            break;
        default:
            break;
    }
}

void EntityFired(Sprite shooter)
{
    Debug.WriteLine(shooter);
    switch (shooter.ToString())
    {
        case "SimpleShooter.Ship":
            if (missles.Count == 0)
            {
                Missle missle = ((Ship)shooter).Fire();
                missles.Add(missle);
                gameRoot.Children.Add(missle.SpriteCanvas);

            }
            break;
        case "SimpleShooter.Alien":
            Bomb bomb = ((Alien)shooter).Fire();
            bombs.Add(bomb);
            gameRoot.Children.Add(bomb.SpriteCanvas);
            break;
        default:
            break;
    }
}

VB

Public Enum GameState
    Ready = 0
    Running = 1
    Paused = 2
    BetweenWaves = 3
    GameOver = 4
End Enum

If keyHandler.IsKeyPressed(Key.Space) Then
    Select Case status
        Case GameState.Ready
            Exit Select
        Case GameState.Running
            EntityFired(PlayerShip)
            Exit Select
        Case GameState.Paused
            Exit Select
        Case GameState.BetweenWaves
            status = GameState.Running
            ctlInfo.GameInfo = ""
            StartWave()
            Exit Select
        Case GameState.GameOver
            Exit Select
        Case Else
            Exit Select
    End Select
End If

Private Sub EntityFired(ByVal shooter As Sprite)
    Debug.WriteLine(shooter)
    Select Case shooter.ToString()
        Case "SimpleShooter.Ship"
            If missles.Count = 0 Then
                Dim missle As Missle = DirectCast(shooter, Ship).Fire()
                missles.Add(missle)

                gameRoot.Children.Add(missle.SpriteCanvas)
            End If
            Exit Select
        Case "SimpleShooter.Alien"
            Dim bomb As Bomb = DirectCast(shooter, Alien).Fire()
            bombs.Add(bomb)
            gameRoot.Children.Add(bomb.SpriteCanvas)
            Exit Select
        Case Else
            Exit Select
    End Select
End Sub

To make it easier to track the state of our game, let's add public properties to our controls and let them be the keepers of those game states. For instance, give the ReamainingLives control a Lives property, and have the setter of that property also update the TextBlock on the control to show the user how many lives are left. Perform a similar task for the other three controls:

C#

public partial class RemainingLives : UserControl
{
    private int _lives;
    public int Lives
    {
        get { return _lives; }
        set
        {
            _lives = value;
            string livesString = string.Empty;
            for (int i = 0; i < _lives - 1; i++)
            {
                livesString = string.Format("{0}{1}", livesString, "A");
            }
            txtRemainingLives.Text = livesString;
        }
    }

    public RemainingLives()
    {
        InitializeComponent();
    }
}

VB

Partial Public Class RemainingLives
    Inherits UserControl
    Private _lives As Integer
    Public Property Lives() As Integer
        Get
            Return _lives
        End Get
        Set(ByVal value As Integer)
            _lives = value
            Dim livesString As String = String.Empty
            For i As Integer = 0 To _lives - 2
                livesString = String.Format("{0}{1}", livesString, "A")
            Next
            txtRemainingLives.Text = livesString
        End Set
    End Property

    Public Sub New()
        InitializeComponent()
    End Sub
End Class

Now we have a number of other things to add to our page class. We are going to have lots of entities to deal with in our game loop. We will have a ship, x number of Aliens, x number of Bombs, and at some points we will have our Missiles, though we restricted that to one at a time in our EntityFired method. We need to, in each game loop check if any missile hits any alien or leaves the game map, if any bomb hits our ship or leaves the game map, and if it is time for a Alien to fire at the ship. Our Page already has a property for our Ship, but the Aliens, Bombs, Missiles need to be collections. When we begin to loop through our Bombs for instance, we will want to remove a bomb if it strikes a Ship or leaves the game map. Since we are going to be iterating these collections, we cannot subtract from those collections as we are looping through them. There are a number of approaches for this, but to keep it simple, we will add a corresponding collection for each of our Bomb, Alien, and Missile collections to track the ones that need to be removed and removed from our game canvas. With this approach, our game loop can add to the "remove these" collection, and then take action on them when we leave the main loop for the parent collection:

C#

List<Alien> aliens;
List<Alien> aliensRemove;
List<Alien> alienShooters;
List<Bomb> bombs;
List<Bomb> bombsRemove;
List<Missle> missles;
List<Missle> misslesRemove;

VB

Private aliens As List(Of Alien)
Private aliensRemove As List(Of Alien)
Private alienShooters As List(Of Alien)
Private bombs As List(Of Bomb)
Private bombsRemove As List(Of Bomb)
Private missles As List(Of Missle)
Private misslesRemove As List(Of Missle)

We also want to add a class to our page that can track the waves of aliens we are going to send one at a time. In addition, we can add a collection to hold these waves. Each wave will track the number of aliens that spawn at once, the total number of aliens faced in the wave, how many aliens get to drop bombs, and how frequently they can fire.

C#

public class WaveData
{
    public WaveData(int count, double fireRate, int atOnce, int fireatonce)
    {
        EnemyCount = count;
        fireRateMilliseconds = fireRate;
        enemiesAtOnce = atOnce;
        fireAtOnce = fireatonce;
        waveEmpty = false;
    }
    public int EnemyCount { get; set; }
    public double fireRateMilliseconds { get; set; }
    public int enemiesAtOnce { get; set; }
    public int fireAtOnce { get; set; }
    public bool waveEmpty { get; set; }
}

VB

Public Class WaveData
    Public Sub New(ByVal count As Integer, ByVal fireRate As Double, ByVal atOnce As Integer, ByVal fireatonce__1 As Integer)
        EnemyCount = count
        fireRateMilliseconds = fireRate
        enemiesAtOnce = atOnce
        fireAtOnce = fireatonce__1
        waveEmpty = False
    End Sub
    Private _EnemyCount As Integer
    Public Property EnemyCount() As Integer
        Get
            Return _EnemyCount
        End Get
        Set(ByVal value As Integer)
            _EnemyCount = value
        End Set
    End Property
    Private _fireRateMilliseconds As Double
    Public Property fireRateMilliseconds() As Double
        Get
            Return _fireRateMilliseconds
        End Get
        Set(ByVal value As Double)
            _fireRateMilliseconds = value
        End Set
    End Property
    Private _enemiesAtOnce As Integer
    Public Property enemiesAtOnce() As Integer
        Get
            Return _enemiesAtOnce
        End Get
        Set(ByVal value As Integer)
            _enemiesAtOnce = value
        End Set
    End Property
    Private _fireAtOnce As Integer
    Public Property fireAtOnce() As Integer
        Get
            Return _fireAtOnce
        End Get
        Set(ByVal value As Integer)
            _fireAtOnce = value
        End Set
    End Property
    Private _waveEmpty As Boolean
    Public Property waveEmpty() As Boolean
        Get
            Return _waveEmpty
        End Get
        Set(ByVal value As Boolean)
            _waveEmpty = value
        End Set
    End Property
End Class

We will need to setup an initialization method for our game, to setup all our collections for sprites, and add progressively difficult waves to our game. The meat of it all now lies in our game loop. For each collection of sprites we call methods that will iterate over its contents and make decisions based on collisions, locations of the sprites. After a loop on each main collection, we can use the 'remove us' collection to clear items from the main collection, remove their canvas from our game canvas, and delete them from the main collection. Finally, we check to see if enough time has elapsed to let the aliens drop another bomb:

C#

void gameLoop_Update(TimeSpan elapsed)
{
    //clear the current Vector so the sprite is not moving unless a keys is pressed
    PlayerShip.Velocity = new Vector(0, 0);
    if (keyHandler.IsKeyPressed(Key.Left))
    {
        PlayerShip.Velocity = Vector.CreateVectorFromAngle(270, 125);
    }
    if (keyHandler.IsKeyPressed(Key.Right))
    {
        PlayerShip.Velocity = Vector.CreateVectorFromAngle(90, 125);
    }
    if (keyHandler.IsKeyPressed(Key.Space))
    {
        switch (status)
        {
            case GameState.Ready:
                break;
            case GameState.Running:
                EntityFired(PlayerShip);
                break;
            case GameState.Paused:
                break;
            case GameState.BetweenWaves:
                status = GameState.Running;
                ctlInfo.GameInfo = "";
                StartWave();
                break;
            case GameState.GameOver:
                break;
            default:
                break;
        }
    }
    PlayerShip.Update(elapsed);

    BombLoop(elapsed);
    MissleLoop(elapsed);
    AlienLoop(elapsed);

    foreach (Alien alien in aliensRemove)
    {
        aliens.Remove(alien);
        gameRoot.Children.Remove(alien.SpriteCanvas);
        AlienShot(alien);
    }
    aliensRemove.Clear();

    foreach (Missle missle in misslesRemove)
    {
        missles.Remove(missle);
        gameRoot.Children.Remove(missle.SpriteCanvas);
    }
    misslesRemove.Clear();

    if (nextShot <= DateTime.Now)
    {
        nextShot = DateTime.Now.AddMilliseconds(enemyShootMilliseonds).AddMilliseconds(elapsed.Milliseconds * -1);

        shotsThisPass = shotsAtOnce;
        if (shotsThisPass > aliens.Count)
        {
            shotsThisPass = aliens.Count;
        }

        if (aliens.Count > 0)
        {
            foreach (Alien alien in aliens)
            {
                alienShooters.Add(alien);
            }
        }

        while (alienShooters.Count > shotsThisPass)
        {
            alienShooters.RemoveAt(GetRandInt(0, alienShooters.Count - 1));
        }

        foreach (Alien alien in alienShooters)
        {
            EntityFired(alien);
        }

        alienShooters.Clear();
    }
}

VB

Private Sub gameLoop_Update(ByVal elapsed As TimeSpan)
    'clear the current Vector so the sprite is not moving unless a keys is pressed 
    PlayerShip.Velocity = New Vector(0, 0)
    If keyHandler.IsKeyPressed(Key.Left) Then
        PlayerShip.Velocity = Vector.CreateVectorFromAngle(270, 125)
    End If
    If keyHandler.IsKeyPressed(Key.Right) Then
        PlayerShip.Velocity = Vector.CreateVectorFromAngle(90, 125)
    End If
    If keyHandler.IsKeyPressed(Key.Space) Then
        Select Case status
            Case GameState.Ready
                Exit Select
            Case GameState.Running
                EntityFired(PlayerShip)
                Exit Select
            Case GameState.Paused
                Exit Select
            Case GameState.BetweenWaves
                status = GameState.Running
                ctlInfo.GameInfo = ""
                StartWave()
                Exit Select
            Case GameState.GameOver
                Exit Select
            Case Else
                Exit Select
        End Select
    End If
    PlayerShip.Update(elapsed)

    BombLoop(elapsed)
    MissleLoop(elapsed)
    AlienLoop(elapsed)

    For Each alien As Alien In aliensRemove
        aliens.Remove(alien)
        gameRoot.Children.Remove(alien.SpriteCanvas)
        AlienShot(alien)
    Next
    aliensRemove.Clear()

    For Each missle As Missle In misslesRemove
        missles.Remove(missle)
        gameRoot.Children.Remove(missle.SpriteCanvas)
    Next
    misslesRemove.Clear()

    If nextShot <= DateTime.Now Then
        nextShot = DateTime.Now.AddMilliseconds(enemyShootMilliseonds).AddMilliseconds(elapsed.Milliseconds * -1)

        shotsThisPass = shotsAtOnce
        If shotsThisPass > aliens.Count Then
            shotsThisPass = aliens.Count
        End If

        If aliens.Count > 0 Then
            For Each alien As Alien In aliens
                alienShooters.Add(alien)
            Next
        End If

        While alienShooters.Count > shotsThisPass
            alienShooters.RemoveAt(GetRandInt(0, alienShooters.Count - 1))
        End While

        For Each alien As Alien In alienShooters
            EntityFired(alien)
        Next

        alienShooters.Clear()
    End If
End Sub

Now we can almost call it a game:

Taking it one step further, I jumped into Expression Blend to create a bit more interesting XAML that our sprites are loading into their SpriteCanvas. This XAML can be found in the full download of the source code. The results make our game just a tad bit more interesting:

Conclusion

Silverlight provides a number of capabilities to enable robust game development. It should be noted that in this game, we are only scratching the surface. The core of this example used a game loop to position sprites on a canvas. Additional capabilities within Silverlight such as animations, styling, templating, and visual states provide even more ways to push the limits of browser based games. If you are at all like me, dabbling with game development is a great way to get your feet wet with Silverlight and learn some basics. Happy coding!

If you want to try this out, the download link for the source code is at the top of the article!

About The Author

Roger Guess is the Director of IT for The Wedge Group where he works with technologies such as Silverlight and WPF. He writes games in his spare time, and blogs at SilverlightAddict.com. He can be reached via email at email@rogerguess.net.

1. How Audio Data is Represented
2. Demystifying the WAV Format
3. Synthesizing Simple Wave Audio using C#
4. Algorithms for Different Sound Waves in C#

Dan goes into depth on how audio is represented, explains how the WAV format works then creates different audio waves like the sawtooth, square and sine wave.  He gives the equation then an implementation with c#.  With part 3 and 4, Dan provides full source code so you too can rock out with some audio dial tones.

Skype: I find your lack of feedback disturbing.

This article will discuss how to build a software tool that makes it easier to talk into Skype with a headset.

Recently my family began experimenting with Skype, but we found that talking over Skype can be exhausting with headphones. After doing a little research, I learned that the problem was that we only heard the other party. You'd think that this is a good thing, but we have a social brain that works hard to gauge our behavior and adjust.

The answer is to feed a little bit of the microphone back into the earpiece, so that our brain knows how loud we're talking. This is called side tone in the telecommunication industry. Without it, we talk louder and louder until we're sure that we'll be heard.

I couldn't find a Skype plug-in to do this… but I had just read a Coding4Fun article, by Mark Heath, about adding audio effects to Skype. I decided I was going to write a tool to add this feedback.

How to use the tool

First, let's take a look at how to use the application. Once it is running, there will be a microphone icon in the lower right hand corner of the screen:

Figure 1: Icon in the system notification tray

Clicking on it, will get the application control window:

Figure 2: The application's controls

Let's look at the side tone controls:

• The “Side Tones” check box enables or disables the side tone. If you have headphones, you'll want this on; if you have loud speakers, you'll want it off.
• The slider controls the volume of the microphone in the headset. You can change it while talking. The volume will vary with microphone and headset.

You can do a “sound check” to see if the feedback is working by clicking on the Sound Check button, and adjusting the volume. I found that the volume setting that works best in a conversation is much, much lower than what works in a sound check.

Next, let's look at the AGC (“Automatic Gain Control”) section. When making a call, the software can automatically adjust how loud you sound to the other party:

• When “Low Pass” is checked, the headphone feedback is at 44100 samples/sec. The microphone sound is filtered to keep only the sounds below 8 Khz, converted to 16000/samples per second and sent to Skype. When it is not checked, headphone feedback is at 16000 samples/sec, and sent to Skype without the low pass filter.
• When "Skype AutoGain" is checked, it signals to Skype that Skype can use its own algorithm. When clear, Skype is told not to apply any adjustments.
• When "AutoGain" is checked, the custom Automatic Gain Control algorithm is used.

The Automatic Gain Control has three sliders:

• The “Cutoff” slider controls the distinction between background noise and conversation. Sound below this level is cutoff and silence is sent to Skype. This will vary with microphone – more sensitive (expensive) microphones will pick more background noise and be better with a higher setting.
• The “Normal” slider controls the volume when you are talking normally. The Gain Control tries to raise the volume to this level.
• The “Loud” slider controls the volume when you talk exceptionally loud. This rarely happens, but when you do talk louder than the Normal level, the Gain Control tries to adjust the volume to this level.

How the Software Works

I originally started this project by creating an effect for the Skype Voice Changer. My plan was to open a WaveStream, begin playing it on the headphones and copy the microphone stream to it.

I quickly found that this was not the way to add feedback. The underlying “WaveOut” system had a huge latency: Everything I heard in the headphones was at least a second or more behind what I was saying. This made it even harder to talk than before, and I had to abandon it.

While researching the problem I found a DirectSound code sample that I could modify into doing what I wanted. (This initial prototype didn't coordinate with Skype – it attached to the microphone and copied the sounds to the output, at a lower volume. But it was on always on.) The sound in the headphones is still slightly behind (the microphone) but is barely perceptible, and we'll muffle it a bit more to make it less distinguishable.

From here on I shall describe the major – or technically interesting – components of the program. We'll look at:

1. DirectSound and Circular buffers
2. Sample Window and Sizing the buffers
3. Using WaitHandle's to Synchronize with DirectSound
4. IIR Filters
5. Automatic Gain Control
6. Estimating loudness

Note: I won't be describing how to connect to Skype. Mark Heath's description is very good.

DirectSound and Circular Buffers

The DirectSound code is in AudioLoop.cs. The module sets DirectSound to capture sound from the default recording device at 16bits / sample at either 16000 or 44100 samples per second. The capture and playback buffers are configured in “looping” mode to act as circular buffers. The capture buffer eventually overwrites samples, and we'll lose them if we don't act fast enough; if we don't update the playback buffer, it will repeat the same sound over and over.

The StartMicrophone() procedure sets up the capture and playback buffers. Then it creates a thread to do the work. The thread is at a high priority so that if the OS has a choice between (say) email or processing sound, it does the sound.

The StopMicrophone() procedure stops the worker thread and cleans up the resources.

The software processes a fixed number of samples at a time, called the “sample window.” The buffers are several times the size of sample window, so that the system can keep capturing and playing while the software is processing them. The sound processing loop is the heart of the application:

1. Wait for the next sample window to be ready
2. Copy the samples from the capture buffer
3. Process the samples and send the results to the playback buffer
4. Process the samples and sends the results to Skype

C#

while (Runnable)
{
    for (int I = 0, offset = 0; Runnable && I < _bufferPositions; I++, offset += bufferSize)
    {
        // Wait for the sample areas to be ready
        notificationEvent[I].WaitOne(Timeout.Infinite, true);

        // Get the sound samples
        byte[] buffer = (byte[]) captureBuffer.Read(offset, typeof (byte), LockFlag.None, bufferSize);

        // Convert samples to 16bit PCM 
        for (int L = buffer.Length, J = 10, K = 0; K < L; K += 2)
            PCM16Buffer[J++] = (Int16) ((UInt16) buffer[K] | (((UInt16) buffer[K + 1]) << 8));

        // Play them out to the ear, if applicable
        if (null != playbackBuffer)
        {
            // Perform a low pass filter to "muffle" the sound
            Butterworth(PCM16Buffer, 10, LPSample, Coefs);

            // put the muffled sample into the output buffer
            // -- The lock flag seems to work, but others may work too
            playbackBuffer.Write(Idx, LPSample, LockFlag.None);
            Idx += buffer.Length;
            if (Idx >= 4*bufferSize)
                Idx -= 4*bufferSize;
            if (!playing)
            {
                playbackBuffer.Volume = _Volume;
                playbackBuffer.Play(0, BufferPlayFlags.Looping);
                playing = true;
            }
        }

        // Process the sound and deliver it to Skype
        if (null != outStream)
        {
            int L = AGC.Process(PCM16Buffer, 10, buffer);
            if (0 != L)
                outStream.BeginSend(buffer, 0, L, SocketFlags.None, SendCallback, null);
            // Note: could send out pink noise if L == 0
        }

        // Move the sliding window of the previous 10 samples into the start
        // of the PCM16Buffer
        for (int K = 0, J = PCM16Buffer.Length - 10; K < 10; K++, J++)
            PCM16Buffer[K] = PCM16Buffer[J];
    }
}

Note the “for” loop at the bottom of the code. This preserves the last 10 incoming samples at the start of the buffer. This is needed to make the sound processing smooth, and will be discussed a bit later.

Sample Window and Sizing the Buffers

How big should the sample window be? This is bit of a trade off in responsiveness and design complexity.

I chose a window big enough to hold 10 milliseconds of sound. Since the ear is sensitive sound to delays of even 30 milliseconds, I cut this done so that a delay wouldn't be perceptible. (When I tried a 50 millisecond window, my voice came out the headphone sounding like an echo... and I found myself talking slower and slower.) The sample window could be made smaller, but I am sure that there is a point where the OS won't schedule the audio loop to wake-n-run more frequently. And, as the sample window gets smaller, the processing may drop in quality, because it doesn't have enough to work with.

The capture buffer is 8 times the size of the sample window. This ratio is arbitrary, but I wanted the buffer to be about an order of magnitude larger. My rationale is that if the processing falls behind, the sound – for the Skype call – won't be dropped. I feel that it is more important to preserve sound quality for the other party than to preserve the quality of feedback.

The playback buffer is four times the sample window. I wanted it small, so that if the processing fell behind, the replaying of a sound will seem to be a continuation of a current sound.

When writing the sound to the playback buffer, we have to track where in the buffer to put the samples. I tried to use GetCurrentPosition() to find where to write to next into the playback buffer; this created terrible sound. Instead, the software uses a local variable to track where to write next.

DirectSound and Notifications

How do we keep in sync with the sound capture – how do we know when a sample buffer is ready?

The application gives a table of buffer indices and WaitHandle's to DirectSound. When the capture buffer's write index reaches one of those indices, it signals the corresponding WaitHandle. The worker thread cycles performs a WaitOne() one each of the WaitHandle's, one at a time. As a convenience, we use a specific kind of WaitHandle called AutoResetEvent. This type of WaitHandle sets itself back to a “wait” state once WaitOne() returns.

If the thread has gotten behind, the WaitOne will return immediately, the loop processes the sample, and begins to catch up with the work.

We must use a separate AutoResetEvent for each of the 8 capture windows. The AutoResetEvent doesn't tell us if it was signaled multiple times. If only one AutoResetEvent handle were used, it wouldn't know that two (or more) sample windows were ready. Instead, it would process just one, falling further behind, adding latency. This would happen randomly overtime, and be hard to test consistently.

IIR: Infinite Impulse Response Filters

This project came together so quickly, so easy – once I found the right approach – that I couldn't resist getting fancy. I added a low-pass filter to muffle the feedback a little. And I added automatic gain control, as an experimental option.

For both of these I used a filtering algorithm called “IIR” (this stands for Infinite Impulse Response – but that term is a confusing mouthful, so let's just call it IIR). IIR is a special purpose virtual machine. Low-pass filters, high-pass filters, combinations of those filters, and even equalizers, can be specified, and use very specific techniques (like a compiler) to convert them into an IIR implementation.

(You could, instead, “compile” the filters to be the resistor values to use in a hardware circuit. That's programming in solder!)

The machine code for these IIR virtual machines is just two list of coefficients, called A & B. The software emulator is code that looks like the following bit of code:

C#

Out[0] = Sample[0];
Out[1] = Sample[1];
for (int Idx = 2; Idx < N; Idx++)
{
    Out[Idx] =
    B0 * Sample[Idx]
 + B1 * Sample[Idx - 1]
 + B2 * Sample[Idx - 2]
        // … more like this …
        // Next, the feed back
 - A1 * Out[Idx - 1]
 - A2 * Out[Idx - 2]
        // … more like this …
 ;
}

IIRs are easy to implement - and take less CPU power than other methods. But sometimes they sound poor; if they sound too bad, you'll want to use a different technique. I found that the low-pass filters in this project work will for some microphones, and add a slight crackle to others.

Example Low Pass Filter

For the low pass filter to create the muffling, I used a Butterworth filter, using the code below. It takes a buffer of signed, 16-bit samples, and then converts the 16-bit values into a byte array suitable for the sound buffer.

The filter code is a bit different than the example code in the previous section. Most of the differences are for speed.

• This code doesn't use a buffer for the old values, instead uses separate variables for the elements of the buffer.  It uses I_0,I_1,I_2 instead of Sample[Idx], Sample[Idx-1], and Sample[Idx-2]. It also uses O_1, and O_2 instead of Out[Idx-1] and Out[Idx-2].
• The A and B coefficients are put into a single array. It also adds two of the sample input values, and is missing a coefficient; this is because the B0 and B2 coefficients are always the same for this kind of filter.

There is one difference that is not for speed. These are tricks done to make the filter smooth, and needed because the sample window is so small. They preserve the state of filter. If we didn't preserve them, the filter would be starting and stopping so frequently that it would add distracting clicks to the output sound. The filters performance would be weakened, because the sample window isn't big enough to hold sounds lower than (about) 200 Hz. Preserving these values, the filter isn't starting and stopping, and doesn't really know about the sample window. All of the IIR filters in this program use similar techniques.

• O_1, O_2 are explicitly preserved across calls by being stored in class variables
• I_1 and I_2 are preserved by the audio loop (remember the warning about preserving 10 samples at the bottom of the loop?) The audio loop preserves the last 10 samples at the start of InBuffer. When this procedure is called, it retrieves the last two samples.

C#

static double O_1 = 0.0, O_2=0.0;

static void Butterworth(Int16[] InBuffer, int Ofs, byte[] OutBuffer, double[] Coefs)
{
  double C0=Coefs[0], C1=Coefs[1], C2 = Coefs[2], C3=Coefs[3];
  double I_1=InBuffer[Ofs-1], I_2= InBuffer[Ofs-2];
  for (int L = InBuffer . Length, J=0, I = Ofs; I < L; I++)
  {
     double I_0 = InBuffer[I];

     // Filter the samples
     double A = (I_0 + I_2) * C0 + I_1 * C1;
     I_2 = I_1;
     I_1 = I_0;

     A = A - O_1 * C2 - O_2 * C3;
     O_2 = O_1;
     O_1 = A;

     // Convert it back to 16 bit
     Int16 S;
     if (A < -32767) S = -32767;
     if (A > 32767)  S = 32767;
     else S = (Int16) A;

     // Store it
     OutBuffer[J++] = (byte)(S & 0xFF);
     OutBuffer[J++] = (byte)(((UInt16)S >> 8) & 0xFF);
  }
}

Automatic Gain Control

I decided next to tackle a problem where my wife's voice did not carry well on calls. This happens a lot to her with cell phones – and answering machines. I was pretty sure that the problem was poor automatic-gain-control (AGC). The typical amplifier in a headset (and in Skype) estimates how loud our voice is, then increases – or decreases – the volume to a reasonable level. It was deciding that my wife's voice was background noise, and cutting her off.

I chose to write an alternate gain control that amplified the sound and passed it to Skype. That way we'd have four to choose from: The one built into the Microphone, the Soundcard's, Mine, and Skype's. (To be fair, these automatic gain controls work well in most cases).

The main portion of the gain control is implemented in the file GainControl.cs. The control algorithm is:

1. Calculate (or estimate) how loud we are currently talking (using the Analyze() procedure)
2. If the loudness is very low, no one is talking… so set the output to zero. (Without this step, the volume of noise and hum would be cranked up)
3. Otherwise, compute the guess-gain by dividing how loud the sound of our voice should be by how loud it currently is
4. Compute the gain (called MaxGain) at which the sound will start clipping. If the guess-gain is louder than this, reduce it to MaxGain.
5. The software adjusts the gain for a gentle transition – especially in the case when we go from absolute quiet, to the start of talking. It does this by tracking the gain (called PrevGain) used in previous sample and the current one.
6. Multiple all the samples by this gain value.
7. If the sample rate is greater than 16000 samples/sec
   1. Do a low pass filter at 8 khz (again in IIR form). This helps prevent artifacts from down sampling
   2. Resample the sound to 16000

The portion of code that calculates the gain looks like (CutOff_dB, LowGain_dB, and TgtGain_dB are the three slider values):

C#

if (!AutoGain)
    Gain = 1.0;
else
{
    double MaxGain;
    double dB = Analyze(InBuffer, Ofs, out MaxGain);

    if (dB < CutOff_dB)
        Gain = 0.0;
    else if (dB < LowGain_dB + 4.0)
    {
        Gain = Math.Exp((LowGain_dB - dB) * db2log);
    }
    else
    {
        Gain = Math.Exp((TgtGain_dB - dB) * db2log);
    }
    Gain = (0.4 * Gain + 0.6 * PrevGain);
    if (Gain > MaxGain)
        Gain = MaxGain;
    PrevGain = Gain;
}

// Skip further process if there is silence
if (0.0 == Gain)
{
    return 0;
}

If you look at the code, you'll see that we don't compare directly with LowGain_db; rather we compare the estimate volume with LowGain_db+4. This gives a little “hysteresis” – if we raise our voice momentarily, the software won't suddenly make it the highest possible volume. Instead, the software lowers the volume a little bit.

When the software changes the sample rate, it basically needs to know how many input samples to skip. At the start of a call, the software computes this, calling it InInc:

C#

// Calculate how we resample to 16Khz
InInc = (int)(1024.0 * SampleRate / 16000.0);

The process of applying the gain adjustment, performing a low pass filter and re-sampling is below:

C#

int NumSamples = InBuffer.Length;
int End = OutBuffer.Length;
int NextIdxForOut = -InInc;
int OutIdx = 0;
for (int I = Ofs; OutIdx < End && I < NumSamples; I++)
{
    // Retrieve the sample
    Int16 S = InBuffer[I];
    double I_0 = S;

    // Apply Gain
    I_0 *= Gain;

    // 8khz low pass filter 
    if (DoLP)
    {
        // Simple Butterworth 8KHz low-pass filter
        double A = (I_0 + DS_I_2) * LP[0] + DS_I_1 * LP[1];
        DS_I_2 = DS_I_1;
        DS_I_1 = I_0;
        I_0 = A - DS_O_1 * LP[2] - DS_O_2 * LP[3];
        DS_O_2 = DS_O_1;
        DS_O_1 = I_0;

        // Change sample rate
        int Tmp = NextIdxForOut / 1024;
        if (I < Tmp)
            continue;
        NextIdxForOut += InInc;
    }

    // Convert it back to 16 bit
    S = (Int16)I_0;

    // Store it
    OutBuffer[OutIdx++] = (byte)(S & 0xFF);
    OutBuffer[OutIdx++] = (byte)(((UInt16)S >> 8) & 0xFF);
}

Estimating Volume

How loud “it should be” is controlled by a slider on the screen. The software estimates how loud the sound is by using an algorithm devised by David Robinson that takes into account how it sounds to a person. This way we can increase the gain on hard to hear sounds, and reduce the gain on sounds that a person is very sensitive to.

The loudness estimator, implemented in the Analysis procedure in GainAnalysis.cs, uses the following algorithm:

1. Uses a combination of two (IIR) filters to make the sound to reflect how our ears hear it.
2. Compute the Mean-Square value of the filtered sound (called MS)
3. Track the last 750 ms of these values.
4. Make a sorted copy of these.
5. Retrieve the first non-zero value at least 95% of the way into the buffer. This is so that we don't take the loudest sample and assume that is how the person is talking.
6. If MS (the value computed in step 2) is much quieter, use that one instead
7. Convert this value into dB scale by performing a logarithm on it.

The code to “normalize” the sound into how a person hears it is below. Along the way it computes the square of the samples (used in step 2). Like the earlier IIR filters, these preserves their variables across the calls. The first IIR is a yulewalk filter, but it preserves it old intermediary values in an array. Like the trick in AudioLoop, where we copy the last 10 samples into the start of the current buffer, the analysis procedure copies the last 10 immediate values into the start of YuleTmp array.

The output of the yulewalk filter is feed into a 150Hz high pass filter. It is essentially the same as the low-pass filter described earlier.

C#

for (int L = Samples.Length, N = Ofs; N < L; N++)
{
    int _V = Samples[N];
    double V = _V;
    if (_V > MaxSample) MaxSample = _V;
    if (-_V > MaxSample) MaxSample = -_V;

    // Perform yulewalk filter
    double S = V * YuleCoefs[0];
    for (int J = N - 1, I = 1; I < 11; I++, J--)
        S += Samples[J] * YuleCoefs[I];
    for (int J = N - 1, I = 11; I < 21; I++, J--)
        S -= YuleTmp[J] * YuleCoefs[I];

    // Store for the feedback into the next stage of the yule walk
    YuleTmp[N] = S;


    // Perform butterworth high-pass filter stage, using S as an input
    double Accum =
       (S + GA_I_2) * HPCoefs[0]
       + GA_I_1 * HPCoefs[1];
    GA_I_2 = GA_I_1;
    GA_I_1 = S;
    Accum = Accum - GA_O_1 * HPCoefs[2] - GA_O_2 * HPCoefs[3];

    GA_O_2 = GA_O_1;
    GA_O_1 = Accum;

    // The square of the filtered results
    Sum += Accum * Accum;
}

// Copy the intermediate yulewalk state for the next
// (this is needed since we are looking at a fairly small time window)
for (int I = 0, J = YuleTmp.Length - 10; I < 10; I++)
    YuleTmp[I] = YuleTmp[J];

The mean-squared is computed:

C#

// The mean square of the filtered results
double MS = Sum / NumSamples;

Tracking the last 750ms of samples is a simple matter of putting it into a circular queue:

C#

MSQueue[QIdx++] = MS;
if (QIdx >= MSQueue . Length)
   QIdx = 0;

Next, is the code to finding the first non-zero value 95% of the way into the buffer. It is a straightforward copy-the-array, sort it, and fetch:

C#

Array.Copy(MSQueue, SortedQ, MSQueue.Length);
Array.Sort(SortedQ);

// Return the 95% 
double X = SortedQ[Q95Idx]; 
for (int I = Q95Idx +1; X < 400.0 && I < SortedQ . Length; I++)
      X = SortedQ[I];

Next, override this value if the current sample window is very, very quiet – that is, the user stopped talking. (If we don't do this, we'll amplify background noise between words)

C#

if (MS < X * 0.40 && MS < 12800.0)
   X = MS;

Finally, convert the result into decibels (or a reasonable approximation of a decibel)

C#

return 10.0 * Math . Log10 (X * 0.5 + double . Epsilon);

Note: The logarithm function takes a positive, non-zero floating point number. However, the value we pass to it can be zero; if we pass a zero, though, the Logarithm function would return a bad value. The simplest thing to do is check to see if the value we are passing is “zero” and not call Logarithm. However, I learned a long time ago to just add “epsilon” to the value to whatever we pass. This can really improve performance on number crunching.

Conclusion

This concludes how to add a little bit feedback and fancy amplification to you Skype phone calls.

If you want to try this out, the download link for the source code is at the top of the article!

If you'd like to experiment further, here are ideas of what can be done:

1. DirectSound has echo cancellation and noise suppression. These seem desirable, esp. if you wanted to try making your own speakerphone. I was not able to get them to work and I would love to learn how.
2. I'm sure it is possible to trim even more latency off of the side-tone playback, and I would be interested in learning better techniques to do so.
3. Another would be to create the ideal equalizer from Robinson's Equal Loudness model, and use that filter and amplify the sound.
4. It might be useful for the other party to control the settings, so that they decide when your voice has the right volume.

Resources and References

• David Robinson (while doing his PhD at University of Essex on human hearing). His algorithm is intended to find the overall loudness of an album. He includes MATLAB scripts to compute coefficients at custom sample rates, as well as some pre-computed coefficients.
• Mark Heath's Skype Voice Changer
• Derek Pierson's DirectX tutorial

About The Author

Randall Maas writes firmware for medical devices, and consults in embedded software. Before that he did a lot of other things… like everyone else in the software industry. You can contact him at randym@acm.org.

They are looking for some feedback and if you want to, please post some over here.

• Creating Sound using MediaStreamSource in Silverlight 3 Beta
• Using Blur to make Dialogs Pop in Silverlight 3
• Silverlight 3 – Perspective 3d Transforms – PlaneProjection
• Silverlight 3 – The Bitmap API / WriteableBitmap
• Silverlight 3 – BasedOn Styles
• Silverlight 3 – Taking Silverlight Out of the Browser
• Silverlight 3 – UI Element to Element Binding
• Silverlight 3 – Creating Video from Raw Bits using a MediaStreamSource
• Silverlight 3 – Drop Shadows and Glows with Pixel Shaders
• Announcing Silverlight 3! SL3 Post Index

Audio and the .NET Framework

Playing back audio in a .NET application is not quite as easy as you might hope it would be. The .NET 2.0 Framework introduced the SoundPlayer component, which allows you to play back an existing WAV file. While this may be fine for many scenarios, as soon as you want to do things even slightly more advanced, such as changing the volume, or playing back from a different file-format, or pausing and repositioning, you must resort to writing P/Invoke wrappers for various Windows APIs.

Back in 2002, as I was getting started learning .NET, I create some audio-related classes of my own to compensate for the lack of audio support in the .NET Framework. I focused initially on reading and writing WAV and MIDI files, as well as playing back audio in a way that allowed real-time mixing and manipulation of the audio at a sample level. As time went by, I made this growing collection of audio classes available as an open source project, called NAudio, now hosted at CodePlex.

Audio Playback in NAudio

NAudio works by constructing an audio playback graph. Audio comes in “streams” which can be connected together and modified before eventually they go to a renderer. This might be your soundcard if you are listening to the audio, or it might be to a file on the hard disk.

In NAudio, all streams derive from WaveStream. NAudio comes with a collection of useful WaveStream derived classes such as WaveFileReader to read from WAV files or WaveStreamMixer to sum together multiple audio streams.

Audio mixing and effects are almost always performed with floating point numbers (32 bit is most common), so one of the first steps after reading audio out of a WAV file is to convert it from 16 bit to 32 bit. NAudio includes the Wave16To32ConversionStream class to do this. If the audio wasn't in PCM in the first place, for example it is MP3, then we make use of a combination of the Mp3FileReaderStream, WaveFormatConversionStream and BlockAlignmentReductionStream to read the audio out and get it into just the format we want. Here's an example showing how to create a WaveStream ready for playback.

WaveStream outStream;
if (fileName.EndsWith(".mp3"))
{
   outStream = new Mp3FileReader(fileName);
}
else if(fileName.EndsWith(".wav"))
{
   outStream = new WaveFileReader(fileName);
}
else
{
   throw new InvalidOperationException("Can't open this type of file");
}                
if (outStream.WaveFormat.Encoding != WaveFormatEncoding.Pcm)
{
   outStream = WaveFormatConversionStream.CreatePcmStream(outStream);
   outStream = new BlockAlignReductionStream(outStream); // reduces choppiness
}

If we simply want to play back the audio without any extra processing, we would use one of the audio output classes provided by NAudio to create an object that implements IWavePlayer. The options are WaveOut, DirectSoundOut, AsioOut and WasapiOut, each representing a different technology for audio playback in Windows. We will use WaveOut, which is the most universally supported. Here we are opening the default output device with a latency of 300ms, and instructing it to use windowed callbacks.

IWavePlayer player = new WaveOut(0, 300, true);
player.Init(outStream);
player.Play();

WaveOut will now repeatedly call the Read method of the output stream to get the next batch of audio samples to play. All we need to do now is to insert our audio effects into the playback chain.

An Audio Effects Framework

To allow us to process sample level audio more simply, I have create a new WaveStream derived class called EffectStream. EffectStream will simply pass each audio sample to one or more audio effects before returning the modified audio in its Read method. The reason I have chosen to host all the effects in a single EffectSteam rather than creating one WaveStream derived class is that I want to avoid the performance penalty of converting between arrays of bytes and arrays of floating point numbers at every step. Some types of effect, particularly those involving Fourier transforms can be processor intensive, so anything we can do to speed up performance will help.

As well as the EffectStream class, we need a base Effect class, from which all of our effects can derive. Here's a simplified version of the base Effect class (minus a whole load of helper mathematical functions):

public abstract class Effect
{
    private List<Slider> sliders;
    public float SampleRate { get; set; }
    public float Tempo { get; set; }
    public bool Enabled { get; set; }

    public Effect()
    {
        sliders = new List<Slider>();
        Enabled = true;
        Tempo = 120;
        SampleRate = 44100;
    }

    public IList<Slider> Sliders { get { return sliders; } }

    public Slider AddSlider(float defaultValue, float minimum, 
            float maximum, float increment, string description)
    {
        Slider slider = new Slider(defaultValue, minimum, 
            maximum, increment, description);
        sliders.Add(slider);
        return slider;
    }

    /// <summary>
    /// Should be called on effect load, 
    /// sample rate changes, and start of playback
    /// </summary>
    public virtual void Init()
    {}

    /// <summary>
    /// will be called when a slider value has been changed
    /// </summary>
    public abstract void Slider();

    /// <summary>
    /// called before each block is processed
    /// </summary>
    public virtual void Block()
    { }

    /// <summary>
    /// called for each sample
    /// </summary>
    public abstract void Sample(ref float spl0, ref float spl1);
}

The bulk of the work of the effect should be done in the overridden Sample method. The spl0 and spl1 parameters contain the current sample values to be modified for the left and right channels respectively. For example, to lower the volume we could halve the amplitude of every sample with the following code:

public override void Sample(ref float spl0, ref float spl1)
{
    spl0 *= 0.5f;
    spl1 *= 0.5f;
}

The Effect class contains Tempo and SampleRate values which are useful for certain types of effect. It also contains a concept of ‘Sliders' for each effect. These are the effect parameters, which allow real-time modification of the effect. So if we wanted to control the volume using a slider, we could write the following code (although bear in mind that normally volume sliders should be logarithmic not linear – see the Volume effect in the sample code for an example of how to do this):

public override void Sample(ref float spl0, ref float spl1)
{
    spl0 *= slider1;
    spl1 *= slider1;
}

To simplify the task of adding, removing and re-ordering effects, I created an EffectChain class, which is a simple wrapper around a List<Effect>. The EffectStream class has an EffectChain that contains all the effects it needs to run. Here is the code for the EffectStream:

public class EffectStream : WaveStream
{
    private EffectChain effects;
    public WaveStream source;
    private object effectLock = new object();
    private object sourceLock = new object();

    public EffectStream(EffectChain effects, WaveStream sourceStream)
    {
        this.effects = effects;
        this.source = sourceStream;
        foreach (Effect effect in effects)
        {
            InitialiseEffect(effect);
        }

    }

    public EffectStream(WaveStream sourceStream)
        : this(new EffectChain(), sourceStream)
    {        
    }

    public EffectStream(Effect effect, WaveStream sourceStream)
        : this(sourceStream)
    {
        AddEffect(effect);
    }

    public override WaveFormat WaveFormat
    {
        get { return source.WaveFormat; }
    }

    public override long Length
    {
        get { return source.Length; }
    }

    public override long Position
    {
        get { return source.Position; }
        set { lock (sourceLock) { source.Position = value; } }
    }        

    public override int Read(byte[] buffer, int offset, int count)
    {
        int read;
        lock(sourceLock)
        {
            read = source.Read(buffer, offset, count);
        }
        if (WaveFormat.BitsPerSample == 16)
        {
            lock (effectLock)
            {
                Process16Bit(buffer, offset, read);
            }
        }
        return read;
    }

    private void Process16Bit(byte[] buffer, int offset, int count)
    {
        foreach (Effect effect in effects)
        {
            if (effect.Enabled)
            {
                effect.Block();
            }
        }

        for(int sample = 0; sample < count/2; sample++)
        {
            // get the sample(s)
            int x = offset + sample * 2;
            short sample16Left = BitConverter.ToInt16(buffer, x);
            short sample16Right = sample16Left;
            if(WaveFormat.Channels == 2)
            {                    
                sample16Right = BitConverter.ToInt16(buffer, x + 2);
                sample++;
            }
           
            // run these samples through the effects
            float sample64Left = sample16Left / 32768.0f;
            float sample64Right = sample16Right / 32768.0f;
            foreach (Effect effect in effects)
            {
                if (effect.Enabled)
                {
                    effect.Sample(ref sample64Left, ref sample64Right);
                }
            }

            sample16Left = (short)(sample64Left * 32768.0f);
            sample16Right = (short)(sample64Right * 32768.0f);

            // put them back
            buffer[x] = (byte)(sample16Left & 0xFF);
            buffer[x + 1] = (byte)((sample16Left >> 8) & 0xFF); 

            if(WaveFormat.Channels == 2)    
            {
                buffer[x + 2] = (byte)(sample16Right & 0xFF);
                buffer[x + 3] = (byte)((sample16Right >> 8) & 0xFF);
            }
        }
    }


    public bool MoveUp(Effect effect)
    {
        lock (effectLock)
        {
            return effects.MoveUp(effect);
        }
    }

    public bool MoveDown(Effect effect)
    {
        lock (effectLock)
        {
            return effects.MoveDown(effect);
        }
    }

    public void AddEffect(Effect effect)
    {
        InitialiseEffect(effect);
        lock (effectLock)
        {
            this.effects.Add(effect);
        }
    }

    private void InitialiseEffect(Effect effect)
    {
        effect.SampleRate = WaveFormat.SampleRate;
        effect.Init();
        effect.Slider();
    }

    public bool RemoveEffect(Effect effect)
    {
        lock (effectLock)
        {
            return this.effects.Remove(effect);
        }
    }
}

When the Read method on EffectStream is called, we first read the requested number of bytes from our source WaveStream. This might be from a WAV or MP3 file, or from a microphone. Then, we convert it from 16 bit to 32 bit floating point audio. 16 bit audio is stored as integers going from -32,768 to 32,767, and 32 bit audio uses the range -1.0 to 1.0 to represent this range. This means we have plenty of headroom to mix together multiple signals without distorting. It is important though to remember that no samples should be greater than 1.0 before converting back to 16 bit.

Porting Effects to .NET

Now we have a basic effect framework, it is time to create some real effects to use. There are many sources of algorithms for digital signal processing (DSP) (try musicdsp.org for a good starting point), but I have chosen to base my effects model on that provided by the REAPER digital audio workstation (DAW). This impressive application, masterminded by legendary software developer Justin Frankel, includes a text-based effects framework. These effects, known as JS effects, allow the use of a C-like syntax to quickly write your own effects. I have modelled my Effect class on the JS syntax, allowing me to quickly port effects across.

public class Tremolo : Effect
{
    public Tremolo()
    {
        AddSlider(4,0,100,1,"frequency (Hz)");
        AddSlider(-6,-60,0,1,"amount (dB)");
        AddSlider(0, 0, 1, 0.1f, "stereo separation (0..1)");
    }

    float adv, sep, amount, sc, pos;

    public override void Slider()
    {
        adv=PI*2*slider1/SampleRate;
        sep=slider3*PI;
        amount=pow(2,slider2/6);
        sc=0.5f*amount; amount=1-amount;
    }

    public override void Sample(ref float spl0, ref float spl1)
    {
        spl0 = spl0 * ((cos(pos) + 1) * sc + amount);
        spl1 = spl1 * ((cos(pos + sep) + 1) * sc + amount);
        pos += adv;
    }
}

Some members in the Effect base class such as cos and slider1 allow me to keep the ported syntax as similar to the original JS script as possible.

REAPER ships with well over 100 of these JS Effects, so I chose about 15 of them and ported them to .NET. They are available in the download that accompanies this article. With some of them, for example pitch shifting effects, you will immediately notice the effect on the sound, while others, such as compressors, require some knowledge of how to adjust the parameters to get good results.

The Test Harness

Obviously we need a way to pass audio through our effects, so our next task is to create a test harness that will allow us to load in audio files and listen to them with the effects applied. For this purpose I created a simple Windows Forms application that allows you to select a WAV or MP3 file to play back. After being converted into PCM using various classes from the NAudio library, the resulting WaveStream is passed through an EffectStream before being passed to the soundcard for playback. The use of an EffectChain allows us to modify the effects loaded, and their order during playback.

To make the loading of effects simpler, I used the Managed Extensibility Framework (MEF), to [DF1] make each effect a “plugin” to the test harness. Each effect is decorated with an Export attribute to indicate that it is a plugin:

[Export(typeof(Effect))]
public class SuperPitch : Effect

Then I can request that MEF auto-populates a property with all the exported effects it can find:

[Import]
public ICollection<Effect> Effects { get; set; }

When the user selects an effect from the list of available effects, we create a new instance of it. This is because you might want to put the same effect into the effect chain more than once:

EffectSelectorForm effectSelectorForm = new EffectSelectorForm(Effects);
if (effectSelectorForm.ShowDialog(this) == DialogResult.OK)
{
    // create a new instance of the selected effect 
    // as we may want multiple copies of one effect
    Effect effect = (Effect)Activator.CreateInstance(
       effectSelectorForm.SelectedEffect.GetType());
    audioGraph.AddEffect(effect);
    checkedListBox1.Items.Add(effect, true);
}

To allow real-time modification of the effect parameters, I created two user controls. The first, EffectSliderPanel allows you to hook up a Windows Forms TrackBar to one of our effect's sliders and manages the minimum, maximum and granularity settings. The second user control, EffectPanel takes an Effect and creates one EffectSliderPanel for each slider in that effect. It also is responsible for calling the Slider method on the Effect whenever the user moves one of the sliders. Here's an example of what it looks like:

Now we are able to test our effects by listening to WAV files and playing them with real-time control over their parameters.

Intercepting Skype Audio

There are many interesting uses for audio effects, but it was suggested to me that I create a “voice changer” for Skype as the example program for this article. At first I didn't think that this would be possible, as you would need access to the audio samples from the microphone before Skype transmitted them over the network.

However, it turns out that Skype has a full featured SDK to allow all kinds of third-party add-ons and enhancements. The Skype API can be used in .NET via a COM object, called Skype4Com. Skype plugins are not loaded directly by the Skype application but attach to it via network sockets. The Skype4Com COM object hides much of this complexity from the user.

Having added Skype4Com as a reference to our application, we then need to connect to Skype. This is achieved by using the following code:

const int Protocol = 8;
skype = new Skype();
_ISkypeEvents_Event events = (_ISkypeEvents_Event)skype;
events.AttachmentStatus += OnSkypeAttachmentStatus;            
skype.CallStatus += OnSkypeCallStatus;
skype.Attach(Protocol, false);

In the CallStatus event handler, we tell Skype that we wish to ‘capture' the microphone. This will cause it to send us the raw audio data from the microphone via a TCP socket. Then we tell it that we will send the audio to be transmitted using another TCP socket.

void OnSkypeCallStatus(Call call, TCallStatus status)
{
    log.Info("SkypeCallStatus: {0}", status);
    if (status == TCallStatus.clsInProgress)
    {
        this.call = call;                  
        call.set_CaptureMicDevice(
        TCallIoDeviceType.callIoDeviceTypePort, MicPort.ToString());
        call.set_InputDevice(
        TCallIoDeviceType.callIoDeviceTypeSoundcard, "");
        call.set_InputDevice(
        TCallIoDeviceType.callIoDeviceTypePort, OutPort.ToString());
    }
    else if (status == TCallStatus.clsFinished)
    {
        call = null;
        packetSize = 0;
    }
}

I found an example Delphi application on the Skype developer website which shows how to intercept the microphone signal to boost the signal level. I used this sample as the starting point for creating my own application to intercept audio samples in Skype. The Delphi application made use of an object called TIdTCPServer, which is a multi-threaded socket server. I created a very simple .NET implementation of this class (without the multi-threading as we will only have one connection at a time):

class TcpServer : IDisposable
{
    TcpListener listener;
    public event EventHandler<ConnectedEventArgs> Connect;
    public event EventHandler Disconnect;
    public event EventHandler<DataReceivedEventArgs> DataReceived;
    
    public TcpServer(int port)
    {
        listener = new TcpListener(IPAddress.Loopback, port);
        listener.Start();
        ThreadPool.QueueUserWorkItem(Listen);
    }

    private void Listen(object state)
    {
        while (true)
        {
            using (TcpClient client = listener.AcceptTcpClient())
            {
                AcceptClient(client);
            }
        }
    }

    private void AcceptClient(TcpClient client)
    {
        using (NetworkStream inStream = client.GetStream())
        {
            OnConnect(inStream);
            while (client.Connected)
            {
                int available = client.Available;
                if (available > 0)
                {
                    byte[] buffer = new byte[available];
                    int read = inStream.Read(buffer, 0, available);
                    Debug.Assert(read == available);
                    OnDataReceived(buffer);
                }
                else
                {
                    Thread.Sleep(50);
                }
            }
        }
        OnDisconnect();
    }

    private void OnConnect(NetworkStream stream)
    {
        var connect = Connect;
        if (connect != null)
        {
            connect(this, new ConnectedEventArgs() { Stream = stream });
        }
    }

    private void OnDisconnect()
    {
        var disconnect = Disconnect;
        if (disconnect != null)
        {
            disconnect(this, EventArgs.Empty);
        }
    }

    private void OnDataReceived(byte[] buffer)
    {
        var execute = DataReceived;
        if (execute != null)
        {
            execute(this, new DataReceivedEventArgs() { Buffer = buffer });
        }
    }

    #region IDisposable Members

    public void Dispose()
    {
        listener.Stop();
    }

    #endregion
}

public class DataReceivedEventArgs : EventArgs
{
    public byte[] Buffer { get; set; }
}

public class ConnectedEventArgs : EventArgs
{
    public NetworkStream Stream { get; set; }
}

Once Skype has been told the port numbers on which to connect, it will attempt to open sockets to our TcpListener classes (one for audio in, and one for audio out). We now simply need to pass the audio through our effect chain. But EffectStream needs a WaveStream derived class for its input, so I created SkypeBufferStream to which we pass the raw data received on the microphone in socket, and it returns it in its Read method. One difficulty I encountered was that Skype offers no way of querying what the sample rate of the incoming data is. On my PC it seems to be 44.1kHz, but I do not know if this is guaranteed on all computers.

class SkypeBufferStream : WaveStream
{
    byte[] latestInBuffer;
    WaveFormat waveFormat;

    public SkypeBufferStream(int sampleRate)
    {
        waveFormat = new WaveFormat(sampleRate, 16, 1);
    }

    public override WaveFormat WaveFormat
    {
        get { return waveFormat; }
    }

    public override long Length
    {
        get { return 0; }
    }

    public override long Position
    {
        get
        {
            return 0;
        }
        set
        {
            throw new NotImplementedException();
        }
    }

    public void SetLatestInBuffer(byte[] buffer)
    {
        latestInBuffer = buffer;
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        if (offset != 0)
            throw new ArgumentOutOfRangeException("offset");
        if (buffer != latestInBuffer)
            Array.Copy(latestInBuffer, buffer, count);
        return count;
    }
}

Now when we receive any data from the microphone socket, we pass it through the SkypeBufferStream which in turn passes it through the EffectStream and finally out on the output socket's data stream. Here's the relevant code (found in the MicInterceptor class):

NetworkStream outStream;
SkypeBufferStream bufferStream;
WaveStream outputStream;

void OnOutServerConnect(object sender, ConnectedEventArgs e)
{
    log.Info("OutServer Connected");
    outStream = e.Stream;
}

void OnMicServerExecute(object sender, DataReceivedEventArgs args)
{
    // log.Info("Got {0} bytes", args.Buffer.Length);
    if (outStream != null)
    {
        // give the input audio to the beginning of our audio graph
        bufferStream.SetLatestInBuffer(args.Buffer);
        // process it out through the effects
        outputStream.Read(args.Buffer, 0, args.Buffer.Length);
        // play it back
        outStream.Write(args.Buffer, 0, args.Buffer.Length);
    }
}

When you run your application for the first time, you will need to grant it permission from within Skype:

To test that the effects are working in Skype is a little tricky as you will not hear the effected sound on your end of the conversation. One good way of checking the effects are working as expected is to use the Skype test call service. This is a number you can dial and it will record what you say and play it back to you.

It is a good idea to test your effect first using a local audio file, as you will not easily be able to determine whether glitches and other audio artifacts were caused by your effect, or simply due to poor network conditions.

There are a few things you should be aware of when selecting effects for use with Skype. First, the audio is mono, so there is no point using any effects such as stereo delay. Second, the audio is almost certainly down-sampled to a much lower sample rate before being transmitted to save on network bandwidth. This means any high frequency components of your sound will be lost. Third, internet telephony applications often have built in echo-suppression, so using delay-based effects might not work quite as well as you were hoping.

For silly voice effects, the most effective is pitch shifting (try the SuperPitch effect and shift either up or down about five semitones). FlangeBaby or Chorus can be used for more subtle voice changing effects. Or if you just want to be annoying, load up a Delay. Feel free to experiment with the other included effects, but bear in mind that many of them are designed with more musical uses in mind, so may not be relevant for internet telephony.

The Sample Code

The source code for all the effects, the EffectStream and the Skype connection code described in this article is available in the provided download. It uses a recent unreleased build of NAudio, so you will also need to visit the NAudio CodePlex site if you want to get access to the full source code. The EffectStream and Effect classes will eventually be made part of the NAudio framework, once I have refined their design a bit.

How to use Effect Tester sample app:

	Click this icon to attach to Skype and await a call. Click it again to disconnect, allowing you to test your effects using audio files instead. The textbox in the top right corner keeps you updated with the status of the connection to Skype
	Click this icon to load a WAV or MP3 file for playback. Files at 44.1kHz work best.
	Rewind, Play, Pause or Stop the current WAV or MP3 file.
	Brings up the effect selector dialog to add a new instance of an effect to the current Effect chain. Loaded effects appear in the CheckedListBox on the left.
	Removes the currently selected effect from the effect chain
	Move the currently selected effects up or down in the signal chain
	Use the checkboxes to enable or disable effects in the effect chain on the fly.
	Use the sliders to adjust the effect parameters in real-time

About the Author

Mark Heath is a .NET developer based in Southampton, UK. When he's not writing .NET audio applications for fun, he enjoys home studio recording, playing football, reading theology books and sword fighting with his four small children. His development blog can be found at http://mark-dot-net.blogspot.com