Summary: A discussion of various ways to benchmark your OAL for both performance and power. Discusses ILTiming for validating interrupt response, OSBench, writing your own benchmarks, monitoring power, and measuring boot and suspend/resume times.

Benchmarking an OAL

This document will focus on techniques and tools useful for benchmarking a Windows Embedded CE OAL. An OAL encompasses the low level platform support and initialization code, and affects the performance and responsiveness of CPU, Memory, Interrupt and I/O subsystems. Driver benchmarking, such as video, network and storage is out of the scope of this document.

Interrupt Latency Timing

Of particular interest to OAL developers are the maximum ISR and IST times. Running these tests under load, preferably with as many drivers active as possible, will determine whether any OAL or driver code runs for excessive periods of time with interrupts disabled, or if high priority ISRs and ISTs consume excessive CPU time.

Interrupt Latency Timing, or ILTiming, measures the response speed of the system to interrupt sources. Although designed as a tool to validate the real-time characteristics of CE devices, ILTiming is an excellent benchmark of the low-level OAL code, in particular the interrupt, memory and cache routines. Further details about ILTiming can be found here: http://msdn2.microsoft.com/en-us/library/aa911504.aspx.

Supporting ILtiming requires a few lines of code to be added to the interrupt and timer routines, and well as a high frequency performance counter that support the PerfCountFreq and PerfCountSinceTick functions. Full details of the pre-requisites can be found here: http://msdn2.microsoft.com/en-us/library/aa914439.aspx.
In the CE Tree, The PXA27x BSP supports ILTiming and can be used as an example implementation.

Interpreting the results

ISR latency should be low and consistent.

The ISR latency is primarily affected by the interrupt code, and driver ISRs running for extended periods. If there is a large spread between the minimum and maximum latency, review any BSP or installed ISR code for expensive operations (i.e. polling), and consider moving code into ISTs. If the minimum latency is too high, review the OEMInterrupt code.
The IST latency is also affected by the context switch performance and any high priority threads. As described in the next section, OSBench can further investigate the performance of these kernel operations. As with ISR latency, look for any threads perfoming expensive operations at high priority or with interrupts disabled.

OSBench

These tests determine the time required to execute various low-level operations, such as context switching or memory allocation. Similar to ILTiming, OSBench requires a high frequency performance counter.

Additional information can be found here: http://msdn2.microsoft.com/en-us/library/aa914830.aspx.

Interpreting the results

Intra-process benchmark times are affected by interrupt and context switch code, while inter-process benchmark times are also affected by cache code. For example, a fast system that requires the cache to be flushed when switching processes would have relatively fast inter-process times and very slow inter-process times.

Synthetic CPU and Memory Benchmarks

In addition to specialized response-time tests, synthetic CPU and memory benchmarks are useful for benchmarking an OAL. Several companies such as SPB offer system-level CE benchmark tools, however few cases are restricted to the OAL.

However, since most metrics of interest to OAL developers require very little OS or driver support, and benchmarks are easily written, simply call QueryPerformanceCounter before and after the operation of interest!

Examples of CPU benchmarks could include summing the first n integers, or determining the nth fibbonaci number.

Example of memory benchmarks could include calculating the sum of a large block of memory accessed sequentially, or accessed with various strides.

Application Benchmarks

Kernel Tracker and Call Profiler can also be used to determine the CPU usage and execution profile of various applications. Running a Remote Desktop Client, or Media Player with the profiler enabled can help to validate platforms for various scenarios. There is significant documentation for these available online at MSDN under Tools for Performance Tuning: http://msdn2.microsoft.com/en-us/library/aa934490.aspx.
Although these tools are of limited use benchmarking OAL code, they are very useful in tracing the source of performance issues.

Additional Benchmarks

Power Consumption

Multimeters, Oscilloscopes and the Microsoft Power Monitor can be used to measure device power consumption.

Since the CPU is often one of the most power hungry peripherals, OAL code, in particular OEMIdle has a direct impact on power consumption.

A good starting point would be to measure power consumption in the loaded, momentary idle, and sleep states.

See http://blogs.msdn.com/ce_base/archive/2006/11/15/optimizing-oemidle-for-lower-power-consumption.aspx for details on optimizing OEMIdle.

Boot and Resume Time

Excessive boot and resume times give the impression of a slow device, irrespective of the actual speed of the hardware.

To start, measure the boot times, resume from hibernation/standby (D3) and resume from idle (D1). In all cases, the device should be responsive and able to function/respond to user input within half a second.

Excessive debug output or busy-waiting for a device to initialize are usually the culprit. Look at the initialization code and see if waits can be reduced or eliminated. Consult component datasheets for details.