Summary: A systematic approach and best practices for running Hopper.

The Hopper Process

Hopper is one of the most challenging tasks ""OEMs"" will have to overcome when developing a Windows Mobile device. Hopper problems are often difficult to understand and fix due to the random nature of the test. Many ""OEMs"" end up extending their development cycle by many months trying to pass Hopper. However, using a good debugging strategy and a systematic approach to running Hopper can help to reduce the effort. This article will outline some best practices for setting up a process for running Hopper.

*Prerequisite
*Running Hopper
*KITL vs Stand-Alone Retail
*Hopper Image
*Release Directory
*Starting Hopper
*Filtering Exceptions
*Collecting Results
*Hopper Logs
*KITL with Debugger Run
*Recording Results
*Run Information
*Filing Bugs
*Additional Resources

Prerequisite

Make sure you have Platform Builder installed. You will need it for downloading images and running the kernel debugger. You should also install the Post-Mortem debugger. The Post-Mortem debugger is a Platform Builder extension that allows you capture and analyze kdmp files. Platform Builder for Windows Mobile 5.0 can be installed from the Windows Mobile 5.0 final RC adaptation kit (build 14343).

Running Hopper

As currently defined by the LTK MTTF test, you need to achieve 20 hours mean and 25 hours median on 10 devices to pass Hopper. A lab environment where all devices can run simultaneously is best. If you have a shortage of devices, you may need to spread each run among several people.

KITL vs Stand-Alone Retail

The best tool for debugging Hopper problems is the kernel debugger over KITL connection. With live debugging, you have a variety of tools available to help diagnose problems. But to pass the LTK requirement, you'll have to eventually run on stand-alone retail devices to obtain the final number.

There are trade-offs between KITL vs stand-alone devices. You should devise a strategy for running on a mixture of both. KITL devices offer more debugging capabilities while Hopper numbers from stand-alone devices are more accurate. Make your decision based on the goal of the run. For example, if your goal is to quickly raise your Hopper runtime, you should focus more on KITL runs. As the project progresses (and hopefully so do the Hopper runtimes), you may start to shift more towards running on stand-alone devices.

Although the LTK requirement says to run 10 devices, you don't always need that many for every run. If your goal is to debug a particular problem, it doesn't really matter how many devices you use. The more devices you use will improve the likelyhood of repro, but 10 is not a hard requirement. When using stand-alone devices to obtain a runtime metric, you need run on 10 devices to get the result.

Hopper Image

Because several people on your team may be running Hopper for a given run, there should be a systematic way to coordinate Hopper images. For each Hopper run, a single generated image should be used to prevent inconsistencies in the run. It is recommended that the Hopper image be generated from a nightly-build system. If this is not feasible, a designated engineer can build and provide the image to the team. The image should be placed on a shared server accessible by all team members. The following information should be kept for each image:

*Build date.
*Source control sync revision or build number from a nightly-build system. You should have some way to keep track of the code revision used to build the image.
*Build settings such as the build script used, which drivers are removed, or special Sysgen settings.
*Goal for the run. For example, if you suspect the battery driver is causing reboots in previous runs, you can choose to remove it from this run in order to see whether the problem goes away. State that as the goal for the run.

Release Directory

The release directory contains valuable information for debugging. It is especially useful when running with the debugger and for viewing kdmp files. The release directory for each generated Hopper image should be made available along with the image.

When using KITL, copy the release directory locally. Each device running with KITL must have its own copy of the release directory. Since the full release directory is very large and contains many files not needed by KITL and the debugger, you can reduce the number of files by only copying ""*.exe, *.dll, *.cpl, and *.pdb"" files.

Starting Hopper

Starting Hopper is as simple as copying Hopper.exe onto the device root directory and launching it from file explorer. If connected over KITL, you can place Hopper in the release directory and launch it from the Platform Builder Target Control window (CESH). Read the MTTF test description in the LTK database for additional test requirements.

Important: Always clean boot the device to factory settings before starting Hopper.

Filtering Exceptions

When running with the debugger, the debugger will break on all exceptions giving you a chance to look at it. This is not always useful for Hopper, as we don’t care about exceptions in non-critical applications that may be handled anyway. The FEX command allows you to filter exceptions. For BSP development, we generally care about exceptions in filesys.exe, device.exe, and gwes.exe. Use these steps to ignore all exceptions other than for those processes listed.

*Use the debugger to break execution.
*From the "Debug" menu, select "Advanced Commands".
*In the command window, enter the following commands and click "Execute" after each. You can also enter "fex ?" for help.
*fex on
*fex ap filesys.exe
*fex ap device.exe
*fex ap gwes.exe
*Close the "Advanced Commands" window and click "Go" to continue exection. You can now start Hopper.

Collecting Results

Collecting good results is crucial for analyzing failures and can save a lot of time by not having to wait for another repro. For each device, capture all of the following information if available.

Hopper Log

Hopper writes to a log file on the device while it’s running. The log file records process info, run time, cause of the failure, and several other things. The Hopper log file is stored on the device at \testlog\hopper.log. If running with a KITL connection, Hopper will also produce a more detailed log file in the release directory called results.log. When Hopper exits, it will attempt to take a shot of the last screen and store on the device at \testlog\LastScreen.bmp.

KITL with Debugger Run

When a device appears hung or an exception occurs on a KITL debugger build, it is preferable to do live debugging while the device is still in this state. If this is not possible, use Platform Builder to capture a kdmp file so the system state can be analyzed at a later time using the Post-Mortem debugger. Make sure you have the Post-Mortem debugger installed on the host machine with the KITL connection to the target device. To capture a kdmp from Platform Builder:

*Use the debugger to break execution.
*From the "Debug" menu, select "Capture Dump File".
*Choose "System dump" or "Complete dump" if necessary. For most cases, "System dump" is sufficient. "Complete dump" captures all of memory and can take very long to transfer over KITL.
*Specify the dump file name to save it.
*Make sure to capture the debug message output from Platform Builder. Usually the last screen-full of lines will be sufficient.

Recording Results

Run Information

Record the runtimes and failures for each run on a spreadsheet. Get the runtime from the Hopper log file. Make a note of the image setting and goal for that run. This will make it easier to view progress and keep the team on track on what was accomplished for each run.

Filing Bugs

Generally, all Hopper failures should be filed. If you believe a failure is a duplicate, add the additional data you have collected to the existing bug. If you’re unsure if a failure is a duplicate, file a new bug. An engineer responsible for analyzing Hopper bugs will need decide if the bug is a duplicate.

Hopper bugs should be filed using a keyword that can be easily searched. For example, you can adopt a convention where Hopper bugs have the keywoard MTTF in the bug description.

Provide as much information as possible when filing the bug. Attach collected results to the bug. If you have call stack info, debug output, or kdmp file, attach them to the bug. Also provide info about the image used and where to get the release directory.

Additional Resources

*Hopper Debugging Strategy
*The article "MTTF Testing - Hopper Demystified" in the Windows Mobile OEM documentation.
*The HoppeRX blog site has useful articles for debugging problems found by Hopper.

Go up to BSP Exit Criteria
Go up to Big Book of BSP