howtoidentifyperftestobjectives

Applies to

* Perormance Testing

Summary

Experience shows that leaving performance testing for last is risky. It also shows that trying to validate or predict production performance too early is essentially pointless. This implies that early performance testing needs to be focused on something other than the prediction of production performance. This How-To addresses establishing objectives for performance testing and technical targets that contribute to, but are separate from, end-user goal and requirements.

* Objectives
* Overview
* Definitions
* Step1: Determine Objectives of Performance Testing
* Step2: Capture or Estimate Resource Usage Targets and Thresholds
* Step3: Capture or Estimate Resource Budgets or Allocations
* Step4: Identify Metrics
* Step5: Communicate Results
* Step6: Stay on Top of Changing Objectives, Targets and Budgets
* Additional Resources

Objectives

* Learn how to establish valuable performance testing objectives at any point in the development lifecycle
* Learn how to communicate those objectives and the value they add to both team members and executives
* Learn how to establish technical, performance related targets (sometimes called performance budgets) that can be validated independently from end-user goals and requirements
* Learn how to communicate those targets and the value that testing against them provides to both team members and executives

Overview

Performance testing objectives can be fairly easy to capture. The easiest way to capture performance testing objectives is simply to ask each and every member of the project team what value you can add for him or her while you are performance testing at a particular point in the project, or immediately following the accomplishment of a particular milestone. While it is not always easy to find and schedule time with each member of the team, especially when you consider that the project team includes executive stakeholder, analysts and possibly even representative users, it’s generally not overly challenging to get them to share the information that will help you establish valuable performance testing objectives. That value may be providing resource utilization data under load, generating specific loads to assist with tuning an application server, or providing a report of the number of objects requested by each web page. While collecting performance testing objectives early in the project is a good habit to get into, so is periodically revisiting them and checking in with members of the team to see if there are any new objectives that they would like to see added.

Terminology

The following definitions are used both within this Explain and throughout the Performance Testing Guidance. These terms also appear in the Glossary {link}, but are repeated here for easy reference. Every attempt has been made to ensure that these terms and definitions are consistent with formal and industrial use; however, some of these terms are known to have certain valid alternate definitions and implications in specific industries and organizations. The uses of these definitions are intended to aid communication and are not an attempt to create a universal standard.
* Performance Testing Objectives: Information to be collected through the process of performance testing that is anticipated to have value in determining or improving the quality of the product, but is not necessarily quantitative or directly related to a performance requirement, goal or stated Quality of Service.
* Performance Thresholds: The maximum acceptable value for a resource of interest, usually specified in terms of response times, throughput (transactions per second), and resource utilization levels. Resource utilization levels include the amount of CPU capacity, memory, disk I/O, and network I/O that your application consumes.
* Performance Targets: The desired value for a resource of interest under a particular set of conditions, usually specified in terms of response times, throughput (transactions per second), and resource utilization levels. Resource utilization levels include the amount of CPU capacity, memory, disk I/O, and network I/O that your application consumes.
* Performance Budgets: Performance budgets are your constraints. Performance budgets specify the amount of resources that you can use for specific scenarios and operations and still be successful.
* Metrics: Metrics are the actual measurements obtained by running performance tests. These performance tests include system-related metrics such as CPU, memory, disk I/O, network I/O, and resource utilization levels. The performance tests also include application-specific metrics such as performance counters and timing data.
* Resource Utilization: Resource utilization is the measure of how much server and network resources are consumed by your application. Resources include CPU, memory, disk I/O, and network I/O. Resource utilization is the cost in terms of system resources. The primary resources are CPU, memory, disk I/O, and network I/O.
* Throughput: Throughput is the number of requests that can be served by your application per unit time. Throughput varies depending on the load. Throughput is frequently measured as requests or logical transactions per second.

Step1: Determine Objectives of Performance Testing

The key to determining the objectives of a performance testing effort is identifying change, risk and opportunity for improvement. The following are methods that you may find effective. Whether you apply these methods exactly or adapt them to fit your project and work environment is not important. What is important is to remember that objectives are intentionally collaborative. They are a tool to help ensure that the performance testing effort is providing as much value to the team, in particular to the Architects, Developers and Administrators, as early as possible during the course of the project.
* Review project plan with individuals or small groups. Remember that a project plan does not have to be in the form of a document. It may be a whiteboard sketch, embedded in a series of emails or a fuzzy idea that lives in the minds of various team members. The point is that no matter how informal it may be, every project has some sort of a plan. While reviewing or extracting the plan, whenever you encounter something that looks like a check point, iteration or milestone ask questions like:
* What is changing here?
* Are there performance budgets or thresholds associated with that change? If so, what are they? Can I test them for you?
* Is tuning likely to be required as a result of this change? Are there metrics I can collect to help you with the tuning?
* Is this change likely to impact other areas that we have tested/collected metrics for previously? If so which? What tests can I run/metrics can I collect to help determine if everything is working as expected?
* Review physical and logical architectures with individuals or small groups. Again, remember that these may not be documented, but someone has at least a concept in mind – or if they don’t, it’s probably valuable to find that out as well. As you review or extract the architectures ask questions like:
* Have you ever done this/used this before?
* How can we determine if this is performing within acceptable parameters early?
* Is this likely to need tuning? What tests can I run/metrics can I collect to assist?
* Individually ask team members what their biggest performance related concern for the project is and how you could detect that problem earliest if it turns out to be a problem. You may need to establish trust with team members before you get the best answers. Reassure the team individually and collectively that you are soliciting this information so that you can better assist them in building a quality product.

Step2: Capture or Estimate Resource Usage Targets and Thresholds

This is a step that is sometimes misapplied. Except in extremely rare circumstances, it is not appropriate for the performance tester to determine targets and thresholds, only to capture and test against them. Even if the performance tester is the most qualified individual to set the targets and thresholds, s/he is not the individual responsible for ensuring they are met, rather s/he is responsible for providing information to those who are responsible for ensuring they are met so that person can make informed decisions. Resist the urge to set targets yourself.
* Ask the architects, or other team members who may be responsible for enforcing and/or making decisions regarding targets and thresholds to share those with you.
* Even though it’s not your job to set targets and thresholds, it’s always a good idea to check and see what the rest of the industry is doing. Do a search or pick up a book to see what recommendations they hold. If they seem relevant to your project, make a note. Think of this information as target and threshold related data of interest that may turn out to be useful in comparison to the actual data you end up collecting during your testing.

Step3: Capture or Estimate Resource Budgets or Allocations

Like the previous step, remember that it’s the performance tester’s job to collect and provide information about budgets and allocations, not to enforce them.
* Ask the architects, or other team members who may be responsible for enforcing and/or making decisions regarding targets and thresholds to share those with you
* Review project documents. Performance testers aren’t always specifically invited to review design and architecture documents, remember to ask.
* Attend developer and architect meetings. Sit in the back and listen. Bring some work to do. Take note of comments like “Suresh, see if you can’t get that object under X memory consumption.” This kind of guidance rarely makes it to paper, but Suresh still might appreciate another set of eyes watching his object’s memory consumption.

Step4: Identify Metrics

Most of the time, this step is rather transparent. For example of an objective states that the CPU utilization of the web server shouldn’t go over 80% for more than 1 second in 10, it is clear that a metric you should be monitoring is the CPU utilization of the web server, polled at not less than 1 second intervals. You may not want to do this during every test, but there is no question what you need to measure. Sometimes, the associated metrics aren’t so clear or are not so simple to collect. Consider the following approach:
* Create a grid or a simple spreadsheet that maps each of the collected objectives to the metric(s) that will indicate if the objective is being met.
* If it isn’t obvious, research or work with the development team to figure out how to collect each metric without skewing the test or any of the other data you hope to collect at the same time.
* Collaborate with the Developers, Architects and Administrators. They know what metrics are valuable to them and how to capture most of them. You know how to put the application in the state that makes those metrics most valuable.

Step5: Communicate Results

Communicating results of tests that capture data related to performance objectives is different than communicating results related to overall performance goals and requirements. Objective related results are intended to be useful information for the team, not to determine an applications overall fitness for release. Share the information freely. In most cases, the fact that an object isn’t being met is not something that gets recorded in a defect tracking system. It is simply information to help the team do their job better.
* Report results vs. targets, budgets, previous measurements as well as your own research. You never know what the team will find most valuable.
* Share reports with the whole team.
* Make the raw data available to everyone;, invite them to parse it in other ways and to show you other ways to present the data that are more helpful to them.
* Be ready, willing, interested and able to re-execute and/or modify and re-execute the tests.

Step6: Stay on Top of Changing Objectives, Targets and Budgets

It is important to remember that objectives are bound to change during the life of a project. As requirements change, features are moved in or out of a particular build, hardware decisions are made, code is refactored etc. performance testing objectives are bound to change as well. Keep up a running dialogue with your team. Ask them what is changing and how it impacts the objectives. Whether you do this in a meeting, at the coffee station or electronically is up to you, just remember that it is your own time that you are wasting if you are testing against an old, no longer relevant objective.