howtomanageperftestcycleagile

How To: Manage the Performance Test Cycle (Agile)

Scott Barber

Applies to

* Performance Testing in an Agile Development Environment

Summary

<<we could say This How To describes a process for managing…
It sounds less preachy>>

This How To describes an industry proven approach to managing performance testing an application that is being developed using an Agile Software Development model. As the term implies, the key to agile development is flexibility. Flexibility, however, does not imply sloppy or inefficient. To remain efficient and thorough in an agile environment, you may need to change the way you are used to managing your performance testing.

* Objectives
* Overview
* Terminology
* Summary of Steps
* Step 1. Identify Performance Testing Success Criteria
* Step 2. Understand the System, the Project Environment and Milestones
* Step 3. Build a Performance Testing Strategy
* Step 4. Create Performance Test Plans for each Development Sprint
* Step 5. Execute the Next Plan
* Step 6. Analyze Results, Report and Retest
* Step 7. Triage, Reprioritize and Update Strategy and Plans
* Step 8. Return to Step 5 or 6
* Additional Resources

Objectives

* Become familiar with an agile performance test management approach.
* Learn how to maximize flexibility without sacrificing control.
* Learn how to provide managers and stakeholders with progress and value indicators.
* Learn how to provide a structure for capturing information that will not noticeably impact the release schedule.
* Learn how to apply an approach that is designed to embrace change, not simply tolerate it.

Overview

<<This approach to managing performance testing is not complex, although it may look like it>>

This approach to managing performance testing is not complicated, although it may seem complicated at first. That is because the approach embraces change during a project's lifecycle, it iterates (not always in a predictable pattern), and it encourages planning to be done just far enough in advance for team coordination, but not far enough in advance that the plan is likely to need significant revision to execute. To simplify these potential challenges, this how to follows a linear path through the logical steps in the approach.

When viewed linearly, the approach starts by determining the success criteria for the performance testing effort. Once the success criteria is understood, an overall strategy is created describing the general approach achieving those criteria by summarizing what performance testing is intended for each key project delivery. On many agile teams, key project deliveries come at the end of what is called a development sprint. With a strategy in place, plans are created for major tests or tasks identified for the first several project deliveries or sprints. When a project delivery is made, the plan executed in priority sequence (based on all currently available information), appropriately reporting, recording, revising, reprioritizing, and improving the application and the overall plan as the work progresses.

Terminology

The following terms are used both within this How To and throughout the Performance Testing Guidance. These terms also appear in the Glossary {link}, but are repeated here for easy reference. Every attempt has been made to ensure that these terms and definitions are consistent with formal and industrial use; however, some of these terms are known to have certain valid alternate definitions and implications in specific industries and organizations. The use of these definitions is intended to aid communication and is not an attempt to create a universal standard.
* Agile Software Development: A lightweight process that adheres to the principles of the Agile Alliance (www.agile alliance.org).
* Test Strategy: The test strategy is the way tests will be designed and executed to support an effective quality assessment.
* Test Logistics: The set of ideas that guide how resources are applied to fulfill the test strategy.
* Test Design: The process of creating tests.
* Test Execution: The process of configuring, operating, and observing a product for the purpose of evaluating it.
* Test Sprint: The testing conducted between iterations or releases. Agile development teams frequently refer to iterations or incremental releases that add new functionality to the product “Sprints”, thus a Test Sprint includes those testing tasks associated with the last completed Development Sprint.
* Test Plan: The set of ideas that guide or represent the intended test process.
* Test Sprint Plan: The combination of a Test Strategy and Test Logistics necessary to prepare for a Test Sprint. This can also be thought of as a “Sprint-level Test Plan”
* *Milestone: *

Summary of Steps

* Step 1. Identify Performance Testing Success Criteria
* Step 2. Understand the System, Project Environment and Milestones
* Step 3. Build a Performance Testing Strategy
* Step 4. Create Performance Test Plans for each Development Sprint
* Step 5. Execute the Next Plan
* Step 6. Analyze Results, Report and Retest
* Step 7. Triage, Reprioritize and Update Strategy and Plans
* Step 8. Return to Step 5 or 6

Step 1. Identify Performance Testing Success Criteri

The expression of intent for a performance investigation can often be accomplished in a single work session involving the lead performance tester, the lead developer, the project manager, and a copy of the project plan, which is used to identify key project deliveries. Because we are articulating and recording intent, not creating another plan, we won’t concern ourselves with dates; rather we want to identify the sequencing of key deliveries and estimate how much time we can expect between these deliveries. The specific deliveries we are interested in relate to hardware components, supporting software, and application functionality becoming available for investigation. For example, it is often the case on a Web development project that the first delivery that lends itself to performance investigation includes bringing up a Web server to present prototypes to stakeholders. In that case, we would want something similar to this in our expression of intent: “Delivery 1: Web server with static prototype available for x days. Investigate various Web server configurations for response time, throughput, and memory allocation. Investigate available network bandwidth and latency. Investigate impact of firewall, proxy server, and load balancer configurations. During investigation, collect configuration data to aid in future tuning and data to assist in validating adequacy of existing network components.” Done. We can move on to expressing the intent of investigation for the next key delivery. These examples may or may not be the top priorities for investigation based on the specifics of your project, but I’m sure you can see how learning more about these areas early in the project can add value. Even if no performance issues are detected, that information will be useful to eliminate variables when trying to pinpoint issues detected later in the project.

Information to include in the performance investigation intent:
* Investigable deliveries
* Duration of each delivery investigation
* Key areas of investigation
Obvious stuff

Step 2. Understand the System, Project Environment and Milestones

Step 3. Build a Performance Testing Strategy

<<Scott try to break this down. Agile is not an understood process at all>>

After sketching out the expression of intent, the performance tester can immediately begin to draft an investigation strategy for each key delivery. This strategy should be limited to a single page per project delivery and should include pictures or diagrams whenever they are more descriptive than text. When text is the medium of choice, lists typically suffice. While there is a wide range of information that may be included in the strategy, the critical components are the desired outcomes of the investigation and the key tasks anticipated to achieve that outcome. The strategies are living documents that are available to the entire team for review, comment, and revision. Depending on the nature of the project, it may make sense to complete strategies a few deliveries in advance to limit potential rework, or it may make sense to create draft strategies for all of the key deliveries while waiting for the first investigable delivery. The following information might appear in the strategy for our example delivery item: “Measure available network bandwidth and latency while increasing the number and/or frequency of page and/or file requests,” and “Under moderate load, measure resource utilization on the Web server using various configuration settings.”

Information to include in a performance investigation strategy:
<<can you give examples for each one. >>
* Intent of the investigation
* Prerequisites of strategy execution
* Tools and scripts required
* External resources required
* Risks to accomplishing strategy
* Data of special interest
* Areas of concern
* Pass/Fail criteria
* Completion criteria
* Planned variants on tests
* Load range
* Tasks to accomplish strategy

Step 4. Develop a Test Sprint Plan for the Next Development Sprint

<<again break it down. It sounds it needs more specifics. Examples>>
As a delivery approaches, we finally create what I call mini-plans. Each task in a strategy now gets up to a single page of its own, often including pictures, to spell out the remaining details needed to complete or repeat the task. The mini-plan should be completed far enough in advance to be shared with the team for recommendations or improvements and for necessary resource coordination to take place. Continuing with our example, a mini-plan might define “moderate load,” identify the exact resources to be monitored (and who will set up the monitors), and specify the configuration settings expected to be varied and by what degree. Once drafted, the mini-plans can be sequenced for execution by priority as determined by key team members.

Information to include in a performance investigation mini-plan:
<<Please give examples for those items>>
* Strategy task execution method
* Specifically what data will be collected
* Specifically how that data will be collected
* Who will assist, how, and when
* Additional information needed to repeat the investigation
* Sequence of tasks by priority

Step 5. Execute Performance Test Work Items

When each delivery is made, the performance investigation begins with the highest priority mini-plan related to that delivery. The most important part of mini-plan execution is to treat it like a one- to two-day exploratory testing session (see the StickyNotes for more information)– modifying the plan intelligently as results analysis leads to new priorities. At the conclusion of each mini-plan execution, share your findings with the team, then reprioritize the remaining tasks, add new tasks, and/or remove planned tasks based on the new questions and concerns raised by the team. When reprioritizing is complete, move on to the next-highest priority task.

Keys to mini-plan execution:
* Analyze results immediately and re-plan accordingly.
* Communicate frequently and openly across the team.
* Record results and significant findings.
* Record other data needed to repeat the test later.
* Revisit investigation priorities after two days.

Step 6. Analyze Results, Report and Retest

Pause shortly (If a Sprint is a week, and a work item is 1-2 days, pause ~ ½ day) to catch your breath, analyze, report and prep for entire team triage

Step 7. Triage, Reprioritize and Update Strategy and Plans

The key to successfully implementing the performance investigation approach, of course, is continual communication among team members. As I've indicated above, it's a good idea not only to keep planning documents available to all members and check back with one another frequently but also to plan time into testing schedules to review and update task lists and priorities. Whether you implement this “three-tier” approach to planning verbatim or with a healthy dose of customization, or whether you do it using documents, spreadsheets, notebooks, or whiteboards and digital cameras (my personal favorite) is completely irrelevant as long as you keep in mind the goals of the approach: structured yet highly adaptable investigation.

Step 8. Return to Step 5 or 6

Lather, rinse, repeat…

Resources

* Manifesto for Agile Software Development (www.agilemanifesto.org)
* The Agile Alliance (www.agile alliance.org)
* “Software Engineering with Microsoft Visual Studio Team System”, Sam Guckenheimer