explainedtypesperftesting

Explained: Types of Performance Testing

Summary

This Explain defines, describes and outlines the benefits and project risks associated with several common types or categories of performance related testing. Many of these terms are frequently misused and misunderstood even within established teams. This Explain is designed to help ensure that your team is using the same terms to mean the same things and to help your team decided which types of performance related testing are most likely to add value to their project based on their current risks, concerns or testing results.

• Summary Matrix
• Performance Testing
• Performance Investigation
• Performance Validation
• Performance Unit Test
• Component Performance
• Performance Smoke Test
• Load Test
• Stress Test
• Capacity Planning Test
• Spike Test
• Endurance Test
• Additional Resources

Objectives

• Define Performance Testing, Load Testing, Stress Testing, Spike Testing and Capacity Planning
• Explain the values and benefits associate with each type of performance testing
• Explain the shortfalls of each type of performance testing
• Demonstrate specific project risks that each type of performance testing address

Summary Table

Performance Test Type Main Benefit(s) Top Risk(s) Addressed
Investigation Collect useful information at various development stages. • Find major performance issues early
• Establishes baselines for comparison
Validation Determine what requirements and goals are and are not being met. • Does the application meet the goals and requirements
• Is this version faster or slower than the last
• Will I be in violation of my contract/SLA if I release
Unit • Immediate feedback about performance of code.
• Help isolate and tune performance issues
• Provide a baseline of performance characteristics • Is this segment of code reasonably efficient?
• Did I stay within my performance budgets?
• Is this code performing as anticipated under load?

Component • Provides information to help them detect and tune component level performance issues.
• Provide a baseline of performance characteristics. • Is this component meeting expectations?
• Is this component reasonably well optimized?
• Is the observed performance issue caused by this component?
Smoke • Provide a quick assessment of current performance.
• Quickly compare one build another.
• Quickly find obvious performance issues.
• Quickly provide data to prioritize next steps. • Is this build/configuration ready for additional performance testing.
• What type of performance testing should I conduct next?
• Does this build have better or worse performance than the last?
Load • Determining throughput of the peak load.
• Determining adequacy of hardware.
• Determining capacity of a database.
• Evaluating the adequacy of a load balancer.
• Collecting scalability and capacity planning data • How many users can the application handle before “bad stuff” happens
• How much data can my database/file server handle?
• Are the network components adequate?

Stress • Determining if data can be corrupted by over stressing the system
• Estimating the conditions under which errors appear
• Establishing application monitoring triggers
• Ensuring that security holes are not opened up by stressful conditions. • What happens if we underestimated the peak load?
• What kind of failures should we plan for?
• What indicators should we be looking for to intervene prior to failure?

Capacity Planning • Provide actual data to the capacity planners.
• Validate capacity planning models.
• Determine current usage and capacity.
• Provide usage and capacity. • Validate that capacity planning models represent reality.
• Ensure capacity planning remains in sync with actual system usage and growth patterns.

Spike • Memory leaks
• Disk I/O (thrashing)
• Slow return to steady - state
• What happens if we underestimated the peak load?
• What kind of failures should we plan for?
• What indicators should we be looking for?
Endurance • Slow memory leaks
• Insufficient file storage capacity
• Slowness due to volume of stored data
• Most realistic scenario to predict production. • Will performance be consistent over time?
• Are there slow growing problems that we haven’t detected?
• Is there external interference that we didn’t account for?

Performance Testing

A technical investigation done to determine or validate speed, scalability and/or stability characteristics of the product under test. Performance is concerned with achieving response times, throughput, and resource utilization levels that meet the performance objectives for the application under test.
Benefits
• The intent of performance testing is to predict the performance characteristics of an application in production.
• These predictions are valuable to the stakeholders who make decisions about whether or not an application is ready for release, or requires performance improvement prior to release.
• Performance testing is general term that covers all of the various sub-sets of performance testing, therefore every value and benefit listed below can also be considered to be a potential benefit of performance testing in general.
• The key benefit to performance testing is having the information to make informed decisions related to the speed, scalability and stability of a product before it is too late to take action if that information demonstrates that the application will not achieve success in production.
Risks Addressed
• User Satisfaction
• Adequate Capacity
• Acceptable Stability
• Production Performance Predicted
• Performance Characteristics understood early
Challenges and Areas Not Addressed
• No single test or type of test.
• Frequently technically complex to do well
• Often time consuming
More Information
• For more information on X, see docs Y
• For more information on Z, see How To A
• For more information on LMNO, see How To P.

Performance Investigation
An activity based on collecting information related to speed, scalability and/or stability characteristics about the product under test that may have value in determining or improving the quality of the product.

Benefits
• The object of Performance Investigation is to collect useful information at various stages during development to catch severe performance issues early and provide information to developers, architects and administrators that will help them tune their part of the application.
• To add value, an investigative task need not necessarily resemble anticipated workloads.
Risks Addressed
• Highlights major performance issues early
• Collects data for retrospective analysis that saves time later
• Establishes baselines for comparison
Challenges and Areas Not Addressed
• No predictions about performance in production
• Can’t tell you if performance is any good, only if it is better/worse than some other measurement or expectation
• Sometimes doesn’t add value until the data is analyzed retrospectively
More Information
• For more information on X, see docs Y
• For more information on Z, see How To A
• For more information on LMNO, see How To P.
Performance Validation
An activity that compares speed, scalability and/or stability characteristics of the product under test to the expectations that have been set or presumed for that product.
Benefits
• The object of Performance Validation is to compare a particular release or configuration of an application against some pre-determined standard or baseline.
• Generally, this is only valuable as the application or system nears completion.
• The primary value of performance validation is to determine what requirements and goals are currently being met or not met and by how much.
Risks Addressed
• Does the application meet the stated goals and requirements
• Is this version faster or slower than the last
• Will I be in violation of my contract/SLA if I release this version
Challenges and Areas Not Addressed
• Pass/Fail style results are of minimal use for tuning
• Very few metrics are captured (relative to investigation) to keep from skewing results
• Seemingly insignificant changes in the test can lead to dramatic changes in results.
More Information
• For more information on X, see docs Y
• For more information on Z, see How To A
• For more information on LMNO, see How To P.
Performance Unit Test
Any performance test that targets a module of code where that module is any logical sub-set of the entire existing code base of the application. Commonly tested modules include functions, procedures, routines, objects, methods and classes. Performance Unit Tests are frequently created and conducted by the developer who wrote the module of code being tested.
Benefits
• Providing immediate feedback to a developer about performance characteristics of the code they just wrote.
• Providing information to developers to help them isolate and tune observed performance issues by enabling their performance unit tests in a special release of the application to execute performance tests against.
• Provide a baseline of performance characteristics for a developer to compare to after changes are made to the code.
Risks Addressed
• Is this segment of code reasonably efficient?
• Did I stay within my performance budgets?
• Is this code performing as anticipated under load?
Challenges and Areas Not Addressed
• No notion of end-to-end response time
• Rarely highlights contention or concurrency issues
• Sometimes leads to hyper-optimizing a non-bottleneck
More Information
• For more information on X, see docs Y
• For more information on Z, see How To A
• For more information on LMNO, see How To P.

Component Performance Test
Any performance test that targets an architectural component of the application. Commonly tested components include servers, databases, networks, firewalls and storage devices.
Benefits
• Providing information to administrators to help them detect and tune performance issues related to their component without the added complication of isolating variables in other components.
• Provide a baseline of performance characteristics for an administrator to compare configurations.
Risks Addressed
• Is this component meeting expectations?
• Is this component reasonably well optimized?
• Is the observed performance issue caused by this component?
Challenges and Areas Not Addressed
• No notion of end-to-end response time
• Sometimes hides the issue the test was designed to exploit
• Sometimes leads to hyper-optimizing a non-bottleneck
• Can lead to misplaced confidence about where the performance problem isn’t.
More Information
• For more information on X, see docs Y
• For more information on Z, see How To A
• For more information on LMNO, see How To P.

Performance Smoke Test
A test designed to determine if your application can successfully perform all the operations under normal load condition for short duration of time.
Benefits
• Provide a quick assessment of whether a build or configuration is performing within a reasonable deviation from expectations.
• Quickly compare one build or configuration to a build or configuration that a performance smoke test was previously executed against.
• Quickly determine if there are obvious and significant performance issues with a build or configuration.
• Quickly provide data to do an initial prioritization of additional performance tests to be conducted against a build or configuration.
Risks Addressed
• Is this build/configuration ready for additional performance testing.
• What type of performance testing should I conduct against this build/configuration first.
• Did this build/configuration change significantly from the last in terms of performance.
• Did we cause an obvious or significant performance related issue with this build/configuration.
Challenges and Areas Not Addressed
• Does not provide a complete data set.
• Many potential performance issues go undetected.
• Not suitable for predictive analysis or drawing detailed conclusions.
• Results can easily be given more weight they deserve if not reported properly.
More Information
• For more information on X, see docs Y
• For more information on Z, see How To A
• For more information on LMNO, see How To P.
Load Test
A performance test focused on determining or validating performance characteristics of the product under test when subjected to workload models and load volumes anticipated during production operations. These tests are all about the “How many’s”, “How bigs” and “How muches”.
Benefits
• Determining the throughput required to support the anticipated peak load.
• Determining the adequacy of a hardware environment.
• Determining the capacity of a database.
• Evaluating the adequacy of a load balancer.
• Detecting concurrency issues.
• Detecting functionality errors under load.
• Collecting data for scalability and capacity planning purposes
Risks Addressed
• How many users can the application handle before “bad stuff” happens
• How much data can my database/file server handle?
• Are the network components adequate?
Challenges and Areas Not Addressed
• Not designed to focus on the speed of response.
• Results should not be used except to compare to other related load tests
More Information
• For more information on X, see docs Y
• For more information on Z, see How To A
• For more information on LMNO, see How To P.
Stress Test
A performance test focused on determining or validating performance characteristics of the product under test when subjected to workload models, and load volumes beyond those anticipated during production operations. Stress tests may also include tests focused on determining or validating performance characteristics of the production under test when subjected to workload models and load volumes when the product is subjected to other stressful conditions, such as limited memory, insufficient disk space or server failure. These tests are all about determining under what conditions an application will fail, how it will fail and what indicators can be monitored to warn of an impending failure.
Benefits
• Determining if data can be corrupted by over stressing the system
• Estimating how far beyond the target load an application can go before causing failures and errors in addition to slowness
• Establishing application monitoring triggers to warn of impending failures
• Ensuring that security holes are not opened up by stressful conditions.
• Determining the side effects of common hardware or supporting application failures.
Risks Addressed
• What happens if we underestimated the peak load?
• What kind of failures should we plan for?
• What indicators should we be looking for to intervene prior to failure?
Challenges and Areas Not Addressed
• Unrealistic by design, making them easy for other’s to dismiss.
• Frequently hard to know how much stress is worth applying.
• Sometimes you break something really expensive to replace
More Information
• For more information on X, see docs Y
• For more information on Z, see How To A
• For more information on LMNO, see How To P.
Capacity Planning Test
Capacity testing is complementary to load testing and related to stress testing in that it determines your server's ultimate failure point, whereas load testing monitors results at various levels of load and traffic patterns and stress testing focuses on intentionally forcing failures due to unexpected circumstances. You perform capacity testing in conjunction with capacity planning. You use capacity planning to plan for future growth, such as an increased user base or increased volume of data. For example, to accommodate future loads you need to know how many additional resources (such as CPU, RAM, disk space, or network bandwidth) are necessary to support future usage levels. Capacity testing helps you identify a scaling strategy to determine whether you should scale up or scale out.
Benefits
• Provide actual data to the capacity planners to validate or enhance their models and/or predictions.
• Conduct various tests to compare capacity planning models and/or predictions.
• Determine current usage and capacity of existing system to aid in capacity planning.
• Provide usage and capacity trends of existing system to aid in capacity planning.
Risks Addressed
• Validate that capacity planning models represent reality.
• Ensure capacity planning remains in sync with actual system usage and growth patterns.
Challenges and Areas Not Addressed
• Capacity model validation tests are complex to create.
• Not all aspects of a capacity planning model can be validated through testing when those aspects would provide the most value.
More Information
• For more information on X, see docs Y
• For more information on Z, see How To A
• For more information on LMNO, see How To P.

Spike Test
A performance test focused on determining or validating performance characteristics of the product under test when subjected to workload models and load volumes that repeatedly increase beyond anticipated production operations for short periods of time. Spike testing is a subset of stress testing. A special case of a stress test that is particularly prone to highlight certain performance issues.
Benefits
• Memory leaks
• Disk I/O (thrashing)
• Slow return to steady - state
Risks Addressed
• What happens if we underestimated the peak load?
• What kind of failures should we plan for?
• What indicators should we be looking for to intervene prior to failure?
Challenges and Areas Not Addressed
• Unrealistic by design, making them easy for other’s to dismiss.
• Frequently hard to know how much stress is worth applying.
• Sometimes you break something really expensive to replace
More Information
• For more information on X, see docs Y
• For more information on Z, see How To A
• For more information on LMNO, see How To P.
Endurance Test
A performance test focused on determining or validating performance characteristics of the product under test when subjected to workload models and load volumes anticipated during production operations over an extended period of time. Endurance testing is a subset of load testing. These tests are not frequently conducted because they are, by definition, extremely time consuming, however they are effective in eliminating surprises in production
Benefits
• Slow memory leaks
• Insufficient file storage capacity
• Performance degradation as a result of an increased in stored data
• Overnight, automatic virus definition updates on a server causing performance degradation
Risks Addressed
• Will performance be consistent over time?
• Are there slow growing problems that we haven’t detected?
• Is there external interference that we didn’t account for?
Challenges and Areas Not Addressed
• Often the time to execute negates the potential for finding important performance issues.
• Frequently difficult to parse and derive meaning from the large volume of data collected.
• Regularly fail unexpectedly due to complications with the load generation environment or scheduled sever maintenance.