How To – Stress Test Web Applications

J.D. Meier, Prashant Bansode, Scott Barber, Mark Tomlinson

Summary

This How To shows you a baseline technique for performing stress testing of Web applications.

Overview

This How To will show a baseline technique for performing stress testing of Web applications. The goal of stress testing is to uncover application bugs that surface under high load or other unexpected stressful conditions. The approach for performing stress testing can be broken down indentifying the following steps:
* The performance critical key scenarios.
* The workload profile for distributing the load among the key scenarios.
* The metrics used against the performance objectives.
* The load test with test cases implementing the key scenarios.
* The execution of the load tests.
* The data analysis of the performance metrics.

This How To will show a baseline technique for performing stress testing of Web applications. The goal of stress testing is to uncover application bugs that only surface under high load conditions. The basic approach for performing stress testing can be broken down indentifying the following steps: the performance critical key scenarios, the workload profile for distributing the load among the key scenarios, the metrics used against the performance objectives, the load test with test cases implementing the key scenarios, the execution of the load tests and the data analysis of the performance metrics

Stress test your application by subjecting it to very high loads that are beyond the capacity of the application, while denying it the resources required to process that load. For example, you can deploy your application on a server that is running a processor-intensive application already. In this way, your application is immediately starved of processor resources and must compete with the other application for CPU cycles.

The goal of stress testing is to unearth application bugs that surface under high load conditions. These bugs can include:

Synchronization issues
Race conditions
Memory leaks
Loss of data during network congestion
Unlike load testing, where you have a list of prioritized scenarios, with stress testing you identify a particular scenario that needs to be stress tested. There may be more than one scenario or there may be a combination of scenarios that you can stress test during a particular test run to reproduce a potential problem. You can also stress test a single Web page or even a single item, such as a stored procedure or class.

The approach involves the following steps:
Identify key scenario(s). Identify the application scenario or cases that need to be stress tested to identify potential problems.
Identify workload. Identify the workload that you want to apply to the scenarios identified in step 1. This is based on the workload and peak load capacity inputs.
Identify the metrics. Identify the metrics that you want to collect about the application. Base these metrics on the potential problems identified for the scenarios you identified in step 1.
Create test cases. Create the test cases where you define steps for running a single test and your expected results.
Simulate load. Use test tools to simulate the required load for each test case and capture the metric data results.
Analyze the results. Analyze the metric data captured during the test.
The next sections describe each of these steps.

Input

The following input is valuable to creating an effective stress test.
* Application characteristics (scenarios)
* Potential or possible problems with the scenarios
* Workload profile characteristics
* Peak load capacity (obtained from load testing)

Summary of Steps

Stress testing can be visualized as a six-step process as shown in Figure x.

Figure x: Stress testing process

The steps are:
Step 1. Identify Objectives
Step 2 – Identify Key Scenarios
Step 3 – Identify Workload
Step 4 – Identify Metrics
Step 5 – Create Test Cases
Step 6 – Simulate Load
Step 7 – Analyze the Results

Step 1. Identify Objectives

Step 2 – Identify Key Scenarios

Identify the scenario or multiple scenarios that you want to stress test. Generally, you start by defining a single scenario that you want to stress test to identify a potential performance issue. There are a number of ways to choose appropriate scenarios:

Select scenarios based on how critical they are to overall application performance.
Try to test those operations that are most likely to affect performance. These might include operations that perform intensive locking and synchronization, long transactions, and disk-intensive I/O operations.
Base your scenario selection on the data obtained from load testing that identified specific areas of your application as potential bottlenecks. While you should have fine-tuned and removed the bottlenecks after load testing, you should still stress test the system in these areas to verify how well your changes handle extreme stress levels.
Scenarios that may need to be stress tested separately for a typical e-commerce application include the following:

An order processing scenario that updates the inventory for a particular product. This functionality has the potential to exhibit locking and synchronization problems.
A scenario that pages through search results based on user queries. If a user specifies a particularly wide query, there could be a large impact on memory utilization. For example, memory utilization could be affected if a query returns an entire data table.
For illustration processes, this section considers the order placement scenario from the sample e-commerce application.

Step 3 – Identify Workload

The load you apply to a particular scenario should stress the system sufficiently beyond threshold limits. You can incrementally increase the load and observe the application behavior over various load conditions.

As an alternative, you can start by applying an anti-profile to the workload model from load testing. The anti-profile has the workload distributions inverted for the scenario under consideration. For example, if the normal load for the place order scenario is 10 percent of the total workload, the anti-profile is 90 percent of the total workload. The remaining load can be distributed among the other scenarios. This can serve as a good starting point for your stress tests.

You can continue to increase the load beyond this starting point for stress testing purposes. Configure the anti-profile needs in such a way that the load for the identified scenario is deliberately increased beyond the peak load conditions.

For example, consider the normal workload profile identified for a sample application that is shown in Table 16.4.

Table 16.4: Sample Workload Profile for a Sample Application

User profile	Percentage	Simultaneous users
Browse	50%	500
Search	30%	300
Order	20%	200
Total	100%	1,000

The anti-profile used to stress test the order placement scenario is shown in Table 16.5.

Table 16.5: Order Use Case Anti-Profile

User profile	Percentage	Simultaneous users
Browse	10%	100
Search	10%	100
Order	80%	800
Total	100%	1,000

Continue to increase the load for this scenario, and observe the application response at various load levels. The time duration of the tests depends upon the scenario. For example, it might take 10 to 15 days to simulate deadlocks on a four-processor machine, but the same deadlocks may be simulated on an eight-processor machine in half that time.

Step 4 – Identify Metrics

Identify the metrics corresponding to the potential pitfalls for each scenario. The metrics can be related to both performance and throughput goals, in addition to ones that provide information about the potential problems. This might include custom performance counters that have been embedded in the application. For example, this might be a performance counter used to monitor the memory usage of a particular memory stream used to serialize a DataSet.

For the place order scenario, where contention-related issues are the focus, you need to measure the metrics that report contention in addition to the basic set of counters. Table 16.6 shows the metrics that have been identified for the place order scenario.

Step 5 – Create Test Cases

Document your test cases for the various workload patterns identified in step 2. For example:

Test 1 – Place order scenario:
1,000 simultaneous users.

Use a random think time between 1 and 10 seconds in the test script after each operation.

Run the test for 2 days.

Test 1 – Expected results:
The ASP.NET worker process should not recycle because of deadlock.

Throughput should not fall below 35 requests per second.

Response time should not be greater than 7 seconds.

Server busy errors should not be more than 10 percent of the total response because of contention-related issues.

Step 6 – Simulate Load

Use performance tools to run the identified scenarios and to simulate load. This allows you to capture the data for your metrics.

Note Make sure that the client computers that you use to generate load are not overly stressed. Resource utilization, such as processor and memory, should not be very high.

Step 7 – Analyze the Results

Analyze the captured data and compare the results against the metric's accepted level. If the results indicate that your required performance levels have not been attained, analyze and fix the cause of the bottleneck. To address the issue, you might need to do one of the following:

Perform a design review. For more information, see Chapter 4, "Architecture and Design Review of a .NET Application for Performance and Scalability."
Perform a code review. For more information, see Chapter 13, "Code Review: .NET Application Performance."
Examine the stack dumps of the worker process to diagnose the exact cause of deadlocks.
For example, to reduce the contention for the place order scenario of the sample application, you can implement queues into which orders are posted and are processed serially rather than processing them immediately after the order is placed.