Return to
PerformanceTestingGuidance
How To: Programmatically Set Think Time Between Test Iterations in VS.NET 2005
Applies to:
* Performance Testing
* VS .NET 2005
Summary:
This How to will explain the process of doing programmatic iteration pacing, with a Load Test Plug-in, in VS.NET 2005 for a Web test, to maintain a steady rate of transactions: pacing can be understood as a think time in between test iterations, and for a load test, you can set pacing time, globally configuring the scenario, through the "Think Time Between Test Iterations" settings. If you need to maintain a steady transaction rate of a test, then you need to conditionally vary your pacing rate, per session and Web test basis.
Contents:
*Objectives
*Overview
*Summary of Steps
*Step 1 – Create Load test containing one or more Web tests in a test mix.
*Step 2 – Define the test iteration delay time to meet business requirements for the Web tests in the load test.
*Step 3 – Create a load test plug-in class to control the pacing time of the Web tests in the test mix.
*Step 4 – Add the created load test plug-in class into the test project.
*Step 5 – Hook up the load test plug-in to the load test.
*Step 6 – Add Context Parameters.
*Step 7 – Execute Smoke test to validate the pacing.
Objectives:
*Learn how to use a Load Test Plug-in to do pacing; timeout iteration of a particular Web test in a load test based on business requirements to simulate real-end users
Overview:
In VS.NET 2005 think time is the delay of time in between http requests in the same session (test iteration). Iteration think time or pacing is the delay of time in between sessions (test iterations). When a Web test finishes it will be delayed characterizing a test delay. This can be set as a general global setting on the Load test for the individual scenario. For test design and planning is required to introduce test iteration pacing based on the test execution thresholds, where the test iteration think time will be based on the difference between threshold and test execution during runtime, if execution time should be less than value of threshold. This allows extra time for the system to respond more slowly while increasing the quantity of users on the system, controlling Load test during runtime execution based on consistent transaction throughput. Below are the key points when using adaptive pacing time during the execution of a load test:
*Use varying iteration pacing technique when abandonment is not an option for the users of the system. For example a corporate sales order application, where employees are goaled based on the number of orders processed per day, or a help desk system that needs to process a number of calls per hour.
*Iteration pacing does not replace think time in between requests. Both think times and iteration pacing should be used in load testing. Think times should be used as in recorded session, replicating the user behavior.
*For stress testing only think times should be used because there are no concerns with emulating a real-end user action or business process transaction. This is common when no test planning and test design steps were done, and recorded scripts are available, or when replaying IIS logs.
*The idea of having interactively calculated iteration pacing is to enable the test script to intelligently maintain an emulation of the business process execution, while adapting to the dynamic system response times.
*System response time is only relevant when the system fails to support the business process demand. Until then system response time is not the primary factor in governing the business process. End-users are tolerant of slower response time, up and to a specific point where the system is so slow it prevents them from doing their business.
*Pacing delay time should be defined as an intervalgreater than the test iteration execution time, when a controlled transaction rate is desired, simulating business goals.
Below table exemplifies the pacing criteria for a Load test with test mix of three Web tests ( browse, search and Place order):
| User | Test Execution | Test Execution | Iteration |
| Scenarios | Time | Delaty | Pacing |
| | (Seconds) | Meeting | Delay |
| | | *Business requirements * | (Iteration Think Time) |
| | | (Seconds) | *(Seconds) * |
| Browse | 300 | 600 | 300 |
| Search | 200 | 400 | 200 |
| Place Order | 400 | 800 | 400 |
Step 1 – Create Load test containing one or more Web tests in a test mix.
First create a test project that will contain your Load Test and Web Tests. Then create your individual Web tests that reflect key business user scenarios. Then create a Load Test, to which you will add your Web tests. While creating Load Test you can set many runtime properties to generate desired load simulation, for example you can specify the load pattern, browser and the network types and add the performance counters to be monitored. For more information on creating load tests please refer "How To: Create a Load Test Using VS.NET 2005" at << to link to how to >>
Step2 – Define the test iteration delay time to meet business requirements for the Web tests.
Start by describing the end user activity and the duration of their activity, qualifying that time as a business requirement. For example, if you were managing a help desk call center you might have an expectation that each employee answers a minimum of 6 phone calls per hour. Therefore, the call center computer system should have response times which are fast enough to support the processing of a single help desk call within 10 minutes, or less. If the average call has 10 steps, you might consider that the system should be able to save the data with no greater than 1 minutes of response time between each step in the process for completing the call. You must also include the end user activity time, where they are talking with their customer on the phone or filling in forms on the page. If the end-user activity time combined with the system response time is more than 1 minute between steps, then the employees will fall behind their goal of processing 6 calls per hour at the help desk.
The matrix below illustrates the relationship between the call center frequency of calls and the system performance required to support the call center employees:
| End User Quantity | End users Activity Time | System Response Time | Total Call Time |
| 5 | 20 seconds | 5 seconds | 250 seconds |
| 10 | 20 seconds | 10 seconds | 300 seconds |
| 20 | 20 seconds | 20 seconds | 400 seconds |
| 25 | 20 seconds | 45 seconds | 650 seconds |
As you can see in the table with the end user activity duration is constant at 20 seconds and the system response time degrades as we put more and more users on the system. When they finally reach a max of 25 users on the system it took the system 45 seconds to save data and move to the next step in the call process. The total call time was 650 seconds. Typically, this is where the employees start to complain that the system is to slow to use and way to slow to meet their target goal of reaching 6 calls per hour, per employee.
If you only used a constant think time between iterations the test throughput would be skewed at the beginning of the test execution and then continue to slow down as the end-user quantity was ramped-up. In the following table you can see that the test simulation starts out simulating more than twice the expected call volume that is observed in the real world. As the system gets slower, the test also slows down to the point where 25 users are processing less than the required call volume that is observed in the real world.
| End User Quantity | Total Call Time (for 10 steps) | Test Iterations ||Test Throughput |
| | (30 second constant pacing) | Total Calls / hour |
| 5 | 250 seconds | 280 seconds | 12.8 |
| 10 | 300 seconds | 330 seconds | 10.9 |
| 20 | 400 seconds | 430 seconds | 8.3 |
| 25 | 650 seconds | 680 seconds | 5.2 |
The difficulty here is that the initial measurements generate too many transactions, and the later measurements generate too few transactions, which makes it increasingly difficult to calculate what the real results. One solution is to provide a programmatic think time between iterations, to keep the test throughput constant during the ramp-up and steady state for the test. You will calculate the duration of the end-user activity, including the system response time, and subtract that time from the desired transactional throughput goal. In our example of the help desk call center, the throughput should be 6 calls per hour or 1 call completed every 10 minutes. The table below illustrates how the calculation is performed.
| End User Quantity | Total Call Time (for 10 steps) | Iteration Time Calculation | Actual Test Iteration Delay Time | *Test Throughput: Total Calls / hour * |
| 10 | 300 seconds | (600 – 300) | 300 seconds | 6 |
| 20 | 400 seconds | (600 – 400) | 200 seconds | 6 |
| 23 | 600 seconds | (600 – 600) | 0 seconds | 6 |
| 25 | 650 seconds | (600 – 650) | -50 seconds | 5.2 |
In the first row, you see 5 users being executed and the Total Call Time is 250 seconds – which includes end-user activity and the system response time. In column 3 you see the calculation for the Iteration Delay Time, which subtracts the Total Call Time of 250 seconds from the desired Test Throughput time of 600 seconds. It is necessary to wait for exactly 280 seconds at the end of the test iteration, before beginning the next iteration.
Step 3 – Create a Load Test Plug-in class to control the pacing time of the Web tests in the test mix.
In this step, you create a plug in class for your load test project. Load Test Plug-in is an extensibility point that allows code to be executed when test iteration is completed. This Load Test Plug-in will introduce the timeout at the end of the test iteration, based on the test execution delay that will meet business requirements. The test interation time out will be configured per Web test basis of all tests in the test mix of a load test. The process of writing a Load Test Plug-in can be summarized:
*Create a plug-in Class.
*Implement Initialize method.
*Implement the call back function.
Creating a Load Test Plug-in Class
To create a load test plug in class, take the following steps:
1. Add a new class to your test. To do so, right click on your project, click Add, and then click Class. Name your class
PacingControl.cs2. Add the
LoadTesting namespace to your project. To do so, add a reference to
Microsoft.VisualStudio.QualityTools.LoadTestFramework by right clicking your test Project and clicking Add Reference. Next, add the following using statement to your class:
using
Microsoft.VisualStudio.TestTools.LoadTesting; 4. Inherit your class from
ILoadTestPlugin.
public class [PacingControl] : [ILoadTestPlugin]
5. Add the following member variables to your class
private [LoadTest] [mLoadTest;]
Your class so far should look like this:
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using
Microsoft.VisualStudio.TestTools.LoadTesting; namespace
LoadPacePlugin{
public class [PaceLoadTestPlugin] : [ILoadTestPlugin]
{
private [LoadTest] [mLoadTest;]
…
}
The purpose of the member variables is as follows:
•
mLoadTest. It is initialized with the
LoadTest object. The
LoadTest object holds the context of a load test settings, runtime variables available and the event handlers to be executed in different stages of load test execution. Below is the description of the event handler used to control pacing time out at the end of the test iteration of the Web test:
•
mLoadTestTestFinished. It is the call back function hooked to the handler
TestFinished of the load test. Every time a test is finished the call back function
mLoadTestTestFinished is executed.
6. Implement the initialize method in your class. To do so, add the following code to your class
public void
Initialize(LoadTest loadTest)
{
[mLoadTest] = loadTest;
[mLoadTest.TestFinished] += new
[EventHandler<TestFinishedEventArgs>(mLoadTestTestFinished);]
}
Implement the Initialize method of the
ILoadTestPlugin interface which takes
LoadTest object as a parameter. In the Initialize subscribe event handler
mLoadTestTestFinished to
TestFinished. public void
Initialize(LoadTest loadTest)
{
….
}
Before starting the test Initialize is called once.
myLoadTest is assigned with
LoadTest passed as a parameter; and the call back function is subscribed to the
TestFinished event handler. This will execute when test iteration is completed.
7. Implement the call back function
void [mLoadTestTestFinished(object] sender, [TestFinishedEventArgs] e)
{
//we get the test name from the context paramater
//if there is none we do not do pacing;
//so for each web test we need to create a
//context paramater with Name=testName
//value=threshold
if [(mLoadTest.Context.ContainsKey(e.TestName))]
{
//the duration of that particular test
[TimeSpan] [mTimeSpan] = e.Result.Duration;
int [toTalTime] = [(int)mTimeSpan.TotalMilliseconds;]
//the threshold of the test
int testMaxduration =
int.Parse((string)mLoadTest.Context[e.TestName]);
//here is conditional wait, if the test duration is less
//than Max test duration as dictated by
//performance requirements
if (testMaxduration > [toTalTime)]
{
//we wait still need to validate this sleeping
Thread.Sleep(testMaxduration - [toTalTime);]
}
else if(testMaxduration < [toTalTime)]
{
//throw an error message, and start the next
//iteration
//right away
//the message goes to the UI
throw new
ApplicationException("Web test "+e.TestName+
" exceeded performance
requirements of "+
"(miliseconds)"+
[testMaxduration.ToString());]
}
}
More Information
Every time a test is completed; the
TestFinished event is raised, which will be handled by this call back function. It checks if there is a context parameter with the name of the test where pacing is to be controlled:
mLoadTest.Context.ContainsKey(e.TestName). If the context parameter does not exist the test continues without doing pacing. The name of the test is passed by the parameter
TestFinishedEventArgs e to the call back function, so when any of the Web tests finishes its iteration the call back function will check if Think time between iterations will happen for that Web test.
The test duration is read from
TestFinishedEventArgs parameter e.Result.Duration and stored in an integer, this is total time execution of the test including think times in between requests. The max duration of the test is read from the value context parameter int testMaxduration =int.Parse((string)mLoadTest.Context
e.TestName). This is the maximum duration of the test allowed specified from the business requirement. If Maximum duration is greater than the total time of test execution, the test waits for the difference between Maximum duration and total time of test execution. If the test execution total time is greater than Maximum duration then the test flags an exception which will be displayed in the UI of VS.NET.
Example Load Testing Plug-In Class
Here is the complete code sample for the plug-in class, which you can simply copy paste.
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using
Microsoft.VisualStudio.TestTools.LoadTesting;namespace
LoadPacePlugin{
public class [PaceLoadTestPlugin] : [ILoadTestPlugin]
{
private [LoadTest] [mLoadTest;]
//we initialize once, setting the handler
//and the context parameter
public void [Initialize(LoadTest] loadTest)
{
[mLoadTest] = loadTest;
[mLoadTest.TestFinished] += new
[EventHandler<TestFinishedEventArgs>(mLoadTestTestFinished);]
}
void [mLoadTestTestFinished(object] sender,
[TestFinishedEventArgs] e)
{
//we get the test name from the context paramater
//if there is none we do not do pacing;
//so for each web test we need to create a
//context paramater with Name=testName
//value=threshold
if [(mLoadTest.Context.ContainsKey(e.TestName))]
{
//the duration of that particular test
[TimeSpan] [mTimeSpan] = e.Result.Duration;
int [toTalTime] = [(int)mTimeSpan.TotalMilliseconds;]
//the threshold of the test
int testMaxduration =
int.Parse((string)mLoadTest.Context[e.TestName]);
//here is conditional wait, if the test duration is
//less than pacing
if (testMaxduration > [toTalTime)]
{
//subtract duration from pacing time
Thread.Sleep(testMaxduration - [toTalTime);]
}
else if(testMaxduration < [toTalTime)]
{
//throw a warning message, and start the next
//iteration
//right away
//the message goes to the UI
throw new
ApplicationException("Web test "+e.TestName+
" exceeded performance
requirements of "+
"(miliseconds)"+
[testMaxduration.ToString());]
}
}
}
}
}
Step 4 – Add the created load test plug-in class into the test project.
Once the Test Plug-in class has been developed add the class project containing the plug-in into the Test Project, it is required for configuring the load test properties for load test plug-in.
*Right Click on the test solution.
*Click on add existing project.
*Browse to project containing the Load Test Plug in.
*Select the Load Test Plug in and it will be added to the Test Project.
Step 5. Hook up the load test plug in to the load test.
In this step we will hook the load test plug-in to the load test so it can be executed. For a Load test to execute the load test plug in, it needs to be configured once the load test plug-in has been added to the test project. The load test plug-in needs to be added in the properties of the root node of the Load test. Below are the detailed steps.
*Once the load test is created click on the root node of your load test.
*Click on the ellipsis on the properties pane of the load test.
*Set Load Test Plug-in dialog is displayed, with available Load Test Plugins.
*Select
PaceLoadTestPlugin.
Step 5 Add Context Parameters.
The context parameters are data object variables you pass to the Load Test Plug-in and that can be assigned to local variables during initialize phase before load test executes, or during the execution of the call back function. Context parameters are used to allow passing variables during execution of a test to make it possible to change the load test settings such as current load, based on conditions such as Web test name and number of test executions.
*In the load test right click on run settings.
*Click on option Add Context Parameter
*In the properties pane there will be the name with default parameter1 name and the value empty. Create parameter name as being the load test name( case sensitive) and as value the Maximum test duration in milliseconds as specified in step 2.
o
HelpDeskCall parameter name value 15000000
Step 6 – Execute a smoke test to validate the pacing.
Increase gradually the load of users and run a short duration test (5-7minutes) to validate your pacing time out and the number of tests that failed to meet performance requirements. Below are the results from running a Load test with a Web test with 4 requests with 1 second think time in between requests and maximum test time set to 15 seconds.
| User | Tests | Total | Test Execution | Test Execution | Iteration | * #Tests* |
| Scenario | Per second | Tests | Time | Delay | Pacing | Execution |
| *Help Desk * | ** | ** | (Seconds) | Meeting | Delay | Not |
| Call | ** | ** | ** | Business requirements | (Iteration | Meeting |
| # Users | ** | ** | ** | (Seconds) | Think Time) | Requirements |
| 1 | 0.063 | 19 | 4.9 | 15 | 10.1 | 0 |
| 5 | 0.32 | 85 | 7.8 | 15 | 8.1 | 0 |
| 10 | 0.65 | 190 | 14.9 | 15 | 0.1 | 72 |
| 25 | 1.6 | 434 | 15.2 | 15 | 0 | 420 |
*During the test run or at the end of the test in the UI pane expand the test and then the Avg. Test Time. Drag it to the graph pane to see the response times for the tests. At the bottom left pane the readings will show the total tests and the tests per second(throughput).
*Click on the errors link in the UI, it will give the reading of number of tests that failed to meet the performance requirements. In the below figure 279 out of 473 total tests failed to meet the 15 seconds test time execution.
* Click on the errors link in the UI, it will give the reading of number of tests that failed
to meet the performance requirements. In the below figure 279 out of 473 total tests
failed to meet the 15 seconds test time execution.
WindowsClient: New WPF Showcase Addition: Enterprise
TechNet Edge: Windows Server 2008 R2 : New Power Management Features
