Return to
PerformanceTestingGuidance
Applies to:
* VS.NET 2005
* Performance Testing
Summary:
This How To explains how to write and use a Load Test plug-in that will control the number of test iterations for a particular Web test or for all Web tests in a test mix of a Load Test. It also demonstrates how to abort the test once the number of required test iterations is reached.
Contents:
* Objectives.
* Overview.
* Summary of Steps.
* Step 1 – Create Load test containing one or more Web tests in a test mix.
* Step 2 – Create a load test plug-in class to monitor the number of tests to be executed.
* Step 3 – Add the created load test plug-in class into the test project.
* Step 4 – Hook up the load test plug-in to the load test.
* Step 5 – Add Context Parameters.
* Step 6 – Run the Lload Ttest with
MaxTestIteration for a specific Web test aborting the test when condition is met
Objectives:
* Learn how to control the execution of a load test based on the number of test iterations in a Web test.
* Learn how to write a load test plug-in to control the number of tests in a Web test to be executed during a load test.
Overview:
Frequently while designing or conducting load tests, it becomes clear that the only way to achieve the test duration that is desired is to increase the number of test iterations (e.g. the number of times that a specific scenario represented by a user session) rather than increasing the duration of individual user sessions. For example, this is desirable in the following scenarios:
* Load Tests that are designed to examine the effects of a certain transaction count of a particular Web test, after which another Load test starts, characterizing. For example a test mix of Browse, Search and Place Order could be run to replicate a scenario that caused performance challenges in a previous release such as placing an order after 20000 orders were entered in the database and still pending shipping status.
* Certain workload characterizations require testers to determine the number of users needed to achieve a particular transaction count of a specific Web Test over given time frame to reproduce anticipated load patterns. For example, during a peak load a website may be expected to process 20000 orders in a 2 hour time frame. To simulate this scenario, the tester likely needs to increase the number of times a transaction are repeated during the given two hour period to achieve the correct number of transactions without additional virtual users or overlapping sessions.
This how to showcases how to control the number of test iterations, when using VSTS.
Scenario
For this How to we will consider a typical commerce application, with following web tests and their workload distribution shown in the Load Distribution table.
Browse: Open Page->Login->Search->Browse->Logout.
Search: Open Page->Login->Search->Logout
Place Order: Open
Page->Login->Search->Browse->AddToCart->Order->Logout Load Distribution
| User Scenarios | % of users | Users |
| Browse | 50 | 250 |
| Search | 30 | 150 |
| Place Order | 20 | 100 |
| Total | 100 | 500 |
Typically each execution of the above case can be understood as the user session, or navigation path through the Web site. In VSTS the user session in Web tests can be also be viewed as test iteration. When the Web test is executed from a load test, during the duration of the test, after finished going through all pages, it executes again the session path, iterating in a loop cycle. Many times when executing a load test it is necessary to control the load test duration based on the number of test iterations(user sessions) of a particular Web test rather than time of test execution. This is desirable in the following scenarios:
Summary of Steps
* Step 1. Create Load test containing one or more Web tests in a test mix.
* Step 2. Create a load test plug-in class to monitor the number of tests to be executed.
* Step 3. Add the created load test plug-in class into the test project.
* Step 4. Hook up the load test plug-in to the load test.
* Step 5. Add Context Parameters
* Step 6. Run the Lload Ttest with
MaxTestIteration for a specific Web test aborting the test when condition is met
Step 1. Create a Test Project with a Load test containing one or more Web tests in a test mix.Create a Test Project.
First create a test project that will contain your Load Test and Web Tests. Then create your individual Web tests that reflect key business user scenarios. Then create a Load Test, to which you will add your Web tests. While creating Load Test you can set many runtime properties to generate desired load simulation, for example you can specify the load pattern, browser and the network types and add the performance counters to be monitored
For more information on creating load tests please refer "How To: Create a Load Test Using VS.NET 2005" at << to link to how to >>
Step 2. Create a load test plug in class to monitor the number of tests to be executed.
In this step, you create a plug in class for your load test project. Load Test plug-in is an extensibility point that allows code to be executed when test iteration is completed. This load test plug-in will allow controlling the duration of a Load test based on the number of test iterations of a particular Web test, or based on the test iterations of all Web tests.
This can help gauging the number of users needed to simulate realistic load patterns for work load characterization of a Web site. For example a Load Test must simulate placing 20000 orders over 2 hours time frame, when the Load test is running a mix of Web tests (Browse, Search and Place Order). This can be particularly challenging to simulate for Load Tests containing many Web Tests. Another case scenario is to run a Load Test based on transaction count, after which another Load Test must be executed characterizing a business scenario. The process of writing a load test plug-in can be summarized:
* Create a plug-in Class.
* Implement Initialize method.
* Implement the call back function.
Create a plug-in Class
To create a load test plug in class, take the following steps:
*
Add a new class to your test. To do so, right click on your project, click Add, and then click Class. Name your class *MaxTestIterationsLoadTestPlugin *
*
Add the LoadTesting namespace to your project. To do so, add a reference to
Microsoft.VisualStudio.QualityTools.LoadTestFramework by right clicking your test Project and clicking Add Reference. Next, add the following using statement to your class:
using [Microsoft.VisualStudio.TestTools.LoadTesting;]
*
Inherit your class from ILoadTestPlugin.
public class [MaxTestIterationsLoadTestPlugin] : [ILoadTestPlugin]
*
Add the following member variables to your class
private [LoadTest] [mLoadTest;]
private int [mMaxTestIterations;]
private int [mTestsFinished;]
private bool [mAbortTest;]
private string [mTestName;]
Your class so far should look like this:
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using [Microsoft.VisualStudio.TestTools.LoadTesting;]
namespace [LoadIterationPlugin]
{
public class [MaxTestIterationsLoadTestPlugin] : [ILoadTestPlugin]
{
#region [ILoadTestPlugin] Members
private [LoadTest] [mLoadTest;]
private int [mMaxTestIterations;]
private int [mTestsFinished;]
private bool [mAbortTest;]
private string [mTestName;]
}
}
More InformationThe purpose of the member variables is as follows:
*
mLoadTest. It is initialized with the
LoadTest object. The
LoadTest object holds the context of a load test settings, runtime variables available and the event handlers to be executed in different stages of load test execution. Below is the description of the context variables used from the
LoadTest object and event handler used to control the number of test iterations:
*
mMaxTestIterations. The number of Test Iterations to be executed. It is read from Context Parameter
MaxTestIterations of the load test,
mLoadTest.Context"MaxTestIterations"). If the plug in is used and the context parameter is not defined the test will run as specified in the time settings of the load test.
*
mAbortTest. Flag to specify if test is to be aborted once the condition is met. It is read from Context Parameter
AbortWhenMaxTestsCompleted of the Load Test
mLoadTest.Context.ContainsKey("AbortWhenMaxTestsCompleted"). If the parameter is not defined test will stop the load but will continue to execute as specified in the time settings of the load test.
*
mTestName. Test name for which the test iteration is to be counted. This name needs to match the name of the Web Test ( case sensitive). For example a Load Test may have Browse, Search, and Orders Web Tests but the Load Test should execute only until when 1000 Orders are placed. It does not count when other tests are executed. It is read from Context Parameter
TestName mLoadTest.Context"TestName". If the parameter is not defined test will count all Web Tests iterations not only the one specified
*
mLoadTestTestFinished. It is the call back function hooked to the handler
TestFinished of the load test. Every time a test is finished the call back function
mLoadTestTestFinished is executed.
*
Implement the initialize method in your class.Implement the Initialize method of the
ILoadTestPlugin interface which takes
LoadTest object as a parameter. In the Initialize method and assigns local variables
mMaxTestIterations, mAbortTest, mTestName, and subscribe event handler
mLoadTestTestFinished to
TestFinished event. To do so, add the following code to your class
public void [Initialize(LoadTest] mloadTest)
{
//load Test object
[this.mLoadTest] = myloadTest;
if (this.mLoadTest.Context.ContainsKey("MaxTestIterations"))
{
try
{
//number of iterations to be executed
mMaxTestIterations =
int.Parse((string)mLoadTest.Context["MaxTestIterations"]);
}
catch [(FormatException)]
{
throw new
ApplicationException(
"MaxTestIterations not in integer format");
}
}
//The function to be executed after every test iteration
[this.mLoadTest.TestFinished] += new
[EventHandler<TestFinishedEventArgs>(mLoadTestTestFinished);]
if (mLoadTest.Context.ContainsKey("AbortWhenMaxTestsCompleted"))
{
try
{
// Bool value to specify if test is to be
// aborted when condition is met
[mAbortTest] =
bool.Parse((string)mLoadTest.Context["AbortWhenMaxTestsCompleted"]);
}
catch [(FormatException)]
{
throw new [ApplicationException(]
"AbortWhenMaxTestsCompleted not in bool format");
}
}
if (this.mLoadTest.Context.ContainsKey("TestName"))
{
try
{
//The name of the test where the test iterations are to be counted
[mTestName] = (string)this.mLoadTest.Context["TestName"];
}
catch [(FormatException)]
{
throw new [ApplicationException(]
"AbortWhenMaxTestsCompleted not in string format");
}
}
}
More InformationBefore starting the test Initialize is called once.
mLoadTest is assigned with
LoadTest passed as a parameter; local variables
mMaxTestIterations,
mAbortTest,
mTestName are assigned from the context parameters read from the load test object. Also the call back function is subscribed to the
TestFinished event handler. This will execute when iteration is completed.
*
Implement the call back function
void [mLoadTestTestFinished(object] sender, [TestFinishedEventArgs] e)
{
if [(mMaxTestIterations] > 0)
{
if [(String.IsNullOrEmpty(mTestName)] ||
[(String.Equals(mTestName,e.TestName)))]
{
if (++mTestsFinished >= [mMaxTestIterations)]
{
[SetLoadForAllScenariosToZero();]
if [(mAbortTest)]
{
[mLoadTest.Abort();]
}
}
}
}
}
More InformationEvery time a test is completed; the
TestFinished event is raised, which will be handled by this call back function. In the
mloadTestTestFinished function checks if
mMaxTestIterations is greater than zero. If this context parameter is not defined the test will continue as specified in the run time settings.
If
mMaxTestIterations is greater then zero, then it checks if
mTestName is null or same as the Test being finished, in this case the condition will be tested for every test including all the tests executed for the load test. If
mTestName is null or is same as the Test being finished, then test executed count is incremented using the
mTestsFinished local counter. If it is null the Load test will count all iterations for all Web tests in the test mix, otherwise only for that specific Web test
If tests counts local counter
mTestsFinished crosses the threshold set by
mMaxTestIterations it calls
SetLoadForAllScenariosToZero which for every scenario retrieved from
LoadTestScenarios collection of
LoadTest, sets the current load to zero. After setting the load to zero if
mAbortTest was defined it will abort the test, otherwise it continues with load set to zero until test finished as defined in the runtime settings.
*
Create the code to set load for the scenarios
void [SetLoadForAllScenariosToZero()]
{
foreach [(LoadTestScenario] scenario in [mLoadTest.Scenarios)]
{
[scenario.CurrentLoad] = 0;
}
}
More InformationLoadTestScenario is the individual scenario designed in the load test. A load test can have one or more scenarios and each scenario can have one or more tests to be executed. The
LoadTestScenario has the
CurrentLoad that can be read or set for the current load running on a scenario for each individual agent that is executing the test. If the test is executing locally without agents then
CurrentLoad expresses the current load running on the scenario on the local machine
It is important to note the
IterationControl Plug-in will hold variables pertaining to the local agent executing the load test. If
mMaxTestIterations is set to 30 and if there are 3 agents executing the test then 90 iterations total will be executed. Also when a scenario is set to zero current load like in
scenario.CurrentLoad =0 the load is set for that particular agent executing the load test plug in.
Example Load Testing Plug-In Class
Here is the complete code sample for the plug-in class, which you can simply copy paste.
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using [Microsoft.VisualStudio.TestTools.LoadTesting;]
namespace [LoadIterationPlugin]
{
public class [MaxTestIterationsLoadTestPlugin] : [ILoadTestPlugin]
{
#region [ILoadTestPlugin] Members
private [LoadTest] [mLoadTest;]
private int [mMaxTestIterations;]
private int [mTestsFinished;]
private bool [mAbortTest;]
private string [mTestName;]
public void [Initialize(LoadTest] loadTest)
{
[mLoadTest] = loadTest;
if (mLoadTest.Context.ContainsKey("MaxTestIterations"))
{
try
{
[mMaxTestIterations] =
int.Parse((string)mLoadTest.Context["MaxTestIterations"]);
}
catch [(FormatException)]
{
throw new ApplicationException("MaxTestIterations
not in integer format");
}
[mLoadTest.TestFinished] += new
[EventHandler<TestFinishedEventArgs>(mLoadTestTestFinished);]
[if(mLoadTest.Context.ContainsKey]
("AbortWhenMaxTestsCompleted"))
{
try
{
[mAbortTest] =
[bool.Parse((string)mLoadTest.Context]
["AbortWhenMaxTestsCompleted"]);
}
catch [(FormatException)]
{
throw new [ApplicationException]
("AbortWhenMaxTestsCompleted not in bool format");
}
}
if (mLoadTest.Context.ContainsKey("TestName"))
{
try
{
[mTestName] = (string)mLoadTest.Context["TestName"];
}
catch [(FormatException)]
{
throw new [ApplicationException]
("AbortWhenMaxTestsCompleted not in bool
format");
}
}
}
}
void [SetLoadForAllScenariosToZero()]
{
foreach [(LoadTestScenario] scenario in [mLoadTest.Scenarios)]
{
[scenario.CurrentLoad] = 0;
}
}
void [mLoadTestTestFinished(object] sender, [TestFinishedEventArgs] e)
{
if [(mMaxTestIterations] > 0)
{
if [(String.IsNullOrEmpty(mTestName)] ||
[(String.Equals(mTestName,e.TestName)))]
{
if (++mTestsFinished >= [mMaxTestIterations)]
{
[SetLoadForAllScenariosToZero();]
if [(mAbortTest)]
{
[mLoadTest.Abort();]
}
}
}
}
}
#endregion
}
}
Step 3. Add the created load test plug in class into the test project.
Once the Test Plug-in class has been developed add the class project containing the plug-in into the Test Project, it is required for configuring the load test properties for load test plug-in.
* Right Click on the test solution.
* Click on
add existing project.
* Browse to project containing the Load Test Plug in.
* Select the Load Test Plug in and it will be added to the Test Project.
Step 4. Hook up the load test plug in to the load test.
In this step we will hook the load test plug-in to the load test so it can be executed. For a Load test to execute the load test plug in, it needs to be configured once the load test plug-in has been added to the test project. The load test plug-in needs to be added in the properties of the root node of the Load test. Below are the detailed steps.
* Once the load test is created click on the root node of your load test.
* Click on the ellipsis on the properties pane of the load test.
* Set Load Test Plug-in dialog is displayed, with available Load Test Plug-ins.
* Select
MaxTestInterationsLoadTestPlugin.
* Click the Ok button .
Step 5 Add Context Parameters.
The context parameters are data object variables you pass to the Load Test Plug-in and that can be assigned to local variables during initialize phase before load test executes, or during the execution of the call back function. Context parameters are used to allow passing variables during execution of a test to make it possible to change the load test settings such as current load, based on conditions such as Web test name and number of test executions.
* In the load test right click on run settings.
* Click on option
Add Context Parameter.
* In the properties pane there will be the name with default parameter1 name and the value empty. Create three parameters one at a time
*
MaxTestIterations value write the number of test iterations. Test will execute normally as specified in the run time settings if parameter is not defined.
*
AbortWhenMaxTestsCompleted value true to abort the test. Test will not abort if parameter is not defined.
*
TestName. Web Test Name to count the iterations only for that test. It needs to match the Web Test Name as in the test mix of the scenario. If the test contains 3 Web tests, Browse, Search and Orders, and parameter is created with value Orders, only the Orders test iteration count is considered for the condition. Is parameter is not defined condition counter will be incremented for all Web test iterations.
Step 6 – Run the Load Test
In this step we will run the Create a lLoad test, which is created with one or more Web tests in the mix setting, with the load test plug- in hooked as shown in above steps. as in before steps.
Below is an example of load test run with 3 web tests (Browse, Orders and Search Products). The load test will run until 2000 orders were placed. The load test was configured to run with 20 users and for 2 hours.
Running Your Load Test
- In the Load Test Editor right click on the Load Test and select Run Test option.
- The test runs until condition is met and an abort message is displayed at the end. By clicking on tables and selecting tests from the drop down list will give the number of tests for each Web test in the test mix, the total number of tests. Below is our depicted result set totaling 2000 orders executed:
Total tests: 10134
Browse tests: 5128
Orders tests: 2000
SearchProducs tests: 3006
Resources
<<To be done>>
PerformanceTestingGuidance
WindowsClient: Several WPF Localization questions
Silverlight: Commitments
Mix Online: When Projects Fall Apart
Channel 10: Adding Flip 3D to Windows 7's Taskbar
