How To: Test Capacity using Transaction Cost Analysis (TCA)

Applies To

* Performance Testing

Summary

This How To shows you how to perform capacity planning for Web applications using transaction cost analysis. Transaction cost analysis measures the cost of a user operation on the available server resource.

* Objectives
* Overview
* Summary of Steps
* Step 1 – Compile a user Profile
* Step 2 – Execute Discrete Tests
* Step 3 – Calculate Cost of Each Operation.
* Step 4 – Calculate the Cost of an Average User profile.
* Step 5 – Calculate Site Capacity.
* Step 6 – Verify Site Capacity.
* Resources

Objectives

* Learn How to Perform Capacity Planning.
* Learn How to use Transaction Cost Analysis for Capacity Planning.

Overview

When the testing projects have limited time or budget for testing resources, it is very common to perform some mathematical estimates in an attempt to predict the performance of the application. Transaction Cost Analysis is a process for conducting just a controlled estimate, based on key measurements of a single transaction through the application. Although TCA does not serve as a replacement for the complete and thorough performance testing of the entire system, it can be very helpful nonetheless to determine if any obvious risks or gross underestimates with the hardware required to support the system. This document will help you to understand the basic process for TCA and how to perform the calculations for extrapolating an estimate of future system capacity.
As you might imagine with any mathematical calculation, it is critical to have very exact inputs to the calculations if you wish to have reliable outputs and results from the TCA model.

Summary of Steps

* Step 1. Compile a User Profile
* Step 2. Execute Discrete Tests
* Step 3. Calculate the Cost of Each Operation
* Step 4. Calculate the Cost of an Average User Profile
* Step 5. Calculate Site Capacity
* Step 6. Verify Site Capacity

Step 1. Compile a User Profile

An individual user profile is also commonly referred to as a Workload Characterization (make link reference here), where information about transaction volumes and application usage patterns are arranged into a matrix or table. You can also compile individual user profiles from analysis of the existing production traffic data, or a transactional workload taken from your load test plan. One possible resource for identifying production usage are the Internet Information Services (IIS) log files.

The Usage Profile table should contain the following key information:
* A list of user profiles – This would be the individual pages or components that are going to be the major consumers of system resources
* The average duration of a user session – How long should a user execute the business process, including any think time or delays due to user behavior
* The total number of operations performed during the session - How many unique and different transactions did a single user execute during the session.
* The frequency with which users perform each operation during the session - How often did a single user execute any unique transaction during the session.

Here is an example of how to compile the user profile from analysis of the IIS log files:

* Calculate the number of user requests for each page and the respective percentages.
The number of user requests for each page can be extracted from the log files. Divide the number of requests for each page by the total number of requests to get the percentage. Following table illustrates a sample profile.

User Requests per Page

ID	URI	# of Requests	Percentage
1	/MyApp/login.aspx	18,234	35%
2	/MyApp/home.aspx	10,756	20%
3	/MyApp/logout.aspx	9,993	19%
4	/MyApp/SellStock.aspx	4,200	8%
5	/MyApp/BuyStock.aspx	9,423	18%
Total	NA	52,606	100%

* Calculate the logical operations and number of requests required to complete the operation.
A user operation can be thought of as a single complete logical operation that can consist of more than one request. For example, the login operation might require three pages and two requests. The total number of operations performed in a given time frame can be calculated by using the following formula:

Number of operations = # of requests / # of requests per operation

The Requests per operation column in the following table shows how many times the page was requested for a single operation.

User Requests per Operation

ID	URI	# of Requests	Requests / Operations	*# of Operations
1	/MyApp/login.aspx	18,234	2	9,117
2	/MyApp/logout.aspx	9,993	1	9,993
3	/MyApp/SellStock.aspx	4,200	2	2,100
4	/MyApp/BuyStock.aspx	9,423	3	3,141
Total	NA	41,850	8	24,351

* Identify the average user profile, session length, and operations per session. You can analyze the IIS log files to calculate the average user session length and the number of operations an average user performs during the session. The session length for the sample application was calculated as 10 minutes from the IIS logs, and the average user profile for the sample application is shown in the following table.
Average User Profile

Operations	# of Operations Executed during an average session
Login	1
SellStock	3
BuyStock	2
Logout	1

Of course, if the application has never been running in a production environment, then you would not have any IIS logs to process. So, another way to create the user profile is to use the documents created from performance test planning. You can find more information about identifying user profiles, see "Workload Modeling" <<We have a how to on this we can provide link to it>>

Step 2. Execute Discrete Tests

Here you must create a very simple test script that will execute discrete tests for each user operation identified in Step 1. You will configure a load test to execute enough stress on your system to reach maximum throughput. For example, you need to run separate tests for Login, BuyStock, and SellStock operations, increasing the load on the system to the point where the CPU resource is at 100% utilization. The test script only fires the requests for a dedicated user operation – such as a single .aspx page.

The procedure for executing the tests consists of the following tasks:

* Set up the environment with the minimum number of servers possible.
Make sure that the architecture of your test setup mirrors your production environment as closely as possible.
* Create a test script that loads only the operation in consideration without firing any redundant requests.
* Define the point at which your system reaches maximum throughput for the user profile. You can identify this point by monitoring the ASP.NET Applications\ Requests/Sec counter for an ASP.NET application when increasing the load on the system. Identify the point at which Requests/Sec reaches a maximum value.
* Identify the limiting resource against which the cost needs to be calculated for a given operation. List the performance counters you need to monitor to identify the costs. For example, if you need to identify the cost of CPU as a resource for any operation, you need to monitor the counters listed in the following table.

Performance Counters Used to Identify Cost

Object	Counter	Instance
Processor	% Processor Time	_Total
ASP.NET Applications	Requests/Sec	Your virtual directory

Note - It is important to have a throughput measure that is reliable and relates specifically to the requests being emulated by the load testing tool. For ASP.NET, you can use Requests/Sec to calculate the processor cost per request.

* Run load tests for a duration that stabilizes the throughput of the application. The duration can be somewhere between 15 to 30 minutes. Stabilizing the throughput helps create a valid, equal distribution of the resources over a range of requests. It is important to use the same range of time for test execution, for each user operation/test you will execute.

Output:

The output from executing this series of steps for each scenario would be a report like the following:

* Number of CPUs = 2
* CPU speed = 1.3 GHz

Following table shows a sample report for the results of the load tests.
Load Test Results comparing the Processor % utilization to the Requests/Sec

User operation	Process\% Processor Time	ASP.NET Applications\Requests/Sec
Login	90%	441
SellStock	78%	241
BuyStock	83%	329
Logout	87%	510

Step 3. Calculate the Cost of Each Operation

Measure the cost of each operation in terms of the limiting resource identified in Step 2. Measuring the operation cost involves calculating the cost per request and then calculating the cost per operation. Use the following formulas for these tasks:

* Cost per request. You can calculate the cost in terms of processor cycles required for processing a request by using the following formula:
Cost (Mcycles/request) = ((number of processors x processor speed) x processor use) / number of requests per second

For example, using the values identified for the performance counters in Step 2, where processor speed is 1.3 GHz or 1300 Mcycles/sec, processor usage is 90 percent, and Requests/Sec is 441, you can calculate the page cost as:

((2 x 1,300 Mcycles/sec) x 0.90) / (441 Requests/Sec) = 5.30 Mcycles/request

* Cost per operation. You can calculate the cost for each operation by using the following formula:
Cost per operation = (number of Mcycles/request) x number of pages for an operation

The cost of the Login operation is:

5.30 x 3 = 15.9 Mcycles

If you cannot separate out independent functions in your application and need one independent function as a prerequisite to another, you should try to run the common function individually and then subtract the cost from all of the dependent functions. For example, to perform the BuyStock operation, you need to perform the login operation, calculate the cost of login separately, and then subtract the cost of login from the cost of the BuyStock operation.

Therefore the cost of a single BuyStock operation can be calculated as follows:

Single cost of BuyStock operation = Total cost of BuyStock – Cost of Login operation

The cost of a single BuyStock operation is:

39.36 – 15.92 = 23.44 Mcycles

Following table shows the cost of each user operation in a sample application using the following scenario.

* CPU Speed = 1300 MHz
* Number of CPUs = 2
* Overall CPU Mcycles = 2,600

*Cost per Operation for Login, SellStock, BuyStock, and Logout Operations *

User Operations	CPU % utilization	Total net CPU Mcycles	ASP.NET Requests / Sec	# of Requests	Operation Costs (Mcycles)	# pages without login	Single Operation cost
Login	90%	2,340.00	441	3	15.92	3	15.92
SellStock	78%	2,028.00	241	5	42.07	2	26.16
BuyStock	83%	2,158.00	329	6	39.36	3	23.44
Logout	87%	2,262.00	510	5	22.18	2	6.26

The operation cost needs to be measured separately for each tier of an application.

Step 4. Calculate the Cost of an Average User Profile

The behavior of actual users can cause random crests and troughs in resource utilization. However, over time these variations even out statistically to average behavior. The user profile you compiled in Step 1 reflects average user behavior. To estimate capacity, you need to assume an average user and then calculate the cost in terms of the limiting resource identified in Step 2.

As shown in following table, during a ten-minute session, an average user needs 147.52 Mcycles of CPU on the server. The cost per second can be calculated as follows:

Average cost of profile in Mcycles/sec = Total cost for a profile / session length in seconds
Thus, the average cost for the profile shown in Table 7 is:

147.52/600 = 0.245 Mcycles/sec
This value can help you calculate the maximum number of simultaneous users your site can support.

Cost of an Average User Profile

Average User Profile	Number of operations executed during an average session	Cost per operation (Mcycles)	Total cost per operation (Mcycles)
Login	1	15.92	15.92
SellStock	3	26.16	78.47
BuyStock	2	23.44	46.87
Logout	1	6.26	6.26
Total			147.52

Step 5. Calculate Site Capacity

Calculating site capacity involves knowing how many users your application can support on specific hardware and what your site's future resource requirements are. To calculate these values, use the following formulas:

* Simultaneous users with a given profile that your application can currently support. After you determine the cost of the average user profile, you can calculate how many simultaneous users with a given profile your application can support given a certain CPU configuration. The formula is as follows:
Maximum number of simultaneous users with a given profile = (number of CPUs) x (CPU speed in Mcycles/sec) x (maximum CPU utilization) / (cost of user profile in Mcycles/sec)

Therefore, the maximum number of simultaneous users with a given profile that the sample application can support is:

(2 x 1300 x 0.75)/0.245 = 7,959 users

* Future resource estimates for your site. Calculate the scalability requirements for the finite resources that need to be scaled up as the number of users visiting the site increases. Prepare a chart that gives you the resource estimates as the number of users increases.
Based on the formulas used earlier, you can calculate the number of CPUs required for a given number of users as follows:

Number of CPUs = (Number of users) x (Total cost of user profile in Mcycles/sec) / (CPU speed in MHz) x (Maximum CPU utilization)

If you want to plan for 10,000 users for the sample application and have a threshold limit of 75 percent defined for the processor, the number of CPUs required is:

10000 x 0.245 / (1.3 x 1000) x 0.75 = 2.51 processors

Your resource estimates should also factor in the impact of possible code changes or functionality additions in future versions of the application. These versions may require more resources than estimated for the current version.

Step 6. Verify Site Capacity

It is very important to qualify the results from TCA analysis as estimates, and extrapoliation of very specific performance measurements. Most test projects use TCA as only an initial estimate of performance, but then continue the load testing effort to execute load tests to verify that the transaction cost analysis model accurately predicts your application capacity and future requirements. In short, TCA is not recommended as a replacement for load testing and you should still execute a thorough and robust performance test if at all possible.

Verify the calculated application capacity by running load tests with the same characteristics you used to calculate transaction cost analysis. The verification script is simply a collection of all transaction cost analysis measurement scripts, aggregated and run as a single script.

The actual values and the estimated values should vary by an acceptable margin of error (+/- 15%). The acceptable margin of error may vary depending on the size of the setup and the budget constraints. You do not need to run load tests each time you perform transaction cost analysis.