LoadRunner Analysis Statistics and Graphs This page explains the sample Summary Report generated by LoadRunner and then, more importantly, explain how to make use of them to make decisions. Pop-up a sample report in a new window (this may take a few seconds)

Topics: Analysis Summary Statistics Summary Transaction Summary HTTP Responses Summary Standard Graphs TPS Explained Changing Graphs & Templates Your comments?

Site Map   About this site   Enter your search terms Submit search form Web   WilsonMar.com

Analysis Summary

Header Time Range

This date is, by default, in the European format of dd/mm/yy (30/8/2004 for August 30, 2004).

Scenario Name:

The filepath to the lrs file.

Results in Session:

The filepath to the lrr file.

Statistics Summary

Maximum Running Vusers:

This number is usually smaller than the number of vusers specified in run-time parameters because of ramp-up time (shown in the Running Vusers graph) and processing delays.

Total Throughput (bytes):

Dividing this by the amount of time during the test run yields the next number:

Average Throughput (bytes/second):

This could be shown as a straight horizontal line in the Throughput graph.

Because network bandwidth is specified in the number of bits per second, dividing this by 8 yields the average bandwidth used, which is less useful than momentary peak bandwidth usage shown in the Throughput graph.

Total Hits:

Dividing this by the amount of time during the test run yields the next number:

Average Hits per Second:

This could be shown as a straight horizontal line in the Hits per Second graph.

View HTTP Responses Summary

This link sends you further down the page.

Transaction Summary

Transactions:

Total Passed is the total of the Pass column. The number of transactions Passed and Failed is the total count of every action defined in the script, multiplied by the number of vusers, further multiplied by the number of repetitions, and also multiplied by the number of iterations.

The numbers to the left of the Pass column are the number of seconds. The Minimum, Average, Maximum columns are illustrated in the Transaction Performance Summary graph.

When analyzing the number of seconds, beware of totaling up all transaction times because that would duplicate the time of actions nested within summary actions.

Average Response Time

Transaction Name

Minimum

This is the fastest time

Average

This is the arithmetic mean

Maximum

This is the slowest time

Std. Deviation

"Std." is an abbreviation of the word "Standard", so called because the mathematical technique used to calculate it (taking the square root of the sum of squared numbers) "standarizes" (makes absolute) both positive and negative deviations from the average. Thus std. deviation is also called "root mean squared deivation". Std. deviation is also referred to as a Greek letter, "sigma".

From Robert Niles' excellent explanation

The larger the std. deviation, the more "dispursed" or volatile the values are around the average.

If the distribution of deviations has a "normal" pattern reflecting what occurs from pure chance — and not "skewed" because of an unnatural influence — we can assume that:

• 68.27% of the values are included within the range of
  one std. deviation on either side of the average.
• 90% of the values are included within the range of
  1.28 std. deviations on either side of the average.
• 95.45% of the values are included within the range of
  1.69 std. deviations on either side of the average.
• 97.88% of the values are included within the range of
  2 std. deviations on either side of the average.
• 99% of the values are included within the range of
  2.33 std. deviations on either side of the average.
• 99.86% of the values are included within the range of
  three std. deviations on either side of the average.

The standard deviation statistic is used to determine whether the difference between two runs could have been caused by pure chance rather than brilliant changes in coding or configuration settings.

According to statisticians, a difference cannot be legitimately called "statistically significant" unless it is greater than what could have occured by chance. For example, we can be 98% sure that averages from two runs are really different only when the other average is more than 2.33 standard deviations away. This is calculated by multiplying the standard deviation by 2.33, then both subtracting and adding that to the average. Try the calculations with your numbers:

Base Average	Std. Deviation	z Desired Level of Significance	Comparison Averages
		90%
+	( x	)	or less or more

90 Percent

This measurement is not shown in the Controller screen during a run because it is calculated from a list of all values. If a run had 200 measurement points, this would be the value associated with the 180th data point (90 percent of 200).

Median

This is not shown in the Controller screen during a run because it is calculated from a list of all values — the middle value from among all measurements ranked by value.

Pass

This is the count of iterations which were successful with no errors.

Fail

This is the count of iterations which ended with an error

Stop

This is the count of iterations which were halted

HTTP Responses Summary

HTTP 200 ("OK") is considered successful.
HTTP 302 highlights a redirection ("Moved Temporarily"), a normal event.
HTTP 404 is a "resource not found" error.
HTTP 500 is a "server busy" error
HTTP 503 is an authentication denial.
W3.org's RFC 2616 Explanation of all HTTP codes

Total

Each redirected page is counted at least twice in the total — once for the 302 redirection and another for the final 200 (or 404 if the page is not found).

Graph Category	Graph	Note
Vusers	Running Vusers This line graph shows the impact of Ramp-Up and Ramp-Down specified for run-time. This also shows the time when Errors begin to occur during the test run. Note the time when the Number of Vusers peaks and when it begins to drop off. This is the "steady state" period of the run.
Transactions	Average Transaction Response Time This line graph is used to determine whether performance is within acceptable minimum and maximum transaction performance time ranges expected of the system. It displays the average time taken to perform transactions during each second of the load test. Unlike the Summary Report, only Transaction End Status of Passed are included (Failed transactions are filtered out by default).
	Transactions per Second This line graph helps you determine the actual load imposed by each transaction action at various spans of time throughout a run. Each line displays the number of transactions performed — both passed(successful) and failed(unsuccessful) transactions. Each point on a line is calculated from the number of transactions that ended within each span of time (the Granularity period automatically set to the lowest allowed relative to the scenario run). For example, an Average data point of 0.5 within the Granularity period of 16 Seconds means that 8 (0.5 x 16) transactions were completed within that 16 second period. TPS (Transactions Per Second) Explained TPS is the count of transactions processed each second. Like Miles Per Hour, it is a measurement of speed. One second A A A A .25 .25 .25 .25 One second B B .5 .5 In this illustration, when each action is performed on its own, action A has a TPS of 4 per second. Action B has a TPS of 2 per second. Note that: Think time decreases TPS Errors increases TPS because failed transactions end sooner than good transactions which complete processing. Multiple actions performed in a linked sequence decreases TPS: One second A B A .25 .5 .25 When actions are performed in sequence, action A now individually has a TPS of 2 per second and action B individually has a TPS of 1 per second. However, the combination of multiple actions average together to 3 per second.
	Total Transactions per Second This graph helps you determine the actual total transaction load on your system at any given moment in time. This line chart displays the total of all transaction actions performed during each second of a load test. An additional line is used to display the total number of Failed (unsuccessful) transactions.
	Transaction Summary This bar graph displays, by transaction action, how many transactions passed, failed, stopped, or ended with errors.
	Transaction Performance Summary This bar chart displays the minimum, average, and maximum response time for each transaction action in the load test. This chart was manually scaled down to make actions easier to interpret. The tall column action on the left summarizes all other transaction actions by a user.
	Transaction Response Time Under Load This XY graph illustrates how the Number of Vusers impacts Response Time. Typically, individual response times follow an exponential curve up as servers strain to process more Vusers. A flat horizontal line means that response time does not vary, usually because the system has enough (or too much) capacity.
Web Resources	Hits per Second The one line on this graph helps you evaluate the amount of load Vusers generate, in terms of the number of hits to all web servers. It displays the number of hits made on all Web servers by Vusers during each second of the load test.
	Hits Summary This pie chart displays the number of hits made on the Web server by Vusers. This graph helps you evaluate, in terms of the number of hits, the amount of load Vusers generate.
	Throughput This line chart helps you to evaluate the amount of load Vusers generate on network resources. It displays the raw amount of bytes that the Vuser client received each second as a result of the hits on the web server during load testing.
	Throughput Summary This pie chart illustrates values in the Total Throughput (bytes) presented in the Summary report.
	HTTP Responses per Second
	Connections
	Connections per Second This line graph displays the number of HTTP & HTTPS Connections requested of all web servers. The graph displays different lines for New Connections and Connection Shutdowns.
	SSLs Per Second This line graph only appears when web apps use HTTPS from servers with SSL certificates installed, commonly associated with login transactions which require added security. This line graph displays the number of New and Reuse Sessions processed per second.
System Resources UNIX Resources: Captured by LoadRunner issuing uptime command.	UNIX CPU Util Each line on this graph displays for each Linux GH9 server the percent of CPUs are busy. It is obtained using the vmstat or sar command. The maximum is 400 for a 4 CPU machine.
	UNIX Avg Load- Last 60 sec (one minute) Background monitoring (rstatd and rup) consumes about 0.02 on loadtest7.
	UNIX Paging rate Each line on this graph displays for each Linux GH9 server the rate pages are swapped in and out of memory. Average of 15-25 is expected from Linux RH9 servers at rest. When loadtest9 is re-populating mysql, this can jump to a max of 25,000.
Web Server Resources Apache: Captured by LoadRunner issuing http://loadtest4/ server-status Large differences between servers may be due to the Load Balancer.	#Busy Servers and #Idle Servers
	Apache CPU Usage Each line on this graph displays the number of CPU Resource used on an individual web server. Differences in usage among web servers may indicate issues with the load balancing method or configuration differences among web servers.
	Apache Hits/sec Each line on this graph displays the number of hits coming through an individual web server each second.
	Apache KBytes Sent/sec Each line on this graph displays the number of raw Kilobytes coming through an individual web server, similar to the UNIX netstat command. At rest, each server averages 0.150 of keep-alive traffic.
Web Page Breakdown	Web Page Breakdown This graph lists the average download time for each transaction action.
	Page Component Breakdown This pie chart shows how much each component is a percentage of the sum of Average download time (in seconds).
	Download Component Size (KB)

Caution on scaling: Please focus on the numbers and avoid comparing lines on the same graph because the automatic scaling of individual metrics currently does not scale like metrics simularly.

The scale of each graph is manually set.

Changing Charts and Templates

• It is necessary to change graphs, such as overlaying Running Vusers over graphs of response time and other results.

• It is often necessary to add graphs, such as to zoom in to the effect of ramp-up at the beginning of a run.

• Changing Global Filter settings erases Filters set for individual graphs.

• When a template is applied, LR honors Global Filter settings.

Multiple Dimmensional Variables

Each run can relate to each other in several ways:
• Repeat runs under the same conditions, necessary to:
  1. determine the amount of variability among runs — the extent differences are due to chance (in baseline runs)
  2. detect creeping changes over time (as in Longevity testing and Production availability monitoring)
• Different loads against an identical configuration of hardware and software:
  4. Different scope (complexity) of user actions (as in speed testing
  5. Different amount of think time between actions by an individual user and
  6. Different amount of pause between iterations of different users
  7. Different number of simultaneous users
  (as in contention and overload testing
  8. Different rate new vusers enter the run.
  9. Different amount of data processed (as in volume testing
• Different runs to compare the impact of different run conditions such as:
  9. changes to application enviornment parameters (such as JVM parameters)
  10. different versions of application or operating system software
  11. additional hardware — memory, CPUs, network connections, etc. — as during scalability testing

Thus, there can be at least a dozen ways run results can differ — needing "mix and match" analysis.

Cross Compare Results

Although LoadRunner Analysis treats the results of each run as a separate sets of data, results from several runs can be compared and contrasted using the Cross Results option under the Analysis "File" menu.

Your rating of this page: Low High Your comments on this topic, please: Publish this comment publicly

Your first name: Your family name: Your location (city, country): Your Email address:  Email me updates

Top of Page Thank you!