A Team Script Framework For LoadRunner

Here is a full-featured framework written in LoadRunner VuGen script that adds agile feature and overcomes several inconveniences and annoyances associated with large-scale performance testing of Web applications

Topics this page:

Site Map  
About this site  

Can Your Script Do All This? The Framework Alternative

You can spend many months scripting reinventing what many others have already figured out.

Or you can leverage the experience of others and focus on your own application by installing and using a script framework that, without additional programming and debugging, can:

1. Precisely control message log verbosity so more meaningful information is presented with less clutter.
2. Capture in each page returned the number of seconds, bytes returned, page title, item counts, etc. so that they can be compared against expected values. Counts of bytes and items in each page provide one of the reasons for differences in latency.
3. Quickly change transaction names and other aspects of how each transaction is run in a spreadsheet rather than digging through (and possibly breaking) code.
4. Specify different behaviors at run-time without changing code and recompiling.

5. Retrieve and report (but not too frequently) metrics (from XML in Mercury Profiler, web pages containing OS/JVM/JMX/Oracle counters from SiteScope or other APM system) and adjust their numeric scale and calculate ratios (such as Apdex) before displaying them as user-defined counters for real-time comparison while LoadRunner runs.
6. Store run information into a detailed log and summary response data in a wiki page, without additional coding.
7. "Timebox" transactions so that each transaction consumes a specified amount of time. This is important as a load testing strategy.
8. Randomly execute transactions (without additional coding but with just changes to a spreadsheet).
9. Retry transactions that return in error or repeat transactions a specified number of times.
10. Execute transactions specific to a Vuser to conduct contention testing using rendezvous points.

Armed with this script library, a professional (trained by me) can create full-featured working load and stress testing scripts in a fraction of the time.

This framework helps any LoadRunner scripter save time and debugging effort.
But the architecture of this script framework is especially helpful to enhance the agility of a team measuring the performance of a complex application.

Annoyances and Solutions

This script overcomes several specific limitations in the LoadRunner product:	Solution
Transaction names must match. One of most common error messages issued by LoadRunner because each transaction name needs to be the same in two places (in lr_start_transaction() and the corresponding lr_end_transaction() functions). They are often many lines apart, which makes them difficult to find and manage.	Automatic Transaction Names
LoadRunner does not automatically capture from each web page returned its "Charset" (such as "iso-8859-1" or "utf-8"), title, and the redirected URL along with the response time and bytes downloaded from each request. LoadRunner does not provide functions that capture such information.	Response Evaluation and Display
Superfluous messages are spit out by LoadRunner regardless of usefulness. I am held hostage. I can't help having LoadRunner's messages fill up output.txt and log files because messages I send out do not appear unless I set the "message level" at the same level which makes superflous messages also appear.	Message Formatting
Script code need to be added in several specific locations to open, format, output, and close custom formatted logs with headings and is readable by a spreadsheet program or wiki page.	Transaction Logging
LoadRunner does not provide a built-in way to obtain Web Services Diagnostics data along with other normal run statistics. It does not provide a built-in way for data to be shared among vusers (load generators) dynamically during a run	Web Services Diagnostic Stats.
Although LoadRunner has a web_set_max_retries function that can be set, when an error occurs, LoadRunner attempts retry only for HTTP Response Messages 500–599 and for network API errors (HttpSendRequest and so forth). LoadRunner does not retry for timeouts or function argument errors.	Retry Logic
LoadRunner applies Run-time settings for the entire run. For example, setting the Message logging to the detail level of "Data returned from server" causes the HTML of every page to output during the whole run. LoadRunner displays HTTP Headers and HTML Body when Run-time Log settings are set to "Extended: Data returned by server" for the entire run. Plus, it adds line numbers, line breaks, and other extraneous text. Also, LoadRunner's "Run Logic" GUI currently makes it difficult to temporarily vary what is executed. There is no way to temporarily NOT run an action other than deleting and re-entering it. Only whole action files can be specified for execution, not individual functions. This leads to a bloated number of action files within a script. I cannot save a change to a random execution percentage in one action until all percentages add up to 100.	Keyword Driven Control Data Run Logic
LoadRunner does not provide a easy way to replicate the percentage mix of requests evident from analyzing production web logs.	Random Execution
LoadRunner provides a lr_think_time() function that adds time around a transaction. Since transaction times vary, LoadRunner does not provide a way to control (within the script) the (transactions per minute) rate that transactions consume the system. LoadRunner provides functions lr_get_transaction_think_time() and lr_get_transaction_wasted_time() at whole second resolution. But this framework maintains its own timers at sub-second fractional resolution.	Transaction Time Boxing
Run Logic is difficult to change. When different people work on scripts, it is difficult to enforce conventions for naming and logic. The same logic is repeated (but in different ways) across different sections of script. Different scripts are crafted for the needs of different types of testing (benchmark speed testing, load testing, etc.), which duplicates the same coding features over several scripts. Coding of reference to other functions within a script make it easier to reuse coding, but if the signature of parameters entering a function needs to change, it is difficult to find out where they are used. LoadRunner does not provide a built-in way for data to be shared among vusers (load generators) dynamically during a run	Keyword Driven Logic

This framework makes irrelevant several limitations to using the advanced Eclipse IDE to edit and compile LoadRunner scripts, because the framework handles run-time and parameter settings from a spreadsheet read by the framework.

The Eclipse platform also cannot step through LoadRunner scripts as they execute. But the framework also makes that mostly irrelevant because the framework enables stop points and changes to verbosity to be specified between any individual action.

What's The Catch - The Down Side?

This framework takes time to initiatlize its variable every time it starts. This takes about 10 seconds if the library is in the Init section and 3 seconds if the library is in a header library.

I agree with Eric Gamma that the bigger the framework becomes, the greater the chances that it will want to do too much, the bigger the learning curves become, and the more difficult it becomes to maintain it.

There is a learning curve to edit the script to what the framework needs and extra complexity in managing different driver files.

However, my belief is that the extra up-front efforts are repaid in time savings from avoiding errors -- from not writing code, and future flexibility.

The up-front effort is lessened by my code generator that creates framework call code from the DriverParms spreadsheet used to run load tests.

Getting Started

You save time by using this shared framework because you only need to specify a few lines of code to benefit from a whole set of features. You don't have to understand how the functions, variables, and attributes interact with each other in the library.

This "LoadRunner on Rails" framework is easy to begin using, and thus also easy to stop using because it sits "on top" of any existing scripts you may have.

In the script's standard vuser_init are two functions:

#include "wi_lib.h"
vuser_init()
{
	wi_vuser_init_vars(); // to initialize utility variables and attributes.
}

This script's vuser_end that LoadRunner runs once at the end is also simple:

vuser_end()
{
	rc=wi_vuser_end(); // function within the wi_lib.c
}

Within LoadRunner's Action section that LoadRunner iterates, insert this framework function:

Action() // invoked by LoadRunner for each iteration.  
{
	int rc=LR_PASS;
	rc=wi_action_Keyword_loop()
	return rc;
}

The workings of these functions are described a scetion below.

Script Framework Library Additional Files

To the standard script files described above, this script framework adds several script files to separate code into files such that responsibilities for maintenance can be divided and better managed.

1. The wi_lib.c Action script file defines generic functions (which LoadRunner stores in a wi_lib.c file). The 3,000 lines of this file is the "brains" of the framework.

   If you don't change this file, you will be able to take advantage of updates to the framework by others simply by replacing it with a new version.

2. The vuser_init.c file or a separate wi_lib.h header file defines two sets of global memory variables: one set of utility variables needed by the library, and one set added for the organization and its applications being tested.
3. The wi_app.c script Action defines application-specific functions (which LoadRunner stores together in the wi_app.c file). This file is changed to define data values application to the organization and for the needs of the application under test. All functions in this file is prefixed with "wi_app_".

If these files above are only changed by the one person responsible for defining or enforcing team standards, changes to provide additional functionality in the core script can be distributed by overlaying the same file in various scripts within the same organization.

In the sample scipt folder are three script Action files created to receive script code generated by VuGen:

• HTMLrecording.c to hold scripts created with the "HTML" recording mode.
• URLrecording.c to hold scripts created with the "URL" recording mode.
• GUIrecording.c to hold scripts created with the "GUI" recording mode.

Using these files means you don't need to worry about remembering to delete them from the Run Logic.

According to the Windows Task Manager, when this framework is loaded, vugen.exe has "Mem Usage" of about 27,116K. During runs, mdrv.exe has "Mem Usage" of about 8,500K.

Editing Recorded Scripts

These are explained in the sections below.

1. Record scripts into a pre-defined Action file.
2. For future flexibility, find and replace recorded URL text with a LoadRunner parameter used by the framework: {pURLHostPath}. More on this at recording and editing LoadRunner scripts
3. Add transaction name functions around each request to the server.
4. If you create new LoadRunner Action, define a new charDriverAct code that function wi_app_act_modules() within wi_app.c can reference.
5. Within each custom LoadRunner Action file, define a new charDriverNode code (such as "Click" or "Update") specified in the Driver file to invoke script functions to carry them out.
6. Within each individual script function, paste captured code into resource , construct URL
7. Add messages, if needed to
8. Define run-time attributes to override default values, as needed.

Easy Transaction Name Manipulation

Unlike LoadRunner transaction commands, this script does not require you to come up with a transaction name used to attach statistics.

Between LoadRunner request functions, instead of inserting lr_start_transaction("w"); and lr_end_transaction("w",LR_AUTO); insert two framework function calls:

wi_startTrans();
rc=web_submit_data(CurrentTrans,
    ... );
wi_endTrans();

Function wi_startTrans() obtains its transaction name from the the "Trans" column within the current Driver Parms row. later section.
Having transaction names in a spreadsheet file provides a quick way to scan and edit transaction names together.

If no transaction name is specified in the driver file, function wi_startTrans() makes up its own transaction name such as "T1" with the TransNamePrefix C character defined in wi_app.c plus a counter incremented every time that function is invoked.

The script framework saves the transaction name specified into the CurrentTrans C character variable (array) so that coders do not need to retype names in transaction end functions.
This helps to avoid perhaps the single most common coding mistake in LoadRunner.

wi_startTrans() also defines LoadRunner web_reg_save_param() functions to capture response text.

The wi_endTrans() accumulate statistics such as the amount of run time and number of bytes downloaded.

Alternatively, if you want to explicitly specify a transaction name in your application-specific script, use this function instead of
wi_startTrans():

wi_startTransName("X1");
rc=web_submit_data(CurrentTrans,
    ... );
wi_endTrans();

Function wi_startTransName("X1") ignores what is specified in the "Trans" column within Driver Parms.

The script framework also provides a set of functions to obtain totals across several transactions:

wi_startTotal();
wi_startTrans();
rc=web_submit_data(CurrentTrans,
    ... );
wi_endTrans();
wi_startTrans();
rc=web_submit_data(CurrentTrans,
    ... );
wi_endTrans();
wi_endTotal();

Currently, only one total is maintained at a time.

Internally, LoadRunner makes available as the {pDriverTrans} LoadRunner parmeter that the script moves into the charDriverTrans C variable. This is illustrated in a

Response Evaluation

{pPlaybackMode}

The wi_startTrans() function also appends
"_txt" to transaction names associated with "MODE=HTTP" and
"_all" to transaction names associated with "MODE=HTML".

This text can be changed to whatever you want in function wi_app_init() within wi_app.c Action script.

sprintf( strGetURLOnlySuffix, "_txt");
sprintf( strGetAllSuffix,     "_all");

Capturing Text Returned From Server

This script framework invokes loadrunner web_reg_save_param commands to capture from each web page returned into LoadRunner parameters:

Object	Captured into Parameter
Page Title	{pageTitle_...}
Redirected URL	{pRedirectedURL_...}
Character Set ("iso-8859-1", "UTF-8")	{pRetCharset}
Response Header	{pResponseHeaders}
Response Body	{pResponseBody}
Response Header and Body	{pResponseFile}

These are in addition to the response time and bytes downloaded from each request.

If no web_... function is defined between startTrans() and endTrans(), LoadRunner issues an error message about "web_reg_param" not used. So in such a case, instead of startTrans() and endTrans(), code this function:

wi_stubTrans();

Clear Cookies

Before the request is made, if the "ClearCookies" attribute is set to "Y", cookies are cleared out.

Output Response Header and Body Returned From Server

If a value of "Y" is specified for run-time advanced attribute named "OutputBody", the transaction functions will capture and display the HTML response body returned from the server (without adding line numbers, line breaks, and other extraneous text).
Similarly, "OutputHeaders" controls the output of HTTP response headers.

Additionally, the wi_endTrans() functions will make a request to the Diagnostic web service if the "ProbeShow" attribute is set to "Y".

Message Formatting

Messages within this script are constructed using the "my_msg" global variable and the printMessage() function:

sprintf( my_msg, "... " );
printMessage("debug", my_msg);

The "debug" in this coding example refer to Java log4j type levels controlling message output.


trace messages are issued within loops. This includes all the message levels below.
debug messages should not be printed when the application is in production. This includes all the message levels below.
info messages are issued to display the condition of major events. This includes all the message levels below.
warn messages are issued when the application is able to carry on without a problem, such as when a default value is overridden. This includes all the message levels below.
error messages are issued when an incorrect administrator-supplied configuration parameter is detected. This includes all the message levels below.
fatal messages are issued before the script run must be aborted, such as when a required value is not found.

User selections in Run-Time Settings Log settings are mostly ignored by this script. I say "mostly" because a "Advanced" settings will result in a dump of internal settings before the script's own message handler take over.

Message Control with Run-time Attributes

Whether a particular message actually appears in the VuGen's Replay Log window and output.txt file (and what appears in log files during script execution within LoadRunner's Controller) depends on the Verbosity Run-time settings Advanced Attributes available in LoadRunner since version 8.0. The meaning of Verbosity attribute values are:

Verbosity	Framework Messages	LoadRunner Messages	LR Log Level	Usage
0	No output whatsoever.	LoadRunner Error messages only.	Disabled	-
1	Error messages only.	LoadRunner Error messages only.	-	-
2	Error & Info messages only,	LoadRunner Error messages only.	Brief	Use this during Production runs.
3	Error, Info, Debug, messages.	LoadRunner Error messages only.	Extended	Use this during VuGen test runs.
4	Error, Info, Debug, Trace messages	LoadRunner Error messages only.	Extended	Use this during VuGen deugging runs.
5	Error, Info, Debug, Trace, messages.	LoadRunner Error messages & Standard log (1).	Extended	-
6	Error, Info, Debug, Trace, messages.	LoadRunner Error messages & Extended log (16).	Extended	-
7	Error, Info, Debug, Trace messages.	LoadRunner Error messages & Extended log with Advanced Trace (16+8)	Extended	-
8	Error, Info, Debug, Trace messages.	LoadRunner Error messages & Extended log with Advanced Trace, Data returned from server (16+8+2)	Extended	-
9	Error, Info, Debug, Trace messages.	LoadRunner Error messages & Extended log with Advanced Trace, Parameter substitution, Data returned from server (16+8+4+2)	Extended	-

This script was written with several "Easter Eggs" (as in many video games) that provide control of run-time behavior.

LoadRunner automatically copies attributes along with other run-time settings from VuGen to load generators so that they are visible within LoadRunner Controller Run-time Settings GUI. This script provides several ways to establish a value for each attribute: First, values are retrieved from user-specified Run-time settings Advanced Attributes. If a value is not defined there, the script contains logic to establish a default value, so no Attributes need to be defined to run this script (a good thing to create new files). These values remain unless overridden by a control parameter setting request.

SiteScope and Web Services Diagnostics Probe Metrics

Metric collection is off by default.
SiteScope statistics are retrieved from the SiteScope server if the framework's "SiteScope" attribute is defined with a value of "Y".

Diagnostics Probe statistics for the instrumented application server under test are retrieved from the Diagnostic server if the "ProbeShow" attribute is defined with a value of "Y". Defaults for run-time attributes ProbeHost and ProbeName need to be specified to avoid an error. Run-time attributes ProbeUserName, ProbePassword do not need to be changed if the diagnostics probe was installed with default values.

Like Oracle STATSPACK reports, SiteScope statistics are obtained at the start and end of the run. But this framework also takes readings after each iteration (if enough time has gone by, as describe below), which provides even more information than running STATSPACK before and after test runs.

SiteScope statistics in the middle of runs are calculated based on incremental change in values elapsed since the last reading. This is necessary because numbers from Oracle V$SYSSTAT are cumulative from when the database was restarted.

Each probe gathering can take a substantial amount of time (up to several seconds) to retrieve and process its raw data.

To avoid overwhelming the server being probed, the "ProbeSecs" attribute is queried to limit the amount of time between probe gathering requests. The wi_SvcDiag_request() function called to make the probe request does not actaully make a request and returns with an LR_FAIL condition if the specified time has not elapsed. Its default value is 60 seconds (one minute) if this attribute is not defined.

The script will bypass making an observation if the number of seconds elapsed since the previous observation is not at least the "BetweenMonSecs" attribute specified. The default is 60 seconds if this attribute is not defined.

Not all statistics reported on the web page retrieved are of interest to this framework, so each metric is retrieved with a separate web_reg_save_param command scanning the web page retrieved.

This framework script adjusts the numeric scale as necessary, such as turning raw bytes to KiloBytes (KB) by dividing with 1024. "k_rows_per_sort" is thousands of rows per sort operation.

The framework issues ratios as a user defined data point in the LoadRunner Controller and in the Wiki summary file.

Other important stats, such as the library cache hit ratio, must be manually setup in SiteScope. According to KB 45014 & 36377, the default SiteScope/templates.applications/commands.oraclejdbc file specifies SQL retrieving Oracle tables V$SYSSTAT V$INSTANCE V$STATNAME V$SESSION dba_data_files

This occurs through jdbc:oracle:thin:@IP Address:1521:Database and Oracle 10g client drivers in classes12.zip expanded into the SiteScope/java/lib/ext direcory.

In the SiteScope/groups/master.config file it helps to cache database connections (which limits the amount of connections and CPU usage) by adding
_oracleJDBCConnectionCaching=true or _databaseConnectionCaching=true for the standard database monitor.

Queries are limited to 4096 bytes. Results can be clipped if these settings are lower than: _databaseMaxColumns=10 _databaseMaxRows=1 _databaseMaxSummary=20

Key ratios DBAs use to manage Oracle are calculated by this framework:

buffer_cache_hit_ratio
chained_fetch_ratio
recursive_call_ratio
redo_space_wait_ratio
k_rows_per_sort (as in Oracle STATSPACK Load Profile reports, but in thousands of rows).

These ratios depend on having SiteScope edited to report:

consistent gets
db block gets
physical reads
recursive calls
redo log space requests
redo writes
sorts (disk)
sorts (memory)
sorts (rows)
table fetch by rowid
table fetch continued row
table scan rows gotten
user calls

SiteScope should also be edited to report Oracle activity volume statistics.

The apdex Threshold Ratio

Several stardard statistics are calculated by the LoadRunner Analysis program:

Minimum	Average	Median	Maximum	90%	Std.Dev.

(The median is not shown by the Controller Run tab because that can only be calculated after all values are available at the end of the run.)

The trouble with these common statistical measures is that they do not reflect the standard specified in SLA (Service Level Agreements), which have definitions of unacceptable response time such as "4 seconds 90% of the time (including WAN traffic time)".

The "90%" reported by LoadRunner is calculated from response times collected throughout the entiraty of a particular run. LoadRunner does not report whether transactions met its performance criteria at various times as loads vary throughout a run.

These shortcommings are resolved by a customer-oriented statistic recently developed by the apdex alliance created and led by Peter Sevcik of NetForecat.

LoadRunner does not calculate this because it's so new. But this framework does calculate it and report it as a user-defined metric.

The apdex (application performance index) reflects the ratio of measurements over a period of time that meet (or nearly meet) performance thresholds. I say "nearly meet" because Apdex gives "half credit" to individual response time measurements that is between two performance standards:

• The point (commonly abbreviated as "T") when users are "Tolerating" slowness, at which the user notices how long it takes to interact with the system. User are no longer "Satisfied".
• The point (commonly abbreviated as "F") when it becomes "Frustrating" to users, when "a casual user is likely to abandon a course of action and a production user is likely to cancel a Task.".

Beyond this point, measurements are not counted as part of the index.

On graphs, excellent performance numbers are shown higher than unacceptable performance numbers.

A 1.0 (100%) value is obtained when all users obtained satisfactory performance all the time.
A 0 (0%) value is obtained when all measurements are frustrating to users.
A 0.6 (60%) value is obtained when 40% of measurements are tolerating or frustrating to the user.

Because the apdex is geared toward production performance monitoring, which yields stats summarized by day, week, month, the Apdex specification v1.1 considers sample sizes of 100 per index value normal. It asks that smaller sample sizes be marked with an asterisk.

Transaction Logging

This script framework provides coding to create two logs to the folder specified in the OutputLogFolder attribute, which contain values such as "C:\\Temp". The double slashes are necessary.

"Y" in the OutputWikiLog attribute will allow the output of rows in a text file formatted with vertical bars to separate columns within a wiki page table after each successful transaction. An example:

||

Time Stamp

||

Transaction

||

Type

||

Seconds

||

Response KB

||

KB/Sec

||

X

||

XKB

||

Note

||

|

070304071113S0GNoneU-1 I2A1D3T1S1

|

img/au.gif

|

image/gif

|

0.617

|

1.406

|

2.3

|

1.4X

|

1.0

|

-

|

For convenience, the elapsed time and amount of data captured from returning transactions is expressed in seconds and KiloBytes rather than the milliseconds and individual bytes. "KB" is 1024 bytes.

The "KB/Sec" column provides a rough calculation for a general estimation. This is of course way low becuase it includes server processing time plus transmission time.

The "X" column is the number of times the actual bytes returned versus the expected number of bytes specified in the Driver Bytes column. For example:

"1.5X" is obtained when 1500 bytes were returned vs. 1000 bytes expected.
"-.5X" is obtained when   500 bytes were returned vs. 1000 bytes expected.

The "XKB" column reports the bytes expected after convertion to KiloBytes.

"Y" in the OutputCsvLog attribute will allow the output of comma separate values in a ".csv" (comma separated) file for import into Excel This is done after every driver control record is processed, whether it was success or not.

Time Boxing

This script framework enables you to achieve a consistent throughput rate as measured by the number of transactions completed per minute.

Time boxing is useful to determine the amount of resources (CPU % utilization, etc.) consumed at a particular level of throughput, usually to mimic a specific mix of production transactions determined by analyzing production web logs.

Simply adding a fixed amount of time after a transaction cannot achieve a consistent throughout rate because actual transaction times vary.

Time boxing ensure that each request takes a consistent amount of time by adjusting the amount of wait time after a transaction to fit a fixed time frame.

This period of time specified two ways:

1. An entry in the "Timebox" column of a DriverParm row specifies the timebox for that request.
2. A "TimeBoxSecs" run-time attribute is added to define the time box period for all transactions; or

Time within each transaction time box includes:

1.requested "Think" time before transaction requests			6.lr_ start trans	-	5.Actual response time	-	7.lr_ end trans	10."Wait" time to fulfill remainder of TimeBox seconds.
2.pause BetweenTriesSecs from previous retry (if any)	3.pre-trans framework "overhead" Time	4.lr_ think time		8.Wasted start time	-	9.Wasted end time		11.System monitoring "probe" time (if any)	12. lr_think time

1. The think time requested in "Think" columns within DriverParms files are a combination of three time frames:
2. The BetweenTriesSecs time specified in attribute satisfied if the transaction was retried;
3. Framework "Overhead" time for the framework to read DriverParam, etc.
4. Time consumed by a lr_think_time() request as needed to fulfill the time not already taken up by the two previously mentioned items above.



5. The transaction Response Time reported by LoadRunner in its Analysis reports should only reflect the time needed to execute a single LoadRunner server request function (such as web_url, or web_submit_data, etc.). This time LoadRunner measures between
6. lr_start_transaction() function (which the framework includes in its wi_startTrans() function) and
7. lr_end_transaction() (which the framework includes in its wi_endTrans() function)
8. To make sure that the time taken by the framework to make calculations is not included in the transaction time,
9. "Waste" time is measured and reported to LoadRunner in a lr_waste_time() function before
10. The Values in TimeBox for a DriverParm request is fulfilled by considering
11. the time already taken by "Probe" Time to obtain diagnostic information.
12. then adding whatever lr_thinktime() is necessary to fill the remaining time.

If a transaction takes longer than the time box time, an error message is issued and time is taken away from the next transaction's wait time. This is an attempt to keep the intended schedule by "making up" for the extra time. An excessive number of these errors of course means that the time boxes should be longer.

Time boxing behaviour is on by default. To disenable it, add a "TimeBox" run-time attribute with a value of "N".

Client Timeouts

When the "ClientTimeout" attribute is set to "max", CONNECT, RECEIVE, and STEP timeout settings on the client are set to their maximums (1000, 1000, and 32000 seconds, respectively).

This is time the script waits if the server does not repond.

A value of "defaults" sets the timeouts to 120 seconds.
A value of "min" sets the timeouts to 10 seconds.

This script overrides these settings to the value of TimeBox Wait seconds to avoid creating time outs.

Execution Control with Keyword Drivers

The call to script framework function wi_action_Keyword_loop() is the "key active ingredient" that makes this script framework save you time by being so powerful and flexible.

Instead of you coding repetitive script code, what happens during a test run is controlled by command requests defined in a "driver" parameter file.

This framework loops through a DriverParms.csv parameter file to control what is executed.

Within each row of the driver parameter file is are keywords that causes the framework to act a certain way.

Rather than changing a series of script code and recompiling, a spreadsheet program is used to define keywords and control data that the script carries out as calls to functions within the script.

For example, instead of hunting down a particular section of script coding to comment out logic (and recompiling), you can simply change the NOT PROCESS flag in the control data and the script will not execute that request.

Using data to control what the script does eliminates the need to repetitively code (and debug) many other features, described below.

Sample DriverParm.csv

This sample script is driven by a parameter (.csv or .dat) file. Its columns are aligned vertically when openned from within Excel

U	V	TT	Rand	Trans	Think	Act	Timebox	Node	Info1	Info2	Info3	Bytes	Tries	Note	Seq
						ParmSet		Version_PubID	1826102						1
Y				01_URL		URL		http://dev...		Hello		12342			2
N				01_URL		URL			http://pe...		Sometimes	12342	3		3
				02_Login		Login	6		me	password1		36823			4
Y			95	Home		Custom		Spec	a21.htm	Cool		56342	X 2	frames	2
			else			Abandon									5
			70	99_Logout	2	Logout			Logout			36213			6

When opened with Notepad or other text processing program, the file looks like this:

U,V,TT,Rand,Trans,Think,Act,Timebox,Node,Info1,Info2,Info3,Bytes,Tries,Seq
Y,,,100,,,ParmSet,Version_PubID,1826102,,,,1
Y,,,100,,,VarSet,OutputResponseFile,Y,,,1
Y,,,100,Login,0,Login,http://127.0.1.1:5001/webapp1/,myself@somewhere.com,Pa$$word12,,,2
Y,,,70,,3,Logout,,,,3

U (Include in Run)

Whenever the Action script encounters any value other than NULL or "Y" in the first column, that row is skipped and processing continues to the next row. This value is limited to 1 character.

This is a great time-saving and error-proofing feature because by changing just one character, you can avoid time-consuming typing mistakes.

V (Vuser ID)

This column specifies Vuser ID used by the LoadRunner Controller to coordinate release of transactions based on lr_rendezvous points during contention testing runs.

TT (Task Type)

This is used to filter execution to requests application only to specific type of tests.
Any character in this column causes the request to be carried out only if it matches what is in the "TaskType" run-time attribute. In other words, a value such as "L" would not be executed unless the TaskType attribute is also set to "L".

A blank in this column specifies that the request should be carried out if TaskType is set to blank in the Info1.

"init" runs only once.

An AttrSet request for TaskType value "EnvID" in Info1 sets the TaskType value at the EnvID attribute value.

This value is limited to 16 characters.

Rand (Random Execution)

The control file contains a "Rand" column to enable the specification of the percentage chance that a particular request will execute.

The "Rand" column specifies the percentage of time that the row is invoked. It can be any number from 0 to 100 (with blanks assumed to be 100). The larger this number is, the more likely its row will be executed. This number is compared against a random number generated at the start of each iteration.
If this is more than or equal to the iteration's internally generated random number, the row is executed.
If this is less than the iteration's internally generated random number, the row is NOT executed.

"else" in the Rand column means that the current request is carried out if the previous transaction was not carried out due to random chance. This essentially provides a type of "if/then" logic.

Trans

The "Trans" column contains the prefix of transaction names which will appear in LoadRunner Analysis. These names do not contain suffixes such as "_txt" which are added automatically depending on what else is specified for the run.
This value is limited to 64 characters.

Think

Think times simulate the length of time a user spends viewing a page or entering data. The amount of time end-users spending "thinking" about what to do next have significant impact on their productivity and their usage of the system. If not enough think time is simulated during load tests, simulated users would navigate the site unrealistically fast, which would yield artificially poor performance results.

To emulate the time users need to look for the data to enter, find the key/mouse location, contemplate, etc., "think time" can be added before a specific request specified in the DriverParms file simply by adding a value in the "think" column of that request.

Unlike the LoadRunner default, this can be a floating point value such as 3.6.

Rather than numbers, the framework has been programmed to recognize several keywords, in both English and Italian tempo names used by musicians. This table is referenced to assign the think time appropriate to each transaction and user group.

Think Time Category	Italian Tempo (bpm)	Description	Casual User Seconds	Expert User Seconds
FASTEST	PRESTO (168-200 bpm)	Clicking a visible menu item.	2-3-4	1-1.5-2
FAST	ALLEGRO (120-168 bpm)	Viewing a page and selecting a drop-down item	5 to 8	2 to 4
MEDIUM	MODERATO (108-120 bpm)	Filling out a data entry screen with 2 to 3 fields.	7 to 12	3 to 6
SLOW	ANDANTE walking pace (76-108 bpm)	Filling out a data entry screen with 4 to 5 fields.	10 to 14	5 to 7
SLOWER	ADAGIO ("at ease") (66-76 bpm)	Filling out a data entry screen with 6 to 9 fields.	14 to 20	7 to 10
SLOWEST	LENTO LARGO (40-60 bpm)	Filling out a data entry screen with 10 to 13 fields.	21 to 25	7 to 10

To add the same amount of think time before every transaction, in Run-Time Settings Additional Attribute add a framework attribute "ThinkSecs" or add a line specifying "AttrSet" in the Action column, "ThinkSecs" in the Node column. The time specified is added to time in Think columns in DriverParms.

To ignore all these settings during a run, specify a value of "N" (any value other than "Y") for attribute "ThinkBefore".

For additional accuracy, overhead "wasted" time that the framework needs to perform internal processing before each transaction (such as reading driver parameters and writing out log header records) are automatically calculated and included as part of think time (if any).

Act

The "Act" column contains the keyword which the script matches with the appropriate function. It is limited to 64 characters. If this field is blank, the value from the previous row is assumed. If no previous value is available, that row is ignored with an error message.

The framework recognizes several act keywords:

Abandon causes the iteration to end, emulating a user who has left the building.

Parmset stores a LoadRunner parameter named in Node with a value in Info1.
This is a tremendously helpful feature because now you can define and instantiate static application-specific parameters very quickly and without cluttering up the Parameter GUI with single-row parameter files.

ParmShow displays the LoadRunner parameter named in Node. The confirmation message:

	INFO: ParmShow of Version_PubID=1826102

AttrSet stores a new value for a pre-defined run-time attribute.

VarSet stores a C variable named in Node with a value in Info1. The app_init function recognizes valid variable names and issues an error if one is not recognized. Unlike parameters, globle variables need to be coded into vuser_init.c.

Braces in the Info1 value "{lrname}" stores that LoadRunner parameter name.

Total with Node value of "start" begins a running total of seconds and bytes returned.
Total with Node value of "end" causes the running totals to be stored in the wiki log.

URL causes a "text/html; file to be retrieved from the address specified in the Node column.
CSS causes a "text/css" file to be retrieved from the address specified in the Node column.
JS causes a "text/javascript" file to be retrieved from the address specified in the Node column.
BMP causes a "image/bmp" file to be retrieved from the address specified in the Node column.
GIF causes a "image/gif" file to be retrieved from the address specified in the Node column.
PNG causes a "image/png" file to be retrieved from the address specified in the Node column.

These are all processed as sub transactions.

Application-specific acts are added as additional LoadRunner Action script files referenced from framework function wi_app_act_modules(). Examples of these include:

• Login
• Logout

Node

"Node" specifies what object or component is affected. If this field is blank, the value from the previous row is assumed. If no previous value is available, that row is ignored with an error message. This value is limited to 128 characters.

TimeBox/Wait

"TimeBox" or "Wait" column specifies the amount of time boxing that the transaction should take.

Info1

This information is specific to each type of request. With Login transactions, this may be a hard-code User ID value. To determine whether this field is blank in the Driver request parameter record, use this coding:

	if( strlen( lr_eval_string("{pDriverInfo1}") ) == ZERO ){
		... it has information ...
	}

This value is limited to 256 characters.

Info2

"Info2" values depend on the request. Usually it contains the title which the framework automatically matches against what is returned between <TITLE> and </TITLE> HTML tags. This value is limited to 256 characters.

Info3

"Info3" values depend on the request. For login requests this column contains the suffix added to subsequent transaction names for that user.

Bytes

"Bytes" specifies the number of bytes expected from a URL request. This field is blank for requests no associated with an interation with a server. This value is typically obtained from a prior run, or from a file directory.

"Y" in the CompareBytes run-time attribute will add comparisons of the bytes returned as a multiple of the expected bytes defined in this field.

The MaxBytesDiff run-time attribute is used as the threshold of bytes difference before issuing an error message.

Tries

"Tries" is the number of times that the transaction is attempted before being declared an error. A blank in this column assumes a single try.

This column is also used to specify repeatition. "x 2" (an X with a space and a number) specifies the number of times that the request is repeated.

Sequence

The last column, "Seq", is a sequence number to uniquely identify each row. This column does not trigger any activity in the script. It's in the last row to make incorrect comma separtors easier to identify. An error message is issued if a "1" is found anywhere else but in the first record. This is done to identify mistakes with specifying the DriverRecs number.

"end" in the Seq column of the last request designates the ending record number, which does not include the header row. This number of records read can be prematurely limited by defining a DriverRecs attribute.

Separating program control data from script code provides you an option to reuse 4) keyword files controlling functional tests

Circular Data Feedback Loop

Having control information in a data file provides tremendous flexibility.

80% of most changes to the script can be accomplished with a change to the driver file.

Changing data in a spreadsheet is
quicker than opening up a program and searching through it.
A data file also enables you to visually spot differences and similaries in the data among various transactions.

The OpenOffice spreadsheet program is not usable for this purpose because it adds double quotes to header records that LoadRunner does not recognize as valid.

File Architecture For Flexibility in Managing Growth in Complexity

Framework script code load Run-time attributes into C global variables defined in the vuser_init.c file or in a .h C header file, which can also contain function definitions.

To determine what to run, framework script functions read Driver fields in the DriverParams.csv file. This file can be viewed and updated using a spreadsheet program such as Microsoft Excel to copy and paste data controlling script execution.

Different behaviours in the script can be specified by different editions of the internal content of files (the DriverParms.csv driver file, .c script files, or the .prm file defining parameters). For example, a different set of files are copied into the script folder for a load test vs. a benchmark test run.

There are several techniques to replace the internal content of files with files of the same name in a different folder.

When VuGen saves a script as a new name, only files which are directly referenced within VuGen script are copied to the new script folder. So it is necessary to have variations of data files stored away from the script folder.

The LoadRunner Controller normally copies to load generators all files in the script folder. But it can be told to send different files to load generators using the "More", "Files" GUI. Such requests can be automated using Shell scripts at the operating system level can
8) use OS copy commands to position the appropriate files, then start LoadRunner Controller automatically to run whatever files are in the folder.

Custom Module Act Distributor

After the framework (within vuser.init.c or wi_lib.c) processes "Act" values it recognizes, function wi_app_act_modules() (at the bottom of file wi_app.c) is called.

wi_app_act_modules() // Called from within wi_lib.c to process custom Act values
{
	int rc=LR_PASS;
	if( strcmp( charDriverAct,"Login") == FOUND ){
		if( strcmp(strProcessWithinInit,"Y") == FOUND ){
			rc=LR_PASS;
		}else{
			rc=wi_app_login();
		}

	}else
	if( strcmp( charDriverAct,"DoSomethingElse") == FOUND ){
		rc=application_specific_module_xyz();

	}else{
		sprintf( my_msg, "Seq=%s Act/Module=%s in not recognized {wi_app_act_modules}"
			,charDriverSeq
			,charDriverAct 
			);
		wi_printMessage("error", my_msg );
		return UNKNOWN;
	}
	return rc; // back to wi_process_driver_tries() in wi_lib.c
}

The framework expects that the "application_specific_module_xyz()" function return a LR_PASS static value if it successfully processed the request or a LR_FAIL if it has not.

This hierarchial architecture of the script framework enables the use of a single script file from unit test through benchmark to load test to scalability tests, with all features built-in.

Several script developers can work at the same time if they each check out from Subversion/CVS the latest version of the script framework which contains coding to recognize Act value "" as a call to application_specific_module_xyz, which can refer to a new LoadRunner application_specific_module_xyz.c file or a function within a script file (or header file) with another name.

The individual developer would then focus only on changing the application_specific_module_xyz.c file, since only this file is checked back into the Subversion/CVS repository for the entire team to use.

Coding Individual Screens

This script sends different requests when the Node is blank (NULL) and when it contains "Spec".
Coding a single function for each request provides flexibility. Act and Node codes enable application functions to be called from the Driver parameter control file, which gives you a way to be selective about what is run simply by changing a value rather than looking for code the code and taking a risk to change it.

At the bottom of the script, if the script cannot recognize the Node provided to it, it returns the "UNKNOWN" static variable back to the framework to handle.

if( strcmp( strGetExtras,"Y" ) == FOUND ){

Avoid coding more than one request per function.

Modular Core Utility Function Libraries

Within the wi_lib.h are definitions of global utility variables used
within the wi_lib.c are utility functions which can be referenced by any framework utility function:

Functionality	In wi_lib.c	Application-specific Functions in wi_app.c
Initialize variables during vuser_init phase	wi_vuser_init_vars() wi_attr_set()	wi_app_init() wi_app_var_set()
Initialize variables during each Action iteration	wi_action_Keyword_loop(), wi_action_init()	wi_app_action_iteration_init()
Close files and cleanup at end of run	wi_vuser_end()	-
Process Driver parameter requests	wi_process_driver_ext_rec() and wi_process_driver_attr_rec() within wi_process_driver_tries() within wi_process_driver_rec() within wi_action_Keyword_loop()	wi_app_act_modules()
control of transactions	wi_stubTrans() wi_assemble_CurrentTrans() wi_startTrans() wi_startTransName() wi_endTrans() wi_startSubTrans() wi_endSubTrans() wi_startTotal() wi_endTotal()	-
printing of messages	wi_startPrinting() wi_stopPrinting() wi_printMessage() wi_print_DriverRec_details()	-
evaluation of responses	wi_startScreen() wi_evalScreen() wi_PrintHttpStats() wi_endScreen() wi_clearScreenInfo() catchResponseToFile() wi_saveImgToFile() wi_replaceCharSeparators()	wi_app_startScreen() wi_app_evalScreen() wi_app_evalScreenTitle()
output of logs	wi_CsvLogOpen() wi_CsvLogClose() wi_WikiLogOpen() wi_WikiLogClose(), wi_CsvLogHeaderWrite() wi_WikiLogHeaderWrite() wi_CsvLogRowWrite() wi_WikiLogRowWrite()	-
Diagnostics	wi_SvcDiag_init() wi_SvcDiag_request()	-
Virtual Table Server	wi_vts_init() wi_vts_request()	-

Note that the names of functions in the script all begin with the name of the action file where they reside.

Functions return values using static variables LR_PASS, LR_FAIL, UNKNOWN.

Catalog of Run-time Attributes

Markers such as {4} have been placed throughout the code to provide a linkage between where global variables are defined in wi_lib.h or vuser_init.c with where they are retrieved, edited, and used in other parts of the script.

Defined in...	Run-time Attribute	Description	Default	Source Marker	Max Chars
wi_app.c	EnvID	A number associated with each particular value of the {pURLHostPath} parameter that includes the protocol (http/https), IP/DNS name, port, and folder of the targeted environment.	1	{21}	4
	ScriptType	"Keyword" for using the DriverParms.csv file, "KeepAlive", "DiagLoo" for only diagnostics.	Keyword	{22}	16
	CleanUpFirst	"Y" to invoke data clean-up during vuser_init phase, if defined.	"N"	{29}	1
	CleanUpAfter	"Y" to invoke data clean-up during vuser_end phase, if defined.	"N"	{30}	1
wi_lib.c	Verbosity	A code which specifies the extent of detail appearing in logs.	4	{4}	1
	TimeBox	"Y" uses the "Tries" column of each transaction control file as the number of seconds that all transaction will take every time (for a controlled processing rate). Vusers are main to wait after each transaction until this time passes.	"N"	{5}	1
	TimeBoxSecs	The number of seconds that all transaction will take every time (for a controlled processing rate). Vusers wait after each transaction until this time passes. Non-negative numbers in the Driver "Timebox" column override this number.	Y	{5}	1
	Randomly	"Y" to use the "Rand" column value to determine whether.	Y	{10}	1
	RandTries	"Y" resets the randomizer for each try repeating a driver row.	"N"	{10}	1
	ProcessPct	"100" to process every row of the control parms, "0" to never process that row.	100	{10}	4
	TaskType	A value such as "LT" would skip all non-blank rows which do not contain that value.	blank	{10}	16
	ThinkBefore	"Y" to add think time before transactions. If a Think value is not specified in the driver row, the ThinkSecs attribute value is used.	"Y"	{10}	1
	ThinkSecs	Number of seconds to pause every transaction before invoking the request. This is overriden by specifications in the "Think" column. This can contain a decimal (such as "3.2").	0	{12}	8
	ThinkTimeRandomRange	The percent that think times defined in the driver Think column can be randomized. "50"(%) means that a 10 second think time can range between 5 seconds and 15 seconds. A range of 100 percent means a range between no seconds and double the think time specified. A zero range (the default) means no randomization and the think time as specified is used.	0	{12}	4
	TransNamePrefix	Text to pre-pend to the transaction name specified in the "Trans" driver parameter column.	blank	{11}	8
	TransNameSuffix	Text to append to the transaction name specified in the "Trans" driver parameter column.	blank	{11}	8
	CompareBytes	"Y" compares the number of bytes returned as a multiple of the bytes expected.	"N"	{13}	8
	MaxBytesDiff	The maximum difference in the number of bytes returned vs. the bytes expected before an error message is issued.	300	{13}	8
	OutputCsvLog	"Y" to write out a csv log for each transaction. Any other value will not.	"N"	{15}	1
	OutputWikiLog	"Y" to write out a txt file containing wiki markup.	"N"	{15}	1
	ShowSubTrans	"N" to not report sub transaction results (only transaction summary and total results).	"Y"	{15}	1
	OutputInfoToWiki	"Y" to write out "info" messages to the Wiki log.	Y	{15}	1
	OutputLogFolder	Blank defaults to C:\\Documents and Settings\\userid\\Local Settings\\Temp	"N"	{15}	1
	OutputHeaders	"Y" to display HTTP Header Response returned from server.	"N"	{16}	1
	OutputBody	"Y" to display HTML Body Response returned from server.	"N"	{16}	1
	ClientTimeout	Rather than "defaults" of 120 seconds. "min" for 10 second minimums during script debugging; "max" for 1000 second maximums during load tests.	min	{17}	8
	BetweenTriesSecs	The number of seconds to wait between retries. Cannot be a negative number. This can be a decimal number (e.g., 3.2) ranther than a whole integer (such as 3).	0	{17}	8
	ClearCookies	"Y" to clear cookies before each iteration, any other value to not do that.	"N"	{18}	1
	SiteScope	"Y" to collect SiteScope statistics.	"N"	{19}	1
	BetweenMonSecs	The number of seconds between SiteScope requests, to avoid overrunning the server under test.	60	{19}	long
	ProbeSecs	The number of seconds between probe requests, to avoid overrunning the server under test.	60	{19}	long
	KeepGoing	"Y" ignores errors and continues processing.	"N"	{19}	1
	ProbeShow	"Y" displays web services Diagnostics statistics in user-defined points on vuser_init, vuser_end, and after each iteration.	"N"	{19}	1
	ProbeName	IP address or computer name.	localhost	{19}	256
	ProbeHost	IP address or computer name.	localhost	{19}	256
	ProbePort	Port used for Dignostics server.	35000	{19}	16
	ProbeUserName	User Name for Diagnostics server.	admin	{19}	32
	ProbePassword	Password for Diagnostics server.	admin	{19}	32
	ProcessWithinInit	"Y" processes DriverParms during initialization (vuser_init). Entries in the "TT" (Task Type) column are used to distinguish whether each request is executed during init or during the regular Action section.	"N"	{23}	1
	ActionLoops	This achieves the same result as setting the Run Logic repetitions for the Action action.	1	{24}	4
	GetURLOnly	"Y" causes LR to make web requests with "MODE=HTTP" (which does not retrieve resources) that works if the script code was recorded in "URL mode" is edited to recognize this. The default, "N", make web requests with "MODE=HTML".	"N"	{25}	1
	GetExtras	"Y" to get EXTRARES extra resources requested from within javascript. "N" saves time during script debugging.	Y	{26}	1
	DriverRecStart	The starting row.	1	{27}	4
	DriverRecs	The number of control parameter file records in the DriverParms.csv file. This is automatically updated to the actual record number when value "end" is encountered in the "Seq" column.	9999	{27}	4
	LineupInit	Activate Rendevous point.	"N"	{31}	1

Parameters

{pURLHostPath}	The front portion of the URL that is common to all requests of the application, the part usually mass replaced with the parameter after a recording. Typically it contains the protocol, Host, and folder, such as: "https://www.yahoo.com/finance/".
{pIteration}
{SSMMHH}	Time stamp.

Time Stamps

This script displays with each message and output log entry a unique identifier to each request:

060211015432S0GNoneU-1I1.1-1R3

This identifier includes a timestamp. The first part of this is used to name log files:

• "06" is the 2-digit year.
• "02" is the 2-digit month.
• "11" is the 2-digit day.
• "01" is the 2-digit hour.
• "54" is the 2-digit minute.
• "32" is the 2-digit second.
• "S0" is the Scenario.
• "GNone" is the Group (GNone when run in VuGen).
• "U-1" is the Vuser number (-1 when run in VuGen).

The second part of this is used to identify processing sequence:

• "I1" is the Iteration.
• "A1" is the ActionLoop count.
• "D1" is the DriverRecs loop count.
• "T3" is the Try loop count.
• "R12" is the Request count.

Trouble Shooting

After you install this framework, run it before making any changes. If you get weird compile messages, you may not have the right version of LoadRunner installed.

This script does not work with LoadRunner 7.8 and earlier versions.

This framework was written by creating a new multi-protocol script of protocol "Web (Click and Script)" using LoadRunner 8.1 FP4. Script created through VuGen recording should have a comment of "Recorder Version : 1289".
The Generation Log should begin with "Web Recorder version : 8.1.0.1451".
The Replay log should contain "Web Turbo Replay of LoadRunner 8.1.0 for WINXP; WebReplay81 build 5495 [MsgId: MMSG-27143]"

Compatibility with Functional Keyword Driver Data

At of this moment, control files in this script is not recognize the control file format of any public keyword engine such as STAF, as described in my page keyword driven (single-user) functional test specification files

STAF has several levels of control files. However, LoadRunner currently only handles a single level of control files.

This can be remedied with a library of functions which retrieves more files and iterates through them.

To fully accept even the first level of driver files, this script will need to be updated to recognize the dozens of commands that STAF recognizes.

Not all commands for functional testing are relevant to load testing.