Test Automation Frameworks

This page compares and describes "data driven" and "keyword" driven automated tests using open source libraries and WinRunner from Mercury Interactive — the hardest won (but easily forgotten) insights not obvious from available documentation.

“Hey, we've got a problem here.”

Topics this page:

Site Map  
About this site  

Why Frameworks for Test Automation?

First, we need to clarify some terms that are used by different authors on this subject.

Starting from the bottom of the graphic, the application under test (AUT) is controlled by a test tool such as WinRunner, which references GUI maps to uniquely recognize objects it wants to control.

Traditionally, WinRunner scripts are usually written as a hierarchy of script functions.

But as the number of scripts increase, individuals may develop a driver which uses data (information in spreadsheets or flat files) to vary test script execution.

This reduces the need to interact with "messy" script code, but does not minimize script coding because the logic to read the data and then perform the desired actions still needs to be coded and debugged.

Organizations wishing for greater productivity can make use of keyword interpreter libraries (such as WRSAFS) which have pre-defined functions to read "high-level" keyword (commands) specified to drive the AUT.

Organizations wishing to manage assets into and out of many tests or need to integrate events, messages, and logs together from various sources (JUnit, grep, WinRunner, etc.) use a package such as open source STAF to provide an information broker infrastructure.

Record/Playback Test Tools

Test tools record user actions as scripts and plays back those scripts to emulate user actions.

Examples of test tool packages for functional testing include:
• Mercury's WinRunner for Web, .NET, Java, PowerBuilder, etc.
• Mercury's Quick Test Pro
• IBM Rational Robot Classic, RobotJ, XDE, etc.
• QARun from Compuware
• Empirix (formerly RSW) e-TEST web testing suite
• JUnit for Java, Nunit for .NET, etc.
• Test Complete
• WorkSoft
• WebFT
• Abbot

Test Tool Driver Frameworks

Test Driver frameworks "drive" the replay of scripts. For example, the SAFS framework uses keywords in plain text files rather than a matrix of function calls.

This makes it faster and easier to use test tools because they:

1. Provide common functions for error detection, logging, etc. not provided by test tools. Because the framework's functions automatically performs validations on every object within every action without additional coding, many hours of script coding and debugging is avoided. For example, the WRSAFS framework includes the CSO libraries of functions for WinRunner users

2. Provide complete examples. For example, WRSAFS installation enables a complete test of the "flight" application that is WinRunner's sample application.

3. Define de-facto naming conventions for common environment variables, functions, messages, and other terminology.

4. Make test specs independent of the test tool. Another tool can potentially be substituted for WinRunner because rather than making calls to the unique syntax of each tool's API, the framework processes generic keywords in plain text files. Otherwise, test scripts crafted for one test tool are usually useless for use in competitive tools.

5. Allow testing work to begin sooner. Unlike capture/playback, which requires the application to be operational, the framework's three-layer test specification hides low-level details. During application design, when use cases are known, Test Analysts can define "what" to test (the "high-level" test cycles) without knowing the scripting language. When specific application variable names are known, analysts can define test suites with specific values. "How" the application's gui map and scripts implement the action keywords is done as the application matures.

6. Enables testing work to progress in a structured, top-down, distributed approach by several specialists working separately.

   Unlike tracing function calls within many pages of test script code, keyword test specs don't have extraneous programming characters, so they are easier to follow what is being tested. This makes it easier to have separate (several) script programmers working on different parts of the app, making the test project easier to scale larger.

   This makes for a much more manageable process to plan and track.

7. Use of public domain frameworks distributes the work. For example, here are the recent changes to the framework source:
   • ANSI wide character support

References:

Keyword Driven Testing.ppt by John Crunk
"Choosing a Test Automation Framework" by Mike Kelly compares five different types of frameworks.
Getting started with Rational Robot automation framework support (RRAFS) by Jirong Hu at the Development Bank of Singapore on 15 Jun 2004

I am aware of these execution engines available:

1. SAFS (Software Automation Framework Support) project safsdev on SourceForge.net was initially developed by Carl Nagle for SAS Institute to leverage:
   RRAFS (Rational Robot Automation Framework) for IBM Rational Robot [FAQ] This is why SAFS is written for Visual Basic (although there are plans for a Multi-Platform test generator).

   WRAF (WinRunner Automation Framework) (also named WRSAFS in installed folders) for WinRunner was added by John Crunk and others since 2002.
   An engine for Abbot (ab-bot) is under development.

   These share the use of

   • SAFS Common (which is unziped to C:\SAFS, containing a .vbs file that performs installation using Java 1.4+ SDK). and
   • STAF
   • the Java 1.4.2_09 SDK, which must be installed prior to any of the above.

   Carl has written tutorials on this.

2. Mercury's "Business Process" Functional Testing product integrated with Test Director.

3. EMOS Framework by Carsten Büche based on work by Dean Rajovic.

4. Keith Zambelich's Totally Data-Driven Automated Testing

5. My own library, which I no longer maintain in favor of WRSAFS. BTW, I used the principles of this page to create a test script framework for LoadRunner.

Infrastructures

An infrastructure such as STAF provides services that standardizes across many applications and utilities:
• Event Scheduling (Cron EventManger)
• Process Invocation
• Dynamic data and configuration Management (Var)
• Queue and Event Messages and Notifications (Email)
• Synchronization of work (Sem)
• Resource Pool (ResPool) Management
• Remote run status monitoring
• Combination of Logs from multiple sources (JUnit logs with grep output with WinRunner reports)
• (Zip) File Packaging and Unpackaging
• File Transfer (FS)

Driver Keyword Test Data Specifications

From SAFS record formats

SAFS reads "test tables" with tabs or commas delimiting fields in fixed positions, such as this file named TestInvalidLogin:

;    , Window, Component,  Action Keyword, Argument values:
T    , Login ,  UserID  ,  SetTextValue  , ^userid
T    , Login , Password ,  SetTextValue  , ^password
T    , Login ,  Submit  ,  Click
T    , Error , Message  , VerifyProperty , ^prop="Text", ^val="Invalid!"
T    , Error ,   OK     ,  Click

These lines are contained in what SAFS calls a Step table because it specifies individual "low-level" step-by-step actions.

SAFS recognizes hat characters (^) to specify variables that are substituted with values in Suite tables such as this "intermediate" file named "VerifyLoginFunctions":

T , TestInvalidLogin    , , ^userid=goodID , ^password=badPassword
T , TestValidLogin      , , ^userid=goodID , ^password=goodPassword
T , TestCancelledLogins , , ^userid=goodID , ^password=goodPassword

Such files are specified by Cycle tables listing test suites:

T , VerifyLoginFunctions, ,
T , VerifyAdminFunctions, ,

Design these "high level" Cycle table entries so that each can run independently of each other in any sequence. That way, they can be run randomly, which can identify additional defects in design or implementation.

SAFS treats the entire line as a comment if it sees at the front of the line a semicolon (; code 59), an apostrophe (' code 39), a pound sign (#), or no characters at all (blank/NULL).

Other Record Types:
C = Driver Command
T = Test Step
B = Block ID
BP = Break Point
S = Skip Test

In Step tables, "Login" and "Error" in the second column refer to screen names which are defined in WinRunner GUI maps. Other testing tools have their own Application mapping schemes

Keywords such as "SetTextValue" in the example SAFS describes in this page with HTML frames and also this "Old-Style" outline format are processed by SAFS Component functions.

For complete details, reference:
Hierarchal Keyword Test Automation.doc
Test Table Naming Conventions.doc

Summary

The architecture of Carl Nagle's data keyword "Core Data Driven Engine" framework is modular and flexible because it uses a hierarchy of 3 levels of drivers: Cycles, Suites, and Steps.

Level	Description	folder	File Suffix
High	Each run starts by opening a Cycle table containing a list of suites to execute.	Cycle	.CDD
Intermediate	Each suite name requested is associated with a Suite table containing values passed to invoke various step tables.	Suite	.STD
Low	Each step table contains a list of actions executed against specific component objects within specified windows.	Step	.SDD




Action keywords such as "SetTextValue" within the StepX.SDD file are associated with functions (generic WinRunner scripts) pre-defined by the WRSAFS framework, with arguments (such as ^user.name) treated by the framework as inputs to framework functions.

Framework Internal Flow Diagram

1. From the Operating System (such as a Run window or a WSH script)
2. a batch file or link to WinRunner can be invoked
3. to invoke WinRunner.
4. The WRUN.ini file may be
5. specified in the batch file
6. or automatically loaded by WinRunner.

7. The add-in during WinRunner initiation specifies
8. run of script wrsfasinit which
9. defines functions from the Mercury and others in the lib\WRSAFS folder.

10. The default startup script is
11. "StartCycleTest" in the C:\WRSAFS folder.
12. This script obtains Windows Registry
13. keys in HKLM Software WRAFS
14. using function "get_registry_value" defined in the CSO library.

15. StartCycleTest invokes the STAFProc.exe application if STAF is enabled, then
16. call the CDCycleDriver function to commence a data-driven test
17. using WinRunner scripts in the C:\WRSAFS folder.

18. The Cycle Driver reads CCD cycle files which begins with driver command "SetApplicationMap" that
19. loads
20. from a project folder (named Datapool by default) that houses
21. gui files initially
22. obtained from WinRunner recordings of
23. manual actions on the
24. application under test (AUT).

25. Objects defined within gui files are referenced by WinRunner. These files are loaded by every driver file.
26. The framework also requires its own application map containing
27. objects referenced within driver specification files cycle,
28. Suite and
29. Step files as well as STAF services. Note: Changes in the Application Map file will require restarting of STAF Services for changes to take effect.

30. These files can be edited by
31. Wordpad
32. Excel spreadsheet program or the
33. SAFS Test Generator utility that saves Excel as tab/comma delimited files

34. The framework's ProcessMapFile script converts WinRunner GUI maps to Framework maps.
35. The driver controls the App
36. and creates text log files (from functions such as SetTextLogEnabled)
37. and well as traditional WinRunner logs (using functions such as report_msg).

Click here to pop this graphic up in a new window.

Reference: CHAPTER 2.doc by John Crunk

SAFS & STAF2 Installation

The most recent version of SAFS Framework I know is 2005.05.12.

From Start > open a Run window:

cd \
cscript SetupSAFS.vbs

Wait for the pop-ups, including the "Installation Complete" at the end.

The SAFSInstall.zip file is used during this install process.

You are given a choice to install STAF. Say "YES" even though an older version 2.5.1 is installed. (it's not compatible with STAF v3.0.3)

After each reboot, your machine will automatically start a DOS window that runs a 3,664K STAFProc.exe which has (for v3.1.0) Windows Task Manager "Mem Usage" of 5,744 K. Don't delete that window or SAFS will not be able to use STAF to read application maps. SAFS uses the opensource STAF project as its multi-platform, cross-process, distributed testing link by implementing a STAF VAR service wrapper called the SAFSVariableService to handle SAFS variables in a STAF environment.

If the "C:\STAF" contains a log.txt file and no useful files if there has been an installation problem (like not being able to find the Java SDK folder).

The SFAS/STAF installer:

Var	Value
CLASSPATH	;C:\STAF\bin\JSTAF.jar; C:\SAFS\lib\jakarta-regexp-1.3.jar; C:\SAFS\lib\safscust.jar; C:\SAFS\lib\safsrational.jar; C:\SAFS\lib\safs.jar; C:\SAFS\lib\safsjrex.jar; C:\SAFS\lib\JRex.jar
CLASSPATH_SAFSBAK	same as above
SAFSDIR	C:\SAFS
STAFDIR	C:\STAF

• adds these Environment variables:

• Adds file DDVariableStore.dll (151,552 byte dated 10/30/2002) to
  • folder %M_ROOT%\arch and
  • folder %SystemRoot%\System32

• Adds within "Control Panel > Add or Remove Programs" an entry for "Software Test Automation Framework".

• Adds within the Windows Registry these entries:
  • HKEY_CLASSES_ROOT
    SAFSSessionResolver.SessionResolver
    SAFSSessionResolver.SessionResolver.1
    Typelib

WRSAFS (WinRunner Software Automation Framework Support) Installation

Documentation is in a prior (2004.8.31) package of WRAFS not included in the current installer package: CHAPTER 1.doc Ok - FAQs.doc PREREQUISITE.doc Work Environment & Demo.doc WRSAFS Install.doc Researching Failures.doc Please search and read through the mail archive before posting new messages to it through emailing to safsdev-wrafs@lists.sourceforge.net

Download and install SAFS (described in the prior section above).

Download the most current (2005.7.11) version of WRAF (WinRunner Automation Framework) (also named WRSAFS in installed folders) for WinRunner was added by John Crunk and others since 2002.

Run the executable (.exe) InstallShield installer. It does the following:

• Puts a "Winrunner SAFS" entry within "Control Panel > Add or Remove Programs".

• Puts into the Windows Registry entries in HKEY_USERS\ ... \ Software \ WRAFS.

• Puts into folder c:\WRSAFS several script folders to define framework functions within WinRunner

  %M_ROOT%\lib\wrafsinit - reloads lib\win32api and 11 libraries in C:\WRSAFS within %M_ROOT%\lib (marked "L" below) that were installed by wrsafs.vbs, listed below in call-chain order:

  • cominit reloads script comlib to run com_initialize()
    • comlib defines ActiveX functions load_com, com_set_prop, com_set_prop_type, com_get_prop_type, com_activate, unload_com, attach_com
  • cso2_init * reloads script cso2load and calls initcso2() which defines ...
    • cso2load functions
  • cso3_init * reloads script
    • cso3load and calls initcso3 which defines cso3load functions
  • cso_init * uses the GUI_get_name() function to determine whether to load for "windows"
    • win32api and
    • cso32lib - loads lib\arch\ cso32lib and lv_fin.dll
    else
    • win_api and
    • cso16lib - loads lib\arch\cso16lib.dll used by "cso functions 3"
  • csoxinit * which loads
    • csoxload - generates functions SMTP_email_x, SMTP_email_file_x, get_last_error, dos_system_x, atoi_x, itoa_x
  • ext_func - loads lib\arch\ext_func.dll and declares extern names to reference it.
  • func_gen * loads from lib\\WRSAFS compiled modules (formerly residing under the samples/utils folder)
    • L - meter *
    • L - misc *
    • L - misc_cs *
    • L - sysinfo *

To avoid the message shown at the right, in Windows Explorer, right click on this folder, select "Properties", and un-select "read-only" to each of these folders:

c:\Datapool
C:\WRSAFS
C:\SAFS
C:\STAF
c:\Program Files\Mercury Interactive\WinRunner\lib\WRSAFS
c:\Program Files\Mercury Interactive\WinRunner\lib (where the wrafsinit script is installed)

Here's the code to do it in batch mode after changing the directory to each folder above:

attrib -R *.* /s

Invoke WinRunner with Add-ins listed:
• Select add-ins ActiveX and Visual Basic. If you don't, calls to functions such as "ActiveX_get_info" will not be recognized and the execution will fail.

• With WinRunner 7.6, you need to request from Mercury a license key to turn on the addin.
  While you're waiting, open wrsafsinit within the %M_ROOT%\lib folder and run it.

• With a previous version, I needed to replace all "WRAFS" to "WRSAFS" in lib\wrsafinit and the C:\WRSAFS library listed above.

• In Tools > General Options > Folders, click the square "+" to add the search paths listed above.

  When a script calls for another function, WinRunner searches for scripts top down in the sequence order which the multiple paths appear above. Search paths allow you to avoid errors such as:

  

• In Tools > General Options > make changes to values specified in the WinRunner Configuration values page.
  Run->Settings:
  "Timeout for checkpoints and CS..." from 10,000 to 5000
  "Fail test when single property check fails" UNCHECK
  "Break when verification fails" UNCHECK
  Run->Synchronization
  "Delay for window synchronization" from 1000 to 0
  "Timeout for synchronization message" from 2000 to 0
  "Drop synch timeout if failed" UNCHECK
  Add-ins:
  These add-ins should be loaded in WinRunner: ActiveX, Visual Basic, WebTest

• Open the C:\WRSAFS\StartCycleTest script and run it.

  This references the C:\Datapool folder containing a sample set of driver specification files to test Mercury's flight sample application:

  • Bench
  • Cycle
  • Dif
  • Logs
  • Step
  • Suite
  • Taborder
  • Test
  • flightvb.gui
  • flightvb.map

  By default, the names of these folders are defined by the WRSAFS installer in these keys under Windows Registry path
  HKEY_LOCAL_MACHINE/Software/WRAFS

  Key	Default Value	Notes
  EnginePath	C:\WRSAFS
  CyclePath	Cycle	Sub folder name under EnginePath
  StepPath	Step	Sub folder name under EnginePath
  SuitePath	Suite	Sub folder name under EnginePath

APPENDIX B: FURTHER REFERENCES

Adding a Driver Spec to the Sample Application

After the demo test completes, the flight app remains at the conclusion of a driver test run using C:\Datapool. So...

Here's an exercise for you. Please add script code to close the application.

The solution is below. But don't look at this until you have tried it yourself!

1. In the Cycle folder, Flight.CDD file, add a line to "CloseApp".

2. In the Suite folder, create a new file named "CloseApp.STD" (by copying and pasting from "LaunchApp.STD")

3. Edit the new "CloseApp.STD" file and add a line:
   T CloseFlight

4. In the Step folder, create a new file named "CloseFlight.SDD" (by copying and pasting from "FindFlight.STD")

5. Edit the new "CloseFlight.SDD" file and add a line:
   T FlightReservation FlightReservation SelectMenuItem "File;Exit"

6. These are determined by seeing what script is recorded by WinRunner as the app is exercised manually.
   win_activate ("Flight Reservation");
   set_window ("Flight Reservation", 4);
   menu_select_item ("File;Exit");

7. To see if the framework can catch errors, change invocation of flighta.exe to flightb.exe.

WRSAFS Custom Configuration

To change WRASAFS samples for your own application:
1. Use WinRunner to record a gui map:
   1. Select Tools|GUI Map Editor in Winrunner
   2. Open the Application to be tested
   3. Click on Learn in the Map Editor
   4. Select the Window of the Application
   5. Answer Yes to the prompt
   6. Navigate to the next screen to learn
   7. Go to Step 3 until all application has been learned
   8. Save the temporary GUI Map File with a permanent name in the project folder.

2. Create an Application Map
   Application Map Files.doc by John Crunk

3. To change the location of your project (from the default "C:\Datapool"), open for edit file STAFWRAFS.INI and WRAFS.INI
   PROJECTDIR = C:\MyDataPool2

   Note: This is done instead of one author's suggestion of editing the C:\WRSAFS\StartCycleTest script, which does not work because called functions reset the value of changes to variable definitions.

4. Instead of coding WinRunner functions, create step files using SAFS generic keywords (Note: There are missing files in the reference installed by my version of SAFS to C:\SAFS\doc\.

SAFS Test Generator

WRAFS accepts tab separated files as input, but can be made to accept files in CSV format by ???

APPENDIX A: USING EXCEL HELPER to convert data in Excel spreadsheets to tab/comma delimited files.

SAFS also provides a utility to manage test driver files: SAFS Test Generator

The MSI installs to C:\Program Files\SAFS TestGenerator\ with several spreadsheets:

• StarterProject.xls - AUT Project Workbook
• StarterOutput_Low.xls - SAFS Engine Workbook
• StarterOutput_High.xls - Sample Ouput Table for Cycles and Suites
• StarterAppMap.xls - Sample Ouput Table for Steps
• DDEngine.xls - Sample Excel AppMap in 3-column format

The installer inserts registry entries in HKEY_USERS\ ... \ Software \ SAS Institute \ Testware \ VBTestGenerator and HKEY_CLASSES_ROOT \ ... \

When VBTestGenerator.exe is invoked, it opens Excel because it needs Excel.

Keywords and Functions

• Driver Framework
• Utility
• Driver Customization
• Window
• GUI Component Test

Here are links directly into Carl Nagle's SAFS Keywords Reference, which identifies the format of each function. (Only keywords relevant to WinRunner are shown, not keywords for RobotJ, etc.). This enables us to search quickly.

Note: ".SBH" files are header names carried over from the predecessor IBM Rational SQARobot product. They are included in functions making use of that library.

Driver Framework Keywords & Functions

Function	Keywords for WinRunner	Note
StartCycleTest	Loads .dlls. This is the top script to invoke, which invokes others listed below.
CycleDriver CycleDriver_X.SBH		[Commonly Used]
CycleDriverSTACK & CycleDriverSTACK_X.SBH	[page currently unavailable]	[Rarely Used]
StepDriver StepDriver_X.SBH		[Commonly Used]
StepDriverSTACK & StepDriverSTACK_X.SBH	[page currently unavailable]	[Rarely Used]
SuiteDriver SuiteDriver_X.SBH		[Commonly Used]
SuiteDriverSTACK & SuiteDriverSTACK_X.SBH	[page currently unavailable]	[Rarely Used]
DDDriverCommands
DDDriverCounterCommands	DeleteCounter, LogCounterInfo, ResetCounter, SetCounterMode, StoreCounterInfo, StartCounter, StopCounter, StartRequirement, StopRequirement, StartCycle, StopCycle, StartSuite, StopSuite, StartProcedure, StopProcedure, StartTestcase, StopTestcase, SuspendStatusCounts, ResumeStatusCounts
DDDriverDatabaseCommands DDDriverDebugCommands
DDDriverDeprecatedCommands
DDDriverFileCommands	[not marked for WR]
DDDriverFlowCommands
DDDriverLogCommands	LogFailureOK, LogMessage, LogTestFailure, LogTestSuccess, LogTestWarning, LogWarningOK
DDDriverStringCommands

Utility Functions

WRSAFS Script	WRSAFS Functions	Note
ApplicationUtilities ApplicationUtilities_X.SBH		[Commonly Used]
BufferUtilities_X.SBH		[Rarely Used] [only .SBH?]
DDEngine.SBH		[WR script not in reference]
DDGUIUtilities DDGUIUtilities_X.SBH		[Commonly Used]
DDUtilities DDUtilities_X.SBH		[Commonly Used]
DDVariableStore DDVariableStore_X.SBH		[Commonly Used]
ExcelUtilities		[Commonly Used] [not a WR script file]
FileUtilities & FileUtilities_X.SBH		[Rarely Used]
FileTestUtilities FileInfoFIFO		[Rarely Used] [Not a WR script file]
LogUtilities LogUtilities_X.SBH		[Commonly Used]
MenuOps		[WR script not in reference]
MenuUtilities MenuUtilities_X.SBH	MUGetMenuItemStrings, MUGetMenuItemIndexID, MUGetMenuItemTextIndex, MUGetMenuItemIndexText, MUGetMenuItemIndexState, MUGetMenuItemTextState MUGetMatchedMenuItemTextState, MUGetMatchedMenuItemTextIndex, MUGetMenuItemTextSubMenuHandle, MUGetMenuItemIndexSubMenuHandle, IsMenuItemEnabled, IsMenuItemGrayed, IsMenuItemChecked, IsMenuItemHiLited, IsMenuItemABitmap, IsMenuItemDefault, IsMenuItemAMenu, IsMenuItemAMenuBarBreak, IsMenuItemAMenuBreak, IsMenuItemAMenuSeparator, MUGetMenuItemStateString MUVerifyTestStateString, GetHierarchyInfo, MUGetMenuInfoFromHierarchy, MUGetMenuHandleInfoFromHierarchy, MUGetMenuHandleInfoFromMenuID,MUGetMenuInfoFromMenuID, ProcessMenuForID, MUOutputMenuHandleStructure, MUOutputMenuStructure, MUGetActivePopupMenu	[Commonly Used]
ProcessContainer	[not a WR script file - for Robot only]	[Commonly Used]
NTCommandUtilities & NTCommandUtilities_X.SBH	WaitForNTCommandCompletion , RunWshShellProgram , ExecWshShellProgram , Run_NTCMPTRE_EXE , Run_SED_EXE , Run_GREP_EXE , Run_NTFileComp_EXE , Run_GNUDiff_EXE , NTFileCompResult , GNUDiffCompResult , Run_RRAFSTextDiff_EXE , Run_RRAFSBinaryDiff_EXE	[Lesser Used]
StackDriverInfo	Declares arrarys StepDriverState,SuiteDriverState, CycleDriverState	[WR script not in reference]
SAFSUtilities & SAFSUtilities_X.SBH	SAFSisToolAvailable, SAFSWaitForTool, SAFSAutoLaunchSAFSTools, SAFSisSAFSAvailable, SAFSisSAFSMAPSAvailable, SAFSisSAFSVARSAvailable, SAFSsetSAFSVariable, SAFSsubmitServiceRequest, SAFSisSAFSLOGSAvailable, SAFSisDriverCommandsAvailable, SAFSLaunchDriverCommands, SAFSAutoLaunchDriverCommands makeQuotedString, makeQuotedPath, SAFSWaitForDriverCommands, SAFSShutdownDriverCommands, SAFSShutdownHookProcess, SAFSisSTAFAvailable, SAFSWaitForSTAF, STAFLaunchSAFS, STAFLaunchSAFSMAPS, STAFLaunchSAFSLOGS, STAFLaunchSAFSVARS, SAFSSTAFpulseEvent, SAFSShutdownSTAF	[WR script not in reference]
STAFUtilities STAFUtilities_X.SBH	STAFInitLogFacility, STAFLogMessage, STAFRegisterWR, lentagSTAFValue, STAFSubmit, STAFSubmit2, STAFFree, STAFCloseLogFacility, STAFUnRegister, IsToolAvailable, ProcessStringCommand, STAFIsSAFSLOGSAvailable, STAFIsSAFSVARSAvailable, STAFIsSAFSMAPSAvailable, STAFGetSAFSVariable, STAFSetSAFSVariable, STAFGetGUIID, STAFAddHeaderFooter, STAFLaunchSAFSLOGS, STAFLaunchSAFSVARS, STAFLaunchSAFSMAPS, STAFOpenAppMap, CurrentAppMapPath, STAFCloseAppMap, STAFLaunchSAFS, STAFVariableCount, STAFClearSAFSVariable, STAFAppMap, SAFSDriverCommands, STAFFileCopy, processToolRequest, getQueueMessage, sendQueueMessage, STAFresetEvent, STAFresetClassicEvents, STAFpulseEvent, STAFpostEvent, STAFwaitEvent, STAFwaitMutex, STAFreleaseMutex, STAFreleaseMutex, callSAFSHook, testvarsub, testvars, prepTestRecordData	[Commonly Used]
StringUtilities	LTrimWhiteSpace , RTrimWhiteSpace , TrimWhiteSpace , LastInStr , GetTrimmedField , GetFixedWidthField , GetTrimmedFixedWidthField , FindAndReplace , GetNextNonBlankFieldIndex , GetNextFieldIndex , GetNextDelimiterIndex , GetFieldIndex , GetFieldCount , GetFieldArray , GetLiteralQuotedField , GetLiteralQuotedRecordField , GetNonBlankField , GetQuotedString , GetTrimmedQuotedString , GetQuotedField , GetTrimmedQuotedField , GetMultiDelimitedFieldCount , GetMultiDelimitedField , ProcessExpression , ProcessFieldExpressions , ConvertHTMLEntities , InitializeRegExp , ConvertHex2Long , RegExpExecute , RegExpTest , RegExpReplace	[Commonly Used]
WIN32.SBH & WIN32_X.SBH	Export header file for advapi32.dll, kernele32.dll, user32.dll, comdlg32.dll, shell32.dll, mpr.dll, winmm.dll, gdi.dll, Shlwapi.dll, occache.dll	[Lesser Used]
WinRegistryUtilities	Misc routines using the WIN32 API -- ConvertTwipsToPixels.	[Lesser Used]
Win32Utilities	[not a WR script file]	[Lesser Used]
ProcessMapFile	Converts WinRunner GUI maps to Framework maps.
Publish	[not a WR script file] Parse SQARobot source and header files that have been properly formatted for publication.	[Commonly Used]
CWPHooker DatabaseUtilities ImageUtilities XMLUtilities	[not a WR script file]	[Lesser Used]

Driver Customization Keywords

Function	Keywords for WinRunner	Note
Custom Driver Commands
Custom Log Utilities
Custom Record Types
CustomTestCommands CustomUtilities CustomStatusUtilities	[not script file] Robot Custom Extensions

Window Keywords & Functions

WRSAFS Function	Keywords for WinRunner	WinRunner Function
Window	Click, CloseWindow, CompareStoredProperties, DoubleClick, GetWinInfo, InputCharacters, InputKeys, Maximize, Minimize, Restore, RightClick, SelectMenuID, SelectMenuItem, SelectMenuItemContains, SetPosition, VerifyImage, VerifyMainMenu, VerifyMenuID, VerifyMenuItem, VerifyMenuItemContains, VerifyOCRValue	desktop_capture_bitmap - Captures a bitmap of the entire desktop or of the specified area of the desktop. set_window - Specifies the window to receive input, according to the window's logical name. set_window - Specifies a window to receive input, according to the window's physical description. win_activate - Activates a window. win_capture_bitmap_- Captures a window or a specified area of a window as a bitmap. win_check_bitmap - Compares a current window bitmap to an expected bitmap. win_check_gui - Compares current GUI data to expected GUI data. win_check_info - Checks the requested window property. win_check_text_- Checks the text of a window or area of a window compared to the specified expected text. win_click_help - Clicks the help button in a window title bar. win_click_on_text - Clicks on text in a window. win_close - Closes a window. win_drag - Drags an object from a source window. win_drop - Drops an object on a target window. win_exists - Checks whether a window is displayed. win_find_text - Returns the location of a string in a window. win_get_desc - Returns the physical description of a window. win_get_info - Returns the value of a window property. win_get_text - Reads text from a window. win_highlight - Highlights a window. win_max - Maximizes a window. win_min - Minimizes a window to an icon. win_mouse_click - Clicks in a window. win_mouse_dbl_click - Double-clicks in a window. win_mouse_drag - Drags the mouse in a window. win_mouse_move - Moves the mouse in a window. win_move - Moves a window to a new absolute location. win_move_locator_text - Moves the mouse to a string in a window. win_open - Opens a window. win_resize - Resizes a window. win_restore - Restores a window from a minimized or maximized state to its previous size. win_type - Sends keyboard input to a window. win_wait_bitmap - Waits for a window bitmap. win_wait_info - Waits for the value of a window property.

GUI Component Keywords & Functions

WRSAFS Function	Keywords for WinRunner	WinRunner Function
CheckBox	Check, Click, UnCheck
ComboBox	Click, Select, SelectIndex, SelectPartialMatch, SelectUnverified, SetTextValue
Database	[not a WR script file]
EditBox	SetTextCharacters, SetTextValue, SetUnverifiedTextCharacters, SetUnverifiedTextValue	edit_activate - Double-clicks an object in an Oracle, Java, or Oracle Developer 2000 application. edit_check_info - Checks the value of an edit object property. edit_check_selection - Checks that a string is selected. edit_check_text - Checks the contents of an edit object. edit_delete - Deletes the contents of an edit object. edit_delete_block - Deletes a text block from an edit object. edit_get_block - Returns a block of text from an edit object. edit_get_info - Returns the value of an edit object property. edit_get_row_length - Returns the length of a row in an edit object. edit_get_rows_count - Returns the number of rows written in an edit object. edit_get_selection - Returns the selected string in an edit object. edit_get_selection_pos - Returns the position at which the selected block starts and ends. edit_get_text - Returns the text in an edit object. edit_insert - Inserts text in an edit object. edit_insert_block - Inserts text in a multi-line edit object. edit_replace - Replaces part of the contents of an edit object. edit_replace_block - Replaces a block of text in a multi-line edit object. edit_set - Replaces the entire contents of an edit object. edit_set_focus- Focuses on an object in an Oracle, Java, or Oracle Developer 2000 application. edit_set_insert_pos - Places the cursor at the specified point in an edit object. edit_set_selection - Selects text in an edit object. edit_type - Types a string in an edit object. edit_wait_info - Waits for the value of an edit object property.
ComboEditBox	Click, Select, SelectIndex, SelectPartialMatch, SetTextValue
GenericFunctions	cvt, GetArgList(table), AddToGUI(), GetObjectType, Space(in count, in thestring), UBound(Array1[])
GenericMasterFunctions with _X.SBH	AssignPropertyVariable, GUIDoesExist, GUIDoesNotExist, InputCharacters, InputKeys, VerifyArrayPropertyToFile, VerifyClipboardToFile, VerifyClipboardVP, VerifyProperty, VerifyPropertyContains, VerifyPropertyToFile, VerifyTabOrder, VerifyValueContains, VerifyValues, VerifyValuesIgnoreCase
GenericObject	Click, CompareStoredProperties, CtrlClick, DoubleClick, JavaMenuSelect, LeftDrag, RightClick, RightDrag, ShiftClick	obj_check_bitmap - Compares a current object bitmap to an expected bitmap. obj_check_gui - Compares current GUI data to expected GUI data. obj_check_info - Checks the value of an object property. obj_check_text_- Checks the text of an object or area compared to the specified expected text. obj_click_on_text - Clicks on text in an object. obj_drag - Begins dragging an object. obj_drop - Ends dragging an object. obj_exists - Checks if an object is displayed. obj_find_text - Returns the location of a string within an object. obj_get_desc - Returns an object's physical description. obj_get_info - Returns the value of an object property. obj_get_text - Reads text from an object. obj_highlight - Highlights an object. obj_mouse_click - Clicks on an object. obj_mouse_dbl_click - Double-clicks on an object. obj_mouse_drag - Drags the mouse within an object. obj_mouse_move - Moves the mouse within an object. obj_move_locator_text - Moves the mouse to a string in an object. obj_type - Sends keyboard input to an object. obj_wait_bitmap - Waits for an object bitmap. obj_wait_info - Waits for the value of an object property.
GraphicControl	Click, ClickButton	button_check_info - Checks the value of a button property. button_check_state - Checks the state of a radio or check button. button_get_info - Returns the value of a button property. button_get_state - Returns the state of a radio or check button. button_press - Clicks a push button. button_set - Sets the state of a radio or check button. button_wait_info - Waits for the value of a button property.
PushButton	Click, ClickButton
RadioButton	Click
HTMLImage	Click, CompareStoredData, CompareStoredProperties
ScrollBar	OneDown, OneLeft, OneRight, OneUp, PageDown, PageLeft, PageRight, PageUp	scroll_check_info - Checks the value of a scroll property. scroll_check_pos - Checks the current position of a scroll. scroll_drag - Drags a scroll to the specified location. scroll_drag_from_min - Scrolls the specified distance from the minimum position. scroll_get_info - Returns the value of a scroll property. scroll_get_max - Returns the value of a scroll at its maximum (end) position. scroll_get_min - Returns the value of the scroll at its minimum (start) position. scroll_get_selected - Returns the minimum and maximum values of the selected range on a slider. scroll_get_pos - Returns the current scroll position. scroll_line - Scrolls the specified number of lines. scroll_max - Sets a scroll to the maximum (end) position. scroll_min - Sets a scroll to the minimum (start) position. scroll_page - Moves a scroll the specified number of pages. scroll_wait_info - Waits for the value of a scroll property.
HTMLFunctions HTMLDocument	[Not WR functions]
HTMLLink	Click, CompareStoredData, CompareStoredProperties
JavaMenuFunctions JavaTableFunctions JavaTree	[Not WR functions]
Label	Click	static_check_info - Checks the value of a static text object property. static_check_text - Checks the contents of a static text object. static_get_info - Returns the value of a static text property. static_get_text - Returns the contents of a static text object. static_wait_info - Waits for the value of a static text property.
ListBox	ActivatePartialMatch, ActivateTextItem, SelectPartialMatch, SelectTextItem, VerifyItemUnselected, VerifyListContains, VerifyListDoesNotContain, VerifySelectedItem, VerifySelectedPartialMatch	list_activate_item - Activates an item. list_check_info - Checks the value of a list property. list_check_item - Checks the content of an item in a list. list_check_selected - Checks that the specified item is selected. list_collapse_item - Hides items in a tree view object. list_deselect_item - Deselects an item. list_deselect_range - Deselects all items between two specified items. list_drag_item - Drags an item from a source list. list_drop_on_item - Drops an object onto a target list item. list_expand_item - Displays hidden items in a tree view object. list_extend_item - Adds an item to the items already selected. list_extend_multi_items - Adds multiple items to the items already selected. list_extend_range - Selects a range of items and adds them to the items currently selected. list_get_checked_items - Returns the value of items marked as checked. list_get_column_header - Returns the value of a ListView column header. list_get_info - Returns the value of a list property. list_get_item - Returns the contents of an item. list_get_item_coord - Returns the dimensions and coordinates of the list item. list_get_item_info - Returns the state of a list item. list_get_item_num - Returns the position of an item. list_get_items_count - Returns the number of items in the list. list_get_selected - Returns the currently selected item. list_get_subitem - Returns the value of the ListView subitem. list_rename_item - Activates an item's edit mode in order to rename it. list_select_item - Selects an item in a list. list_select_multi_items - Selects items in a multiple-selection container object. list_select_range - Selects all items between two specified items. list_set_item_state - Sets the state of an icon of the specified ListView or TreeView. list_wait_info - Waits for the value of a list property.
ComboListBox	Select, SelectIndex
ListView	ActivateTextItem, SelectIndexItem, SelectRandomItems, SelectTextItem, VerifyItemUnselected, VerifySelectedItem
TreeView	CollapseTextNode, ExpandTextNode, SelectTextNode, VerifyNodeUnselected, VerifySelectedNode
PopupMenu	SelectPopupMenuItem, SelectPopupMenuID
TabControl	Click, ClickTab, MakeSelection, SelectTab, UnverifiedClick	tab_get_info - Returns the value of a tab property. tab_get_item - Returns the name of a tab item. tab_get_selected - Returns the name of the selected tab item. tab_select_item - Selects a tab item. tab_wait_info - Waits for the value of a tab property.
TableView	ActivateTableRow, GetCellValue, GetRowCount, SetSelectedColumn, SetSelectedRow, SetTableCellData, VerifyCellValue	tbl_activate_cell - Double-clicks the specified cell in a table. tbl_activate_col - Double-clicks the specified column. tbl_activate_header - Double-clicks the specified column header in a table. tbl_activate_row - Double-clicks the specified row. tbl_deselect_col - Deselects the specified column. tbl_deselect_cols_range - Deselects the specified range of columns. tbl_deselect_row - Deselects the specified row. tbl_deselect_rows_range - Deselects the specified range of rows. tbl_extend_col - Adds a column to the currently selected columns. tbl_extend_cols_range - Adds columns to the currently selected columns. tbl_extend_row - Adds a row to the currently selected rows. tbl_extend_rows_range - Adds rows to the currently selected rows. tbl_get_cell_data - Retrieves the contents of the specified cell from a table. tbl_get_cols_count - Retrieves the number of columns in a table. tbl_get_column_name - Retrieves the column header name of the specified column in a table. tbl_get_column_names - Returns the names and number of columns in a table for PowerBuilder applications. tbl_get_rows_count - Retrieves the number of rows in the specified table. tbl_get_selected_cell - Returns the cell currently in focus in a table. tbl_get_selected_row - Returns the row currently selected in a table. tbl_select_cells_range - Selects the specified range of cells. tbl_select_col_header - Selects the specified column header of a table. tbl_select_cols_range - Selects the specified range of columns. tbl_select_rows_range - Selects the specified range of rows tbl_set_cell_data - Sets the contents of a cell to the specified text in a table. tbl_set_cell focus - Sets the focus to the specified cell in a table. tbl_set_selected_cell - Selects the specified cell in a table. tbl_set_selected_col - Selects the specified column in a table. tbl_set_selected_row - Selects the specified row in a table.
HTMLTable	Click, ClickCell, CompareStoredData, CompareStoredProperties, VerifyCellValue
VSFlexGrid	[Not a WR script folder]
ToolBar	[WR functions defined, but for Robot Classic only?]	toolbar_button_press - Clicks on a toolbar button. toolbar_get_button - Returns the name of a toolbar button. toolbar_get_button_info - Returns the value of a toolbar button property. toolbar_get_button_num - Returns the position of a toolbar button. toolbar_get_buttons_count - Returns the number of buttons on a toolbar. toolbar_select_item - Selects an item from a menu-like toolbar, as in IE 4. 0 or the Start menu in Windows 98.
SSTree	CollapseNode, ExpandNode, SelectNode, VerifyExpandedState, VerifySelectedText	[Sheridan Class=SSTree]

FIT/FITNesse/Abbot

FIT (Framework for Integrated Test) was developed by Ward Cunningham,

It's classified as an acceptance testing tool at http://www.xprogramming.com/software.htm because it can be used by (non-technical) customers to build tests easily using spreadsheet tables saved as html.

FIT parses html pages looking for Action Fixtures that describe user actions and verifications. FIT converts into html tables the commands it recognizes.

FIT then executes those tests by referencing a test fixture written in Java.

Abbots (Ab- bots) are inner classes that find components using Java reflection classes.

Extreme Programming Adventures in C# by Ron Jeffries

XP Testing a GUI with FIT, Fitnesse, and Abbot by Joseph Bergin at Pace University

STAF Version 3

The first version of STAF came out in 1998 from IBM Austin (where Tivoli and Linux test is designed). It went open source in 2001. It's now used by 150+ IBM teams and 100+ organizations worldwide.

download the Software Testing Automation Framework as Sourceforge project STAF, which provide the software to power services (http, email, event, eventmanager, namedcounter, cron, timer, fsext, stax, sxe) on most of the major OS platforms.

The installer creates these folders and files:

bin / .exe files, STAF.cfg and folders IPv4, IPV6, perl56, perl58
codepage / alias.txt, ibm-437.bin, ibm-1252.bin
data /
docs / .htm and .pdf files
include / .h header files
lib / STAF.lib used by Java programs
samples / c, cpp, demo, genwl, java, misc, rexx

External services need to be downloaded into folder manually created:

services

At the STAF root folder, file STAFReg.inf can be deleted unless you are on IBM's internal network and thus need to register with IBM.

Also at the STAF root folder, file C:\STAF3\STAFEnv.bat sets up an instance of the STAFproc.exe daemon service on every machine involved in the peer-to-peer STAF environment. It establishes the PATH to executables needed to invoke these commands to start and stop STAF:

STAFproc.exe

To shutdown in Windows, open another command window:

STAF local shutdown shutdown

This uses STAF.exe with the general syntax of:

STAF <Endpoint Machine> <Service> <Request>

Alternately, STAF can be installed as a Windows service. Once done, STAFProc.exe is only used for startup debugging.

Getting Started with STAF v3.0.3
STAF FAQ
staf-news for low traffic, read-only list for public news and announcements
staf-users for public questions, suggestions, support, and general discussion of STAF
staf-devel for public development use only

XP Testing a GUI with FIT, Fitnesse, and Abbot by Joseph Bergin at Pace University

STAF Services

STAF is used as a collection of web services communicating with HTTP protocol over port 80.

I/E	#	Service Name	Description	STAF 3.0 Location: {STAF/DataDir}/service/<Service name (lower-case)	STAF 2.x
Ext. Java	4.	CRON		Directory	<System.getProperty(user.home)>/crondata
Internal		DELAY
		DIAG
Internal		ECHO
Ext. Java		EMAIL
Ext. Java	5.	EVENT		Files /GenManager.out and RegManager.out	<Directory where STAFProc was started>/GenManager.out and RegManager.out
Ext. Java	3.	EVENTMANAGER		Directory	{STAF/Config/STAFRoot}/data/eventmanagerdata
Internal		FS File System
Internal		HANDLE
Internal		HELP	Allows for robust data-logging and log file querying and manipulation.
Ext. Java		HTTP
Ext. C++	1.	LOG	Allows for robust data-logging and log file querying and manipulation.	Directory	{STAF/Config/STAFRoot}/data/log
Internal		MISC	Provides miscellaneous services such as VERSION, WHOAMI, and WHOAREYOU information, and allows for listing and querying enabled interfaces.
Ext. C++		MONITOR	Allows test cases the ability to log and query status messages.
	6.	NamedCounter		File ... /namedCounter.ser File	<Directory where STAFProc was started>/.serFile
Internal		PING	ping other STAF clients to determine their availability.
Internal		PROCESS	start, stop, and manage processes.
Internal		QUEUE	Manipulate and manage queues awaiting processing.
Ext. C++	2.	RESOURCE POOL (ResPool)	manage exclusive access to entries within resource pools.	Directory	{STAF/Config/STAFRoot}/data/<Service name (lower-case)>
Internal		SEM SEMAPHORE	Manipulate and manage mutex and event semephores.
Ext. Java		STAX workflow - Provides an XML-based execution engine.
Internal		SERVICE	Manage STAF services and requests.
Internal		SHUTDOWN	Manage the STAFProc daemon process.
	7.	Timer		Files /tlist.ser and wlist.ser	{STAF/Config/STAFRoot}/bin/tlist.ser and wlist.ser
Internal		TRACE	Turn tracing on and off at the service and tracepoint level.
Internal		TRUST	Manipulate and manage trust levels (security).
Internal		VAR VARIABLE	Manipulate and manage system, shared and process specific variable pools.
Internal		Zip	Allows for unzipping, listing, and adding/deleting entries in Zip archives which are PKZip, WinZip and Jar compatible.

To change the list of external STAF services, change the bin\STAF.cfg configuration file. Example in the UG:

service Event library JSTAF execute c:/staf/services/STAFEvent.jar

STAF.cfg has these entries by default immediately after a fresh install:

# Turn on tracing of internal errors and deprecated options
trace enable tracepoints "error deprecated"

# Enable TCP/IP connections
interface tcp library STAFTCP

# Set default local trust
trust machine local://local level 5

# Default Service Loader Service
serviceloader library STAFDSLS

The above establishes a trust level of 5 to itself. To this you can add:
TRUST LEVEL 2 DEFAULT

To it you can add static system variables such as:

SET SYSTEM VAR Test/TestABC=websphere
SET SYSTEM VAR Test/TestXYZ=150

STAF.cfg is loaded only after STAFProc.exe is restarted (after being shutdown).

STAF Requests

Commands explained in the STAF FAQ

staf local service list
STAF local handle query all
staf local handle list handles
staf local var list
staf local trust list
STAF local PROCESS START SHELL COMMAND "dir *.*"
STAF local VAR RESOLVE STRING {STAF/Config/CodePage}
staf local var resolve string {STAF/Config/Machine}

More sample requests at STAF Command Reference

System Variables are available to all STAF machines.

Requests between STAF principals are in XML files.

A handle is assigned each communication between machines.

STAF log files are stored in a binary format which the FmtLog.exe utility writes to an output file in a readable format.