Testing Tools

QAT was developed to ease the issues encountered by having to perform Quality Assurance tests across a variety of hardware and software combinations. The way to achieve this is to abstract the process of running the test on a machine. So, from a tester's point of view, there is one test suite, which the tester executes, and then gets the results for further analysis. The specifics of the machine on which the test will be run is completely hidden, and is necessary for running the test.

The QAT tool can be divided into two main sections, the Agent, responsible for actually running each test or group of tests, and the Harness, which is responsible for test selection, management, results, and agent co-ordination. It can be accessed via a browser interface if remote access to the harness is required.

First, each test case is written in a platform-independent way in a simple scripting language, very similar to C shell, although it could be implemented as any language you want. All references to platform-specific items are delayed by referring to one or more configuration files. A configuration is then built for each test based on the architecture and operating system of the target machine. So, instead of directly executing a command within the test, we refer to variables, which will only be defined at the time of running the test.

Once the test case is ready, the test is transferred to whatever machine we want to run it on (agent). This is achieved by developing a relatively stupid agent that can respond to a limited set of requests from the harness via TCP sockets. Some examples are GETFILE, SENDFILE, DELFILE, EXECUTE, etc. The test script running on the harness may then use this command set to build a desired environment on the agent. We are able to transfer files, unzip files, delete files, set environment variables, and any of the other good things we need to run a test. Once the test is run, a result is decided on and sent back to the harness for processing.

Since the agent is written in Java, it runs on any architecture supporting a JVM, including restricted ones running pJava, kJava, and various other slimmed-down versions.

Ease of Use

* Learning curve

* Easy to maintain the tool

* Easy to install--tool may not be used if difficult to install

Tool Customization

* Can the tool be customized (can fields in tool be added or deleted)?

* Does the tool support the required test procedure naming convention?

Platform Support

* Can it be moved and run on several platforms at once, across a network (that is, cross-Windows support, Win95, and WinNT)?

Multiuser Access

* What database does the tool use? Does it allow for scalability?

* Network-based test repository--necessary when multiple access to repository is required

Defect Tracking

* Does the tool come with an integrated defect-tracking feature?

Tool Functionality

* Test scripting language--does the tool use a flexible, yet robust scripting language? What is the complexity of the scripting language: Is it 4 GL? Does it allow for modular script development?

* Complexity of scripting language

* Scripting language allows for variable declaration and use; allows passing of parameters between functions

* Does the tool use a test script compiler or an interpreter?

* Interactive test debugging--does the scripting language allow the user to view variable values, step through the code, integrate test procedures, or jump to other external procedures?

* Does the tool allow recording at the widget level (object recognition level)?

* Does the tool allow for interfacing with external .dll and .exe files?

* Published APIs--language interface capabilities

* ODBC support--does the tool support any ODBC-compliant database?

* Is the tool intrusive (that is, does source code need to be expanded by inserting additional statements)?

* Communication protocols--can the tool be adapted to various communication protocols (such as TCP/IP, IPX)?

* Custom control support--does the tool allow you to map to additional custom controls, so the tool is still compatible and usable?

* Ability to kick off scripts at a specified time; scripts can run unattended

* Allows for adding timers

* Allows for adding comments during recording

* Compatible with the GUI programming language and entire hardware and software development environment used for application under test (i.e., VB, Powerbuilder)

* Can query or update test data during playback (that is, allows the use of SQL statements)

* Supports the creation of a library of reusable function

* Allows for wrappers (shells) where multiple procedures can be linked together and are called from one procedure

* Test results analysis--does the tool allow you to easily see whether the tests have passed or failed (that is, automatic creation of test results log)?

* Test execution on script playback--can the tool handle error recovery and unexpected active windows, log the discrepancy, and continue playback (automatic recovery from errors)?

* Allows for synchronization between client and server

* Allows for automatic test procedure generation

* Allows for automatic data generation

Reporting Capability

* Ability to provide graphical results (charts and graphs)

* Ability to provide reports

* What report writer does the tool use?

* Can predefined reports be modified and/or can new reports be created?

Performance and Stress Testing

* Performance and stress testing tool is integrated with GUI testing tool

* Supports stress, load, and performance testing Allows for simulation of users without requiring use of physical workstations

* Ability to support configuration testing (that is, tests can be run on different hardware and software configurations)

* Ability to submit a variable script from a data pool of library of scripts/data entries and logon IDs/password

* Supports resource monitoring (memory, disk space, system resources)

* Synchronization ability so that a script can access a record in database at the same time to determine locking, deadlock conditions, and concurrency control problems

* Ability to detect when events have completed in a reliable fashion

* Ability to provide client to server response times

* Ability to provide graphical results

* Ability to provide performance measurements of data loading

Version Control

* Does the tool come with integrated version control capability?

* Can the tool be integrated with other version control tools

Test Planning and Management

* Test planning and management tool is integrated with

GUI testing tool

* Test planning and management tool is integrated with requirements management tool

* Test planning and management tool follows specific industry standard on testing process (such as SEI/CMM, ISO)

* Supports test execution management

* Allows for test planning--does the tool support planning, managing, and analyzing testing efforts? Can the tool reference test plans, matrices, and product specifications to create traceability?

* Allows for measuring test progress

* Allows for various reporting activities

Pricing

* Is the price within the estimated price range?

* What type of licensing is being used (floating, fixed)?

* Is the price competitive?

Vendor Qualifications

* Maturity of product

* Market share of product

* Vendor qualifications, such as financial stability and length of existence. What is the vendor's track record?

* Are software patches provided, if deemed necessary?

* Are upgrades provided on a regular basis?

* Customer support

* Training is available

* Is a tool Help feature available? Is the tool well documented?

* Availability and access to tool user groups

• SilkTest derives its initial startup configuration settings from its partner.ini file. This though is not important because SilkTest can be reconfigured at any point in the session by either changing any setting in the Options menu or loading an Option Set. An Option Set file (*.opt) permits customized configuration settings to be established for each test project. The project specific Option Set is then be loaded [either interactively, or under program control] prior to the execution of the project’s testcases. The Options menu or an Option Set can also be used to load an include file (*.inc) containing the project’s GUI Declarations [discussed in section 2.6 on page 5], along with any number of other include files containing library functions, methods, and variables shared by all testcases.

• WinRunner derives its initial startup configuration from a wrun.ini file of settings. During startup the user is normally polled [this can be disabled] for the type of addins they want to use during the session [refer to section 2.3 on page 3 for more information about addins]. The default wrun.ini file is used when starting WinRunner directly, while project specific initializations can be established by creating desktop shortcuts which reference a project specific wrun.ini file. The use of customized wrun.ini files is important because once WinRunner is started with a selected set of addins you must terminate WinRunner and restart it to use a different set of addins. The startup implementation supports the notion of a startup test which can be executed during WinRunner initialization. This allows project-specific compiled modules [memory resident libraries] and GUI Maps [discussed in section 2.6 on page 5] to be loaded. The functions and variables contained in these modules can then be used by all tests that are run during that WinRunner session.

Both tools allow most of the configuration setup established in these files to be over-ridden with runtime code in library functions or the test scripts.

Test Termination

• SilkTest tests terminate on exceptions which are not explicitly trapped in the testcase. For example if a window fails to appear during the setup phase of testing [i.e. the phase driving the application to a verification point], a test would terminate on the first object or window timeout exception that is thrown after the errant window fails to appear.

• WinRunner tests run to termination [in unattended Batch mode] unless an explicit action is taken to terminate the test early. Therefore tests which ignore this termination model will continue running for long periods of time after a fatal error is encountered. For example if a window fails to appear during the setup phase of testing, subsequent context sensitive statements [i.e. clicking on a button, performing a menu pick, etc.] will fail—but this failure occurs after a multi-second object/window “is not present” timeout expires for each missing window and object. [When executing tests in non-Batch mode, that is in Debug, Verify, or Update modes, WinRunner normally presents an interactive dialog box when implicit errors such as missing objects and windows are encountered].

Addins and Extensions

Out of the box, under Windows, both tools can interrogate and work with objects and windows created with the standard Microsoft Foundation Class (MFC) library. Objects and windows created using a non-MFC technology [or non-standard class naming conventions] are treated as custom objects. Dealing with truly custom objects is discussed further in section 2.8 on page 6. But objects and windows created for web applications [i.e. applications which run in a browser], Java applications, Visual Basic applications, and PowerBuilder applications are dealt with in a special manner:

• SilkTest enables support for these technologies using optional extensions. Selected extensions are enabled/disabled in the current Option Set [or the configuration established by the default partner.ini option settings].

• WinRunner enables support for these technologies using optional addins. Selected addins are enabled/disabled using either the Addin Manager at WinRunner startup, or by editing the appropriate wrun.ini file prior to startup. Note that (1) some combinations of addins [WinRunner] and extensions [SilkTest] are mutually exclusive, (2) some of these addins/extensions may no longer be supported in the newest releases of the tool, (3) some of these addins/extensions may only support the last one or two releases of the technology [for example version 5 and 6 of Visual Basic] and (4) some of these addins and extensions may have to be purchased at an addition cost.

Visual Recorders

SilkTest provides visual recorders and wizards for the following activities: • Creating a test frame with GUI declarations for a full application and adding/deleting selective objects and windows in and existing GUI declarations frame file. • Capturing user actions with the application into a test case, using either context sensitive [object relative] or analog [X:Y screen coordinate relative] recording techniques. • Inspecting identifiers, locations and physical tags of windows and objects. • Checking window and object bitmaps [or parts thereof]. • Creating a verification statement [validating one or more object properties].

WinRunner provides visual recorders and wizards for the following activities: • Creating an entire GUI Map for a full application and adding/deleting selective objects and windows in an existing GUI Map. It is also possible to implicitly create GUI Map entries by capturing user actions [using the recorder described next]. • Capturing user actions with the application into a test case, using either context sensitive [object relative] or analog [X:Y screen coordinate relative] recording techniques. • Inspecting logical names, locations and physical descriptions of windows and objects. • Checking window and object bitmaps [or parts thereof]. • Creating a GUI checkpoint [validating one or more object properties]. • Creating a database checkpoint [validating information in a database]. • Creating a database query [extracting information from a database]. • Locating at runtime a missing object referenced in a testcase [and then adding that object to the GUI Map]. • Teaching WinRunner to recognize a virtual object [a bitmap graphic with functionality]. • Creating Data Tables [used to drive a test from data stored in an Excel-like spreadsheet]. • Checking text on a non-text object [using a built-in character recognition capability]. • Creating a synchronization point in a testcase. • Defining an exception handler. Some of these recorders and wizards do not work completely for either tool against all applications, under all conditions. For example neither tool’s recorder to create a full GUI Map [WinRunner] or test frame [SilkTest] works against large applications, or any web application. Evaluate the recorders and wizards of interest carefully against your applications if these utilities are important to your automated testing efforts.

Object Hierarchy

• SilkTest supports a true object-oriented hierarchy of parent-child-grandchild-etc. relationships between windows and objects within windows. In this model an object such as a menu is the child of its enclosing window and a parent to its menu item objects.

• WinRunner, with some rare exceptions [often nested tables on web pages], has a flat object hierarchy where child objects exist in parent windows. Note that web page frames are treated as windows, and not child objects of the enclosing window on web pages that are constructed using frames.

Object Recognition

Both of these tools use a lookup table mechanism to isolate the variable name used to reference an object in a test script from the description used by the operating system to access that object at runtime:

• SilkTest normally places an application’s GUI declarations in a test frame file. There is generally one GUI declaration for each window and each object in a window. A GUI declaration consists of an object identifier—the variable used in a test script—and its class and object tag definition used by the operating system to access that object at runtime. SilkTest provides the following capabilities to define an object tag: (1) a string, which can include wildcards; (2) an array reference which resolves to a string which can include wildcards; (3) a function or method call that returns a string, which can include wildcards, (4) an object class and class relative index number; and (5) multiple tags [multi-tags] each optionally conditioned with (6) an OS/GUI/browser specifier [a qualification label].

• WinRunner normally places an application’s logical name/physical descriptor definitions in a GUI Map file. There is generally one logical name/physical descriptor definition for each window and each object in a window. The logical name is used to reference the object in a test script, while the physical descriptor is used by the operating system to access that object at runtime. WinRunner provides the following capabilities to define a physical descriptor: (1) a variable number of comma delimited strings which can include wildcards, where each string identifies one property of the object. [While there is only a single method of defining a physical descriptor, this definition can include a wide range and variable number of obligatory, optional, and selector properties on an object by object basis]. The notion behind this lookup table mechanism is to permit changes to an object tag [SilkTest] or a physical descriptor [WinRunner] definition without the need to change the associated identifier [SilkTest] or logical name [WinRunner] used in the testcase. In general the object tag [SilkTest] or physical descriptor [WinRunner] resolve to one or more property definitions which uniquely identify the object in the context of its enclosing parent window at runtime. It is also possible with both tools to dynamically construct and use object tags [SilkTest] or physical descriptors [WinRunner] at runtime to reference objects in test scripts.

Object Verification

Both tools provide a variety of built-in library functions permitting a user to hand code simple verification of a single object property [i.e. is/is not focused, is/is not enabled, has/does not have an expected text value, etc.]. Complex multiple properties in a single object and multiple object verifications are supported using visual recorders:

• SilkTest provides a Verify Window recorder which allows any combination of objects and object properties in the currently displayed window to be selected and captured. Using this tool results in the creation, within the testcase, of a VerifyProperties() method call against the captured window.

• WinRunner provides several GUI Checkpoint recorders to validate (1) a single object property, (2) multiple properties in a single object, and (3) multiple properties of multiple objects in a window. The single property recorder places a verification statement directly in the test code while the multiple property recorders create unique checklist [*.ckl] files in the /chklists subdirectory [which describe the objects and properties to capture], as well as an associated expected results [*.chk] file in the /exp subdirectory [which contains the expected value of each object and property defined in the checklist file]. Both tools offer advanced features to define new verification properties [using mapping techniques and/or built-in functions and/or external DLL functions] and/or to customize how existing properties are captured for standard objects.

Custom Objects

Note: The description of this feature, more than any other in this report, is limited in its scope and coverage. An entire white paper could be dedicated to exploring and describing how each of these tools deal with custom objects. Therefore dedicate several days to evaluating how each of these tools accommodate custom objects in your specific applications. To deal with a custom object [i.e. an object that does not map to standard class] both tools support the use of class mapping [i.e. mapping a custom class to a standard class with like functionality], along with a variety of X:Y pixel coordinate clicking techniques [some screen absolute, some object relative] to deal with bitmap objects, as well as the ability to use external DLL functions to assist in object identification and verification. Beyond these shared capabilities each tool has the following unique custom object capabilities:

• SilkTest has a feature to overlay a logical grid of X rows by Y columns on a graphic that has evenly spaced “hot spots”[this grid definition is then used to define logical GUI declarations for each hot spot]. These X:Y row/column coordinates are resolution independent [i.e. the logical reference says “give me 4th column thing in the 2nd row”, where that grid expands or contracts depending on screen resolution].

• WinRunner has a built-in text recognition engine which works with most standard fonts. This capability can often be used to extract visible text from custom objects, position the cursor over a custom object, etc. This engine can also be taught non-standard font types which is does understand out of the box. Both tools offer support for testing non-graphical controls through the advanced use of custom DLLs [developed by the user], or the Extension Kit [SilkTest, which may have to be purchased at an addition cost] and the Mercury API Function Library [WinRunner]. SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2.9 Internationalization (Language Localization)

• SilkTest supports the single-byte IBM extended ASCII character set, and its Technical Support has also indicated “that Segue has no commitment for unicode”. The user guide chapter titled “Supporting Internationalized Applications” shows a straightforward technique for supporting X number of [single-byte IBM extended ASCII character set] languages in a single test frame of GUI declarations.

• WinRunner provides no documentation on how to use the product to test language localized applications. Technical Support has indicated that (1) “WinRunner supports multi-byte character sets for language localized testing…”, (2) “there is currently no commitment for the unicode character set…”, and (3) “it is possible to convert a US English GUI Map to another language using a [user developed] phrase dictionary and various gui_* built-in functions”.

Database Interfaces

Both tools provide a variety of built-in functions to perform Structure Query Language (SQL) queries to control, alter, and extract information from any database which supports the Open Database Connectivity (ODBC) interface.

Database Verification

Both tools provide a variety of built-in functions to make SQL queries to extract information from an ODBC compliant database and save it to a variable [or if you wish, an external file]. Verification at this level is done with hand coding.

WinRunner also provides a visual recorder to create a Database Checkpoint used to validate the selected contents of an ODBC compliant database within a testcase. This recorder creates side files similar to GUI Checkpoints and has a built-in interface to (1) the Microsoft Query application [which can be installed as part of Microsoft Office], and (2) to the Data Junction application [which may have to be purchased at an addition cost], to assist in constructing and saving complex database queries.

Data Driven Testing

Both tools support the notion of data-driven tests, but implement this feature differently:

• SilkTest’s implementation is built around an array of user defined records. A record is a data structure defining an arbitrary number of data items which are populated with values when the array is initialized [statically or at runtime]. Non-programmers can think of an array of records as a memory resident spreadsheet of X number of rows which contain Y number columns where each row/column intersection contains a data item. The test code, as well as the array itself, must be hand coded. It is also possible to populate the array each time the test is run by extracting the array’s data values from an ODBC compliant database, using a series of built-in SQL function calls. The test then iterates through the array such that each iteration of the test uses the data items from the next record in the array to drive the test or provide expected data values.

• WinRunner’s implementation is built around an Excel compatible spreadsheet file of X number of rows which contain Y number of columns where each row/column intersection contains a data item. This spreadsheet is referred to as a Data Table. The test code, as well as the Data Table itself, can be created with hand coding or the use of the DataDriver visual recorder. It is also possible to populate a Data Table file each time the test is run by extracting the table’s data values from an ODBC compliant database using a WinRunner wizard interfaced to the Microsoft Query application.. The test then iterates through the Data Table such that each iteration of the test uses the data items from the next row in the table to drive the test or provide expected data values. Both tools also support the capability to pass data values into a testcase for a more modest approach to data driving a test.

Restoring an Application’s Initial State

• SilkTest provides a built-in recovery system which restores the application to a stable state, referred to as the basestate, when the test or application under test fails ungracefully. The default basestate is defined to be: (1) the application under test is running; (2) the application is not minimized; and (3) only the application’s main window is open and active. There are many built-in functions and features which allow the test engineer to modify, extend, and customize the recovery system to meet the needs of each application under test.

• WinRunner does not provide a built-in recovery system. You need to code routines to return the application under test to its basestate—and dismiss all orphaned dialogs—when a test fails ungracefully.

Scripting Language

Both tools provide proprietary, interpreted, scripting languages. Each language provide the usual flow control constructs, arithmetic and logical operators, and a variety of built-in library functions to perform such activities as string manipulation, [limited] regular expression support, standard input and output, etc. But that is where the similarity ends: • SilkTest provides a strongly typed, object-oriented programming language called 4Test. Variables and constants may be one of 19 built-in data types, along with a user defined record data type. 4Test supports single- and multi-dimensional dynamic arrays and lists, which can be initialized statically or dynamically. Exception handling is built into the language [via the do… except statement].

• WinRunner provides a non-typed, C-like procedural programming language called TSL. Variables and constants are either numbers or strings [conversion between these two types occurs dynamically, depending on usage]. There is no user defined data type such as a record or a data structure. TSL supports sparsely populated associative single- and [pseudo] multidimension arrays, which can be initialized statically or dynamically—element access is always done using string references—foobar[“1”] and foobar[1] access the same element [as the second access is implicitly converted to an associative string index reference]. Exception handling is not built into the language. The only way to truly understand and appreciate the differences between these two programming environments is to use and experiment with both of them.

Exception Handling

Both tools provide the ability to handle unexpected events and errors, referred to as exceptions, but implement this feature differently:

• SilkTest’s exception handling is built into the 4Test language—using the do… except construct you can handle the exception locally, instead of [implicitly] using SilkTest’s default built-in exception handler [which terminates the currently running test and logs an error]. If an exception is raised within the do block of the statement control is then immediately passed to the except block of code. A variety of built-in functions [LogError(), LogWarning, ExceptNum(), ExceptLog(), etc.] and 4Test statements [raise, reraise, etc.] aid in the processing of trapped exceptions within the except block of code.

• WinRunner’s exception handling is built around (1) defining an exception based on the type of exception (Popup, TSL, or object) and relevant characteristics about the exeception (most often its error number); (2) writing an exception hander, and (3) enabling and disabling that exception handler at appropriate point(s) in the code. These tasks can be achieved by hand coding or through the use of the Exception Handling visual recorder.