REST: Reducing Effort in Script-based Testing
The (narrow but very important) problem: Test scripts used for version 1.0 of an application will probably break when applied to version 2.0 of that application. Testers try to edit old test scripts so that they won't need to create new ones from scratch, but the process is slow, tedious and error-prone. It involves running a script until it breaks (this may take a while), figuring out whether it was a test script error and, if so, correcting it, and then starting over...potentially hundreds of times.
An example: Imagine what would happen to a test script if version 1.0 of the application used a combo box while version 2.0 used a text box for the same function. Obviously, any script that touches (what used to be) the combo box will break in 2.0. This is an example of a Changed Object (CO) error. The other type of error is Wrong Path (WP), which occurs when a script tries to access a Graphical User Interface (GUI) element (button, check box, etc.) that's been removed in 2.0. Either error will bring the test run to a screeching halt.
A solution would look like this: Given a set of test scripts and two versions of an application, automatically identify all the problematic areas of the GUIs and the scripts, then help the tester change the scripts so that they'll work with 2.0. REST does this.
REST accomplishes its magic through a combination of program analysis and the accessibility layer, an Application Programmer Interface that supplies a wealth of information about the structure and actions of a running GUI program. The accessibility layer normally is used to provide services to users with various kinds of physical deficit (sight loss, motor problems,...)--but it can also be used to analyze a GUI application from another program--in this case, from REST.
REST relies on the tester to navigate to the applications' various main screens, then uses the accessibility layer to build trees of those screens' GUI components (a small sample tree: Window #61 owns a dialog box which in turn owns six buttons and a text field). For each main screen, the two GUI trees (1.0's and 2.0's) are compared. When differences are found, REST engages in semi-automatic (i.e., the tester helps) analysis to decide whether the various mismatches are WP errors or CO errors, and in either case guides the tester in changing the affected test scripts such that they'll work with version 2.0.
REST was tested(!) on versions of four large open source applications and found to be efficient and effective in reducing the time necessary to update test scripts. For more information on REST, get in touch with Accenture Technology Labs' Mark Grechanik.