Automated Testing Tools
11.6 THE BENEFITS THAT TESTING TOOLS CAN PROVIDE Based on the previous sections in this chapter, you may conclude incorrectly that it is
a good idea for a tester to always use automated testing tools. This would cause the tester to worry fi rst about which test tool to use. Quite the contrary, one of the early-
11.6 The Benefi ts that Testing Tools can Provide 169
development lifecycle decisions a test team leader must make is whether to include automated testing tools in the test environment plans. Only after the appropriateness of test tool usage has been determined does the question of tool choice arise. The re- mainder of this chapter deals only with identifying development project circumstances that determine the appropriateness or inappropriateness of automated test tool use.
11.6.1 Repeatability—a Return on Investment Issue
If a particular test script will be executed three or more times, a test tool can begin to provide an economy of scale. The effort, resources, and cost to build the automated test script the fi rst time are substantially higher than a single manual test that per- forms the same script steps. The second time the automated script is executed, none of the build costs are incurred. The second manual test of the same test script costs as much as the fi rst manual test to perform.
After three or more test executions of the same script, the accumulated cost of performing these executions using a test tool is less than the accumulated cost of performing these same executions manually. The more times the automated test script is repeated beyond three times, the less expensive the total execution ef- fort becomes relative to the same manual testing activity. Because manual testing of- fers no economies of scale, the total cost of manual testing continues to grow linearly by the same incremental cost for each subsequent test execution.
11.6.2 Repeatability Precision—a Technology Issue
One of the most frustrating and time-consuming situations for a software developer is for a tester to report a suspected defect that the developer cannot recreate. This situation can arise from a number of different causes, the most common of which is incomplete manual test case execution script documentation. In other words, the tes- ter cannot tell the developer the exact sequence of actions and events that led to the defect discovery. The developer must use trial and error to fi ll in the tester’s informa- tion gaps and try the most likely actions and events to recreate the suspected defect.
Another possible cause of defect recreation diffi culty for the developer lies with the tester’s ability to follow directions. The manual test execution script may be very crisp and clear about the steps to be completed and the expected results. Testers, being human, may inadvertently skip a step or complete only part of the step action because the instruction continued on the next page. The likelihood of this human element creeping into test execution increases with the number of times the tester must repeat the test due to a false sense of familiarity with the steps.
Automated testing tools directly address both of these repeatability precision issues. Every action to be taken by a testing tool is included in the tool’s test script.
The state of test data prior to script execution is defi ned in the test case. The testing tool executes its script with well-defi ned test data without human intervention. The combination of explicit script, explicit data state, and human non-intervention guar- antees repeatable test results with very high precision. This precision translates into more expedient developer diagnosis of discovered defects.
11.6.3 Hidden Action and Results Verification—a Technology Issue
Although many application responses are revealed directly to the end-user, there may be many application responses that are hidden from the end user’s view. Exam- ples of revealed responses are new screen messages indicating “successful submis- sion,” new screens indicating successful submission with processing details, or new screens with requested search results. An example of hidden responses is a control item in a screen object like a “submit” button that becomes hidden from the end user afterwards to inhibit duplicated submissions.
Another example of hidden responses is database changes resulting from the end user submitting some kind of response (search request, purchase order, and so on). The revealed response is “successful submission.” The hidden response is a set of database updates that may not be directly displayed to the end user or viewable by the end user. Developers go to great lengths to keep these actions and results hidden from the end user, either because revealing them will not help the end user do a better job or because revealing them will allow the end user to put the appli- cation in unexpected states of processing with potentially disastrous results to the business.
Testing tool designers are aware of this duality: end user versus developer view of the application responses. Knowing that both views are necessary for complete testing results, the testing tool designers have produced testing tools that can observe and act on both views of the AUT.
11.6.4 Timing Point Precision and Distribution—a Technology Issue
One of the test tool capabilities arguably most benefi cial for software develop- ment in the 21st century is precision timing points. Prior to having this capability, testers were relegated to using a stopwatch, a wall clock, or a sweep second hand on their wristwatch. If the application activity took 15 or 20 min to complete as was common in the 1970s, then chronometers provided suffi cient timing point precision.
As application response speeds became much faster, the Rule of 8 (see Chapter 9) refl ected new sub-minute response time measurement needs. The tester needed and was provided with tool timing point precision to the millisecond. This leap in preci- sion afforded the tester extremely precise response time measurements for both the total application process and subprocesses previously unattainable from stopwatch accuracy. The testing benefi t of subprocess timing point precision is the ability to identify bottlenecks in the response path as illustrated in the Round Trip Perfor- mance Testing section of Chapter 9.
The trend toward distributed computing in the 1980s led to an additional chal- lenge for performance testers. The application was no longer spending time just on the end-user computer and a large host computer. The application time was distributed across a number of specialized midsize computers called servers. Even 11.6 The Benefi ts that Testing Tools can Provide 171
if application response times had remained in the multiple-minute range, the tester would have been challenged to apply a stopwatch to several servers that would most likely be dispersed geographically, one of the primary advantages of distrib- uted computing. The advent of viable Internet business applications has caused the span of these geographically distributed computing environments to stretch around the world. Test tools can and do provide millisecond response time measurements across these widely distributed computing environments without the tester leaving his chair.
11.6.5 Achievement of Impossible Manual Tests—a Return on Investment Issue
Choosing to do manual performance testing in preference to tool testing is likened to going hunting for elephants with a rock instead of a rifl e. Although stopwatch timing may not be the most precise way to measure an application’s response times, stopwatch timing can still get the job done if no testing tools are available. The true limitation of this approach resides in the size and complexity of the workload to be tested as described in Chapter 9.
Consider the situation of a workload to be tested in the range of 10–100 active business transactions and no test tools available. This situation can be tested manu- ally by calling in the entire offi ce staff on a Saturday, give them manual scripts to run, feed them pizza and beer, and have them pound on the system all day. The results are not particularly precise or repeatable, but the job does get done. The authors know of large companies who still do “Saturday Special” timing of their applications.
Any performance workload requirement beyond 100 active business transac- tions cannot be tested manually. The good news is that performance test tools have matured to the point that they can test workloads well beyond the 100 transaction breakpoint for manual testing. It is commonplace to achieve performance tool test- ing of workloads in the thousands of transactions. Some of the more specialized Internet performance testing tools can drive workloads of 500,000 transactions or more.
11.6.6 Reusability (Regression Testing)—a Return on Investment Issue
Test tools have been designed from the beginning to easily and reliably reuse test scripts. As testing processes have matured, this reusability feature has greatly en- abled the concept and practice of regression testing described in Chapters 7 and 8.
If the tester manages and maintains all test scripts produced for the fi rst ver- sion of an application, then this set of test scripts for Version 1 can be applied to an early copy of Version 2. Code untouched by Version 2 should pass the Version 1 script tests. Code touched to accommodate Version 2 changes should also pass the Version 1 script tests. New code for Version 2 will require test script additions or
modifi cations for whatever new functionality that Version 2 provides. Functionality in Version 1 no longer supported in Version 2 will require test script retirement.
11.7 THE LIABILITIES THAT TESTING TOOLS