Test Planning
5.3 TEST CASES
One outcome of preparing the test plan is a list of test cases needed for each development phase. Test cases are initially identifi ed by the testing team as high- level testing requirements from the new application’s business requirements, business risks, or software specifi cations. As the developers identify more and more details about the new application, the testers expand the description of the test case to include very detailed testing instructions and expected outcomes.
The test team manager might say, “at this phase of development, we need to test all the data entry screens.” This would cause the test team to identify all the data entry screens to be developed and the general business functions these data entry screens will support. For example, if the application under test was a payroll system, then there might be six groups of data entry screens: employee personal information, employee benefi ts information, employee tax information, state tax information, and federal tax information. To keep testing execution management 5.3 Test Cases 83
as simple as possible, the test team would defi ne six test cases, one for each group of data entry screens—regardless of the actual number of data entry screens to be tested in each group. The writing schedule of these test cases then needs to be coor- dinated with the developer schedule for these data entry screens.
Another common example of test case planning is performance testing. Al- though we have not yet discussed the particulars of performance testing, hopefully the reader can identify at least four potential test cases for most applications: a base- line performance test case, a batch workload test case, an online workload test case, and an Internet workload test case.
5.3.1 Test Case Details
The purpose of the test plan is to collectively document the testing “what” and “why”
for this application. We focus on the testing “how” for this application by drafting test cases. As we saw with the test plan terminology, you will fi nd references to these
“how” test documents as test scenarios, test cases, and test scripts elsewhere. As long as you understand the purpose of these test documents and confi rm that you are col- lecting all the information necessary for successful testing, the terminology choice is arbitrary. For the purposes of discussion in this textbook, the detailed test execution document will simply be called the test case.
The contents of the test case take on the narrow focus of just one of many testing activities that collectively support the overall test plan. Defi ning the boundaries of a good testing case is still very much an art that is learned over a number of years of testing experience. A good starting place is to identify some of the smallest or simplest business activities that the new software needs to support and defi ne test cases for each of these activities. Then, defi ne test cases that portray either larger business activities or specifi c sequences of smaller business activities that tend to approximate useful business tasks. Check the software business requirements and confi rm that you have accounted for all required business activities with your test cases and add test cases for the business activities that are not covered yet by testing.
We will examine some test cases in Chapters 8 and 9 that cover other aspects of the software besides business activities, but business activities are the logical starting place for designing test cases. Test cases should include the following:
unique test case ID (from test plan) unique title (from test plan) brief description (from test plan)
development phase in which this test case is executed (from test plan) specifi c testing goals and their achievement measures
suggested test data suggested test tools test startup procedure 1.
2.
3.
4.
5.
6.
7.
8.
test closedown procedure test reset procedure for rerun Repeated test execution steps (a). step number
(b). step action (c). expected results
(d). Placeholder actual results
Placeholder fi rst attempt to execute this test case date: time
Placeholder number of attempts to successfully execute this test case Placeholder list of software defects discovered by this test case Placeholder is this test case executed successfully ? (yes/no)
<Placeholder>
Some of the items above are prefi xed by <Placeholder>. This denotes the need for some type of tracking and reporting capability to provide that item from external sources. Some test teams opt to do this tracking and reporting with simple tools like spreadsheets or simple databases. Other test teams opt to do this tracking and re- porting with more complex test management tools. At the end of the testing day, the approach you choose must tell you in detail what testing was attempted, what testing succeeded, and what testing failed.
Here is a brief description of each test case item:
Unique test case ID—a string of letters and numbers that uniquely identify this test case document. The ID is defined during test plan design and is en- tered on test plan item 9(d).
Unique title—a short descriptive label that clearly implies the purpose of this test case. The title is defined during test plan design and is entered on test plan item 9(d) beside the test case ID.
Brief description—a paragraph that expands on the title to give more details about the purpose of this test case. The description is defined during test plan de- sign and is entered on test plan item 9(d) beside the test case ID and unique title.
Development phase in which this test case is executed—this is a cross- reference to the development phase section of the test plan that mentions this test case. It is meant to be a reminder of the development state of the code to be tested as a reality check against the kind of testing this test case is intended to accomplish.
Specific testing goals and their achievement measures—this item takes the purpose of the test case stated in the title and brief description and quantifies specific testing goals as well as how the test team can measure the achieve- ment of these goals.
9.
10.
11.
12.
13.
14.
15.
1.
2.
3.
4.
5.
5.3 Test Cases 85
Suggested test data—this item identifies the minimal amount of data and most useful data sources for executing this test case in the test environment.
For example, if the application under test is a payroll system, then a copy of the current employee master file would be one of the most useful data source for testing. The number of employee records needed for testing this test case would depend on the testing goals. Usually, a representative sample of em- ployee records will be adequate for testing (see Generic black box testing techniques in Chapter 7). Sometimes it requires less effort to use a copy of the whole file rather than to carve out a representative sample.
Suggested test tools—if the tester is knowledgeable about automated test tools, this is where the tester suggests one or two that have capabilities particularly appropriate for executing this test case. A recap of all tool suggestions across all test cases in the test plan becomes the basis of the test manager’s rec- ommendation regarding the automated test tools in the testing environment.
Not all test cases lend themselves to automated test tool usage. The cost of automated test tools makes them prohibitively expensive for a relatively small number of executions. Test tools will be examined further in Chapter 11.
Test startup procedure—this documents what hardware, software, and data must be available to start the test execution. It further identifies how the required software including the application under test must be launched or made active in order to allow the first execution step of this test case to be accomplished.
Test closedown procedure—this documents how to gracefully stop all the software launched or made active including the application under test. A graceful software stop allows the software activities to idle down and the software to terminate normally.
Test reset procedure for rerun—this documents what must be done between the most recent test execution closedown and the next test startup in order to allow this test case to be executed again. Most times, this section deals with restoring data to its previous state prior to adds, changes, and deletes caused by executing this test case after the startup.
RepeatedTest execution steps—this list of items is repeated for each new step number. One execution attempt of this test case is achieved by perform- ing all the steps once in step number order.
(a). Step number—a unique number that sequences the test case steps for- execution.
(b). Step action—the specifi c, fully described action taken by a tester to evoke a desired software behavior. Examples of step actions include keystrokes, mouse movements, button clicks, drop-down list selections, and voice com- mands.
(c). Expected results—the particular software behavior or response expect- ed from the step action. Examples of expected results include screen responses, printer responses, fi le responses, and network responses.
6.
7.
8.
9.
10.
11.
(d). Placeholder Actual results—the particular software behavior or response actually observed from executing the step action. Examples of actual results include screen responses, printer responses, fi le responses, and network responses.
PlaceholderFirst attempt to execute this test case date: time—this item documents the date and time when the steps of the test case are fi rst attempted. This metric helps the test manager measure testing progress toward 100% test cases at- tempted.
PlaceholderNumber of attempts to successfully execute this test case—
this item measures how many attempts become necessary to achieve a successful test case execution.
PlaceholderList of software defects discovered by this test case—the log of genuine, correctable software defects discovered by this test case. If the software development process does not change signifi cantly from project to project, then this log will begin to reveal defect patterns that may forecast the approximated number of defects to be discovered by the next development project. If the software development process does change signifi cantly over time, then this log can demon- strate how much the new development process increases or decreases the software quality (fewer defects) by development phase. This kind of defect log interpretation is the subject of an entire section in Chapter 12.
PlaceholderIs this test case executed successfully? (yes/no)—this item is answered only after the fi rst test case attempt. The answer can subsequently be changed from “no” to “yes” after any number of attempts that achieve successful test case execution. Usually, some of these attempts occur after software correction in order to confi rm the correction as well as achieve successful test case execution.
5.3.2 Step Action
Step actions are the documented keystrokes, mouse movements, and screen render- ing for all of the end user steps to complete a business task. Using the data entry screen testing example, a sequence of step actions would describe the mouse or key- board commands to move the cursor to the next data fi eld to be tested. The next step actions would document the specifi c data entered by keyboard or selected by mouse from a dropdown menu. After a series of these keyboard entries and mouse move- ments on the screen, the next step actions might describe a “request completed” mes- sage that appears below the submit button or in a separate window. The overt intent of the step actions is to cause the system under test to behave as if the end users were performing routine business activities.
One of the major issues confronting the test case writer is the degree to which the test case is dependent on visual screen positioning of objects. Screen objects will change position (even by a pixel or two), change label, change label font, change color, or change some other property. If screen position validation is not a testing objective, then the test case author needs to avoid specifying the expected results
12.
13.
14.
15.
5.3 Test Cases 87
based on absolute screen position. If screen position is part of the software require- ment or specifi cation, then be alert to how different kinds of screen monitors and screen display resolution will probably distort that positioning. This issue arises more when using automated testing tools than when using manual scripting due to the precision that the tools bring to the expected versus actual result comparisons.
5.3.2.1 Expected versus Actual Results in a “Failed” Execution Step When the expected results match the actual results, the step is said to have “passed”
the test. When the expected results do not match the actual results, the step is said to have “failed” the test and further diagnostic analysis is required. There are at least two possible causes of a “failed” step: the expected results are incorrect or the actual results are incorrect. Incorrect expected results can come from several sources starting with the tester’s incorrect understanding of what the expected results should be and continuing to the developer’s incorrect understanding of what the expected results should be to the business expert’s incorrect understanding of what the expected results should be. If the tester, the developer, and the busi- ness expert all agree on the expected results, then the “failed” step has revealed a software defect. Who, when, and how the software defect is corrected is one of the topics in Chapter 12.
5.3.2.2 Validate Test Case Step Actions
Test case validation is a double check of the step actions and expected results with the software requirements, specifi cations, and business expert. The risk of not validating test cases is the increased likelihood that the test team will incorrectly report a number of “failed” step actions as software defects instead of step action design defects ( testing defects). Put yourself in the shoes of a developer who is under much pressure to deliver a complex application on a tight deadline. Your work has just been interrupted a third time today to attend a meeting with the test team to discuss another suspected major software defect … and the last two meetings were halted 15 min into the discussion when you pointed out that the tester’s expected results did not match the software specifi cations. How many more of these kinds of meetings will you attend before you tell your development manager that testing is a waste of time?
5.4 WRITING YOUR TEST PLAN AND TEST CASES