Principle 11. Testing is a creative and challenging task [12]
6. The columns in the decision table are transformed into test cases
The following example illustrates the application of this technique.
Suppose we have a specification for a module that allows a user to per-form a search for a character in an existing string. The specification states that the user must input the length of the string and the character to search for. If the string length is out-of-range an error message will appear. If the character appears in the string, its position will be reported. If the character is not in the string the message “not found” will be output.
The input conditions, or causes are as follows:
C1: Positive integer from 1 to 80 C2: Character to search for is in string The output conditions, or effects are:
E1: Integer out of range
E2: Position of character in string E3: Character not found
The rules or relationships can be described as follows:
If C1 and C2, then E2.
If C1 and not C2, then E3.
If not C1, then E1.
Based on the causes, effects, and their relationships, a cause-and-effect graph to represent this information is shown in Figure 4.5.
The next step is to develop a decision table. The decision table reflects the rules and the graph and shows the effects for all possible combinations of causes. Columns list each combination of causes, and each column represents a test case. Given n causes this could lead to a decision table with 2n entries, thus indicating a possible need for many test cases. In this example, since we have only two causes, the size and complexity of the decision table is not a big problem. However, with specifications hav-ing large numbers of causes and effects the size of the decision table can be large. Environmental constraints and unlikely combinations may re-duce the number of entries and subsequent test cases.
A decision table will have a row for each cause and each effect. The entries are a reflection of the rules and the entities in the cause and effect graph. Entries in the table can be represented by a “1” for a cause or effect that is present, a “0” represents the absence of a cause or effect, and a “—” indicates a “don’t care” value. A decision table for our simple example is shown in Table 4.3 where C1, C2, C3 represent the causes, E1, E2, E3 the effects, and columns T1, T2, T3 the test cases.
The tester can use the decision table to consider combinations of inputs to generate the actual tests. In this example, three test cases are called for. If the existing string is “abcde,” then possible tests are the following:
C1
C2
E1
E2
E3
F I G . 4 . 5
Cause-and-effect graph for the character search example.
Inputs Length Character to search for Outputs
T1 5 c 3
T2 5 w Not found
T3 90 Integer out of range
One advantage of this method is that development of the rules and the graph from the specification allows a thorough inspection of the spec-ification. Any omissions, inaccuracies, or inconsistencies are likely to be detected. Other advantages come from exercising combinations of test data that may not be considered using other black box testing techniques.
The major problem is developing a graph and decision table when there are many causes and effects to consider. A possible solution to this is to decompose a complex specification into lower-level, simpler components and develop cause-and-effect graphs and decision tables for these.
Myers has a detailed description of this technique with examples [1].
Beizer [5] and Roper [9] also have discussions of this technique. Again, the possible complexity of the graphs and tables make it apparent that tool support is necessary for these time-consuming tasks. Although an effective set of test cases can be derived, some testers believe that equiv-alence class partitioning—if performed in a careful and systematic way—
T 1 T 2 T 3
C1 1 1 0
C2 1 0 —
E1 0 0 1
E2 1 0 0
E3 0 1 0
T A B L E 4 . 3
Decision table for character search example.
will generate a good set of test cases, and may make more effective use of a tester’s time.
4 . 8 . 2 S t a t e T r a n s i t i o n T e s t i n g
State transition testing is useful for both procedural and object-oriented development. It is based on the concepts of states and finite-state ma-chines, and allows the tester to view the developing software in term of its states, transitions between states, and the inputs and events that trigger state changes. This view gives the tester an additional opportunity to develop test cases to detect defects that may not be revealed using the input/output condition as well as cause-and-effect views presented by equivalence class partitioning and cause-and-effect graphing. Some useful definitions related to state concepts are as follows:
A state is an internal configuration of a system or component. It is defined in terms of the values assumed at a particular time for the variables that characterize the system or component.
A finite-state machine is an abstract machine that can be represented by a state graph having a finite number of states and a finite number of transitions between states.
During the specification phase a state transition graph (STG) may be generated for the system as a whole and/or specific modules. In object-oriented development the graph may be called a state chart. STG/state
charts are useful models of software (object) behavior. STG/state charts are commonly depicted by a set of nodes (circles, ovals, rounded rectan-gles) which represent states. These usually will have a name or number to identify the state. A set of arrows between nodes indicate what inputs or events will cause a transition or change between the two linked states.
Outputs/actions occurring with a state transition are also depicted on a link or arrow. A simple state transition diagram is shown in Figure 4.6.
S1 and S2 are the two states of interest. The black dot represents a pointer to the initial state from outside the machine. Many STGs also have “er-ror” states and “done” states, the latter to indicate a final state for the system. The arrows display inputs/actions that cause the state transfor-mations in the arrow directions. For example, the transition from S1 to S2 occurs with input, or event B. Action 3 occurs as part of this state transition. This is represented by the symbol “B/act3.”
It is often useful to attach to the STG the system or component vari-ables that are affected by state transitions. This is valuable information for the tester as we will see in subsequent paragraphs.
For large systems and system components, state transition graphs can become very complex. Developers can nest them to represent different levels of abstraction. This approach allows the STG developer to group a set of related states together to form an encapsulated state that can be represented as a single entity on the original STG. The STG developer must ensure that this new state has the proper connections to the un-changed states from the original STG. Another way to simplify the STG is to use a state table representation which may be more concise. A state table for the STG in Figure 4.6 is shown in Table 4.4.
The state table lists the inputs or events that cause state transitions.
For each state and each input the next state and action taken are listed.
Therefore, the tester can consider each entity as a representation of a state transition.
As testers we are interested in using an existing STG as an aid to designing effective tests. Therefore this text will not present a discussion of development and evaluation criteria for STGs. We will assume that the STGs have been prepared by developers or analysts as a part of the re-quirements specification. The STGs should be subject to a formal inspec-tion when the requirement/specificainspec-tion is reviewed. This step is required for organization assessed at TMM level 3 and higher. It is essential that
A/act-1
S1 S2
B/act-3
A/act-2
C/act-5 B/act-4
C/act-6 F I G . 4 . 6
Simple state transition graph.
testers be present at the reviews. From the tester’s view point the review should ensure that (i) the proper number of states are represented, (ii) each state transition (input/output/action) is correct, (iii) equivalent states are identified, and (iv) unreachable and dead states are identified.
Unreachable states are those that no input sequence will reach, and may indicate missing transitions. Dead states are those that once entered can-not be exited. In rare cases a dead state is legitimate, for example, in software that controls a destructible device.
After the STG has been reviewed formally the tester should plan ap-propriate test cases. An STG has similarities to a control flow graph in that it has paths, or successions of transitions, caused by a sequence of inputs. Coverage of all paths does not guarantee complete testing and may not be practical. A simple approach might be to develop tests that insure that all states are entered. A more practical and systematic ap-proach suggested by Marik consists of testing every possible state tran-sition [10]. For the simple state machine in Figure 4.6 and Table 4.4 the transitions to be tested are:
Input A in S1 Input A in S2 Input B in S1 Input B in S2 Input C in S1 Input C in S2
The transition sequence requires the tester to describe the exact inputs for each test as the next step. For example the inputs in the above
tran-S 1 S 2 Inputs
Input A S1 (act-1) S2 (act-2)
Input B S2 (act-3) S1 (act-4)
Input C S2 (act-5) S2 (act-6)
T A B L E 4 . 4
A state table for the machine in Figure 4.6.
sitions might be a command, a menu item, a signal from a device or a button that is pushed. In each case an exact value is required, for example, the command might be “read,” the signal might be “hot” or the button might be “off.” The exact sequence of inputs must also be described, as well as the expected sequence of state changes, and actions. Providing these details makes state-based tests easier to execute, interpret, and main-tain. In addition, it is best to design each test specification so that the test begins in the start state, covers intermediate states, and returns to the start state. Finally, while the tests are being executed it is very useful for the tester to have software probes that report the current state (defining a state variable may be necessary) and the incoming event. Making state-related variables visible during each transition is also useful. All of these probes allow the tester to monitor the tests and detect incorrect transitions and any discrepancies in intermediate results.
For some STGs it may be possible that a single test case specification sequence could use (exercise) all of the transitions. There is a difference of opinion as to whether this is a good approach [5,10]. In most cases it is advisable to develop a test case specification that exercises many tran-sitions, especially those that look complex, may not have been tried be-fore, or that look ambiguous or unreachable. In this way more defects in the software may be revealed. For further exploration of state-based test-ing the followtest-ing references are suggested, [5,10,11].
4 . 8 . 3 E r r o r G u e s s i n g
Designing test cases using the error guessing approach is based on the tester’s/developer’s past experience with code similar to the
code-under-test, and their intuition as to where defects may lurk in the code. Code similarities may extend to the structure of the code, its domain, the design approach used, its complexity, and other factors. The tester/developer is sometimes able to make an educated “guess” as to which types of defects may be present and design test cases to reveal them. Some examples of obvious types of defects to test for are cases where there is a possible division by zero, where there are a number of pointers that are manipu-lated, or conditions around array boundaries. Error guessing is an ad hoc approach to test design in most cases. However, if defect data for similar code or past releases of the code has been carefully recorded, the defect types classified, and failure symptoms due to the defects carefully noted, this approach can have some structure and value. Such data would be available to testers in a TMM level 4 organization.
4 . 9
B l a c k B o x T e s t i n g a n d C o m m e r c i a l O f f - t h e - S h e l f ( C O T S ) C o m p o n e n t sAs software development evolves into an engineering discipline, the reuse of software components will play an increasingly important role. Reuse of components means that developers need not reinvent the wheel; instead they can reuse an existing software component with the required func-tionality. The reusable component may come from a code reuse library within their organization or, as is most likely, from an outside vendor who specializes in the development of specific types of software compo-nents. Components produced by vendor organizations are known as com-mercial off-the-shelf, or COTS, components. The following data illustrate the growing usage of COTS components. In 1997, approximately 25%
of the component portfolio of a typical corporation consisted of COTS components. Estimates for 1998 were about 28% and during the next several years the number may rise to 40% [12].
Using COTS components can save time and money. However, the COTS component must be evaluated before becoming a part of a devel-oping system. This means that the functionality, correctness, and reli-ability of the component must be established. In addition, its suitreli-ability for the application must be determined, and any unwanted functionality
must be identified and addressed by the developers. Testing is one process that is not eliminated when COTS components are used for development!
When a COTS component is purchased from a vendor it is basically a black box. It can range in size from a few lines of code, for example, a device driver, to thousands of lines of code, as in a telecommunication subsystem. It most cases, no source code is available, and if it is, it is very expensive to purchase. The buyer usually receives an executable version of the component, a description of its functionality, and perhaps a state-ment of how it was tested. In some cases if the component has been widely adapted, a statement of reliability will also be included. With this limited information, the developers and testers must make a decision on whether or not to use the component. Since the view is mainly as a black box, some of the techniques discussed in this chapter are applicable for testing the COTS components.
If the COTS component is small in size, and a specification of its inputs/outputs and functionality is available, then equivalence class par-titioning and boundary value analysis may be useful for detecting defects and establishing component behavior. The tester should also use this ap-proach for identifying any unwanted or unexpected functionality or side effects that could have a detrimental effect on the application. Assertions, which are logic statements that describe correct program behavior, are also useful for assessing COTS behavior [13]. They can be associated with program components, and monitored for violations using assertion sup-port tools. Such supsup-port tools are discussed in Chapter 14.
Large-sized COTS components may be better served by using random or statistical testing guided by usage profiles.
Usage profiles are characterizations of the population of intended uses of the software in its intended environment [14].
More information about usage profiles will be provided in Chapter 12.
Other approaches to testing COTS components have been described, for example, by J. Voas [15]. These are not strictly black box in nature.
As in the testing of newly developing software, the testing of COTS components requires the development of test cases, test oracles, and aux-iliary code called a test harness (described in Chapter 6). In the case of COTS components, additional code, called glue software, must be devel-oped to bind the COTS component to other modules for smooth system
functioning. This glue software must also be tested. All of these activities add to the costs of reuse and must be considered when project plans are developed. Researchers are continually working on issues related to test-ing and certification of COTS components.
Certification refers to third-party assurance that a product (in our case a software product), process, or service meets a specific set of requirements.
Some articles on these topics can be found in References 13, 15, and 16.
4 . 1 0
B l a c k B o x M e t h o d s a n d T M M L e v e l 2 M a t u r i t y G o a l sSince the TMM is the guiding framework for this text, an association between chapter material and TMM levels will be a feature at the end of most of the book chapters. Activities, tasks, and responsibilities (ATRs) for the three critical groups (developers/testers, managers, users/clients) that bind the chapter material to a given set of maturity goals will be described.
In this chapter we are particularly interested in exploring the nature of the association between black box testing methods and the TMM level 2 maturity goal that reads as follows: “Institutionalize basic testing tech-niques and methods.” The strong association of Chapter 4 material with this goal arises from the fact that black box testing methods are consid-ered to be a part of this basic group of techniques and methods. The methods that were discussed in this chapter provide a systematic approach to test case design that (i) has roots in testing theory, (ii) supports a be-havioral view of the software, (iii) is applicable to most levels of testing, (iv) provides sets of practical repeatable procedures for an organization to adapt and reuse, and (v) has associated notations that link testing with other development activities.
An organization that wishes to improve its testing process needs to adopt and consistently apply black box testing methods to software that is being developed and/or maintained. This needs to be accomplished early in the maturity level framework. However, not every black box (or white box) method needs to be used for every project. In fact, some black box methods require more in-depth training and sophisticated tools that the organization may not be ready for at TMM level 2. As the
organi-zation grows in maturity it can adapt those black box methods that meet both its needs and the needs of its clients.
Black box methods have ties to the other maturity goals at TMM level 2. For example, use of the methods must be prescribed in the testing goals/policy statements, applied in test plans, and included in training material. Use of black box methods also supports achievement of higher-level TMM maturity goals, for example, integration of testing into the life cycle, software quality evaluation, and quality control.
We now have determined that there are associations between black box testing methods and TMM maturity goals at level 2. Now we need to look at how the key players in the testing process, developers/testers, managers and users/clients can support achievement of this goal. For the remainder of this discussion the reader should keep in mind that at TMM level 2 there is no requirement for an independent test organization. De-velopers will often take on a dual role of both programmer and tester. As a consequence, note that the ATRs associated with this TMM level refer to a critical view called developer/tester.
As a developer/tester, you can play an important role in achieving satisfaction of maturity goals at TMM level 2. Your activities, tasks, and responsibilities include attending classes and training sessions, reading materials, acquiring tools, working with knowledgeable colleagues, and gaining hands-on experience in the application of the black box testing methods. After you master the black box methods in this chapter, you need to insure that a balance of these approaches is used for test design.
The black box methods you select as a tester depend on the type of soft-ware being tested and the conditions and constraints on testing. For ex-ample, if the software-under-test is event or command-driven, then state-based testing should be one of your choices. Object-oriented systems also benefit from this technique. However individual methods in an ob-ject/class may also benefit from use of equivalence class partitioning and boundary value analysis. The latter also works well for standard pro-cessing systems that are procedural in nature and have well-defined input domains. Error guessing may be used to augment any of the other black box approaches. As a developer/tester you need to insure that white box testing methods are also applied especially for unit and integration test, and that both black and white-box based test cases are included with the test plan. White box testing methods will be described in the next chapter.