a routine that has been adequately tested in one environment may not have been adequately tested to work in another environment, the

Principle 11. Testing is a creative and challenging task [12]

1. a routine that has been adequately tested in one environment may not have been adequately tested to work in another environment, the

Axiom 5 says that semantic closeness is not sufficient to imply that two programs should be tested in the same way. Given this definition of shape, Axiom 6 says that even the syntactic closeness of two programs is not strong enough reason to imply they should be tested in the same way.

7 . A n t i d e c o m p o s i t i o n P r o p e r t y

“There is a program P and a component Q such that T is adequate for P, T⬘ is the set of vectors of values that variables can assume on entrance to Q for some t in T, and T⬘ is not adequate for Q.” This axiom states that although an encompassing program has been adequately tested, it does not follow that each of its components parts has been properly tested. Implications for this axiom are:

1. a routine that has been adequately tested in one environment may

the fact that all instances of an identifier, let us say a in Q have been replaced in P by an identifier, let us say b, where “b” does not occur in Q, or if there is a set of such renamed identifiers.” This axiom simply says that an inessential change in a program such as changing the names of the variables should not change the nature of the test data that are needed to adequately test the program.

1 0 . C o m p l e x i t y P r o p e r t y

“For every n, there is a program P such that P is adequately tested by a size n test set, but not by any size n ⳮ 1 test set.” This means that for every program, there are other programs that require more testing.

1 1 . S t a t e m e n t C o v e r a g e P r o p e r t y

“If the test set T is adequate for P, then T causes every executable ment of P to be executed.” Ensuring that their test set executed all state-ments in a program is a minimum coverage goal for a tester. A tester soon realizes that if some portion of the program has never been executed, then that portion could contain defects: it could be totally in error and be working improperly. Testing would not be able to detect any defects in this portion of the code. However, this axiom implies that a tester needs to be able to determine which statements of a program are executable. It is possible that not all of program statements are executable. Unfortu-nately, there is no algorithm to support the tester in the latter task, but Weyuker believes that developers/testers are quite good at determining whether or not code is, or is not, executable [2]. Issues relating to infea-sible (unexecutable) paths, statements, and branches have been discussed in Section 5.4.

The first eight axioms as described by Weyuker exposed weaknesses in several well-known program-based adequacy criteria. For example, both statement and branch adequacy criteria were found to fail in satis-fying several of the axioms including the applicability axiom. Some data flow adequacy criteria also failed to satisfy the applicability axiom. An additional three axioms/properties (shown here as 9–11) were added to the original set to provide an even stronger framework for evaluating test adequacy criteria. Weyuker meant for these axioms to be used as a tool by testers to understand the strengths and weaknesses of the criteria they

select. Note that each criterion has a place on the “subsumes” hierarchy as shown in Figure 5.5. A summary showing several criteria and eight of the axioms they satisfy, and fail to satisfy, is shown in Table 5.2 [11].

Weyuker’s goal for the research community is to eventually develop criteria that satisfy all of the axioms. Using these new criteria, testers will be able to have greater confidence that the code under test has been ad-equately tested. Until then testers will need to continue to use exiting criteria such as branch- and statement-based criteria. However, they should be aware of inherent weaknesses of each, and use combinations of criteria and different testing techniques to adequately test a program.

As a note to the reader, there are existing studies that discuss issues relating to when to apply specific test adequacy criteria, and whether satisfaction of stronger criteria does in fact reveal more defects than weaker criteria. The effectiveness of tests based on the criteria relative to those derived from the random test data generation approach is also dis-cussed. The studies are both theoretical and empirical in nature. The key researchers in this area include Frankl, Weyuker, Zhu, Parrish, and Gutjahr [1,2,11,13–17].

5 . 7

W h i t e B o x T e s t i n g M e t h o d s a n d t h e T M M

In the previous chapter we discussed various black box–based testing methods and established a connection between these methods and TMM level 2 maturity goals. A similar argument can be made for the important role of white box methods in the evolution of test process maturity and for their association with TMM maturity goals. As in the case of black box–based test design, white box methods also provide a systematic way of developing test cases. However, white box methods have a stronger theoretical foundation and are supported by test adequacy criteria that guide the development of test data and allow evaluation of testing goals when tests are subsequently executed. In addition, white box methods have adequate tool support, and they depend on the use of notations that allow a connection to other development activities, for example, design.

Use of black box as well as white box methods needs to be specified in the organizational testing goals/policy statement, and both need to be

S t a t e m e n t B r a n c h M u t a t i o n

Axiom 1 No No Yes

Axiom 2 Yes Yes Yes

Axiom 3 Yes Yes Yes

Axiom 4 Yes Yes Yes

Axiom 5 Yes Yes Yes

Axiom 6 Yes Yes Yes

Axiom 7 No No Yes

Axiom 8 No No Yes

T A B L E 5 . 2

Sample test data adequacy criteria and axiom satisfaction [11].

applied in test plans. Managers should ensure that testers are trained in the use of both for consistent application to all organizational projects as described in the TMM. The Activities/Tasks/Responsibilities (ATR’s) as-sociated with adapting and implementing black box methods also apply to white box methods as well.

Several white box testing techniques were discussed in this chapter, and once again the role of smart developer/tester is to choose among them.

In addition to the selection of properties to test based on test adequacy criteria, the developer/tester must also decide on the degree of coverage that is appropriate in each case. When making a choice among white box testing methods the tester must consider the nature of the software to be tested, resources available, and testing constraints and conditions. For example, a tester might choose to develop test designs using elements of control flow such as branches. In this same example, to insure coverage of compound conditions the tester may decide that multiple decision cov-erage is a wise testing goal. However, if the code is not complex, and is not mission, safety, or business critical, then simple branch coverage might be sufficient. The tester must also apply this reasoning to selection of a degree of coverage. For example, for a simple nonmission critical module, 85% branch coverage may prove to be a sufficient goal for the module if testing resources are tight. Remember that the higher up on the

ordering hierarchy you climb, and the greater the degree of coverage you specify, the more testing resources are likely to be required to achieve testing goals.

In all cases the tester should select a combination of strategies to develop test cases that includes both black box and white box approaches.

No one test design approach is guaranteed to reveal all defects, no matter what its proponents declare! Use of different testing strategies and meth-ods has the following benefits:

1. The tester is encouraged to view the developing software from several