Functional Testing
7.5 DETAILED WHITE BOX TESTING TECHNIQUES
The objective of white box testing is to verify the correctness of the software’s statements, code paths, conditions, loops, and data fl ow. This objective is often referred to as logic coverage. The prerequisites for white box testing include the software requirements, use cases, the executable program, its data, and its source code. If this topic sparks your interest, there are a number of good textbooks that can give you more details and the results of current academic research.[29–33]
The software developer normally does white box testing as an extension of code debugging activity early in the development cycle. Software developers usually focus on “making the code work” according to use case activities, which gives them the tendency to debug only the code they know works (selective logic test coverage).
Testers add value to developer debugging activity by helping the developer plan and debug more of the code than usual (more thorough logic test coverage). The more the logic test coverage you attain while debugging, the fewer the defects will be discovered later by other kinds of testing.
As Capers Jones concludes, the earlier these defects can be discovered, the less expensive they are to correct. The business motivation behind white box testing is ex- pected economies of testing. Much of the research you will fi nd in white box testing will relate to hypotheses, algorithms, and procedures that attempt to achieve 100% logic test coverage under certain, very controlled circumstances. Research has not yet produced a white box approach that guarantees 100% logic test coverage for all situations.
We will briefl y discuss six generic white box testing techniques.
7.5.1 Statement Coverage Technique
Statement coverage techniques focus on determining what percentage of the source code lines in a program has been executed. If there are 5,000 lines of 7.5 Detailed White Box Testing Techniques 107
source code in a program and you can determine manually or with a tool that you have executed 4,537 lines of source code, then you have achieved a 90.7%
statement coverage (exceptionally high for complex programs). The underlying hypothesis is that the higher the source code test coverage, the fewer will be the defects found later. The practical conclusion is that new, unexecuted code lines are just a software time bomb waiting to explode at the most inopportune mo- ment in production. The question is not “if” they will explode, only a question of “when.”
7.5.2 Branch (Simple Condition) Coverage Technique Branch coverage techniques focus on determining what percentage of the source code branch (true/false) logic in a program has been executed. If there are 1,500 source code branch points in a program and you can determine manually or with a tool that you have executed 1,145 branches (count true and false branch executions separately), then you have achieved a 76.3% branch point coverage (exceptionally high for complex programs). The underlying hypothesis is that the higher the branch point test coverage, the fewer will be the defects found later. The practical conclusion is that unexecuted branches, true or false, are just a software time bomb waiting to explode just like unexecuted statements. The co-occurrence of unexecuted branches with unexecuted statements is found most often in untested error recovery logic.
Because all computer logic conditions resolve to either true or false, you may wonder about the stipulation of simple conditions in this technique. Choosing to test the simple condition branches before the compound condition branches requires fewer initial test actions. All the developer needs to do is choose any test value that will force a true branch and any test value that will force a false branch, just two test values per branch.
7.5.3 Compound Condition Coverage Technique
The compound condition coverage technique extends the branch coverage technique to branches with compound conditions, ones that contain combinations of Boolean operators AND, OR, and NOT along with pairs of parentheses, possibly nested.
The challenge is to identify all the test value combinations that will evaluate to true and false for every simple condition and every Boolean combination of simple conditions. Truth tables are normally employed at this point in the white box test planning to enumerate all the possible condition permutations. Here is a compound condition containing two simple conditions.
(AGE 18 AND SEX M)
In order to identify the test data for 100% compound condition coverage, the following truth table is constructed.
From this truth table, you can see that a simple branch test that evaluated to the only true condition and one false condition would miss the two additional test data input combinations that would evaluate to false … and possibly discovering a logic error.
The truth table becomes larger and more complex with additional Boolean conditions, allowing more opportunity for missing logic coverage when using only simple branching coverage. Here is a compound condition with three simple conditions.
((AGE 18 AND SEX M) OR HEIGHT 6 ft)
There are standard programming language rules for the order of evaluating complex expressions such as this example. In order to identify the test data for 100% com- pound condition coverage, the previous example would expand in the following way using standard Boolean evaluation rules.
From this truth table, you can see that a simple branch test that evaluated to one true con- dition and one false condition would miss the six additional test data input combinations that would evaluate to both true and false … and possibly discovering a logic error.
7.5.4 Path Coverage Technique
Path coverage techniques focus on determining what percentage of the source code paths in a program have been traversed completely. There are a number of defi nitions of source code paths in the literature. In its simplest form, a source code path is the sequence of program statements from the fi rst executable statement through a
AGE SEX
ANDed condition 18 False
18 False 19 True 19 True
F False M True F False M True
F False F False F False T True
ANDed ORed
AGE SEX Condition HEIGHT Condition
18 False 18 False 19 True 19 True 18 False 18 False 19 True 19 True
F False M True F False M True F False M True F False M True
F False F False F False T True F False F False F False T True
16 False 16 False 16 False 16 False 17 True 17 True 17 True 17 True
F False F False F False T True T True T True T True T True
7.5 Detailed White Box Testing Techniques 109
series of arithmetic, replacement, input/output, branching, and looping statements to a return/stop/end/exit statement. If there are 943 different paths through a program and you can determine manually or with a tool that you have executed 766 of them, then you have achieved an 81.2% path coverage (exceptionally high for complex programs). The underlying hypothesis is that the higher the path test coverage, the fewer will be the defects found later. The practical conclusion is that unexecuted paths are just a time bomb waiting to explode at the most inopportune moment in production, even though all of the statements and branch points in the path have been individually tested.
7.5.5 Loop Coverage Technique
Loop coverage techniques focus on determining what percentage of the source code loops in a program has been cycled completely. There are several loop constructs in programming languages like DO, FOR, WHILE, and UNTIL. Some loops are a clever construct of IF statements and subsequent returns to these IF statements.
Regardless of the loop construct, the objective of loop testing is to force the program through the loop zero times, one time, n/2 times (where n is the terminal loop value), n times, and n 1 times. The one-time loop, the n/2-time loop, and n-time loop validate expected loop response at the beginning, middle, and end of the longest loop.
The zero-time and n 1-time loop test for unexpected and inappropriate looping conditions. We will see this end-point/mid-point testing tactic again in black box boundary value testing. If there are 732 loops in a program and you can determine manually or with a tool that you have executed 312, then you have achieved a 42.6%
loop coverage (about average for complex programs). The underlying hypothesis is that the higher the loop test coverage, the fewer will be the defects found later. The practical conclusion is that unexecuted loops and loops execute only within expected loop limits are just a time bomb waiting to explode at the most inopportune moment in production.
7.5.6 Intuition and Experience
This section is devoted to those software aspects that experienced testers have found to be troublesome areas of coding that the more formal debugging and testing techniques tend to miss. Here is a summary of those troublesome aspects.
7.5.6.1 Dates
Dates present three unique data challenges to the developer: valid formats, sort sequence, and calculations. The original mm/dd/yyyy format of dates remains problematic because for each mm (month) there is a specifi c range of values for dd (day) depending on yyyy (years like leap years). The validation of mm versus dd versus yyyy remains complex and grows more because we crossed the century boundary with computer systems (the infamous Y2K problem). The global nature
of the Internet has caused software to support a variety of date input formats. For example, the U.S. date standard is mm/dd/yyyy, whereas the British date standard is dd/mm/yyyy. Throw in the spelling or abbreviation of the month represented by mm, and the date input formats become a sizeable format coding challenge.
Date sort sequencing can also be a signifi cant application problem. If the application wishes to maintain its records in date order (either ascending or descending), the application must convert the input date to a yyyy/mm/dd sort format in order for the year to be the primary sort key, the month to be the secondary sort key, and the day to be the tertiary sort key.
An alternative approach to converting dates back and forth between sort and report formats is to calculate the number of days elapsed since some “anchor date”
in the past. If the data are really historical, then systems choose a very old anchor date like October 15, 1582, the advent of the Gregorian calendar and leap year. If the data just span a century or two, then January 1, 1800 might be an appropriate anchor date. Then, every date is sorted in two formats, a display format and number of days elapsed since the anchor date. The number of days elapsed then becomes a much simpler sort key to manage, just a positive integer. A secondary advantage of this calculation and storage of days elapsed is that you have the data structure suffi cient to readily calculate “date x days” or “date y days” or day of the week (Monday, Tuesday, and so forth).
All of these date calculations are possible sources of date processing error or failure. The tester can plan and execute most of the date testing by collaborating with the developers to identify and understand where dates and date calculations are imbedded in the software input/output design, fi le designs, and database designs.
7.5.6.2 Zero Length Anything
There are a number of programming situations that can produce zero data or counts or process steps. Here is a partial list.
arrays blank inputs divide by zero loops
pointers record lengths records (empty fi les) sorts
As with the dates, many of these zero items can be forced by pressing the “send”
key when you have supplied no data. Also, where calculations are known to exist, entering blank or zero for some or all of the input values also force these behaviors.
More specifi c testing might require collaboration with the developer to identify other potential zero-causing areas of the software.
7.5 Detailed White Box Testing Techniques 111
This is one of the areas in which your background and experience as a software developer will tend to guide you toward zero-causing areas of the software because that is where you would place it if you were coding it.
7.5.6.3 Buffer Overflow
Buffer overfl ow is a particularly insidious programming problem because it does not manifest itself until the software is relatively busy. The problem arises when an area in memory called a buffer is set aside to manage transient data like user input or report output or database records or internet packets. When that buffer area becomes full of data, something must happen to empty part of the buffer and accommodate more data. Some refi ll strategies dictate that a full buffer must be completely emptied before more data can be accommodated. Other refi ll strategies wrap the data around the buffer, accommodating more data while emptying what it has. So, you can be fi lling and emptying a buffer a number of different ways when the buffer overfl ows its area. This can make buffer overfl ow errors exceptionally challenging to repeat.
The symptoms of buffer overflow are as insidious as the event itself because when a buffer overflows, it overwrites adjacent areas in the computer’s memory.
If that adjacent area also holds data, then the data become corrupt for no apparent reason. If that adjacent area holds instructions, then the software begins bizarre behavior that might implicate perfectly correct code under normal conditions.
The buffer overfl ow problem takes on a higher level of testing criticality when you realize that many computer viruses are introduced by forcing buffer overfl ow into the software’s instructions.
One of the most effective ways to test for buffer overfl ow is to drive the software with a large volume of input data, force a large volume of internal fi le processing or database processing, and force a large volume of output for specifi c buffer areas.
As with the date calculation situation, the tester must collaborate with the devel- oper to identify the specifi c inputs, outputs, fi le processing, and database processing that rely on specifi c buffers that need to be tested. It may make sense to combine some of the buffer overfl ow white box testing with later performance black box testing described in Chapter 9 to get the volume of buffer traffi c necessary to cause overfl ow.
All of the white box testing techniques presented here attempt to increase the logical coverage of source code debugging/testing. The underlying premise is that unexecuted source code is one of the most frequent and predictable sources of defects.