9.2 Test Suite Generation
9.2.2 Test Case Generation Metric Adaption
Based on the information obtained from the analysis of the Scoped Test Models or the evaluation of the feedback from the Mutation Analysis, a suitable coverage metric can be determined for the subsequent test case generation. To understand the procedure
explained in the further course, a short preview of the realization of test case generation through data flow analysis is given here. Whenever we talk about a coverage criterion that is used for test case generation, we implicitly refer to a corresponding data flow analysis of the MAF that produces a test suite that covers the associated criterion best.
In particular, this can be done for all the criteria shown in figure 5.5. If each of the criteria specifies a corresponding data flow analysis, the decision space for the criteria adaptation process is obtained, which is shown schematically in figure 9.3.
Node Coverage Edge Coverage Edge-Pair Coverage Prime Path
Coverage
Complete Round Trip Coverage
Simple Round Trip Coverage Complete Path
Coverage
All-du Paths Coverage
All-Uses Coverage
All-Defs Coverage All-C-Uses/
Some-P-Uses
All-P-Uses/
Some-C-Uses
All-P-Uses Coverage Data flow
Analysis Data flow
Analysis
Data flow Analysis Data flow
Analysis Data flow
Analysis Data flow
Analysis
Data flow Analysis Data flow
Analysis
Data flow Analysis Data flow
Analysis Data flow
Analysis Data flow
Analysis Data flow
Analysis
Figure 9.3: Coverage Criteria Subsumption Hierarchy including connected data flow analyses (based on figure 5.5)
The entry points for test case generation determined in the previous section usually represent the leaves of the illustrated tree structure. An exception can be the start cri- terion manually specified by the expert since it is possible to select any criterion of the tree structure. Such aStart Criterionmay quickly lead to a test suite that meets the spec- ifiedExit Criteria, but might be not optimal concerning the amount and complexity of included test cases.
Therefore, when selecting the Start Criterion by analyzing the relevant Scoped Test Model, the Node Coverage criterion is usually set as default. This criterion is one of the weakest criteria since a high degree of coverage can usually be achieved by a rela- tively small number of paths through the EGPP Test Model. However, it is not possible to draw any conclusions about the respective coverage of the connected EGPP System Model. This in turn is one of the advantages of having these models created by differ- ent experts, since the first differences between the target and actual system are revealed early. Once a Start Criterion is determined, the data flow analysis is applied, which is discussed in the following section.
In the context of further iterations of the MCSTLC (see 2 of figure 9.1) the feedback is furthermore included. In this case, the hierarchy introduced above comes into effect.
However, before the algorithm for the evolution of the applied coverage metric is dis- cussed, another quantity is introduced, which is used for decision-making. This mea- sure considers the evolution of theExit Criteriaover the last two iterations of the MC- STLC. In particular, a so-calledmetricsTrendis determined, which indicates whether
Iteration
Domain
system requirements safety 0 0.2(edge cov.) -(node cov.) 0.3(node cov.) 1 0.35(edge cov.) 0.9(node cov.) 0.32(node cov.)
δ +0.15 +0.9 +0.02
Table 9.1: ExampleExit Criteriaevaluation to determine themetricsTrend the last adjustment of the Coverage Criterion for test case generation has lead to an improvement. For this purpose, the change of each atomic criterion involved in the Exit Criteriais examined and the median of the changes is further processed asmet- ricsTrend. To show this by a concrete example, the following evaluations of theExit Criteriaof two past iterations of the MCSTLC are utilized.
As the table shows, the values of theExit Criteriahave improved throughout. For the modeling domainssystemandsafetyby+0.15and+0.02respectively. For the model- ing domainrequirementseven by+0.9, which is since after the iteration0theExit Cri- teriawas adjusted by including the respective domain. This is the reason why instead of the mean value (+0.357) the median (+0.15) is chosen to determine themetricsTrend, because it better represents the actual change in such a situation.
To select a better criterion for the following step, three ranges are defined on the scale of themetricsTrend variable (from-1 to 1), which are used to determine the future criterion using different methods (see figure 9.4). The values (-0.02and+0.02), which determine the three ranges, are only to create a certain blur around the neutral value0 and do not have any special meaning.
searchForNextCriteriaBelowOrReverseLastStep searchForNextCriteriaSideways searchForNextCriteriaAbove 1.
Last Step Criteria Current Criteria
Future Criteria
1.
2.
3.
2.
1.
2.
Figure 9.4: Ways of determining the future Coverage Criterion applied
On the one hand, the range from-1 to-0.02is used as an indicator for poor perfor- mance in the previous step. In this case, the future criterion is selected by the method searchForNextCriteriaBelowOrReverseLastStep. The method searches for a new criterion at a lower level, giving preference to criteria that were not used in previous steps. However, if there is no alternative, the criterion of the previous step can be rese- lected, which stops the adaptation process.
The second range covers the values between-0.02 and+0.02, which reflects a weak increase/decrease in the value of the metric. Such values are seen as an indication of stagnation of the increase/decrease, so in this case, the methodsearchForNextCrite-
riaSidewaysprovides the future criterion. As you can see in the graph, it is possible to determine criteria on the same level by utilizing lower as well as higher-level elements of the structure.
The third and last area ranges from+0.02to+1and reflects very positive development in the past choice of the coverage criterion. In such a situation, we try to carry on the positive trend and determine the future metrics by thesearchForNextCriteriaAbove method. This method moves up in the structure but excludes criteria that have already been applied in previous steps. If no new criteria are found, the process ends, because either the top-level of the structure has been reached, or all higher criteria have given worse values for theExit Criteria.
Regardless of the method that produced the set of future criteria, a final criterion is randomly selected. For our Running Example, we assume that theNode Coveragecrite- rion has initially (Iteration 0) been selected, and in the current step (Iteration 1) the Edge Coveragecriterion is evaluated. Based on themetricsTrenddetermined in table 9.1 +0.15, the algorithm uses the methodsearchForNextCriteriaAboveto determine the set of future criteria. If we search for new criteria in figure 9.3 according to this method starting with theEdge Coveragecriterion, we obtain the criteriaAll-P-Uses Coverage,All- Uses Coverage, andEdge-Pair Coverage. The criterion selected from this set is used in the next step of the processing chain to perform the connected data flow analysis. Details about this processing step are conducted in the following section.