TOWARDS A MODEL-CENTRIC SOFTWARE TESTING LIFE CYCLE FOR EARLY AND CONSISTENT TESTING ACTIVITIES

The constant improvement of the available computing power makes the execution of more and more complex tasks possible these days. These quality requirements are achieved by using a mutation-based analysis of the identified test cases, which builds on the model base.

Problem Statement and Research Questions

How can better adaptability and context sensitivity of test generation based on coverage criteria and model-level measures reduce overall complexity. How can mutation-based testing concepts together with a mechanism for test case execution applied at the model level improve the quality of automatically generated test suites.

Concepts and Objectives

In addition to the application of the abstraction concept, we emphasize the advantages of breaking down the problem. The content related to details of the technical implementation is attributed to the second author.

Figure 1.2: Illustration of general approach (see objective 1)

Research Projects

The subsequent integration into a toolchain represented the completion of the desired work in the context of the project. Personal contribution: In addition to the conceptual work in the context of the ReTeC project, I have extended the Omni Model approach.

Supervised Thesis

In section 8.4, research work is elaborated in the context of the approach before a summary with an outlook is given (section 8.5). First, the case studies considered are discussed and how they are realized in the context of the Omni model (chapter 12).

Figure 3.1: Overall structure of the thesis

Model Transformations

If the source and target models are specified according to the same MM, an endogenous model transformation takes place. In the context of the A3F (see section 7.3), the Query View Transformation Operational (QVTO) plays a central role for horizontal endogenous M2MTs, making missing semantics explicit within the transformation.

Figure 4.2: Relation of basic concepts in M2MT (based on [56])

Model-Driven Architecture

Fundamentals of Testing

Here, in particular, the testability of the underlying test base is challenged and, if necessary, this causes a change request for the test base. A central role of the test design phase is to end up with a focused set of tests for the targeted parts of the SUT.

Figure 5.1: The common software testing life cycle

Standardization

ISO/IEC, the last part of the standard, elaborates on a specific testing technique, namely keyword testing. The final activities deal with the documentation and communication of the developed artifacts within the framework of the test plan (TP7 - TP9).

Figure 5.2: Important standards in the context of Software Testing and Quality Assur- Assur-ance

Test Design Techniques

However, once the set of mutants P′ is created, the set of test cases represents the central artifact of the mutation testing process. Here, all the test cases included are run against each of the previously created mutantsP′.

Figure 5.5: Subsumption hierarchy of structural and data flow criteria (based on [19]

Model-Based Testing

Scenarios of Model-Based Testing

The model itself is not derived from the requirements base, but is semi-automatically derived from the target code. The logical next step from the previous scenario to automated test verdicts is provided by manual modeling rather than deriving it from the code base (see Figure 5.8c).

Model-Based Testing in Practice

The original set of test cases can be optimized and handed over to the next stage of MCSTLC. This phase of MCSTLC deals with the execution/interpretation of model artifacts from the integrated model foundation.

Figure 6.1: The Model-Centric Software Testing Life Cycle

Running Example: Ceiling Speed Monitor

System Structure and Behavior Metamodels

A complete mapping of the concepts covered in the SYSML specification is possible, but not intended in the context of this thesis. An alternative modeling language for the system model parts of the Omni Model is UML.

Test Metamodels

U2TPDataPartitionInstance Concept for the specification of an instance of the defined U2TPDataPartition, allowing the derivation of concrete test cases from a U2TPBehaviourModel. This is especially important when test cases are run against the same instance of the system.

Integration Metamodel

IMtreeEdge (im_tt) Abstract concept for a connector of the tree IMPartOf (im_po) Concept reflecting the inclusion of the tree node. AnIMTraceElement allows a model element of the integration model to specify connections to artifacts of other domain-specific models.

Figure 7.11: Integration Model (IM) metamodel

Analysis-Specific Models

Execution Graph++ Metamodel

EGPPElement (el) is the most abstract concept of the Execution Graph++ Meta-model (EGPPMM), to determine the domain of the concept. EGPPFinalNode (fn) is a special node that represents the end of the control flow captured by the graph structure.

Figure 7.18: Execution Graph++ (EGPP) metamodel

Model to Model Transformations

A test modeling of this component is shown on the right side of the figure. A representation of the EGPP test model for the system test level is shown on the left side of the figure.

Figure 7.22: Pattern for EGPP instance transformed from structure and behavior System Model artifacts

Architecture And Analysis Framework

Framework Architecture

The combination of such metamodels together with the integration model represents the Omni Model approach in the context of A3F. The aforementioned REST API is used here, which guarantees the decoupling of the CASE tool and the framework.

Figure 7.27: Internal architecture of the A3F

Working with the Framework

Future extensions - In principle, further useful extensions are conceivable, but these were not realized during the prototypical implementation. In our case, the processed data from the analyzesea_db_loader anddata_transformer can be examined mainly through the Model Elements view and the Analysis Results view.

Related Work

In the context of our approach, the missing/false semantics of the original artifact can be enriched so that the characteristics of flexibility and early applicability are preserved. This causes problems especially in the context of creating specific model sections (see Chapter 8), but is explicitly covered by our implementation in the form of EGPP.

Conclusions and Outlook

Test Focus Specification

In combination with the Aspect Specification Language, the name of the Aspect serves as a reference to the relevant definition. Depending on the type of the underlying aspect (scoped vs. set), different operators can be used to specify the constraint.

Excerpt of the Omni Model

An example application of the aspect constraint language can be seen in the flow of the presentation of the Scoping Test Pattern in section 8.2. Further, there are no limits on the number of integration levels in Test Models.

Test Model Scoping

Integration Model based Filtering

The evaluation of Aspect Specification Snippets starts at the root node of the tree structure. Based on this, the model elements in the tree structure specify concrete values for the respective Aspects.

Test Model Mapping and Reconstruction

The set of possible paths is calculated in advance based on the full-fledged Test Model. After applying the Unit and Integration test levels, the Filtered Test Model is obtained, which is located on the right side of the figure.

Test Model Split and Enrichment

On the left side of the figure, the starting point is given by the filtered test model. In particular, it is possible to specify the information for individual model elements of the Test Model.

Figure 8.5: Algorithmic approach for Test Model Mapping and Reconstruction applied to the Running Example

Technical Realization within A3F

Nodes with rounded corners are EGPPNodes, containing atomic declarations of the resulting test cases. In Figure 8.7 these remaining parts of the graph are represented by the nodes labeled "|other paths|" or.

Related Work

Through an upstream analysis of the Test Model, similarity values (fitness function) can be derived and annotated to the model in the form of Aspects. In general, the literature research carried out in the area of test case selection, prioritization and reduction could not identify a matching approach.

Conclusions and Outlook

Expert’s Configuration Parameters

The choice of a specific threshold value must be based on empirical values. However, adjustment of the threshold value by the expert during later iterations is possible.

Machine-Interpretable Mutation Analysis Results

The expert defines a threshold value for the test case quality, which is based on his Mutation Score (see section 11.2.3). This is necessary because the application of a dynamic threshold in early iterations of this feedback cycle would have negative effects on the resulting set of test cases, which is due to missing knowledge of the average quality of the initial test suite.

Excerpt of the Omni Model

The mentioned criteria are specified on the domain-specific model artifacts involved in the Omni model and evaluated in the context of section 9.2.1. Furthermore, the expert determines per domain which coverage metric should be applied and what threshold the metric should exceed.

Test Suite Generation

Artifact and Feedback Evaluation
Test Case Generation Metric Adaption
Data Flow Analysis-Based Test Case Generation
Feedback-Oriented Test Suite Creation

During subsequent MCSTLC iterations (see 2 in Figure 9.1), feedback is further incorporated. However, in the first iteration of test case generation (see 1 in Figure 9.1), the mentioned activities are not useful and therefore omitted.

Figure 9.2: Exit Criteria specification via multi-domain metric evaluation

Technical Realization within A3F

The parameter integration models and test models were obtained from the results of the analysis of the analyses, which are detailed in Figure 8.8. Only the autotip parameter, which evaluates the Mutation Analysis results of the previous A3F analysis, is a special feature.

Figure 9.6: Analyses dependency graph for the egpp_path_generation analysis The parameters integrationmodel and testmodels are taken from the analysis results of the analyses, which are detailed in figure 8.8

Related Work

The criteria used in test case generation can be evaluated in combination to increase the effectiveness of the resulting test suite [73]. Such a combination of criteria is included in our approach, making the quality of the resulting test suite more transparent and resilient.

Conclusions and Outlook

Execution Graph++ Characteristics Analysis

Under this category are subsumed EGPP cases that are advanced with respect to the structuring model elements, but still have fragmented control flow information in the context of the behavior descriptive model components. However, with respect to the included detailed information in the form of code fragments of our minimal language presented in code fragment 7.1, these graphs do not show any meaningful information.

Abstract Test Execution Engine Configuration Parameters

This includes EGPP instances that do not contradict any rules of the underlying metamodel down to their structural nature and the behavioral descriptions they contain in the form of graphs. The concrete characteristics of the configuration parameters are discussed in the context of the technical realization in the A3F (see section 10.3).

Excerpt of the Omni Model

TheControl Flow-Aware ATE mainly relies on the structural information of the graph at hand and therefore provides configuration parameters that affect this type of analysis. Specifically, the expert can influence the thresholds of the internally used search algorithms, such as breadth-first and depth-first search, or define termination criteria that make the execution more efficient.

The Abstract Test Execution Approach

Digression into a Control Flow-Aware ATE Approach
Overall Concept for Data Flow-Aware Abstract Test Execution
Omni Model-Based Path Merging
Evaluation of Path Space
Evaluation Result to Test Verdict Mapping
Result Selection and Test Report Generation

TI represents the set of Test Pattern pattern elements included in a considered interval. The order of System Model and test case model elements is preserved.

Figure 10.2: Omni Model foundation for the ATE approaches

Technical Realization within A3F

The top part of the report contains general information about the input artifacts and the determined result. If detailed information is desired, the middle section of the report presents the continuous model elements of the test case (lines 8-11) and the model elements of the system (lines 15-19).

Figure 10.6: Analyses dependency graph for the egpp_excute analysis

Related Work

In the context of our approach, the components of the test cases correspond to the possible states of the system, but using structural and data flow-based relationships. An examination of the system is performed by considering all possible execution paths, implemented by different approaches in the context of Symbolic Execution.

Conclusions and Outlook

The last challenge related to the presented GOE concepts is the scalability of the approach. Further work in the context of the proposed concepts includes, for example, a more specific preprocessing of the available model information.