5.2 Model-Based Testing
5.2.2 Model-Based Testing in Practice
When it comes to the practical application of MBT, the availability of appropriate tool- ing marks an essential point. Furthermore, adequate modeling languages are needed to make such technologies widely applicable and acceptable. Over the years, a very broad spectrum has been created in both areas.
From formal modeling languages to languages that emphasize visual presentation, there are a variety of solutions, often with application context-specific content. Specifi- cally, certain model artifacts of the existing modeling scope of widely used languages, such as UML, are often used for MBT purposes, e.g. statecharts or activity charts for the specification of the SUT’s behavior. [168]
At this point, another standard of the OMG, namely the UML Testing Profile (UTP), introduces concepts for carrying out MBT in the UML ecosystem. Technically, this stan- dard is therefore realized as a UML profile, which covers large parts of the process steps known from the STLC. [4]
Zander et al. [178] and Schieferdecker et al. [154] therefore extracted the following groups of concepts from the mentioned standard.
• Test architecture
This group covers the basic elements of a test and its relations. Essential parts are the so-calledtest configurations andtest components, which create the basic setup for later test execution.
• Test behavior
In this group, all elements regarding the dynamic aspects of test case execution are included. The most important concept is thetest caseitself, whose execution is specified bytest actions, and the outcome is represented bytest verdicts.
• Test data
Concepts of this group cover the data handling and specification, which conse- quently lifts the set of abstract test cases to a set of concrete test cases. This func- tionality is implemented by model elements such as thedata pool,data specification, data partitionordata items.
• Test planning
This group of model elements combines constructs of test analysis and test de- sign. Essentially, this introduces structure-giving model elements that, for exam- ple, combine test cases intotest sets, or the so-called test context, which logically combines multiple information in the context of the tests.
Altogether, this selection of modeling concepts covers almost all scenarios of MBT. The standard itself draws some possible usage scenarios, ranging fromthe specification and
design of test systemsover the creation of model-based test specifications for existing System Modelstomodeling test cases, test environments, and test data[4].
Apart from the modeling language perspective, an excerpt of available tooling is pre- sented throughout the rest of this section. First, a tool developed in an academic context by the TU Graz is mentioned, which combines model-based testing with mutation test- ing as presented in section 5.1.3. Here, either based on UML statemachines or Object Oriented Action Systems (OOASs), a set of test cases is derived using mutation op- erators. The tool called MoMuT represents a highly automated test case generation approach, which is still maintained and applied in a practical context. [115]
In contrast to the academic context, the commercial Conformiq Creator & Designer is briefly introduced. Conformiq supports various input formats. Possible modeling lan- guages for processing range from UML activity charts and statemachines, to propri- etary DSLs. The mentionedConformiq Creator & Designerrepresent an excerpt of a tool suite enabling testers to further execute the specified test cases. Besides, many other formats of other tools in the context of MBT are supported, e.g. Testing and Test Con- trol Notation (TTCN-3). [96]
The last tool presented here is called mbtSuite which is developed by AFRA GmbH, as well as sepp.med GmbH. In the context of the mentioned research project ReTeC [166] this tool was extended by functionality. The modeling basis is provided by UML activity charts and statemachines. Based on these, the test cases can be derived using different coverage criteria or even genetic or randomized algorithms. A connection to test management and execution tooling is also available. [156]
TOWARDS A MODEL-CENTRIC SOFTWARE TESTING LIFE
CYCLE
6
General Approach and Running Example
Targeting the challenges emerging from a steadily rising level of development com- plexity, previously identified in section 1.1, we further draw a more concrete and holis- tic picture of our approach sketched in section 1.2. Finally, the running example, used throughout the main chapters of our work, is introduced.
To overcome these challenges, an MCSTLC is developed based on the process steps of a classic STLC (see section 5.1.1). The goals are the greatest possible automation of the process steps and the applicability in the early phases of development. This enables the use of an MCSTLC for quality control as well as a tool during development to consistently get hints for improvement.
6.1 General Approach
Throughout the rest of this chapter the phases of the MCSTLC are explained (illustrated in figure 6.1). These phases were originally sketched in one of our conference contribu- tions [146]. The specified life cycle steps are related to the process steps of the original STLC, serving as the starting point. Besides, specific artifacts produced/consumed by the processing steps are addressed but not included in figure 6.1.
1. Model Creation/Modification marks the entry point of the MCSTLC and at the same time represents the interface to all disciplines in the development. The main tasks of this process step are given by theIntegrated Model Basis(also calledOmni Model) and the specification of configuration parameters like applicable coverage metrics, for the automated processing chain. Before executing the MCSTLC, all artifacts of the mod- eling domains involved in the development process have to be available and suitably integrated by theIntegration Model. The Integration Model links information across the different development domain-specific model artifacts. Further, this Integration Model can be extended by adding content to align the different models (for details see chap- ter 7). This model artifact is the basis for all further process steps to obtain a fully auto- mated iteration of the life cycle. However, at the end of each iteration of the MCSTLC manual adaptions are necessary.
Model Creation/
Modification Model-Based Test Case Management
Abstract Test Execution Integrated
Model Basis
Model-Based Test Suite Generation
Model-Based Mutation Analysis 2
1
3
5
4
Figure 6.1: The Model-Centric Software Testing Life Cycle
From a tester’s point of view, having the standard STLC in mind, activities from dif- ferent process steps are combined in the context of the MCSTLC phase. ThePlanning and Controlprocess step represents the counterpart, but there are also activities from Analysis and Design, as well asTest Closure, carried out in the current MCSTLC phase.
In particular, the creation of a Test Model, representing a part of the Integrated Model Basis and forming the basis for the automated derivation of test cases are conducted in this phase. Further, measures either taken after completion of test activities or at the end of an MCSTLC iteration, are summarized under this phase.
Overall, the initial and the final phase of the life cycle represents the interface between humans and the automated processing chain. In particular, the information provided by the processing chain, namely the Human-Interpretable Test Resultsand the Human- Interpretable Mutation Analysis Results, should provide the user with sufficient support to a targeted test phase in the early stages of development. The elaboration in chapter 7 provides details and illustrates the application with a practical example.
2. Model-based Test Case Management represents the first phase of the automated processing chain. In this phase, different activities of the classic STLC are combined.
Aspects ofTest PlanningandTest Analysisare realized by transforming the fully-blown Test Model into a strongly focused submodel. This is achieved by a multi-stage pro- cess, where model artifacts are pruned and projected to models of other development domains. All of the mentioned stages reside on the Integrated Model Basis or expert’s configuration input. Moreover, aspects ofTest Design and Test Implementation are ap- plied at this phase. Due to the focussedness of the resulting Test Model excerpt, con- cepts like prioritization, selection, and reduction of test suites may be realized. Notice, that the concepts mentioned above are carried out before the complexity exposes in the context of the concrete set of derived test cases.
To realize this functionality, all available models are taken into account to derive a suit- able portion of the original Test Model reflecting the tester’s mindset. The carefully linked model data from various development domains of the Integrated Model Basis significantly impacts the quality of subsequent processing results. Details are presented in chapter 8 with an example.
3. Model-based Test Suite Generation takes as input the resultingScoped Test Model and performs activities, which are encapsulated in theTest Designand theTest Implemen- tationprocess steps of a classic STLC. Specifically, properties of the underlying models like contained data flow information, are evaluated and assessed together with configu- ration parameters, like exit criteria or a start metric, specified in the introductory phase of the life cycle. To determine an appropriate algorithm for the test case generation, a subsumption hierarchy of coverage metrics and respective implementations is utilized as a decision basis. The resulting test cases reflect the current test focus and represent the starting point for later iterations of the life cycle.
However, if there are test cases derived from this Test Model instance or similar Scoped Test Models in the form ofMachine-Interpretable Mutation Analysis Results, an improved starting point for determining an appropriate test case extraction metric is given. The original set of test cases can be optimized and handed over to the next phase of the MCSTLC. The feedback from previous iterations enables theevaluation of the exit criteria, as in the classic STLC. Metrics on the model artifacts are evaluated after each iteration and provide a basis for termination of the cycle. Chapter 9 details and illustrates the application using an example.
4. Abstract Test Execution is the counterpart to the process activities Test Execu- tion and partlyTest Reportingof the standard STLC. This phase of the MCSTLC deals with the execution/interpretation of model artifacts from the integrated model basis.
Therefore, the quality-proven set of test cases is analyzed about the integration level of test cases and the level of abstraction of the targeted System Model part. Based on the results of this analysis, two different kinds of abstract test case execution mechanisms are available. In case of model artifacts including no data flow information, a structural conformance check between the input test cases and the System Model linked via the Integrated Model Basis is performed. In the case of more concrete data embedded in the System Model, model interpretation based on derived paths through the SUT is carried out. Both approaches have in common, the valuable Human-Interpretable Test Resultswhich further expand the expert’s knowledge during theModel Creation/Modi- ficationphase. This process step enables testers to perform a test execution on model artifacts, whereby insights to the SUD at an early stage of development are gained. The elaboration in chapter 10 provides details and illustrates the application with a practical example.
5. Model-based Mutation Analysis represents a new process step not included in a classic STLC. The goal is to evaluate the quality of the created test cases concerning
a certain set of faults and thereby improve the resulting test suites. In contrast to the previous manual static checks, this is achieved by a flexible realization of mutation analysis on models. I.e. the test cases created in theModel-based Test Suite Generation phase are executed against targeted mutants of the existing System Model. Using the Integrated Model Basis the evaluation is carried out mainly automatically and is based on concepts that can be assigned to theTest ExecutionandTest Reportingof the STLC.
The results are used in other phases of the MCSTLC. This includes theModel-based Test Suite Generationphase, which can adjust the metrics for generating test cases based on Machine-Interpretable Mutation Analysis Results. However, the results can be made avail- able to domain experts, i.e. Human-Interpretable Mutation Analysis Resultsare provided for theModel Creation/Modificationstep. Chapter 11 details and illustrates the applica- tion with a practical example.