4.3 Model-Driven Architecture
5.1.1 Fundamentals of Testing
In addition to the insight on themissing ability for showing the absence of defects, testing maynot be carried out exhaustively. Further, there is a strong need for anearly application and tight focus of the testing activities, in order to reduce the overall costs for testing.
Especially in late phases of testing, e.g. in a pre-release phase, a tester should always keep in mind the phenomena ofdefect clustering, which says that most of the remaining defects are encapsulated in a small number of modules of the SUD. In a row with the pesticide paradox, which complains about the repeated application of the same test suite and its expressiveness over time, these principles are based on experiences over the past decades of testing software. Finally, testing is heavily context-dependent and therefore needs to be adapted according to requirements imposed by the development context as well as regulatory instances. Further continuing the thought of the previous section, testing is not a pure verification nor a pure validation technique and therefore does not give any causal relation between the absence of faults in the product and its usability for the customer. [80]
All the presented insights on testing commonly lead to a specific variant of how testing is carried out during development. However, all different variants follow the same basic structure, shown in figure 5.1 and commonly known as the Software Testing Life Cycle (STLC). Although the figure shows a linear process, the linearity is not a necessity, i.e. certain steps of the process may either overlap or be iterated during the process.
Throughout the following sections, we detail activities included in the process steps, while introducing important definitions for the field of software testing. The contents detailed in these sections are based on the widely accepted definitions of testing activi- ties from the ISTQB [80] [174].
Test planning and control
As illustrated in figure 5.1, the first step of the STLC deals with planning and control ac- tivities. In particular,Test Planningaims at understanding the goals and objectives of the customer. Furthermore, all the involved stakeholders need to be addressed. Overall, the project context needs to be identified and understood in detail. For a test engineer, this results in atest mission, sometimes called atest assignment.
Planning and Control
Evaluating Exit Criteria and
Reporting Development
Artifacts Analysis and
Design
Implementation and Execution
Test Closure
Figure 5.1: The common software testing life cycle
Based on this intermediate version of a test mission, a conformity check with the com- paniestest policy as well as the overalltest strategyneeds to be performed. In case of conformance issues, the affected parts of the test mission are adjusted.
Definition 7(Test Policy[170])
A high-level document describing the principles, approach, and major objectives of the organi- zation regarding testing.
Definition 8(Test Strategy[170])
A documentation aligned with the test policy that describes the generic requirements for testing and details how to perform testing within an organization.
As soon as the general planning is finished, the basic testing approach and therefore necessary resources are determined. The range of available approaches is usually di- vided into functional, non-functional/quality, structural, and regression testing ap- proaches. The first two categories are sometimes subsumed under the termBlack-box Testing, while structural tests are often calledWhite-Box Testingapproaches. Apart from the mentioned non-regression categorizations, sometimes there is an additional cate- gory calledGrey-Box Testing, reflecting partial or uncertain knowledge of the SUT.
Definition 9(Black-box Testing[170])
Testing, either functional or non-functional, without reference to the internal structure of the component or system.
Definition 10(White-box Testing[170])
Testing based on an analysis of the internal structure of the component or system.
One aspect that is closely related to the chosen test approach is the definition of an exit criterion. This criterion determines whether testing activities are completed or not.
Completing the set of test planning activities, the subsequent process steps as well as included tasks are scheduled according to the project timeline.
In contrast to the planning part, theTest Controlactivities are usually carried out in par- allel to all remaining process steps included in figure 5.1. At this point, the structure of the figure seems misleading, but above all tackles the definition of how the con- trol activities are performed. An essential aspect of the test control process step is the steady comparison of the actual testing progress against the planned progress. This is followed by a continuous reporting of the derived status to all the previously identi- fied stakeholders. In case of unacceptable deviations from the plan, an adjustment of the plan based on the monitoring information is triggered. Two types of triggers are conceivable. On the one hand, corrective actions due to misconceptions in the planning phase, e.g. change of exit criteria. On the other hand, expanding actions through a more detailed view of the SUT.
Test analysis and design
Based on the developed test plan, test engineers further detail the intended test setup by analyzing artifacts and thereby developing a concrete design. In particular, devel- opment artifacts like requirements, architecture documents, design specifications, and included interfaces, also known as thetest basis, are studied. At this point, the test basis serves as sufficient information for the design of black-boxtest cases. Even if no black- box tests are intended, thetest conditionsfor the relevanttest itemsare developed in this phase of the STLC. Here, especially the testability of the underlying test basis is chal- lenged and, if necessary, triggers a change request for the test basis. A central role of the test design phase is to end up with a focused set of tests for the targeted parts of the SUT.
Definition 11(Test Condition[170])
A testable aspect of a component or system identified as a basis for testing.
Definition 12(Test Item[170])
A part of a test object used in the test process.
Definition 13(Test Case[170])
A set of preconditions, inputs, actions (where applicable), expected results and postconditions, developed based on test conditions.
Besides the analysis and design of test cases, the test environmentmarks an essential point of investigation in this phase. On the one hand, the necessary infrastructure to perform subsequent test phases is determined. On the other hand, appropriate tooling
for the test engineer is set up and configured to integrate to a sufficient toolchain that operates on the previously defined infrastructure. Overall, the goal of this STLC-phase is a tangible set of test conditions and test procedures with regard to the predominant set of test objectives.
Definition 14(Test Environment[170])
An environment containing hardware, instrumentation, simulators, software tools, and other support elements needed to conduct a test.
Test implementation and execution
Following the analysis and design view on testing, the STLC-phase of this section deals with the concreteimplementation of testsand the thereon-based execution. In particular, duringtest implementation, the previously specified set ofabstract test casesis detailed by developing appropriate test data to derive a set of concrete test cases. Moreover, test procedures are implemented by scripts to support the execution of tests in the test envi- ronment. In order to make the execution more efficient, the test cases are grouped into logical collections, calledtest suites, sharing test data or test objectives. Furthermore, the test suites are prioritized and a schedule for the subsequent execution is derived.
Any missing components of the test environment are implemented and verified in this phase to create the best possible starting point for the following phases.
Definition 15(Abstract Test Case[170])
A test case without concrete values for input data and expected results.
Definition 16(Test Suite[170])
A set of test scripts or test procedures to be executed in a specific test run.
Following the implementation of test-related artifacts, thetest executionis performed. In particular, the previously defined test suites are executed against the SUT. The record- ing of the execution traces and results is the most important task, since based on these so-calledtest logs, documents or even instructions for action are derived for stakehold- ers. Especially in cases, where observed behavior differs from intended behavior, the test log marks the basis of the decision to develop appropriate countermeasures.
Definition 17(Test Log[170])
A chronological record of relevant details about the execution of tests.
Evaluating exit criteria and reporting
Continuing the STLC presented in figure 5.1, an important point of decision in the pro- cess is reached. As its name implies, theevaluation of exit criteriadetermines whether
sufficient testing has been performed. Two different conclusions can be drawn from this check. Either that the previous tests were not sufficient and further iterations are necessary, or that the criterion chosen in the planning phase is not suitable and must be adapted. This step applies to each of the definedtest levelsfromUnit/Componenttesting up toAcceptancetesting. Regardless of the current test level, atest reportis generated, which discloses the current testing status for stakeholders.
Definition 18(Test Level[174])
Depending on the constructive phases for the test basis, the test levels are commonly divided into the following levels: unit/component testing, integration testing, system testing, and acceptance testing. A more fine-grained division into levels is conceivable if required by the test policy or the product.
Definition 19(Test Report[170])
Documentation summarizing test activities and results.
Test closure activities
If the exit criteria take effect and these are considered useful, the final actions of the STLC, namelytest closure activities, are performed. The primary objective is to compare planned test artifacts with artifacts that have been created. Furthermore, this includes a review of recorded problems and knowingly remaining errors to ensure appropriate defect handling and documentation.
Apart from test result-specific closure activities, some test environment-related tasks need to be performed. This includes a handover of test-related artifacts to the mainte- nance department as well as archiving all the test tools and infrastructure relevant for the previously detailed process steps. Moreover, improvements for future variants of the STLC are identified at this point and necessary adjustments are documented and initiated.
As already mentioned in the introductory part of this section, the concrete instance of a STLC is subject to many influences such as the type of STLC, the companies test policy or general strategy, empirical values, and standards set by the application context.