- the Intended Support, its concept, elaboration and underlying assumptions;
- the Actual and Intended Introduction Plan including introduction, installation, customisation, use and maintenance issues;
- the Actual and Intended Impact Model;
- the Reference Model;
- the criteria used.
Section 6.1 discusses the importance of evaluation, the different types of evaluation in DRM and some evaluation approaches from other areas that are relevant for design research. The types of DS-II are described in Section 6.2, the proposed Systematic DS-II process is introduced in Section 6.3 and its steps discussed in detail in Sections 6.4 to 6.8.
6.1 Evaluation
6.1.1 Importance of Evaluation
When developing products, it is good practice to evaluate results throughout the process, in particular to determine whether the product (or its description) can be released to the next stage. In the same way design support should be evaluated at various stages in its development process irrespective of the type of support and the extent to which the support has been developed. The Institute of Electrical and Electronics Engineers (IEEE) defines what they call validation as the “confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use are fulfilled” (IEEE 1998). For a further discussion on validation of methods and in particular of design methods, see Frey and Dym (2006); Seepersad et al. (2006).
Evaluating design support is essential because its effects can only be assumed while developing the support. This is due to the fact that
• the support is a creation, involving various assumptions introduced during the translation or extrapolation of what was found in reality or in a theory, and during the development of the initial idea;
• the introduction of the support creates a new situation that did not exist before and about which only assumptions exist: many unexpected effects may occur;
• the context in which and for which the support is being created is dynamic.
Developing support is a creative process. From an investigation into the needs and problems – the current situation – and available theories, ideas and technologies, a new situation is envisaged. Support is developed that should change the current situation into the desired situation. This is a design step, the results of which can not necessarily be derived directly from an analysis of the current situation.
The introduction of support in the current situation is a change in its own right that has an effect: a new situation is created. The ‘old’ Reference Model is no longer valid. The introduction of the support will and is indeed intended to influence the factors in the Reference Model in order to realise the desired situation. Some effects, however, might have been unforeseen. These side-effects might be negative as well as positive. Side-effects that have a negative impact on the outcome are, e.g., the time required to apply a support or the method used to introduce the support. Examples of positive side-effects are a temporary increase of motivation due to the introduction of something new and increased attention for the problem at hand. Side-effects can obscure the effects of the support: the evaluation could suggest a more positive effect than the support actually has, or even suggest a positive effect, where the actual effect is nil or negative. Because of the many facets involved in design (see Figure 1.1) positive as well and negative side-effects are likely to occur during the evaluation and when the support is actually used.
The context in which the development process takes place changes, irrespective of the introduction of design support: people learn, markets change, organisations evolve, new technologies emerge, new knowledge becomes available and new regulations are put in place. As a consequence, the original needs of the designers may no longer exist, the support’s concept might be overhauled, the acceptance of the support may change and with it the ease of introduction. For example, some years ago Internet technology to implement design support would have been regarded with scepticism and would have required more resources for training and for building an infrastructure. This is no longer the case.
Evaluation of design support in itself is difficult because:
• the level of implementation of the Actual Support usually does not cover all functionalities of the Intended Support (as described in Chapter 5);
• the effects of more heuristic support, such as guidelines, methodologies or approaches can be difficult to assess;
• it can take time until the effects occur if proper application requires learning and de-learning processes, and changes of mind set and working habits;
• design processes can be long compared to a research project so that the actual success-related effects might not emerge until many years after introduction;
• the expected outcome, i.e., the desired situation, depends not only on whether the support is functioning (as evaluated in PS), but also on the validity of the description of the current situation and its problems, the quality of the description of the envisaged situation, the conceptualisation of the support, the realisation of the concept, its introduction into the current situation (which may require training, specific resources, organisational changes and customisation), its users and use, its application and maintenance.
These issues emphasise the need to focus not only on the outcome, but also on the process of applying the support in order to be able to interpret the evaluation results – and thus determine the true strengths and weaknesses of the support – and to suggest improvements.