A systems approach to the evaluation of an academic department as a service provider at a University of Technology.

1

Issues of service quality affecting Universities 3

The need for a Systems Approach to evaluation at a University 6

Winberg (2004:40) believes that the fear that universities of technology will lower academic standards stems from a narrow vision of knowledge and scholarship. In addition, it was found that the concept of a university of technology in South Africa is not well known.

Goal and Sub-goals of the Research 10

Scope and delimitations of the Research 10

Research Methodology 11

In physical systems, quality is often an absolute measure of the properties of the physical artifact. The study will implement action research to develop a mixed methods framework for evaluating an academic department as a service provider, based on the service organization extension of the Work System Method (Alter, 2007b) and elements of soft systems methodology (Checkland, 1981) . ).

Figure 1.1 A triad of the Justification of Research (adapted from Landry and Banville 1992:79; Robey, 1996:402)

Timeline of the Development of the Conceptual Framework 16

Importance of the Research 19

The purpose of the workshops was to identify the relevant stakeholders in the evaluation of service quality at a UOT. The first is "the determination of the actual properties to be assessed" (Joseph and Joseph, 1997:16).

Overview of the structure of the thesis 20

21

What is Evaluation? 22

Outcome evaluation: Has the program or technology produced any demonstrable effects on the specified target(s). Impact assessment: Has the program or technology created broader or unintended effects beyond the specific targets.

Theory of Evaluation 25

Evaluation Strategies 27
Fourth Generation Evaluation 30
Participatory Evaluation 34
Empowerment Evaluation 38

The evaluator's role is therefore to act as a trainer and teacher in the evaluation process. The importance of this phase is that the setting of goals is proportional to the current state of the evaluation object.

Purposes of Evaluation 41

Programme Improvement 41
Accountability 42
Hidden Agendas 44

Another criticism of the method is that there is insufficient empirical evidence to indicate that it satisfies the intended purpose. It has also been argued that programs should be evaluated in terms of the knowledge they generate.

Current Practices of Evaluation at Universities 45

European Foundation for Quality Management (EFQM) 49

Good Management Project (GMP) 200 51
Good Management Project (GMP) 143 52

Baldrige National Quality Programme Education

Criteria of the Baldrige National Quality

Data Envelopment Analysis (DEA) 59
Servqual Model 61
Servperf and Higher Education Performance Model 62
Importance-Performance Analysis (IPA) 64
The Balanced Scorecard 65

Johnes and Taylor (1990) identified another problem in applying the production model to the university sector. Based on the foundations of the SERVPERF framework, the Higher Education Performance model (HEdPerf) was developed. However, a number of issues need to be resolved before the technique can be applied.

Figure 2.1 The EFQM Model Source: Dahlgaard-Park (2008)

Evaluation Practices in South Africa 68

Higher Education Quality Committee (HEQC) 69
South African Excellence Model (SAEM) 72

One of the objectives of the South African Higher Education Act of 1997 is to provide quality assurance and quality promotion in higher education. Ad hoc implementation of quality assurance outside of general planning and management procedures is not successful. Ownership of the quality assurance process can be increased through well-designed self-evaluation mechanisms.

Figure 2.4 The South African Excellence Framework (Source: IRCA, 2004)

Service Quality 76

The Gap Model 77
Managing Customer Expectations 81
Zones of Customer Tolerance 82
Measuring Service Quality 83

Zeithaml and Bitner explain customer satisfaction as the degree of fit between customer expectations of service quality and the quality of service as perceived by the customer. This is the difference between the organization's quality specifications and management's perception of consumer expectations of the service and its quality. The test instrument is based on the premise that service quality is the difference between customers' expectations and their assessment of the service they have received.

Analysis of Non-Systemic Approaches of Evaluation 87

Cronin and Taylor (1992) question the validity of the P – E introduced in the disconfirmation paradigm, suggesting that this concept is a potentially misleading measure of service quality perceptions. There has been considerable debate that the performance-minus-expectation construct may be a flawed and inconsistent measurement of the assessment of service quality (Cronin and Taylor, 1992:58). A detailed analysis of these models shows an underrepresentation of the construct of customer satisfaction in relation to service quality.

Table 2.2: Evaluation Models applied to Higher Education

Conclusion 93

The second important part of the Work System Method is the Work System Life Cycle. Who is the intended beneficiary of the evaluation of an academic department as a service provider to a university? The following characteristics of SSM motivated its inclusion in the development of the conceptual framework.

94

What are services? 95

Young and Burgess (2010) state that approximately 75% of the economic activity generated in the US economy is represented in the service sector and 76% in the United Kingdom is services. A service is any act or performance that one party can provide to another that is essentially intangible and does not result in the ownership of anything." (Kotler and Keller, 2006:402). Kandapully, Mok and Sparks (2004:6) state that service is defined as "any activity or benefit that one party can provide to another that is essentially intangible and does not result in the ownership of anything".

The unique characteristics of services 97

Intangibility 99
Inseparability 99
Heterogeneity 100

Zeithaml, Bitner & Gremler (2009:22) state that services cannot be alienated from the person of the seller. Also, the buyer cannot be removed from the producer, since it is produced and consumed at the same time. Kotler and Armstrong (2010) and Zeithaml et al. 2009:22) state that perishability refers to the fact that services cannot be saved, stored, resold or returned.

Figure 3.1: Goods and services continuum. Source: Palmer and Cole (1995:24)

Services Marketing mix and its applicability to tertiary institutions 101

Product elements 102
Place and time 102
Price and other user outlays 104
Promotion and education 105
Process 106
Physical Evidence 108
People 109
Productivity and quality 109

In addition to the nature of the product, place and time and its price, promotion of the product is an important aspect of the marketing mix. Universities therefore require that different elements of the marketing mix be used for different publics. A student's first impression of a tertiary institution is often based on the interaction with the staff of the institution.

Contemporary issues on service quality management 110

The challenges of managing service quality 112
Changes in conceptions of service quality management 114

In light of the fact that all aspects of the service mix are part of service encounters, it is important to understand service quality and determinants of service quality. The location of a customer's perception of service quality on the continuum depends on the nature of the discrepancy between the expected service versus the service perceived by the consumer. Service quality (SQ) is thus operationalized as performance (P) – minus – expectation (E) (calculated rejection) to provide service providers with a tool to evaluate and manage their service quality levels by working on the two important parameters of customer perception ( P ) and expectations (E).

Figure 3.2 Value creation and service-dominant logic

Service as a system 117

Conclusion 122

Flood and Jackson (1991:19) defined emergent properties in light of the classical concept of synergy. These models are used to predict the system's response to changes in the environment. Some principles of VSM will be adopted in the development of a systemic framework that forms the basis of this study.

124

Why systems thinking ? 126

Checkland (2000:S11) believes that complex problems involve richly interconnected sets of "parts" and that the relationships between the parts may be more important than the nature of the parts themselves. In order to change the overall behavior of the system, the structure of the system must be changed. This cyclic current causes delays, large gain and damping effects, which affect the overall behavior of the system.

Sociological paradigms 128

Furthermore, system solutions should never be created in isolation from the environment to which they belong (Reisman and Oral, 2003:8). Reynolds and Holwell (2010:10) mention that CST shares the same epistemological shift as the soft systems tradition, but addresses some of the perceived shortcomings in both hard and soft systems thinking, and in particular the inadequate attention to power relations. Critical systems thinkers believe that the world is by definition not harmonious.

Table 4.1 System of Systems Methodologies (adapted from Jackson 1991:29)

Critical Systems Heuristics 134

Total Systems Intervention (TSI): Versions One and Two 137

Decision Maker Who should be/is in control of the success conditions of S. What success conditions should be/are outside the decision maker's control. The critical reflection mode uses the three stages of the TSI to reflect on the appropriateness of the problem-solving mode.

Figure 4.1 The process of Total System Intervention TSI (Wilby, 1996:233)

System Dynamics 144

SD views all concepts in the real system as continuous quantities connected in loops of information feedback and circular causality. Second, it is imperative that the framework developed is sensitive to the fact that the university, by design, is a system that must take into account system dynamics. The next popular systems approach is Soft Systems Methodology, a cyclical iterative research approach for formulating and structuring thinking about problems in a situation where people have different views of the world.

Soft Systems Methodology 146

Critique of Soft Systems Methodology 150

These models represent "systems of human activity," and Checkland recognized this as one of the most important advances in the development of SSM. The formulation of SSM Mode 2, known as the two-threaded version of SSM, was the result of the original seven-stage layout being too restrictive. In Analysis 3, the politics of the problem situation and how power is acquired and exercised are examined.

Figure 4.2 The Learning Cycle of soft systems methodology (adapted from Checkland (1989:84)

The Viable System Model 151

Jackson reveals that System 1 deals with the implementation of the tasks that the organization should be doing. The values and beliefs adopted by System 5 must be shared with all other elements of the organization. Checkland argues that VSM does not provide adequate ways to design the process of negotiation between the different viewpoints that make up the organization.

Figure 4.4 The Viable System Model (adapted from Beer 1985)

Work System Method 154

Elements of a Work System and its applicability to this study 156
The Work System Life Cycle 159

The final methodology that will be analyzed in this study will be the work system method. The framework of the work system consists of 9 elements, of which 4 are internal and 5 are external. The Work System Life Cycle (WSLC) is a useful model to help understand how change occurs and how a system evolves over time (Alter, 2006b:89).

Figure 4.5: The Work System Framework (adapted from Alter, 2006b:13)

Linking Evaluation to Systems Thinking 163

The need for Multiple Criteria Decision Analysis in the Evaluation of

The Analytic Hierarchy Process and its extensions 167
AHP and group decision making 169

The next step in developing the framework involved researching and analyzing the systems methodologies and techniques most appropriate for the study. Elements of the work systems method, coupled with some elements of the systems approaches mentioned earlier, will be used in developing the framework of this study. Based on the above requirements, a concerted effort was made to meet these requirements during the development of the framework in this study.

A combination of the above arguments provides a solution to the issue of paradigm incompatibility in the developing framework. The purpose of the conceptual framework is to provide systematic guidelines for understanding the assessment of service quality in a university.

171

The need for a multi-methodological approach to address the

Approaches to be included in a conceptual framework for the

Critical Systems Thinking (CST) 180
Soft Systems Methodology (SSM) 184
The Viable System Model (VSM) 185
System Dynamics (SD) 187
Work System Method (WSM) 190

A Conceptual Framework for the Evaluation of an Academic

Conclusion 197

A review of the literature indicates that no approach has successfully addressed the complexities associated with evaluating the quality of university service delivery. Based on the complexity of the nature of evaluating an academic department as a service provider to a university, coupled with previous discussions, it can be expected that a combination of methodologies from Options F to I could be relevant to the development of the framework. The following approaches, shown in the figure below, are included in the framework and justification for the selection of the approaches is provided.