and definition of the problem. For example, in the ontological sense the problem is taken as a given without other possible interpretations, whereas in an epistemological sense there may be multiple ways of understanding
‘the problem’ based on the alternative worldviews of those involved. The mental construct and adopted paradigm of the problem solvers are important factors that serve to define the context of ‘the problem’ and determine what information is collected and the methods of examination.
4.7 Policy evaluation
Bardsleyet al. (2001) proposed that natural resource policy was based on political expediency and identified four characteristics of natural resource policy that contributed to its failure.
1. Environmental policy treated each segment of the environment separately as individual concerns. Each separate policy was judged on its merits within a particular context. Unless the chain of contextual factors is iden- tified and the decision-making rules changed, there will be little change in actions and outcomes (Edwards and Steins, 1999). Change will require a divergence from normal science methodology and the adoption of an alter- native philosophy and methodology that identifies the contextual factors.
2. Policy has not taken account of how incentives change behaviour.
3. Policy formulation and implementation have not been dealt with together.
4. There has been little or no monitoring and auditing of environmental outcomes resulting in lack of accountability of public money.
In general, natural resource issues were a secondary consideration to the primary productive purpose of a region and consequently the former were dealt with in a reactive manner rather than a proactive preventative manner (Clausen and McAllister, 2001). For example, on private land in the WA agricultural region the primary objective was agriculture, and issues of natural resource management, other than for productivity, were add-ons when and if agricultural profit permitted.
Following the Rural Boom, concern over the alleged failure of public policy led to a resurgence of policy analysis studies in the 1980s. Under the general heading of policy research, there are many ways in which an exami- nation of the problem of policy failure might be approached. Accordingly, the intent can vary greatly, depending on the aim, the methods and the affiliation of the researcher (Hogwood and Gunn, 1992). Mobbs and Dovers (1999) made a broad distinction between descriptive and analytical policy research, consistent with the categories of Hogwood and Gunn (1992) who distin- guished between policy studies and policy analysis respectively, although these typologies are by no means definitive or exhaustive. Mobbs and Dovers (1999) also identified some common methodological approaches that might be applied to natural resource management: for example, political science approaches; psychological/sociological approaches; policy/program evalua- tion; public choice approaches; legal policy research; institutional analysis;
policy cycle analysis; and decision process approaches. There is no agreed approach to applied policy analysis and policy evaluation has been criticised as too often coming from a single disciplinary perspective (Syme and Sadler, 1994).
In public policy there are two overarching schools of thought. The dominant of these is the positivist, rational and empirical school in which there are five commonly used methods: surveys, experimentation, interview, use of data and the use of case studies. Post-positivist or subjective research methods include history, naturalistic inquiry, and stories. Post-positivists such as Sabatier and Jenkins-Smith (1993) and Fox (1990) argued that the analysis of policy implementation had become too complex for the rational positivist approach and suggested that a more intuitive or participatory approach of the post- positivist or subjective school was required.
Whether or not specific actors have been attributed with a role in the policy, the nature of policy implementation studies depends on the fundamental epis- temological or ontological premises of the researcher (Howlett and Ramesh, 1998). In a review of policy change, Howlett and Ramesh (1998) proposed that most early positivist policy analysis studies argued that actors and their behaviour made very little difference to policy outcomes. These studies would occupy a position within the functionalist paradigm in the Burrell and Morgan framework shown in Figure 4.2. In contrast the more recent post-positivist approaches to the study of policy making focus on the way in which the language of politics ‘constructs’ public policy. In particular, the language of politics is predisposed with interpretations of what the policy ‘problem’ is, in an ontological sense.
Recent policy analysis models have made greater attempts to integrate the broader suite of important factors that link policy to performance or effectiveness. To achieve better integration the models were designed to take a macro-level perspective focussing on the tractability of ‘the problem’ with reference to the social and political context rather than becoming immersed in the detail (Sabatier and Mazmanian, 1980). The changes between the macro- and the micro-level analysis frameworks highlighted the perennial problem of the tension between theory and practice in policy studies. Lester and Goggin (1998) argued for the need for reform in policy implementation studies, suggesting that fundamental changes were required that moved the orientation away from the reductionist positivist tradition of normal science towards a ‘post-positivist epistemology’.
Dissatisfaction with the outcomes of implementation research led to the development of the Advocacy Coalition Framework, evolved from an earlier implementation analysis framework (Sabatier and Mazmanian, 1980). The new approach emphasised the need for new conceptual approaches that included policy dynamics and learning over time within a systems frame- work (Sabatier and Jenkins-Smith, 1993). The approach focussed on the establishment and maintenance of competing ‘coalitions’ of interest groups,
government institutions/agencies, and key individuals, each attempting to realise their preferred policy options in a subsystem of political pressure, strategy, and information sharing called the ‘policy subsystem’. Sabatier and Jenkins-Smith (1993) proposed that values underpin the guiding instruments or strategies that influence decision making and provide the causal theory upon which policy positions are constructed. Shared values can be used to map the relationship between various organisations within the subsystem. This latter policy analysis method is appropriate for the evaluation of the integrated and adaptive management approaches that have dynamic characteristics, involve coalitions of groups at the local, regional, state and national levels, and involve knowledge acquisition, since the Advocacy Coalition Framework predicts that policy core beliefs hold coalitions together. Coalition survival during policy implementation, especially in the face of unanticipated complexity and uncer- tainty, requires learning and the ability to adapt to changing policy conditions.
This approach is also consistent with the theoretical changes taking place in natural resource management described in Chapter 6.
We suggest that the epistemology of CCP of natural resource management of the twentieth century was based on the normal science paradigm (posi- tivist orientation) and is concerned with the direct cause and effect level of understanding within the concept of single-loop learning in which information feedback is interpreted by existing mental models and assumptions. In single- loop learning, feedback operates in the context of existing decision rules, strategies, culture and institutions, which in turn are derived from our mental models. In contrast double-loop learning is concerned with feedback from the ‘real world’ in which it can stimulate changes in mental models. Such learning involves new goals and new decision rules, not just new decisions (Sterman, 2000) (Figure 4.6).
In the United Kingdom some of the key lessons of the Economic and Social Research Council’s (ESRC) Global Environmental Change Programme (ESRC Global Environmental Change Programme, 2000) are:
1. that all policy domains need to incorporate environmental and social aspi- rations and hence involve human values;
2. decisions have to deal with conflicting opinions about environmental prob- lems;
3. the risks environmental problems pose to different groups in society; and 4. the need to build trust and engage citizens in solutions.
Although the programme originally focussed on global issues, it increasingly turned its attention to the implementation of sustainable development and in doing so research investigated smaller-scale measures and processes – at
Real world
Information feedback Decisions
Strategy, structure, decision rules
Mental models of the real world
Real world
Information feedback Decisions
Strategy, structure, decision rules
Mental models of the real world
(a) (b)
Fig. 4.6. Single- and double-loop learning. Single-loop learning (a) occurs when information feedback is interpreted by existing mental models. The learning feedback operates in the context of existing decision rules, strategies, culture and institutions, which in turn are derived from our mental models. In contrast, double-loop learning (b) is concerned with feedback from the real world in which it can stimulate changes in mental models. Such learning involves new goals and new decision rules, not just new decisions.Source:Sterman (2000)
local, national and regional levels – which contribute to the achievement of sustainability at the global scale.
In addition to policy failure as a cause for natural resource degradation there is now an extensive literature proposing that natural resource manage- ment problems stem from the interdependencies of natural, political and social systems, and technology, often being described as ‘messy’ or ‘wicked prob- lems’ (Mason and Mitroff, 1981; Bellamyet al., 1999; Bellamy and Johnson, 2000), and will require very different ways in which to conceptualise and understand them (discussed in Chapter 5).
4.7.2 Integrated natural resource management
Although the rhetoric of integrated and adaptive approaches to natural resource management is embracing general theoretical developments in systems theory, ecology and the participative and adaptive management philosophies (Gundersonet al., 1995; Patterson and Williams, 1998; Bellamy et al., 2001), there are few methodologies to evaluate policy implementation based on the same foundations. Bellamyet al. (2001) believed this limitation contributed to the failure of policy initiatives to address sustainable and equitable resource use. Bellamyet al. (2001) identified a lack of evaluating frameworks to help guide continuous program development in the way
natural resource management initiatives contribute to ongoing improve- ments in resource use, sustainability and social wellbeing of communities concerned. Bellamy et al. (2001) reviewed previous evaluation techniques and found conventional economic methodologies to be inadequate to meet the multiple objectives of natural resources management. In addition, two specific models of policy implementation evaluation, the Mazmanian–Sabatier model (Sabatier and Mazmanian, 1980) and the Planning, Research, Implementa- tion, Monitoring and Evaluation Framework (PRIME) (Syme et al., 1994) for the development, implementation and evaluation of whole catchment management plans, were criticised for deficiencies in addressing instrumental assumptions (that is, causal theory in terms of the model) and deficiencies in problem context or formulation and structures, respectively. The more recent Advocacy Coalition Framework of Sabatier and Jenkins-Smith (1993) may have been a more appropriate framework to use to evaluate these approaches but was not included in Bellamy’s research.
In order to fill the gap of lack of comprehensive evaluating frameworks for policy implementation, Bellamyet al. (2001) devised an integrated systems- based framework for the evaluation of natural resource management policy initiatives with three objectives to provide:
1. a basis for an integrated evaluation of the different perspectives (for example, social, economic, environmental, institutional and technical) on the performance of the natural resource management initiative;
2. a framework for guiding implementation; and 3. a rigorous basis for synthesising findings.
We suggest that a fourth objective might be realised in the construction of an evaluation framework. It may also act as a tool to help understand ‘the problem’ in the epistemological sense.
The framework of Bellamy et al. (2001) comprises the identification of seven components:
1. the context, which is composed of environmental, economic, social, insti- tutional and technological factors;
2. the issue characterisation;
3. the object or intent;
4. instrumental assumptions;
5. process of implementation;
6. products; and
7. outcomes in an iterative process that includes review.
It marks a major shift towards incorporating the theory of systems into natural resource management. It does so by emphasising the importance of the context or ‘problem situation’ and by providing for two important steps in the process, the implementation stage and a rigorous basis for synthe- sising the findings. However, this framework omitted one important compo- nent. Although Bellamyet al. (2001) identified in the text a range of factors that reflect the worldviews or mental constructs of the evaluation team (for example, the values, priorities, experiences and organisational culture) which will influence the formulation of the problem and the analytical models used, it was not incorporated explicitly as a step in the framework. Consequently this framework suppresses the importance of mental constructs to the process and the potential impacts that this may have on the outcomes of the eval- uation. Because of this the mental constructs including assumptions of the researcher are not made explicit. Hullet al. (2002) showed empirically that assumptions about nature were embedded in people’s preferences for envi- ronmental policy and management and constrained people’s vision of what environmental conditions could and should exist, thereby constraining the future that could be negotiated, further emphasising the need for the practi- tioner’s assumptions to be made explicit.
In this book a general systemic framework for understanding problem- solving (Jayaratna, 1994) is used and described in Section 5.2. There are three contexts in which any method is used and their identification is important in understanding how they are used and whether or not they are effective (Jayaratna, 1994). The three contexts are creation, selection/interpretation and action. The constructed method reflects the mental constructs of the creators, whereas the people who select the method to solve their perceived problem may try to interpret the method through their own mental constructs.
Those who ultimately have to use the method may apply it according to their mental constructs; consequently the context in which the method users interpret the method may change its nature, form, structure and content, and therefore its effectiveness (Jayaratna, 1994). A fuller examination is given in Chapter 6.