Not Forgetting: Monitoring, Evaluation and Learning
It may seem commonsense to emphasize the importance of learning from experience in order to improve future policy responses and institutional capacities. Surely this is normal practice? Ideally, as policies are implemented, routines for capturing the necessary data are put in place, the effectiveness of policy interventions are monitored, and formal evaluations feed into the redesign of polices. However, in the emergencies area and elsewhere in public policy, careful harvesting of insights from past and current experience and purposeful application of the knowledge thus gained to adapt and improve capacities are too often not evident. This chapter identifies key issues that, if addressed, will enable policy learning and improvement, allowing for the design and maintenance of adaptive policy processes and institutional settings. It goes beyond the more familiar and well-documented practice of monitoring and evaluation, considering the nature of policy learning, conducive institutional characteristics, basic forms of information and their routine capture, and the development of human capacities.
The point of ‘not forgetting’ is to learn and improve. The usual language describ- ing this activity is monitoring and evaluation (M&E), representing an ever-growing enterprise in government and associated consultancies, and targeted at operational project and programme evaluation. Consequently, our coverage of M&E is a brief summary later in this chapter, with most space devoted to the broader concept of policy learning, since this is more relevant to the themes of this book.
administrative boundaries. This complicates understanding of, preparedness for and response to disasters by spreading roles and information across time and space, and making cross-event and cross-context transfer of lessons difficult.
• Pervasive uncertainty surrounding the causes and magnitude of possible events, and vulnerabilities and capacities under different disaster scenarios. Importantly, no matter how good the knowledge base and the understanding of vulnerabilities and capacities, the existence of inevitable residual uncertainty and the possibility of surprise throw doubt on the efficacy of the hardest-won lessons.
• Imperatives for wide community participation in management and policy. Posi- tively, this broader engagement (see Chapter 4) widens the catchment of experi- ence and knowledge available to inform learning. However, the more inclusive and broad the membership of the policy community, the more challenging policy learning becomes: communication, organization, and development of mutual understanding among a diversity of actors with a diversity of values, interests, information-processing capacities and organizational strengths.
• The high and often urgent ‘stakes’ in disaster situations – lives lost, livelihoods ruined and environments degraded – are a strong argument for sophisticated policy learning. But combined with uncertainty in various forms, they also make lesson-drawing a complicated task, more contested and politically sensitive, and thus a more hazardous activity for the policy analyst. In a professional and organ- izational sense, close attention to the performance of policy and institutional settings involves risks to individuals, agencies and political leaders, especially at times when attribution of blame is sought in post-disaster debates. The lulls of attention and resources between disasters can be difficult times in which to main- tain interest.
• Vulnerability to disasters is determined by factors located deep in social and economic situations (indirect or systemic causes) – patterns of settlement, resource dependency, economic condition, livelihood security, infrastructure and health systems – as well as more immediate causes such as building quality. Analysis of how well policy and institutional measures attend to these causes and to prescrip- tions is complicated and difficult, but is needed if vulnerability is to be reduced.
• Emergencies and disasters entail cross-sectoral and whole-of-government responsi- bilities and implications. While this is reasonably well accepted in more immedi- ate emergency planning and response, it is less apparent in terms of integrated policy and institutional settings to ensure that pre-event and ongoing disaster policy brings together different parts of government and society. The generation of relevant information and policy lessons across sectors and portfolios is neces- sarily harder than in policy domains where responsibilities are more narrowly contained.
Despite these challenges, emergency management probably learns from experience more than most professional fields: perhaps only health and medicine puts as much effort and thought into lesson-drawing from experience. This is not surprising since both share the attributes of high stakes, political and moral imperatives and sensitivities, and complex systems with multiple cause–effect linkages. Emergency management agencies and individuals review events, debrief staff and communities, communicate
lessons and warnings, and take accreditation procedures and competency standards built on accrued experience very seriously. They are held routinely accountable and are closely evaluated for their performance by coronial courts, commissions of inquiries and the like, as well as in the larger ‘court’ of public and political opinion.
Nevertheless, most of this information-gathering and learning focuses on preparedness for likely or known disaster events, and on related and more immediate response and recovery capacities. The emphasis in this book is on the policy and institutional capacities within which operational emergency management operates and is either enabled or constrained. There is less formalized evaluation of these settings, and the ways in which this can be more clearly thought about is the focus of this section.
Warning signs of impending disaster are very often ignored for a wide range of reasons clearly set out by authors such as Turner (1978); and Turner and Pidg- eon (1997) and Perrow (1984), as well as others who work on risk management in a corporate environment (e.g. Hopkins, 2005) and on many post-disaster public inquiries. It is frequently the case that it is very difficult for indicators, whether for a dam collapse, major industrial or transportation accident or other incident, to become part of emergency decision-making that leads to action. Such problems and failure to learn arise from organizational attributes and cultures that inhibit problem acknowledgment, learning and even minor change. They also generally arise because of near-term economic imperatives in consideration of emergency management, most readily seen in the development of areas prone to flooding, wildfire, landslide and so on. Often, the people concerned may be aware of the risks but may feel that they have no option, such as those crowded around the Union Carbide plant in Bhopal.
The case of a 1966 mining spoil heap collapse in Aberfan, south Wales, illustrates some of the organizational issues. The problems were well known in the village and had been brought to the attention of authorities on several occasions through differ- ent channels. No action was taken and the tip collapsed onto the village school, killing 144 people, including many of the area’s school children. The post-disaster inquiry condemned the British Coal Board for, among other, things ignoring warn- ings well before the disaster (McLean and Johnes, 2000). Following the Aberfan disaster, US authorities examined similar sites across the US. One such site was a tail- ings dam at Buffalo Creek, West Virginia. The local coal industry had a long history of safety-related problems; but even though the risks were clear, activities continued as normal. On 26 February 1972, the tailings dam collapsed, eliminating villages, killing 125, injuring over 1000 and making 4000 homeless (Erikson, 1976, 1979).
One test for whether information from monitoring and warnings – formal or informal – is likely to be useful is whether information on near-misses comes to the attention of decision-makers in real time. As the human-made contributing factors to emergencies and disasters are scattered across portfolios, sectors and places (spatial planning, transport, chemical approvals, engineering standards, and many more), emergency managers can hardly be expected to gather information from across government, society and industry. The whole-of-government nature of comprehen- sive disaster policy is emphasized.
If policy and institutional learning is rarer than it should be – and the disasters field is not alone in facing this situation – then the first step in advancing
the endeavour should be a clarification of what such learning is, why it is pursued and by whom.