WHY: The Monitoring and Evaluation Framework – An overview
- Introducing the Monitoring and Evaluation (M&E) Framework
- Developing the M&E Framework
- Structure of the M&E Framework
- Rationale for an M&E Framework
- Principles underpinning M&E
- M&E in the context of performance management
The City has been implementing M&E activities as part of its performance management process for many years. The city has decided to adopt these principles and ensure implementation through the design and use of an M&E framework.
WHAT: Key concepts included in the M&E Framework
- What is ‘monitoring’?
- What is ‘evaluation’?
- Comparing ‘monitoring’, ‘evaluation’ and ‘audit’
- M&E practices outside South Africa
- Learnings from international benchmarks: Requirements for sound M&E
- M&E approaches in context: An area for learning and improvement
- Driving outcomes across South Africa: The national approach to M&E
- The ‘outcomes approach’: A foundation for meaningful M&E
- Ensuring alignment with the national perspective
Further details in this regard – in the context of the city – are provided in Section 3.6 below. The diagram below describes the interrelationship between national, provincial and local government commitments – in the context of the results approach.
M&E within the context of the City’s planning mechanisms
- Guiding principles for the City’s M&E Framework
- Values framing the City’s delivery approach
- The Joburg 2040 GDS: the ultimate compass for M&E in the City
- Locating M&E activities associated with the five-year IDP and cluster planning
- Locating M&E activities in relation to the annual IDP, SDBIP, Business Plans and
Genuine, focused interactions with all city stakeholders – no matter who they are or where they are from – in all M&E activities, in the interests of construction. Also included here is the City's Code of Values, which is identified as applicable in the context of the City's performance management system, given the applicability of the values to the M&E framework, which itself supports effective performance management. This therefore serves as a starting point for understanding the focus of M&E in the city.
Delivery of the four outcomes is dependent on a concerted effort and focus from everyone in the city – especially as long-term outcomes often require the achievement of targeted outputs and supporting activities in the short and medium term. In the case of the city, these are framed within the framework of the Mayor's flagship programmes, placed within five-year IDPs, containing cluster-specific priorities and plans.
HOW: Putting the M&E Framework into practice
Placing M&E activities within the City’s planning cycle
M&E activities cannot be separated from the overall strategy or project within which they take place. This is reflected in the figure below, in the context of different 'loops' of learning that flow between the M&E system and operational delivery plans and processes. Two levels of planning are therefore needed in the context of an outcomes approach to M&E.
Given the fact that monitoring and evaluation activities are of little use when carried out within a poorly designed organization, cluster, department or project plan, an overview of operational planning and the steps involved is provided in Annex 3. Steps specifically related to planning for monitoring and evaluation are described below, followed by details on other key elements of the monitoring and evaluation cycle (ie monitoring, evaluation, reporting, communication and feedback).
Phase 1: Planning for M&E in the City
These GDS indicators should be reviewed in the context of the above criteria and confirmed where appropriate. When identifying baselines, the focus should be on a measure of the status quo in the context of the defined indicator. Targets should be established for each indicator level that apply within the framework of the city's planning timeframes (ie
For information resources to be used appropriately, MVs will be created early in the M&V cycle. The figure below provides an overview of the M&V planning elements addressed within the planning phase, in the context of the process of broader organizational, group, department/entity or individual planning.
Phase 2: Conducting monitoring
Monitoring activities will also need to be supported by a monitoring system, which in the case of a city will include an information technology database through which indicators, background information, data and analysis can be stored, maintained and easily accessed. The focus should be on ensuring the continuous collection of quality data and the appropriate flow of information across the city if required. Findings and analyzes must be reported in all cases – both positive and negative findings are reported transparently.
As with evaluation, the nature of data and information to be collected in monitoring will be defined already in the planning phase. When collecting data, we must focus on data that is relevant, accessible, timely, understandable and accurate.
Phase 3: Conducting evaluations
Mid-term evaluations carried out on the five-year cluster plans and the long-term GDS; Many of the principles that apply to the collection of data for monitoring purposes apply in the case of evaluation. Feedback: Providing timely, appropriate and user-friendly feedback is crucial to the credibility of the evaluation itself.
Value for money: The cost of the evaluation should be commensurate with the size of the intervention itself. While the analysis in the context of a monitoring exercise will be done in relation to the predefined indicators included in the M&V plan, an evaluation exercise can address different questions, as depicted in the figure above.
Phase 4: Reporting on M&E
Legislature (which includes the Executive Mayor [OB], and is supported by Group Strategy Policy and Relations or GSPR) Five-year IDP 5-year Assessment Report;. Performance Audit Committee (JPAC) – which includes the OB, and is supported by GSPR Five Year. Performance Audit Committee (JPAC) – which includes the OB, and is supported by Group Strategy Policy and Relations (GSPR).
Monitoring reports such as the quarterly review report help build an understanding of progress and achievement in the context of business plans and SDBIP, thereby ensuring ongoing strategy implementation. A more refined M&V system in the city will allow for a nuanced reflection of reporting types, audience, purpose, format and frequency – thus helping to deliver more optimal results that are consistent with their intended use.
Phase 5: Communication and feedback
Integrated planning requires a focus on the entire process of delivery – from the simple, short-term actions specific to individual performance, to the creation of the end result or the intended 'whole', and thereby, the desired impact. Maintaining understanding among all players within the City of the organization's goals, and how each small but aligned action contributes to the outputs and outcomes associated with these goals, is critical to achieving City-wide results. When building a car, if even one small part (eg a bolt) is missing, the quality of the end result (and the ability to function) will be affected.
Building this understanding of delivery in the context of the city can lead to improved team focus while deepening the connection between individual roles, indicators, business and cluster plans and the city-wide IDP. If the city's M&E mechanisms are able to build a better understanding of how everyone is part of this larger 'whole' with constant feedback on implementation regarding all levels within the results approach, a greater number of stakeholders can be more fully involved in a shared - set vision of the future .
Applying analysis and findings emerging from the M&E cycle
Building a culture of learning requires everyone, across the city, to understand and actively participate in action, analysis and reflection - while demonstrating ownership of the M&E concept and system. The application of findings will be encouraged by involving a wider range of stakeholders in the M&E and analysis process itself - e.g. In addition, reinforcing bodies involved in oversight within the city and those directing the M&E system itself will also provide the platform for better application of findings.
The chapter that follows discusses the way in which M&E issues relate to the field of governance and control in the city. The final chapter discusses the requirements for the successful implementation of the M&E framework as a whole.
M&E in the context of governance and oversight
The role of M&E in supporting good governance and stakeholder engagement
Users and uses of information emerging from the M&E Framework
It builds on the National Treasury's MFMA Circular (2006, p.1) in respect of oversight (Circular No. 32), which notes that "While it is initially left to the Mayor or Executive Committee to address any performance failures, ultimately the council is vested with the power and responsibility to oversee both the executive and administration.Understanding the nature of responsibility, accountability and oversight across the city helps build a clearer sense of the role players for whom M&E information has critical value The figure below further elaborates on these arrangements, drilling down into the various roles that the CEO has – in terms of oversight, decision-making and delivery – thereby clarifying the nature of the M&E information required.
It is critical that frameworks such as the M&E framework outlined here align with the City's institutional form. Further input on the practical ways in which alignment will occur between the steps included in the M&E framework, and the various role players across the City's various governance and delivery structures, will be provided within the M&E handbook to follow.
Next steps in implementing the M&E Framework
Requirements for implementation
Confirm roles and responsibilities in relation to both monitoring and evaluation-related activities (e.g. in terms of: process establishment; deployment; quality control; . training; data collection, analysis for monitoring versus evaluation, reporting, storage, management and oversight) – and communicate through a clear matrix. Identify capacity gaps (across management, technical, administrative, counsellor, communication and other roles) – and address them through capacity building. Design and establish an M&E system to enable easy data collection, storage and retrieval – and ongoing knowledge management.
Identify if additional systems or tools are required for sound data collection and data credibility – and if so develop/establish/acquire. Focus training and management on building capacity and interest in gathering and analyzing information – and applying insights.
Concluding comments
Monitoring and Evaluation in the Government of Canada: 'Lessons Learned' from 30 Years of M&E Development. Available at: http://www.treasury.gov.za/publications/guidelines/FMPI.pdf The Presidency (2010) Guide to the outcomes approach. Available at: http://www.info.gov.za/view/DownloadFileAction?id=94547 SAICA (The South African Institute of Chartered Accountants) (2009).
Available at: http://www.undp.org/evaluation/documents/HandBook/part_2.pdf Wageningen UR (2010) PPMM&E in projects and programmes. United National World Food Program (UNWFP) (n.d.). Monitoring and evaluation guidelines: What is RBM-oriented M&E.
Acronyms and abbreviations
Proposed M&E roles and responsibilities matrix
Organisational, cluster, departmental or project planning, in the context of the
Input is provided on the application of M&E activities to short-, medium- and long-term planning mechanisms – in the context of the outcomes approach. These lessons and those regarding the steps deemed necessary in setting up an M&E system – as described below – will be applied in the context of the city's M&E framework and its roll-out. The figure below depicts and defines these concepts, within the context of the city's long-term planning horizon.
The above distinctions and the wider application of the M&E Framework within the |City take place in the context of a tightly regulated and managed annual and five-year planning plan. As is evident, M&E activities undertaken in relation to the ultimate objectives of GDS should emphasize the delivery of impacts and to some extent outcomes. Ensure continued awareness of the importance of everyone in the city working as a team in support of our long-term vision; and.
The theory of change should be made explicit and questioned – testing the adequacy of the results, in the context of the intended impact; These issues will be more fully addressed in a review of the city's Performance Management Framework.
Templates – An outcomes chain
Templates – Indicator planning framework
Templates – A City-wide M&E Plan (timeframe specific)
An example of the M&E Plan in action
Templates – M&E Plan at the cluster level
Template – Annual M&E Plan at the inter-cluster level
Template – M&E Plan at the inter-cluster level
Template – City-wide reporting template
Template – Cluster-specific reporting template
