7. SBIA Stage 5: Identification of Indicators – What Should We
For example:
IF the income resulting from the sale of carbon credits (the outcome) is spent on schooling and more nutritious food (the assumption or intermediate state), there should be a reduction in poverty (the impact).
In this case, the outcome indicator could be the net carbon income per family, and the impact indicator could be the proportion of carbon income spent on poverty-related goods or services.
Possible examples of output, outcome, and impact indicators are presented in Table 3. As is apparent from this list, indicators can be quantitative (numerical) or qualitative – in practice it is important to have a mixture of both. Quantitative or numerical indicators (usually for outputs) are easier to understand and compare between projects and more convincing for decision-makers. On the other hand, qualitative indicators can help us understand complex social processes. This being said, there are also ways of generating numerical indicators of qualitative indicators (Social Toolbox Section 7).
Table 3. Examples of Possible Output, Outcome, and Impact Indicators Indicator Types Possible Examples
Output
Indicators - numbers of jobs created - number of people trained - number of trees planted
- number of participants in environmental education workshops Outcome
Indicators - number of households adopting a new livelihood activity
- percentage or absolute increase in household income from carbon payments - reduction in hours spent by women collecting firewood or water
- percentage of carbon beneficiaries agreeing that they get a fair payment (this implies a viable project and an effective benefit-sharing system)
- percentage of women on the project stakeholder committee - number of village management committees functioning effectively - ecological and economic zoning completed
- establishment of improved monitoring systems for protected areas Impact Indicators - percentage of reduction in infant mortality
- percentage of reduction of households living on < $2 per day
- percentage of local population changing from a negative to a positive attitude to forest conservation measures
- significant increase in female participation in decision-making - reduction in domestic violence
- percentage of increase in the population of an endangered species - number of hectares of a rare ecosystem preserved
The theory of change provides the best basis for selecting indicators since attribution is factored in: as noted by USAID (2006) it makes little sense to use indicators that do not capture key linkages in a project’s underlying causal chain. Secondly, deriving indicators from a theory of change promotes cost-effectiveness, since it focuses
monitoring efforts on the most important change factors. If a theory of change is coherent and has been validated with the stakeholders, it should be relatively easy to identify the indicators. Conversely, if the indicators are difficult to establish, this could mean that the theory of change needs more work.
7.3 Community-Based or Self-Evaluation Indicators
The CCB Standards require an “an evaluation of the [project] impacts by the affected groups” (CCB Criterion CM1.1). It is therefore important to identify stakeholder-identified or community-based indicators, both as part of the theory of change approach – when improved well-being, poverty alleviation, or similar impacts are sought – and as a complement to the theory of change approach in order to get stakeholder perspectives on project impacts. Communities have their own priorities for improving their lives and their own definitions of what constitutes a success or failure, and these are often different than outsiders’ criteria. Within communities, women usually have different criteria than men.
Community-based indicators can be identified by asking some simple questions18
• How do you hope this project will improve your lives?
to focus groups (including a women-only group):
• How could this project make your lives worse?
• What benefits are you hoping for from this project for your family and/or community?
• What would make this project a success for you?
• What would make this project a failure for you?
General questions can be followed up by more specific ones. For example, focus groups might say that they hope the project will improve the health of their families; further probing could result in an indicator related to a reduction in dysentery or other water-related health problems (see Catley et al. (2007) for further guidance on community-based indicators).
7.4 Criteria for Indicator Selection
The criteria for a good indicator are very similar but not identical to the criteria for objectives – indicators should be SMARS:
• Specific: the indicator should be defined and understood by all stakeholders in the same way – for this to be the case it must be transparent and an unambiguous measure of change
• Measurable: ideally it should be possible to record quantitative as well as qualitative changes in the indicator
• Achievable: the indicator should be realistic in terms of the cost and complexity of data collection methods
• Relevant and Reliable: the most relevant indicators are those that form part of a causal chain; a reliable indicator is one that gives consistent answers or numbers
• Sensitive: the indicator should change in proportion to changes in the condition or variable which it is measuring
18 At the verification stage, the questions can be even more direct such as “Do you think this project was successful? Why? (Or why
The cost-effectiveness of indicators depends largely on the data collection process necessary to measure them.
This is covered in more detail in SBIA Stage 6. One way of keeping costs in check is to use proxy indicators which are less precise and sometimes less objective, but are easier to record or observe. This is another instance where the principle of appropriate imprecision can be invoked. Proxy indicators are always recommended for very sensitive data such as income or wealth. An example of a proxy indicator for wealth is the quality of housing and/or, in some societies, the number of cattle; for income, a commonly used proxy indicator is expenditure.
7.5 Indicators for Negative Impacts
For potential negative consequences (SBIA Stage 4), indicators can be selected much as they would be for the expected positive results. For most negative outcomes or impacts, there will be symptoms (or indicators) to make one suspect a deeper problem, for example, lack of attendance at meetings, arguments, conflicts, disunity, desertion from the project, allegations of inequity in benefit-sharing mechanisms, an unexpected increase in local land or food prices, etc. If such signs are detected, participatory methods should be used to explore the underlying problem.
Negative impacts on biodiversity are often connected with project leakage, so that ongoing monitoring of land cover changes within and outside the project area is essential. Any loss of native vegetation will have a negative impact on biodiversity; therefore, a basic biodiversity indicator is the amount of native vegetation lost outside the project area due to the project.
7.6 SBIA Workshop Guidance
Our experience from several impact assessment workshops suggests that four (or ideally 3.5) days is the maximum time for an SBIA workshop, and conducting SBIA Stages 2, 3 and 4 can easily consume these days.
Identifying the indicators and developing a detailed monitoring plan (SBIA Stages 5 and 6) are demanding and quite technical tasks. It is therefore suggested that SBIA Stages 5 and 6 are undertaken by a carefully selected sub-group (say 6-10 people) from the main workshop with one or two additional M&E experts, if available. This should happen right after the main part of the workshop, possibly following a weekend break, while the information is fresh and the wall charts are still available.
It is suggested that two Monitoring Plan teams are formed, with each team working simultaneously on a focal issue monitoring plan. They should complete the following steps:
• Revise and possibly modify the results chain, checking especially on the cause-and-effect logic and looking for causal chains between outputs, outcomes, and impacts
• Review the earlier assessment of the most important or pivotal results and prioritize them (given that it is not practical or viable to identify and measure indicators for all the results)
• Identify causal chains between project outputs, outcomes, and impacts, as well as any key
assumptions, linkages, or intermediate results in these chains, especially between the outcomes and impacts
• Specify the most important results, especially intermediate results, as SMART objectives.
• Identify at least one indicator per objective using the criteria discussed above.
• Swap with the other Monitoring Plan team to review the objectives and indicators (or this swap could be done at the end of SBIA Stage 6).
Annex 1 Section 6 provides further guidance, and Table 4 presents some examples of objectives and indicators.
Table 4. Examples of Objectives and Indicators (SIA Case Study: GuateCarbon REDD Project)
Focal Issue Objectives Indicators
Strengthened Governance
By January 2012, the Coordinating Committee of the Maya Biosphere Reserve has the mechanisms to implement the environmental security strategy in at least 70% of the area
- Mechanisms approved
By March 2012, an effective program of community
leadership is being developed in 10 concessions - Community leadership program designed and implemented - Number of people trained By June 2014, at least 50% of judiciary operators in
the Petén are applying their specialized understanding of environmental legislation
- Number of judiciary operators trained
By December 2014, at least 80% of environmental
actions result in criminal sentences - Number of criminal sentences
Gender-Social Equity
By the end of 2011, [number] of project area communities and families are receiving training to strengthen shared family responsibilities
- Number of trainings received - Number of communities trained - Number of women, youth, and
others trained By the end of 2013, [number] of women have
finished primary education in the project area - Number of women who completed 6th grade
- Number of women reincorporated into primary education
By the end of 2013, at least three production projects are implemented by youth and women in the project area
- Number of projects being implemented
- Number of new initiatives - Number of women/youth
implementing projects By the end of 2013, multiple ethnic youth and
women are involved in community organizations and training courses in the project area
- Number of women and youth participating in community organization activities
- Percentage of annual increase in youth and women participants By the end of 2013, the management boards and
community committees are composed of 25-30%
women and children participating in decision- making
- Number of women and youth on community committees
7.7 Other Methods
Social Impact Assessment (SIA)
Another basis for establishing indicators is the Sustainable Livelihoods Framework (SLF) described in more detail in Social Toolbox Section 6; the SLF is used by the Brazil-based Social Carbon Standard as a basis for identifying indicators. A practical and participatory method of identifying indicators derived from the SLF is the Landscape Outcomes Assessment Methodology (LOAM) method (Aldrich and Sayer 2007). The LOAM method is described in Social Toolbox Section 6.4.
While a merit of indicators derived from the SLF approach is that they measure progress towards project sustainability, limitations, at least as regards meeting the CCB Standards, are that they do not factor in attribution and do not include a theory of change. An option could be to combine SLF indicators (for example, using LOAM) with matching methods or, more economically, participatory impact assessment (PIA) ranking and scoring methods to establish attribution (Catley et al. 2007). PIA methods are described in Social Toolbox Section 7.
In situations in which poverty reduction is a major social objective, the Basic Necessities Survey (BNS) is a systematic and cost-effective approach to measuring changes in poverty over time. The BNS method is explained in Social Toolbox Section 8. But again, it would be necessary to combine BNS with matching methods or PIA techniques to establish attribution.