This paper examines the characteristics of community health promotion and the challenges they present for evaluation. A review of health promotion evaluation leads to an exploration of newer approaches based on the ideas of complexity theory and development. A reflexive analysis of three program evaluations previously undertaken as an evaluation consultant is used to develop a conceptual model to assist in the planning and implementation of health promotion evaluations.
The findings suggest that the context-dependent nature of health promotion programs; turbulence in the community context and players; multiple stakeholders, goals and strategies; And. The use of the model should contribute to building cumulative evidence and knowledge to identify principles of health promotion effectiveness that may be transferable to new situations. A brief history of approaches to health promotion and evaluation sets the tone and leads to a discussion of opportunities for future development.
Health promotion writers have described a set of principles to guide practice that reflect the values of the Ottawa Charter (see Box 1). Community health promotion encompasses activities in communities based on the principles of the Ottawa Charter (Baum, 1998). However, it is argued that health promotion principles should guide the evaluation of health promotion initiatives (Poland, 1996; Tremblay et al., 2013).
The complexity of health promotion is illustrated by the interconnectedness of issues affecting health and well-being (Norman, 2009).
Reflections
The review of health promotion and evaluation presented above guided a reflective analysis of three program evaluations that I had previously undertaken in my role as an evaluation consultant (see Table 3 for summaries). Following this thinking, I drew on the relatively new ideas of development evaluation and complexity theory to build a conceptual model that could help health promotion evaluators recognize and deal with complexity. I then applied the retrospective nature of the model to one of the program evaluations to illustrate key learnings.
A number of implications become clear from this: the health promotion program and its design must be flexible to accommodate and respond to stakeholder input; the evaluator must take on the role of negotiator or mediator to bring different interests to a workable consensus, and stakeholders need the skills, resources, and interest to be meaningfully involved in the evaluation. Evaluation should be part of the planning and implementation cycle, providing rapid feedback and monitoring the changing context and outcomes of the initiative.
A conceptual model for CBHP evaluation
The layout of the model is not intended to suggest that planning, implementation and evaluation are linear. The model suggests that the evaluator, or evaluation team, should be involved in all phases rather than being brought in when the initiative is well established or towards the end of its life. These two sources of information are then used to produce a macro program theory and logic model, including predicted outcomes, to provide overall guidance for program development.
Ideally, program planners, funders, managers, and evaluators should work together to develop program theory. Program theory at this stage is an overarching model of how the program is expected to work and what outcomes are likely to be achieved, based on prior evidence and knowledge. The program theory must then be adapted to local conditions (Rey et al. 2012), which requires two filters: context and resources.
Consideration of the local context and community includes, for example, the identification of the population group for the program, the geographic area, the needs and strengths of the community. Applying these filters leads to the development of a more locally specific microprogram theory, or perhaps a set of theories. The power of the model is to then bring them together to make the theory of the program explicit and to draw a logical model of the program in a joint exercise.
In addition, the model enables identification of the impact of local contextual factors (micro) and likely resource constraints. The result of this filtering process is that a context-specific and realistic logic model of the program can be incorporated into the execution stage. The model illustrates that the implementation of complex interventions usually does not proceed in a linear fashion.
The program theory and the components of the logic model are continuously adapted to take these new issues into account, and in this way the criticism of program logic models as being too static and linear is addressed. Findings from the evaluation are context specific as they relate to the microprogram theory as it is adapted during implementation. The ability to identify principles of effectiveness can be enhanced as results from similar programs are built up through repeated application of the model.
Applying the Planning, Implementation and Evaluation Model to Healthy Ageing – Nutrition
Identifying the complex components of the project would have made it possible to plan its evaluation more effectively. This would have allowed existing evidence to play a greater role in program planning and implementation and would have provided a framework for developing a program theory and program logic model to guide evaluation. Documenting the project manager's and key stakeholders' working knowledge of how to drive and support organizational change in the context of healthy aging would also have contributed to the understanding of the program's underlying theory.
These changes in the planning phase all require the evaluator to have input during the early planning stages of the program and are likely to increase the level of evaluation resources. Complexity theory emphasizes the need to document emerging interactions, feedback loops, adaptations of the planned program and activities rather than focusing exclusively on processes and impacts expected from the original program logic model. Participants and stakeholders will also need an interest and understanding of evaluation and the ability to engage in this aspect of the project.
Changes in this phase will mean that the evaluator analyzes the impact of the internal and external environment on the implementation and subsequent impacts from the project. In the model, environmental analysis is a critical component of evaluation and should help the program remain relevant in a changing context. This should increase the usefulness of the estimate as it reflects actual events and not just what was predicted.
For example, in the Healthy Aging - Nutrition project, collaborative links between participating organizations were established that were not anticipated but were likely to continue after the project ended. The Healthy Aging–Nutrition Evaluation involved leaders of the participating organizations and several other stakeholders, but no input was sought directly from the workforce or clients. This was outside the scope of the evaluation but would have added to the comprehensiveness of the evaluation findings.
Developmental evaluation and complexity theory suggest that it is important to ask reflective questions as a way of assessing the quality of the evaluation and the evaluator's own. Consideration of reflective questions would serve as a stimulus to examine the overall quality and effectiveness of the healthy aging–nutrition evaluation process. While analysis of the evaluation data for Healthy Aging – Nutrition led to a model of facilitators and barriers to change in participating organizations, returning to refine the program's original logic model in light of this would strengthen and increase transferability. valuation finding.
Conclusions
Finally, the model proposes to identify principles of effectiveness that can be transferred to other contexts. Regarding the issue of transferability of findings, a further question can be asked: to what extent can the learning from evaluation of this initiative be applied in other situations. This means trying to identify the principles and assumptions underlying the program theory that contribute to effectiveness and that may be applicable in a different context.
In this way, cumulative learning could be used to build knowledge about what works in different contexts. The model therefore aims to contribute to the reduction of complexity and to the prediction of effective practical principles that could be applicable in a different setting or context.
Lessons learnt
Focus on fairness Equality of access and outcomes should be assessed as part of the evaluation. Interactions between stakeholders and subsystems of the initiative lead to new ways of implementation. As interacting and adaptive agents self-organize, ongoing connections emerge and agents co-evolve as parts of the system over time.
