FINAL EVALUATION OF POST
EMERGENCY PREPAREDNESS
OPERATION PROGRAMME
2013-2015
Report prepared for the International Red Cross and Red Crescent Societies and Thai Red Cross Society
ABSTRACT
This report serves to provide a glimpse of the spectrum of interactions and activities implemented by The Thai Red Cross Society in response to the prolonged and widespread flood in 2011. These scenarios of interaction show how the different stages of the project are structured will go a long way to serve the needs and alleviate the pain-points of different stakeholders who are part of the process.
P
ACRONYMS AND ABBREVIATIONS ... iii
EXECUTIVE SUMMARY ... iv
High project delivery based on disbursement rate ... v
Impact of strengthening network and partnership is much broader through social capital ... v
Project impact based on proportion of expenses by activities ... v
INTRODUCTION ... 1
1.1 Project Milestone ... 1
EVALUATION PROCESS ... 2
2.1 Evaluability Assessment ... 2
2.2 Consultative process ... 2
2.3 Transact walk/Field visit ... 3
2.4 Lessons learnt workshop ... 3
2.5 Experts’ opinions ... 4
2.6 Integration of information ... 4
2.8 Limitations ... 4
KEY FINDINGS ... 6
3.1 Relevance ... 6
Engaged partners within existing structure: ... 6
Procedures and tools were streamlined according to international and national standards: ... 6
Broadening scope of risk reduction activities: ... 7
3.2 Effectiveness ... 7
Increased confidence and capacity: ... 7
Associated with external organizations nationally and regionally: ... 8
Created new and retraced old networks through the volunteer group: ... 8
Importance of user centered design ... 8
3.3 Efficiency ... 9
Facilitated and enabling process:... 9
High project delivery efficiency based on disbursement rate: ... 9
Better coordination: ... 10
Light reporting process without compromising accountability: ... 10
Flexible funds reallocation within the TRCS level: ... 10
Guided and consistent engagement in project design and implementation: ... 10
3.4 Impact ... 11
Disaster resilient community: ... 11
Enhanced institutional preparedness and responsiveness through network: ... 11
Project impact based on proportion of expenses by activities: ... 11
Impact of strengthening network and partnership is much broader through social capital: ... 12
Cross-sectional communications: ... 13
3.5 Sustainability ... 13
Buy-in from leadership: ... 13
Centralized database: ... 13
Lessons learnt workshop: ... 13
3.6 Key take-away from Lessons Learnt Workshop ... 14
3.7 Process evaluation (based on findings from Lessons Learnt Workshop) ... 15
Conclusion and Recommendations ... 17
Indicators for project monitoring and evaluation purpose ... 17
Setting user centric project tone. Seeding collaborations at the onset. ... 17
Helpful templates, strong coordinator and guidelines to set the expectation in place. ... 17
Periodic review for staff to reflect on the project and partnership. ... 17
Regular monitoring to troubleshoot and support activities in the right direction ... 17
Clearly delineated roles to guide working relationships and reduce process pain-points ... 18
P
ag
e
ii
ACKNOWLEDGEMENT
This study has only been possible through the generous support from Mr Sanjeev Kafley, Regional DRR Advisor, Regional Community Safety and Resilience Unit (Disaster Management & Health) and Ms Rommanee Klaeotanong, Regional Disaster Risk Reduction Officer, Community Safety and Resilience Unit of the IFRC Southeast Asia Regional Delegation in Bangkok.
We take this opportunity to thank the Project Management Office (PMO) – Mr Chamnong Sangmahachai Ms Rongrong Khemarangsi, Ms Natrada Notananada, and Ms Ratjiporn Muangjan of the Thai Red Cross Society for their support, advices and guidance in ensuring the smooth implementation of the evaluation.
P
ag
e
ii
i
ACRONYMS AND ABBREVIATIONS
CBDRR Community Based Disaster Risk Reduction
DM Disaster Management
DMERT Disaster Medical Emergency Response Team GIS Geographical Information Systems
IEC Information Education and Communication
IFRC International Federation of Red Cross and Red Crescent Societies MEA Metropolitan Electricity Authority
MOU Memorandum of Understanding NGO Non-governmental organization
NS National Societies
PMO Project Management Office PNS Participating National Societies PEA Provincial Electricity Authority
PR Public Relations
RC/RC Red Cross/Red Crescent
SAO Sub-district Administrative Organization THB Thailand baht
P
ag
e
iv
EXECUTIVE SUMMARY
In response to the prolonged and widespread flooding in 2011, the Thai Red Cross Society (TRCS) provided emergency relief support to 1.3 million most affected people. No international assistance was requested. With support from IFRC, the TRCS has launched a project to increase preparedness capacities through activities relating to capacity building and community based disaster risk reduction. All activities under this project started from June 2012, and has recently received a one-year extension until end of 2015. The project consists of 59 sub-projects which include capacity building for the TRCS’ staff, improvement of infrastructure, community based disaster risk reduction activity, and development of Information Education Communication (IEC) materials. Refer to Annex 1 for a list of activities and their status. This consultancy was commissioned to facilitate the TRCS in developing lessons learned, and knowledge-base to shape and influence broader and longer-term disaster management governance in Thailand.
EVALUATION PROCESS
The evaluator conducted an evaluability assessment using the illustrated theory of change to guide the evaluation of the project. The process was followed with a five-day consultative process involving in-depth consultations, and transact walk/field visit (7-11 September 2015) in Phang Nga and Bangkok. A lesson learnt workshop (15-16 October 2015) was also conducted by a Thai-speaking consulting firm. Consultation and integration of information were conducted together with IFRC and the Project Executive Management team for insightful and useful learnings.
Evaluator’s perceived theory of change
It is important to note that the intent of this exercise is not to generalize findings, but to form a unique interpretation
of events from the TRCS’s perspectives. Thus this report should be understood as an exploratory study.
KEY FINDINGS
The key findings were synthesized from the 50 participants we met over16 conversations (one-to-one and group) as well as summary from the lessons learnt workshop. They were from different parts of the ecosystem – key decision makers, implementing officers and community members. IFRC also weighed in throughout the engagement to share interpretations of what was learned throughout the process. To make sense of the process, we use the following five themes – relevance, effectiveness, efficiency, impact and sustainability.
Key achievements Lessons learnt
Relevance Engaged partners within existing structure
Procedures and tools were streamlined according to international and national standards
Broadening scope of risk reduction activities
Effectiveness Increased confidence and capacity
Associated with external organizations nationally and regionally
Created new and re-connect with old networks through volunteers
Importance of user centered design approach
Efficiency Facilitated and enabling process Better coordination
Light reporting process without compromising
Effective emergency preparedness and
response
Set up technology, infrastructure and
equipment
Trained staff, volunteers and
community
Developed guidelines and IEC
P
ag
e
v
accountability
Flexible funds reallocation within the TRCS level Guided and consistent engagement in project design and implementation
High project delivery based on disbursement rate Impact Disaster resilient community
Enhanced institutional preparedness and responsiveness
Impact of strengthening network and partnership is much broader through social capital
Cross-sectional communications
Project impact based on proportion of expenses by activities
Sustainability Commitment from the leadership to assign a management-level staff to lead the project together with a dedicated team of PMO
Vision of building Volunteer Bureau’s centralized database (VITRIS) as well as consolidation of organizational information management through lessons learnt workshop and external evaluation was an indication of their commitment to shape and influence broader and longer-term disaster
management governance
The key take-away from Lessons Learnt Workshop were as follows:
Outputs What works What needs better support and coordination
Set up technology, infrastructure and equipment
Develop infrastructure (warehouse, sterile room, GIS and war room) by learning from best practice. For example, learning through exposure visits, training survey and consult specialist.
GIS project is an allied network of 13 government and semi- governments agencies including the TRCS. At the TRCS request, the procurement process was carried out by IFRC. The challenge of raising the rest of the funds from the rest of the allied network limited the potential of this network.
Trained staff, volunteers & community
Opportunity for large group communications through collaborations, recruitment, training and network building.
Insufficient resources (budget and materials), tight implementation turnaround, diverse target groups, inadequate understanding of community needs. Developed
guidelines and IEC
Comprehensive communication tool - Mr. Radar Cartoon and Animation, easy to read and understand; and opportunity for public education on numerous topics
IEC materials need to be contextualized further, and be delivered through appropriate strategy and more engaging communication tools.
Strengthened network and partnership
Red Cross branding and association with the Royal Family. Opportunity for cooperation across bureaus.
Streamline processes and procedures to improve cooperation and coordination.
CONCLUSIONS AND RECOMMENDATIONS
This report serves to provide a glimpse of the spectrum of interactions and activities offered by this project, how different stakeholders interacted with one another, and how the project benefitted the community, volunteers and staff. These scenarios of interaction show how the different stages of the project are structured and can go a long way to serve the needs and alleviate the pain-points of different stakeholders who are part of the process.
Introduction of disaster resilience organization balance scorecard for a more systematic approach to design ways of support and resourcing in a more targeted way that meet the organizational needs.
Setting user centric project tone and seeding collaborations at the onset using neutral platform
Helpful templates and guidelines to set the projects expectation in place. Different assessment criteria for different projects, and fast track approvals for “standard” RC activities like volunteer recruitment or training.
Periodic review for staff to reflect on the project and partnership and creating opportunities to adapt activities for
better outcomes
Regular technical and management monitoring to troubleshoot and support activities in the right direction
through a spirit of an equal, mutually supportive relationship.
P
ag
e
vi
P
ag
e
1
INTRODUCTION
In response to the prolonged and widespread flooding in 2011, the Thai Red Cross Society (TRCS) provided emergency relief support to 1.3 million most affected people. No international assistance was requested by the TRCS. Therefore, no emergency mechanisms were launched. Instead, the IFRC Disaster Management Annual Appeal was revised with an additional project component to accommodate the contributions provided by various donors in support to the TRCS Relief and Post Emergency Preparedness operation.
With support from IFRC, the TRCS has launched a project to increase preparedness capacities through activities relating to capacity building and community based disaster risk reduction. All activities under this project started from June 2012, and has received a one-year extension till end of 2015. The project consists of 59 sub-projects which include capacity building for the TRCS’ staff, improvement of infrastructure, community based disaster risk reduction activity, and development of Information Education Communication (IEC) materials. Refer Annex 1 for list of activities and their status.
1.1 Project Milestone
- Project started its inception phase
- Submitting activity plan to IFRC year until signing of agreement timeline to 1 years
- Continue with to clear and settle all financial matters
Initial Project Agreement signed on 14 June 2013 for the duration from
June 2012 – 31 December 2014
Sign 2nd project amendment to extend project agreement for 1 year in 2015
P
ag
e
2
This consultancy was commissioned to facilitate the TRCS in developing lessons learned, and to build the knowledge-base in shaping and influencing a broader and longer-term disaster management governance in Thailand.
EVALUATION PROCESS
This exercise was understood to be both retrospective and forward-looking, in that it will describe how resources have been used and make recommendations to improve the process for the future.
2.1 Evaluability Assessment
The evaluator conducted a cross-stakeholder document synthesis of IFRC’s documents and generated a perceived overarching program rationale, Theory of Change, and causal pathways to guide the evaluability of outcomes and impacts over the duration of the project.
Figure 1: Evaluator’s perceived theory of change
2.2 Consultative process
A five-day consultative process with both field (Phang Nga) and headquarter (Bangkok) staff was planned over 7 to 11 September. Rather than a formal interview, the session was conducted more as a casual conversation. This allows time to open up and for us to explore deeper to glean insights of the situation. They were invited to share their experience and to openly give feedback on what works and what doesn’t. It directed the evaluation to explore deeper into the disaster management process which impacts the community directly, and pointed towards a learning process and experience. This approach also fit into the time and resource constraints of IFRC and the TRCS.
The team spent about 30 to 90 minutes to chat face-to-face. A total of 7 group discussions with the communities, and 22 discussions with staff either in group or individually were conducted in total. Insights gathered through several informal discussions and observations were also included.
Effective emergency preparedness and
response
Established technology, infrastructure and
equipment
Trained staff, volunteers and
community
Developed guidelines and IEC
Strengthened network and
P
ag
e
3
2.3 Transact walk/Field visit
When we conduct in-depth conversations in the context of the people’s environment, it adds experiential data to the exercise and helps us to ask better questions. A team, formed by the consultant, IFRC Regional Disaster Risk Reduction Officer, 2x TRCS headquarter and 4x TRCS health station staff, visited two communities - Phang Nga and Koh Mai Pai Island - to better understand their situation; and how the TRCS intervention had met their needs.
Figure 2: Table top exercise at Koh Mai Pai Island
2.4 Lessons learnt workshop
Colleagues from different departments within the TRCS were invited to spent two days, 15 and 16 October 2015, reflecting on design process, the knowledge they have learnt and experiences they acquired around 3 key themes: People, Project Management and Project Implementation.1 Getting the team to converge on a common platform helps to encourage learning and insights by borrowing elements of experience through listening to one another and developing a sense of mutual
1
People: Encourage people to see individual’s value and ability, learning from experience of working with others to implement
activities. It is the result of understanding oneself and the interpersonal relationship through “flower in my mind” activity, the review of
P
ag
e
4
understanding. This process was contracted to another external consulting agency with highly skilled facilitators who are skilled in Thai language.
Figure 3: Lessons learnt areas
2.5
Experts’ opinions
The consultants work through the information with the IFRC Regional Disaster Risk Reduction Advisor and Officer as well as the TRCS project coordinator. Key comments that were surprising, insightful or useful were discussed and synthesize as rich learnings.
2.6 Integration of information
Data and inputs from the consultative phase and lessons learnt workshop was used to triangulate the external and internal validity of observed trends. This mixed methods approach leans strongly on integrating dynamic and participatory approach with a strong utilization focus. Given the various implementing models and community dynamics contexts across bureaus, the process documentation will be especially key in understanding the project.
In order to quantify the qualitative findings according to the theory of change (refer section 2.1), a simple scoring matrix was developed to indicate the level of contribution towards each component of the theory of change which is:
Table 1: Scoring matrix NA Not applicable + Contributed a little ++ Contributed quite a bit +++ Contributed a lot
2.8 Limitations
The possible biases for this evaluation are as follow:
Hawthorn effect (respondents’ bias knowing that they are being observed),
Interviewer/Translator bias (knowledge of evaluator and translator which influences the way the question was asked and how the responses were interpreted), and
Social desirability bias (when respondents answer in a way to please the interviewer) Technology &
infrastructure
Network and partnership Guidelines
and IEC Training
Project Management
Relationship
Skill/Experience
Mind
Attitude Project
P
ag
e
5
Additionally, it should also be noted that the evaluator was unable to reach all the project sites to validate the information documented for this project. It is important to note that the intent of this exercise is not to generalize findings, but to form a unique interpretation of events from the TRCS’s perspectives. Thus this report should be understood as an exploratory study.
P
ag
e
6
KEY FINDINGS
The findings in this section is synthesized from the 50 participants we met over16 conversations
(one-to-one and group) as well as summary from the lessons learnt workshop. They were from
different parts of the ecosystem – key decision makers, implementing officers and community members. IFRC also weigh in throughout the engagement to share interpretations of what was learned throughout the process. To make sense of the process, we structure into five themes – relevance, effectiveness, efficiency, impact and sustainability.
3.1 Relevance
It is assumed that the primary objective is to reflect on the way which the TRCS and IFRC proposed to achieve its selected goals and objectives. Secondly, relevance refers to the extent to which tasks have actually been achieved, as reported in documents and interviews. Important aspects of this analysis are how plans and actions take the internal and external contexts into consideration.
Table 1: Relevance score
Theory of change/Outputs Existing
structure
Standards Scope of
content
Set up technology, infrastructure and equipment NA +++ NA
Trained staff, volunteers and community +++ +++ ++
Developed guidelines and IEC NA +++ ++
Strengthened network and partnership +++ +++ NA
3.1.1 Key achievements
Engaged partners within existing structure: The TRCS highlighted that all activities were conducted in cooperation through partners within existing structures, be it schools or villages. Needless to say, involving local government throughout the process will boost the public confidence in the government’s disaster response and integrate the TRCS into the community eco-system.
Procedures and tools were streamlined according to international and national standards: During the 2011 emergency, it was reported2 that existing national policies developed were not implemented. It was found that the government applied the Prime Minister’s Policy to manage the event instead of the Disaster Prevention and Mitigation Act. There were gaps in management and coordination among government agencies from national to local levels. Additionally, IFRC impact largely depends on the strength of individual National Societies (NS). As such, it was pertinent to promote the TRCS development based on international and national standards. An extensive list of training and exposure visits were made available to enhance learning for staff and volunteers:
Development of tools and materials to disseminate information
Exposure visits and learning from IFRC Kuala Lumpur and private firm
Promotion of collaborative training for the TRCS, government, other NGO and PNS staff
Development of policies and minimum standards to guide disaster response o SPHERE standards – Water and Sanitation
o IFRC Financial, Procurement and Logistics guidelines
2
P
ag
e
7
Disaster Medical Emergency Response Team (DMERT) – participated together with other public and private actors; deployed staff for Nepal Earthquake response.
3.1.2 Lessons learnt to strengthen project relevance
Broadening scope of risk reduction activities: Community Based Disaster Risk Reduction (CBDRR) activity in Phang Nga benefited five out of the 14 communities which they had previously worked with during the 2004 tsunami. Since the 2004 tsunami, many of their livelihoods have switched from being fishermen to boatmen, canoeing, island visits among other ecotourism activities. Both communities visited during this evaluation highlighted that they were “physically not affected” – no lives lost or serious infrastructure damages in both 2004 and 2011 events - except for loss of income for a period of 3-6 months. It was shared that CBDRR could include early recovery component like alternative livelihood support as part of preparedness to enable them to cope with and recover from disasters quicker. At the same time, staff also reported that there is potential of broadening CBDRR to integrate other relevant Red Cross components like environment consciousness, climate change, garbage management and preservation of mangrove forest.
3.2 Effectiveness
The main results will be compared with the goals set in the initial proposal. Factors contributing towards “significant changes” will be defined as proxy success criteria.
Table 2: Effectiveness score
Theory of change/Outputs Confidence &
capacity
Connection User-centered
Set up technology, infrastructure and equipment +++ +++ +
Trained staff, volunteers and community +++ +++ +
Developed guidelines and IEC +++ ++ +
Strengthened network and partnership +++ +++ NA
3.2.1 Key achievements
Increased confidence and capacity: This is the first time where all the bureaus in the TRCS were engaged and participated directly in a large grant. Despite some of the challenges and frustrations, there was a resounding sense of satisfaction and achievements.
“There may be politics within the TRCS. But whenever something happens, we come together and work
together. In this project, there is shared resources, shared work and shared honour.”
“We visited Modern Pharma Co., Ltd - one of our medicine suppliers to learn about their production process, quality control, transportation, warehouse management. Additionally, through the funds of this project, we also got the opportunity to visit the Port Authority of Thailand to learn more about custom clearance process as well as to learn how to manage space for large items. It was also an opportunity to build our relationship with them.”
P
ag
e
8
Associated with external organizations nationally and regionally: Types of linkage ranged from purchasing goods and services from providers – engaging them in the needs and requirements of disaster work – to opening up internal training opportunities for government and NGO staff and volunteers as well as engaging them in learning trips.
“We set up the disaster database and IT system and the main objective is to assess damage level accordingly and to help disaster affected people based on their needs. TRC is working on the information integration cooperation with allied network like MEA, PEA and a few other organizations to hire a university team to develop the GIS. This type of cooperation is first time in Thailand.”
“The TRCS send a team of 13 persons for 3-week training at Emergency Medical Institute of Thailand - 1 week on theory and twice field exercises for 1 week each. It was conducted in English because it is an international training. This team is a certified DMERT team. After the training, they activate our trainees to respond to Nepal earthquake. One doctor and one assistant nurse went for two weeks which costs were all borne by MOH Thailand. We recreated a similar training for five days and trained 40 TRC staff and volunteers from the medical service division and health station as well as 35 external people. For this in-house training, we organized it in Thai language using the same trainer team as the
one organized by Emergency Medical Institute of Thailand”
Created new and retraced old networks through the volunteer group: Volunteers are the backbone of RC/RC Movement. This project allowed the TRCS to re-connect with old and build new networks within the communities and academic institutions through recruitment and training. Some of these activities were implemented by the Youth Bureau where they created the Youth Volunteer Network. They stepped up more volunteer trainings in Bangkok, as well as in the provinces – Chantaburi (100), Nan (150), Surathani (190). Other similar activities were also implemented by the Administration Bureau/Communication where they worked with the TRCS Public Relations and Youth Volunteers, as well as the TRCS Nursing College where they created the Flood Disaster Preparedness Volunteer Network for University and College Level Students.
3.2.2 Lessons learnt to strengthen project effectiveness
Importance of user centered design3 approach: It is never an easy feat to design a course that allows optimal absorption of the material to be similarly achieved by different profiles of participants. This is specifically the case when learning was viewed as a process of transferring knowledge.
“There was no common selection criteria. Each bureau was allowed to send whoever they want. There were different levels of English command and different seniority. I didn’t get much out of the course after 5
weeks… it’s a pity because I was looking forward to it.”
Without a user centered design approach, mixed effectiveness results are also observed across a number of activities.
“I don’t think the community are particularly interested in the guideline and IEC materials produced…
except Mr. Radar Cartoon and Animation. It is easy to read and understand. We need communications that could capture the attention of different segment of the community.”
The following two examples also showed the importance of user centered design approach:
Example 1: The rack storage of flat-bottomed boats at Angthong Warehouse was designed to be three layers. Due to budget constraint, the design did not take into consideration the human safety
3
P
ag
e
9
aspects such as height of ceiling, standing space, and level of difficulty in pulling or pushing the racks around. Currently, only bottom rack is used to store the boats while the other two layers were used for other storage purpose. The hoist equipment is also not ideal as it sways when pulling up the boats.
Example 2: While constructing the structure of Angthong Warehouse, the engineer did not take into consideration of the volume of water which will go into the water purification units. Eventually, the Administrative bureau, RCHB had to use their own budget (instead of project fund) to make changes to ensure the structure is able to take the weight of the units.
3.3 Efficiency
To assess efficiency, we will use the internal organization and management and then discuss if the systems are in place to support efficiency.
Table 3: Efficiency score
Theory of change/Outputs Facilitated Coordination Reporting Funds Governance
Set up technology, infrastructure and equipment
NA NA NA NA NA
Trained staff, volunteers and community +++ + + + +
Developed guidelines and IEC NA NA NA NA NA
Strengthened network and partnership +++ + + + +
3.3.1 Key achievements
Facilitated and enabling process: Despite the initial hiccups, the TRCS staff acknowledged the helpfulness of the facilitated and enabling project implementation process led by the PMO and the Executive management led by Mr Chamnong Sangmahachai, Assistant Secretary General for Human Resource/Project Chairman, and Ms Rongrong Khemarangsi respectively noted this. They felt supported and were able to approach them to negotiate terms to ensure work is completed according to plan and objectives
“Transparency and accountability should be mutual between the TRCS & IFRC. We rely on IFRC important role as facilitator and negotiator. The competency of individual delegate is crucial to get things
done.”
High project delivery efficiency based on disbursement rate: The high disbursement rate for both the TRCS, that is (96.0%) and IFRC (84.0%) indicated a high project efficiency in terms of delivering the outputs expected. Refer Figure 5.
P
ag
e
10
3.3.2 Lessons learnt to strengthen project efficiency
Better coordination: It was understood that there are two ways of volunteer recruitment – either through the Volunteer Bureau or directly through the operating bureau. This project created opportunities for all bureau to recruit and train old and new volunteers, ranging from Administration Bureau-Communication, Youth Bureau, Relief and Community Health Bureau, TRC Nursing College, Human Resources Bureau, and Volunteer Bureau. There was neither coordination nor oversight by any bureau including the Volunteer Bureau.
Light reporting process without compromising accountability: An onerous reporting process focused on accountability may affect the best interests of achieving the strategic outcomes in terms of better and cost-efficient impact.
“This project is administration intensive. Unlike other activities we had to do monthly instead of quarterly cash request. So basically we lost about a week in a month to do financial reporting for a few days in the first week of each month and cash request for a few days in the last week.”
Flexible funds reallocation within the TRCS level: IFRC has an agreement with the TRCS that new tranche of funds will only be disbursed if the previous tranche of funds has been 80 percent utilized. IFRC left it to the TRCS to manage the funds internally between bureaus and activities.
“Normally our unit provide reasonably accurate cash forecast and utilize them within each quarter. However, some other units might have their activities delayed either due to difficult procurement or some other reasons and not able to use their funds according to their projection. This affected our unit as we could not receive any funds for three months. If the activities are urgent, sometimes we use our own funds from the bureau and replace them later. And sometimes we delay our activities by one quarter… it is not easy to manage the activities like this specifically if the activities involve the community.”
Guided and consistent engagement in project design and implementation: While the staff expressed a strong sense of ownership and achievements towards the end of the project, they had also expressed confusion over the initial objective of the project.
“In the beginning, we were not clear about the main objective of this funding. First, they told us that the
end beneficiaries should be the community… but the project started too late so we have to change project to our own capacity building… It was very confusing…”
“For tsunami, money goes through International Relations for disbursement at Relief bureau. ADB’s money went through finance bureau to support recovery activities for all bureaus This time, we have a
recovery committee which comprises of a chairman and staff of different bureaus… nobody understood
the process in the beginning…”
It was apparent from several conversations that the project was intended to be a bottom-up approach with minimum interference from the Project Management Office (PMO). The perceived role of PMO role which comprised of a chairman, focal person from each bureau and secretariat team was that of an enabler through the process of screening, approving and coordination.
The quarterly coordination meetings were platforms for reporting rather than opportunities for feedback, discussion and learning. To some extent, the political environment in the TRCS and IFRC encourages populist approach, which provides space for misunderstandings and miscommunications to take place. Lack of clear established guidelines led to frustrations, duplicative work and tension.
P
ag
e
11
implementing partners. This could also include introducing various control mechanisms to demonstrate transparency, fairness and value-for-money between different activities.
It was also mentioned that the Memorandum of Understanding (MOU) between IFRC and the TRCS has no detail information or instruction concerning the objectives, financial and procurement process. They did not have anything to fall back on to guide them. In their perspective, procedures changed according to individual staff preference.
3.4 Impact
Figure 7: Evaluator’s perceived theory of change
3.4.1 Key achievements
Disaster resilient community: Community members shared that they are children of the sea and they have acquired acuity in weather forecast through observing the wind direction and waves movements. They have also their own acquired system to deal with these stresses. However, CBDRR had brought them together as a community to create a systematic disaster management plan. Specifically, community members from Koh Mai Pai mentioned that they benefit more from CBDRR as they live on an island with limited access to the resources on mainland.
Enhanced institutional preparedness and responsiveness through network: The basic task of humanitarian logistics comprises acquiring and delivering requested supplies and services, to the places and times when help is needed, whilst ensuring best value for money. In the immediate aftermath of any disaster, these supplies include items that are vital for survival, such as food, water, temporary shelter and medicine, among others.
“With the new pallet and replacement of conveyor belt, there was improvement in speed and correctness of the storage, preparation and disperse of items for relief operations. We had an effective warehouse management. There was also reduction in reports of injury for warehouse staffs during the operation”
“All equipment installed post-tsunamic 10 years ago. No maintenance since then. With new budget to improve the communication system – cable signal. Improvement. Prevent lightning strike – prevention in the system. Communicate between health station and signal go further.”
“Also use as war room and where decision is made. Before renovation, it was just one of the meeting
rooms. Equipment was not good so difficult to gather information. Technology connection enables them to get information to make decision from internal and external. Video conference with field staff. Communication and vehicles. Command system. Currently, they are also able to pull out information from online metrological and video conference. Mobile communication with vehicle back through system.
They are able to run on Android mobile if staff download Vidyo”
Project impact based on proportion of expenses by activities: Unsurprisingly, close to 50% of the expenses went towards setting up technology, infrastructure and equipment, followed by a quarter of
Effective emergency preparedness and
response
Set up technology, infrastructure and
equipment
Trained staff, volunteers and
community
Developed guidelines
P
ag
e
12
expenses for developing guidelines and IEC materials and a fifth of budget for training staff, volunteers and community.
The proportion of the budget is relatively small in terms of strengthening network and partnership, however every component of this project contributed towards profiling the TRCS; hence strengthens their networks and partnership at national level.
Figure 7: Proxy project impact based on proportion of expenses by activities
3.4.2 Lessons learnt to strengthen project impact
Impact of strengthening network and partnership is much broader through social capital: Based on an earlier study on “Role of Social Capital in strengthening resilience in Thailand”4, Thai communities are characterised as a society built on strong ties, albeit ones primarily comprised of family, together with other locally-based ties with friends and neighbours. To an extent, it is clear that in their immediate networks, people care and look out for each other, which include women, the disabled and the elderly. This resonates in line with the TRCS specific education activity which targets the older people to address their health and social protection needs. However, the study had also highlighted that for the minority who lack social capital, life can be extremely difficult. As such, further investments in social capital needs to be broad-based and inclusive, with a focus on integrating marginalised populations, ranging from documented and undocumented migrant communities to political and religious minorities.
The administrative bureau created a PR Volunteer network comprised of working adults. It was reported that most of them used to be communication with each other and with external media bodies using letter writing and fax. However, through intensive workshop supported by this project, they learned new communication approaches like email, Facebook, phone, use of video and camera. This network helped the headquarter staff to keep in touch with the provincial chapter and work with them to reach a wider audience through the formal media. So far, a total of 20 regional and national articles
4 http://www.ifrc.org/en/publications-and-reports/evaluations/?c=&co=SP351TH&fy=&mo=&mr=1&r=&ti=&ty=&tyr=&z=
Trained staff, volunteer and communiy
19%
Developed guidelines and IEC
26%
Strengthened network and
partnership 4% Set up technology,
infrstructure and equipment
46%
P
ag
e
13
has been generated by PR volunteers. The bureau expressed high interest to scale up this activity as well as to educate them about use of RC emblem.
Cross-sectional communications: It was reported that interactions within the TRCS are mostly ‘vertical’ and confined between a bureau itself. This project was instrumental in seeding lateral connections for more cross-collaborations.
3.5 Sustainability
We propose to assess the TRCS and IFRC’s commitment to and record of building community’s capacity to sustain and perpetuate achievements.
3.5.1 Key achievements
Buy-in from leadership: In order to ensure project sustainability specifically in the community, it is important to get the buy in from community leaders and local government. Health station staff highlighted that one of the main strengths of their activities was that they chose to work with communities whom they had established trust and working relationship during 2004 tsunami. It was a continuation to previous efforts that will further strengthen the social capital and community resilience. For example, when there was small earthquake recently, the TRCS was able to establish communication and get information from the community quickly even though there was no planned intervention. It was documented that early warning and evacuation was carried out according to what was agreed and practice when the TRCS introduced CBDRR.
Centralized database: The challenge of volunteer management is not about recruiting or training volunteers. The challenges are about coordination and mobilization. As such, the TRCS invested in a centralized database – VTRIS – managed by the Volunteer Bureau. The first phase to create the database is completed and they are working on the second phase which is to integrate the volunteer databases in all bureaus. There was apprehension about the challenges ahead but all agreed that this is the way forward to ensure a sustainable volunteer engagement and management.
P
ag
e
14
3.6 Key take-away from Lessons Learnt Workshop
Table 4: Key take-away
Theory of change/Outputs What works Need better support and
coordination
Set up technology, infrastructure and equipment
Renovate and develop infrastructure by learning from best practice. For example, field visit, training survey and analyze problems, consult specialist.
The list of technology, infrastructure and equipment includes:
Warehouse: Renovate to improve the conditions with equipment to speed up the process and reduce staffs’ workload
communication system for disaster relief operations/work with allied network
War Room: Renovate and
upgrade communication system with new technology to provide information support and real time situation updates during crisis/disaster situation
The GIS project is an allied network of 13 government and semi- governments agencies including the TRCS. At the TRCS request, IFRC invested 300,000 CHF for procurement of software. The challenge of raising the rest of the funds from the rest of the allied network limited the potential of this network.
Trained staff, volunteers and community
Communication with target group works
Form network through community
and academic institution
collaboration
Training material, training media, good training courses and training schedules
Train youths and build their volunteer spirits together with competency building (knowledge, capability, and attitude)
Train the trainer courses to roll out training to youths under volunteer network
Monitor trained trainers and roll out the training
Small budget (some activity) Not enough equipment
Short timeline for implementation Availability of target group due to
different schedule
Some target group could not attend due to work priority
Assessment process for community based initiative need to be further comprehensive to identify the appropriate support in order to address the underlying causes.
Developed guidelines and IEC Mr. Radar Cartoon and Animation are easy to read and understand Produce communication media with
P
ag
e
15
Lacking knowledge management Outreach did not target teenager
like use of on-line game
Strengthened network and
partnership
RC image and public confidence was enhanced. Also strong association with the Royal Family Positive culture in terms of receptive
to other opinions, respect, open
Work continuity and supports
Need to organize relationship improvement activities
Systematically set up database of allied network like financial database, improve access to resources database, etc.
Share and exchange database within the TRCS organizations and come to a mutual understanding Improve/set up policy that support
the collaborations with other high potential organizations
3.7 Process evaluation (based on findings from Lessons Learnt Workshop)
Table 5: Process evaluation
Process What works Need better support &
coordination the project (financial viability to run the project)
Delay transfer of funding affecting the project implementation
The project financial procedure does not follow the TRCS financial procedure
Change procurement
and financial procedure during the project procurement from small shop without proper receipt, the TRCS accept the certificate of receipt while IFRC would only accept if amount is small. Different understanding
between IFRC and the TRCS concerning the procurement clauses on agreement of this project brought some confusions in the initial phases. It was resolved after a number of orientation sessions to the relevant colleagues of the TRCS on the procedure of
IFRC financial
management and
procurement system. The agreement was also amended accordingly.
Procurement Procedure Transparent and
verifiable
Procurement condition
creates confusion
because of differences in implementation process
Provide management fee Outsource Accounting
and Finance
P
ag
e
16
procedure before signing agreement enough resources for basic needs
The TRCS and IFRC supported creation of community network and social capital
Knowledge and skills from CPR training helped disaster victims and created volunteer spirit. Participants commented that there were changes in community behaviors in terms of helping each other as compared to before
Beneficiaries said they would like the TRCS to understand their needs (people, organization,
Beneficiaries suggested to involve local government like SAOs during field work. This way SAOs will understand the context, needs of communities, and actually be a part of disaster preparedness and response. Communities need support from SAOs as their work will be tougher without the help from SAOs.
Project Management Stakeholders gain more benefits
A good project for under privileged group, increase volunteer network, and communicate on what the TRCS does to public Work according to plan
and objectives
The project allows the TRCS’ bureaus to plan and implement activities according to their together with consultant to provide advice
Set up specific timeline for project
Project Management
Office to coordinate between the TRCS and IFRC
Good liaison person helping with project
implementation and
solving problems
There is a good team to coordinate between the TRCS and IFRC
The project is supported
by Executive
Management
Funding viable to support
Unclear procedure and
Delay in budget approval Change in work process
by always adding new process
Different understanding during coordination with relevant bureau/unit Evaluator came without
prior discussion and planning together with the team
There should be better orientation to the agreement between IFRC and the TRCS about MoU; clear information and
direction for the
implementation prior to the start of project - like financial procedure, report, etc.
Set up common platform
where project
implementing teams can share and exchange and acknowledge problem on a regular basis
Based on local country implementing team to submit project proposal Recruit dedicated project
team
Consider to continue project on long term basis
Consider project
P
ag
e
17
the project
implementation
for the TRCS
Conclusion and Recommendations
This report serves to provide a glimpse of the spectrum of interactions and activities offered by this project, how different stakeholders interacted with one another, and how the project benefitted the community, volunteers and staff. These scenarios of interaction show how the different stages of the project are structured and can go a long way to serve the needs and alleviate the pain-points of the different stakeholders who are part of the process.
Indicators for project monitoring and evaluation purpose.
Most of the investment of this project went towards strengthening institutional capacities – mainly financial, administration, logistics, information systems and human resources. It should be noted that a qualitative evaluation is only as useful as the quality of insights shared by the participants. This was partly due to a lack of project assessment indicators5 - outcomes and output - to guide the direction for implementation as well as to set a baseline for this evaluation. A quantitative framework is indicative and should be significant for a more systematic approach to design ways of support and resourcing in a more targeted way that meet the organizational needs.
Setting user centric project tone. Seeding collaborations at the onset.
When the TRCS receives information about an exciting new funding, IFRC will be engaged to facilitate an inception workshop for all the TRCS’ bureaus in discussing about project design, project implementation plan and to support needs assessment, if needed, to be conducted in the community. This will set the project tone to be user centric where stakeholders at each level are engaged right from the beginning. It also seeds collaboration opportunities.
Helpful templates, strong coordinator and guidelines to set the expectation in place.
To avoid bottlenecks, templates and guidelines should be developed to guide the process. Secondly, the initial concept of embedding the PMO team with a strong project coordinator within the TRCS has contributed positively towards the success of this project. Perhaps project coordinator having a direct reporting line to IFRC could have helped the communication process between the TRCS and IFRC. Lastly, different assessment criteria could be recommended for different projects, as well as fast track approvals for “standard” RC activities like recruitment or training of volunteers.
Periodic review for staff to reflect on the project and partnership.
Instead of waiting until the end of the project to have Lessons Learnt workshop and an external evaluation, the project could budget for a mid-term review workshop to bring all stakeholders together to discuss what works and what doesn’t. If specific part of the project or partnership is not working, alternative should be considered and be taken up promptly. For more complex projects, annual review may also be required. Collaboration between different bureaus or health stations should be encouraged especially if activities are similar.
Regular monitoring to troubleshoot and support activities in the right direction.
Besides having regular coordination meetings with main members of bureaus to discuss issues, there could also be a network of technical and management staff who will conduct periodic monitoring visits to see activities in action, to offer advice and consultancy and troubleshoot on the spot together.
5
P
ag
e
18
Implementing staff should not feel like they are being monitored or micro-managed. Instead they should feel like an equal, mutually supportive relationship.
Clearly delineated roles to guide working relationships and reduce process pain-points.
IFRC and the TRCS to position their roles as both enabler and regulator throughout the process. There should be clear delineation of roles and responsibilities, as well as a streamlining of processes. Cash request and disbursement from IFRC to the TRCS on a six-monthly basis whereas the TRCS to bureaus on a three-monthly basis is recommended. Two-way communication to be encouraged so that staff would receive feedback based on the report submitted.
P
ag
e
a
ANNEX 1
LIST OF ACTIVITIES
No. Proposed Activities Implemented Location Process
Remark
Started
date Ended date
1 Relief Community Health
Bureau
1.1 English Language
Development for TRCS Staffs
1.1.1 - Asian University Program (5-week duration)
1.2 Technical Knowledge Training Program (s) for Finance Staffs
(Public Training) Cancelled
1.3 "Finance in Emergency" Training Program - IFRC standard emergency finance
1.4 Incident Command System (ICS) Training - 2 workshops, 40-50 people/workshop
1.5 Donation Management Training
Cancelled
1.6 Build Warehouse/Storage Location for Water Purification Units and Flat-bottomed boats
Angthong 14 June
2014
9 January 2015
Completed 1.7 Increase Warehouse
Operation System to Support RCHB's Flood/disaster Relief Assistance Operation
1.7.1 - Improve RCHB's Warehouse Physically to support the operation during flood/disaster 1.7.2 - Improve Medical Equipment
Preparation and Disperse Room
Bangkok September
2013
December 2013
Completed 1.7.3 - Purchase Office Equipments
(Stationery and Office Asset) to support the operations 1.8 Logistic Training and
development Program
April 2014 Completed/Modern Pharma & Port
ie. T-shirts, caps, waistcoats, backpacks
Q4/2013,
Q1/2014
Q4/2014
Completed
P
1.13 Develop and improve communication system
1.14 Develop and improve Disaster Operation center
Bangkok 26 March
2014
Q4/2014 completed
1.15 Improve and upgrade Disaster Database and IT system (GIS)
Bangkok/Ayutthaya (Phase 1)
Q4/2013 Jan 2016 Phase 4 of 4
hand-over and payment in Jan 2016
1.16 Rapid Action Team training (NDRT)
1.17 Central Region Disaster Response Team Seminar - a collaboration session with central region authority (3 days)
Ayutthaya 21-23 July
2014
21-23 July 2014
completed
1.18 Proactive Disaster Relief Program - Strategic Planing
Level Cancelled
1.19 Training 30 Nurses and TRCS staffs on how to use Water Sanitation and Hygiene Operational
Q4/2013 Q4/2013
Completed 1.20 Disaster Relief Operation
Response Team Seminar - a collaboration session with 1.21 Family Kits for Disaster
P
1.22 Water and Sanitation Project for Water Purification Unit Staffs
Ayutthaya Q4/2012 Q1/2013 completed
1.23 Disaster Management Tool : HDV cam recorder with memory recording unit
Bangkok 7 December
2012
7 December 2012
Completed
1.24 International Day for Disaster Risk Reduction (2012-2014)
1.25 Building Capacity for TRC Drivers of Large Vehicle Driving Skills
1.26 Staffs to attend public training Program on Disaster Medical Emergency Response Team (DMERT) and procure equipment for the field training session
knowledge training session on Disaster Medical Emergency Response Team (DMERT) and procure equipment for the field training session
1.28 Translate, edit, develop manual and organize the introduction program on trend of Epidemic in Thailand / Sphere 2011 Book translation and printing
Q4/2014 Q4/2015
completed
1.28.2 Sphere 2011 Book translation
and printing (new activity)
Q4/2014 Q2/2015
Printing process
Pilot Community Safety and Resilience (CSR) Initiative - Project Period end in December 2014 (1.29-1.33)
1.29 Community and school teachers mobilization and
1.30 Training and capacity building Chacheungsao September
2014
November 2015 1.31 Assessment : cross sectorial
P
ag
e
d
1.33 Set up Early Warning System at community level
- Conduct a learning session for 120 community people - Procure equipments that the 2 pilot communities use for their EWS
1.34 Institutional support for implementing RC branch
Cancelled
1.35 Mitigation and preparedness
intervention Cancelled
1.36 Half Year Meeting Review Cancelled
1.37 Disaster Preparedness for Senior Citizens
Chainat/Ayutthaya/Ba ngkok
Q4/2014 Q1/2015 Completed
1.38 IEC Materials
1.38.1 - Diseases happening in each
season of Thailand
July 2013 September
2013
Completed
1.38.2 - Disaster Information Q4/2014 Q4/2015 Completed
1.39 Telephone System
Improvement Project (RCHB)
February
2014
April 2014 Completed
1.40 Generator Installation Project Cancelled
1.41 Renovate emergency response room on the 3rd floor of Disaster Relief Building
1.43 Project meeting (Quarterly meeting/mid-year/year-end review)
June 2012 21 Dec 2015
Completed
1.44 Lesson Learnt workshop Oct 2015 Oct 2015 Completed
2 Administration bureau -
Communication
2.1 TRC Young Reporter Project 2-day training for 4 groups, 80-100/group
4 Regions Q4/2013 Q3/2014 completed/4
workshops
2.2 TRC's PR Volunteer Project 2 days training 70
people/group
4 Regions Q4/2013 Q3/2014 completed/4
P
ag
e
e
2.3 Develop, Improve and Maintain TRCS website - English Version
Cancelled
3 Youth Bureau
3.1 First Aid Training for TRC Youth Volunteers (disaster preparedness)
Bangkok Q1/2013 Q4/2013 completed
3.2 TRCS Youth Volunteer Project Promotion on Disaster
Preparedness
Bangkok Q1/2013 Q4/2013 Completed
4 Nursing College
4.1 Flood Disaster Preparedness Volunteer Network for University and College Level Students
Bangkok Q3/2013 Q1/2015 Completed
5 Personnel Bureau
5.1 Managerial Grid - Result Oriented Practical Workshop (3-day in house workshop)
Nakorn Pathum Q1/2014 Q1/2014 completed
New Activities
Angthong 22 April
2015
21 June
2015 Completed
2015-1.2
Brochure - Water Purification Unit Sirindhorn Red Cross Health Station - Phang Nga
Phang Nga Health Station
Q2/2015 September
2015
Completed
2015-1.4
Psycosocial Train of Trainer (TOT) workshop
Water Sanitation and Hygiene Operation Workshop building for students - be a safe
school Cancelled
2015-1.7
Community Preparedness Programm - Phang Nga
Phang Nga Q1/2015 Sept 2015 and Capacity Building Activities - a continuous program from 2015-1.7 Community
preparedness program - Pang-nga
Phang Nga September
2015 Support - continuous
implementation from 2015-1.4 Psychosocial Train of Trainer (TOT) workshop (printing PSS manual)
TRC intensive Communication - TRC PR Volunteer
BKK or Ayutthaya 3-9 August
2015
3-9 August
P
TRC intensive Communication - TRC Young reporter
BKK or Ayutthaya 3-9 August
2015
First Aid Youth Volunteer Network with good morals
4 Regions Q2/2015 Q3/2015
Completed
2015-3.2
CPR and Chocking training for youth network
The Developing of Thai Red Cross Volunteer Inf.Syt
Bangkok Q1/2015 Q3/2015
Completed
2015-6.2
Training volunteer in community area
4 Regions Q2/2015 Q3/2015
Completed
ANNEX 2
STAKEHOLDERS INTERVIEWED
Project Management Unit
Mr Chamnong Sangmahachai Ms Rongrong Khemarangsi Ms Natrada Notananada
Ms Ratjiporn Muangjan
Phang Nga Health Station
Ms Boonyarat Pornchan Ms Ketsara Raksanak
Mr Suppachok Sobrit Ms Arom Kankiam Ms Angoon Wongcharoen
Community 1
Mr Prateep Chairam (36) Mr Sucha Shuchart (64) Mr Prayong Suchart (33)
Ms Phakhini Siden (54) Ms Boonsong Daengpiek (60) Mr Thanong Kongbutr (47)
Ms Praphai Kongbutr (49)
Community 2
Ms Mriya Densiri (44) Ms Farida Warisri (39) Ms Dolkornee Martraksa (48)
Mr Anand Sansumut (34) Mr Nakorm Wirtphong (33) Mr Harold Yeesanae (53)
Youth Bureau
Ms Sarunya Jompang Ms Ornpapar Chatnomphet Ms Vatusiri Swatjun
Ms Metinee Jongsri Mr Komkrit Vijituiriyakul
Relief Bureau
Lt. Gen. Dr. Amnat Barlee Ms Sukamya Supudommungmee Ms Pavinee Yuprasert
Ms Wallapa Suksirimuch Ms Nawaporn Piewkhum Ms Darin Hasachoo
Dr Varangkana Tungkhamsai Ms Sirikanya Sirirangsi Ms Kittitouc Klinloy
Ms Pongsak Khalatat
Disaster communication centre
Mrs Yaowaluk Yodsopha Mr Samart Nakyanyong Mr Caatchaval Ngamsirilert
Communication Bureau
Ms Sasithorn Rattanaprawit Ms Yunsinei Jirayotiu Ms Watcharjya Iamthananont
Ms Krongthong Phetwong
Logistics
Ms Jirawan Santipitak Ms Sumalin Silpipat Ms Kantima Chotrungruang
Ms Sudsiri Channarong Ms Lumpar Karntikoon
Administration Bureau
P
ag
e
g
ANNEX 3
DOCUMENTS REVIEW
Monthly reports - October to December 2013
Monthly reports – January to December 2014
P
ag
e
h
ANNEX 4
INTERVIEW CHECKLIST
Process documentation
What are your roles and responsibilities in the project?
What are the deliverables of your activities funded through this project?
What are the key achievements?
What are the challenges?
If you have a chance to do a similar project again, what will you do differently?
Output and outcome assessment
What are your roles and responsibilities in the project?
Can you tell me more about the activities?
What were the criteria for selecting the communities or types of activity?
What are the Strength, Weakness, Opportunities and Threats (SWOT)?
P
ag
e
i
ANNEX 5
EVALUATION TIMELINE
Initial documents review
Submission of inception report 7 September
Field visit to Pang Nga province 8 – 11 September
Consultation with TRCSS bureau 14-16 September
Lessons learn workshop 15-16 October
Submission of draft report 24-25 November
Submission of final report 1 December
Field Data Collection in Phang Nga (9-11 September 2015)
Date Time Description Remark
8 sept 2015 16.30-17.55 Flying from Bangkok SL, Rommanee &
18.20-19.30 Travel to hotel in Phang Nga - TRCS airport pick up
9 sept 2014 07.00-08.00 Travel to community By car
08.00-12.00 Observe the presentation of comm. TRCS and evaluation team
12.00-13.00 Lunch
13.00-16.00 Observe the table top exercise Evaluation team
Interview community members 16.00-17.00 Travel back to hotel
10 sept 2015 08.30-09.00 Travel to TRCS Health station Phang Nga
09.00-11.00 Group/individual interview TRCS member 11.00-12.00 Individual interview TRCS member/volunteer
–
who attend the meeting/workshop/training 12.00-13.00 lunch
13.00-14.00 Evaluate the communication system in Phang
14.00-15.00 Open discussion with TRCS members
11 sept 2015 07.00-09.00 Travel to Koh Mai pai Island By TRCS’s vehicle
09.00-12.00 Educate Incident Command System to Community
12.00-13.00 lunch
13.00-16.00 Table top exercise and interview community 16.00-17.00 Travel back to TRCS
