Department of Health Library Services ePublications - Historical Collection
Please Note: Aboriginal and Torres Strait Islander people should be aware that this publication may contain images, voices or names of deceased persons in photographs, film, audio recordings or printed material.
Purpose
To apply preservation treatments, including digitisation, to a high value and vulnerable Historical collection of items held in the Darwin and Alice Springs libraries so that the items may be accessed without causing further damage to the original items and provide accessibility for stakeholders.
Reference and Research Disclaimer
Please note: this document is part of the Historical Collection and the information contained within may be out of date.
This copy is a reproduction of an original record. Please note that the quality of the original record may be poor and cannot be enhanced with the scanning process.
Northern Territory Department of Health Library Services Historical Collection
DL HIST 362.1207 MIL 1983
REFLECTIONS ON A WORKSHOP
INTERCOUNTRY WORKSHOP ON EVALUATION OF TRAINING PROGRAMMES FOR PRIMARY HEALTH CARE - FEBRUARY 7 - 18, 1983.
C.J.MILLER
T 1l1[1i1~1~1t1l1f 1111~11~11~~1~111i1l1[1~1l11!1~1i
3 0820 00019298 4
REFLECTIONS ON A W.H.O. INTERCOUNTRY WORKSHOP ON
EVALUATION OF TRAINING PROGRAMMES FOR PRIMARY HEALTH CARE - FEBRUARY 7-18, 1983
Between 7-18 February, 1983 I attended, as one of two Australian representatives, an intercountry workshop on evaluation of training programmes for primary heal th care sponsored by the World Health Organisation (W.H.O.), and held at the WHO Regional Teacher Training Centre, University of New South Wales, Sydney.
My attendance was made possible jointly by the Federal and Northern Territory Departments of Health. It was in my capacity· as co-ordinator of the curriculum for the Northern Territory Aboriginal Health Worker training programme that I attended.
The workshop was attended by reprsentatives of Japan, The Trust Territories of the Pacific Islands, Kiribati, Malaysia, Phillipines, Cook Islands, Vietnam, Papua New Guinea, Tonga, Korea, Australia, and a WHO advisor from Fiji. Two members of the training centre co-ordinated and facilitated the workshop.
The general aim of the workshop was to develop and improve the skills and knowledge of participants with respect to the evaluation of training programmes for various categories of health personnel engaged in primary health care. More specific objectives were that by the end of the workshop participants would be able to do the following:-
(a) define and describe the role of programme evaluation in the planning and conduct of training programmes;
(b) determine the uses of the evaluation results;
(c) determine the scope of the evaluation;
L.. ... l
1 8 SEP 2001 LIBRARY
(
(d) choose a conceptual approach for the evaluation;
(e) decide who should be involved in the evaluation activity;
( f) define the problem areas and identify parameters which need to be investigated and information which will be acceptable as evidence;
(g) identify the nature of the data to be collected in relation to those parameters;
(h) design the evaluation plan and choose methods of collection and analysis;
(i) administer the evaluation activity, budgetary and staffing components;
(j) report on the evaluation outcomes;
including its
(k) identify methods of facilitating decision-making based on the results, taking into account organisational, professional and political factors.
Format
The workshop format, in general, consisted of morning plenary sessions devoted to the introduction of new material, and afternoon small group or individual work on personal projects.
The accent was on full participation by all people, drawing on their experience in the primary heal th care programmes of their countries.
A number of after-hours social events greatly facilitated the daily participation of group members.
The Programme A ten working-day programme, the opening day of which revolved around introductions and a discussion of the definition of evaluation.
3
What is Evaluation? a systematic process to determine the worth of something. This distinguishes i t from research which is principally designed to add to the body of the knowledge.
For the purposes of this workshop, programme evaluation was defined as a process of collecting information for decision- making, in order to assess and improve the worth of the programme. Evaluation is often seen as a rather nebulous concept more concerned with placating examiners or appeasing the conscience of programme planners who feel, rather guiltily, that they ought to at least be asking some sought of questions about how well they are doing.
In this dynamic, decisions
workshop, programme logical, stepwise and judgements about
evaluation was presented as a management tool for making
the need for change. This was based on the premise that decisions made in the presence of relevant, reliable information must be better than arbitrary decisions.
What kinds of decisions? Management decisions have to be made at all phases of a programme's operation. A model (systems) was presented to illustrate the elements of a programme which require decision-making:-
A shoe production line
jcontext
eg market information
inputs
materials manpower capital
process
production line
outcome
shoes
Improvements in the worth of this shoe factory would depend on management decisions made at all phases - related to context, input, process, outcome issues. Evaluation of the programme therefore occurs at all stages. Ou~come evaluation alone (ie did the factory make a good pair of shoes?) is inadequate (the shoes may have been produced at a price which meant they could not be sold at a profit, or dissatisfaction on the shop floor may mean there is about to be a strike.)
In a training programme, the same applies. Evaluation must be carried out at all phases:-
I
inputsI
~I
processI >
J outcomeI I
contextI
teachers the teaching/ studentstudents learning pro- success culture, political syllabus programme. student and economic teaching aids teacher deploy- performance
environment time ment. Teaching
methods.
By evaluating CONTEXT - i t is possible to determine the needs of potential consumers and the feasibility of the programme in the context of particular economic, politic al or cultural conditions. This gives rise to the formulation of objectives - the INTENDED RESULTS.
By evaluating INPUTS i t is possible to select resources, staff and facilities and design activities for the implement- ation of the programme. This gives rise to a plan of action - the INTENDED MEANS.
By evaluating PROCESS - i t is possible to decide whether the plan works, to change the plan or identify unintended consequenc~s. This gives rise to programme modification and feedback to planners - the ACTUAL MEANS.
By evaluating OUTCOMES i t is possible to determine deficiencies, unintended results, and see if actual results
00'2....'L"Z_.
5
match intended results. This gives rise to an assessment of overall effectiveness of the programme - the ACTUAL RESULTS.
These intended and actual means and ends can be put conveni- ently together on the following grid -
INTENDED ACTUAL
Evaluation of needs Evaluation of results
ENDS and feasibility (OUTCOME)
(CONTEXT)
Evalua-tion of plan Evaluation of the
MEANS (INPUT) running program
(PROCESS)
With this model in mind, the group then identified the many components of a training programme which could be the foci of evaluation.
1. Goals, aims, philosophy.
2. Students, trainees, eg literacy level, class size.
3. Teachers, trainers, eg attitudes, teaching skills.
4. Teaching and learning objectives, eg balance of know- ledge, skills and attitudes.
5. Content, syllabus, eg does i t reflect the heal th needs and Health Worker job requirements?
6. Timetable, teaching input.
7. Teaching methods used, learning objectives effectiven~ss, eg use of role play, A-V aids.
8. Assessment methods - formal or informal.
their
9. Site and facilities, eg centralised versus peripheral teaching, buildings etc.
10. Administrators, managers.
11. Budgets, funds.
12. Programme evaluation methods.
TyPes of evaluation
By determining the purpose of the evaluation, two types of evaluation can be identified:- one which is designed primarily to assess the overall effectiveness of a programme.
This is· a "one-off" evaluation to determine whether a programme should continue or not, or to make major decisions at some key stage in the programme (it is called SUMMATIVE evaluation). The other type is designed primarily to facilitate decisions for the improvement of the ongoing programme (it is called FORMATIVE or DIAGNOSTIC evaluation).
This means building into a programme information-collecting and analysing systems so that on-going management decisions can be made making best use of this data.
By the end of the first day of the workshop, the following steps to be taken in any programme evaluation had been outlined:-
1. DECIDE WHAT TO EVALUATE
Identify the program to be evaluated Describe the program
Determine areas of concern in relation to which information is required
Identify the audience and participants concerned with the evaluation
Set priorities for evaluation in consultation with audience
7
2. DESIGN THE EVALUATION PLAN
Formulate the question( s) to be addressed by the evaluation
Determine the information required to answer the particular questions
Specify indicators/criteria for determining worth of program, and making necessary choices
Determine when, how and from whom to obtain required information
Identify evaluation
resources/constraints
3. CONDUCT OF THE EVALUATION
for conduct of
Locate and appraise information already available Select/construct information-gathering instruments Pilot test scheme of evaluation and make necessary adaptations to design
Administer evaluation scheme to obtain data
4. DEALING WITH THE INFORMATION
Design and implement methods of data analysis, interpretation and presentation
Draw recommendations
Determine recipients of evaluation report Determine method of reporting
Prepare and disseminate report(s)
5. DECISION MAKING
It must be stressed that in setting up any evaluation of a programme, each step must be the subject of consultation with those who will i::eceive the evaluation report. In this way, the validity and-reliability.of the data will not be in such doubt that the usefulness of the report as an aid to decision- making could be in jeopardy.
The remainder of the workshop consisted of each participant focussing on his/her own training programme, alone or in small groups, following these steps of evaluation so as to produce an example of a small evaluation project. The outlines of these projects were presented at the end of the workshop.
The morning sessions were devoted to the introduction and discussion of the successive steps in the sequence.
Individual presentations, group exercises and role play were used to highlight or demonstrate important issues raised, eg nominal group technique to determine priorities, fish-bowl role play to highlight the vested interests of those involved in training programmes, group exercises in consensus decision- making and reading evaluation reports. On the concluding day participants were invited to fill in a short questionnaire as a way of evaluating the workshop and its facilitators.
COMMENT
The workshop was an extremely valuable experience for me, providing a very clear, logical and practical framework upon which the often vexed question of evaluation could be approached. In times of resource constraints, i t was clear that managers of training programmes must identify the key concerns that they have about their programme and institute on-going information - collecting and analysing systems which will provide answers to the questions that these concerns raise. Only in this way will managers' decisions regarding programme change be rational rather than arbitrary.
The Regional Teacher Training Centre, and more specifically, Dr's Rotem and Bandaranayake and Mr Somerton, are to be congratulated on a most practical, informative workshop which was both efficiently and sensitively run.
The experience of sharing- the Northern Territory Aboriginal Health Worker training programme with others involved in primary health care in the Western Pacific, and the insights
9
gained into programme evaluation, I would not like to have missed.
RECOMMENDATIONS
I recommend that the managers of the two major training centres in the Northern Territory ( and in fact, any other programme managers in the department) identify the major on-going concerns related to their training programmes and institute simple information collecting and analysing systems such that management decisions can be rationally made to improve their programmes.
I suggest that the step-wise process of evaluation outlined in this workshop report be adopted as an aid to setting up these systems.
APPENDICES
Attached is a brief outline of the small evaluation project undertaken by me during the course of the workshop. It was nothing more than an example to experience first hand, part of the evaluation process presented during_the two week workshop.
The concern expressed and therefore the questions asked may, however, be shared by others and will therefore be of interest and use.
C.J. Miller
Curriculum Co-ordinator
Aboriginal Health Worker Training Programme Northern Territory Department of Health
TITLE:
STUDENTS:
OBJECTIVES:
APPENDIX
PROGRAMME PROFILE
Northern Territory
Aboriginal Health Worker Training Programme
270-300 Health Workers, aged between 16-50, selected by their communities on tribal, family or social grounds, not on the basis of educational achievement. They have a great range of literacy and numeracy levels from illiterate to secondary school level. The average would be the reading and writing in English of a 9-10 year old in a European schooling situation. Women outnumber men by a ration 2-3:l
To improve the overall life of Aboriginal people to ensure their survival and growth - by Aboriginal involvement in management and decision-making
To preserve Aboriginal health care systems in an atmosphere of exchange thereby creating a "two-way medicine"
To equip Aboriginal people with the know- ledge and skills to manage and run their own primary health care
JUSTIFICATION: . Buying more medicines, non-Aboriginal nurses and doctors will not improve the health of the people
CONTENT:
DURATION:
11
Transferring knowledge and skills to the Aborigines will help give them more power and be involved in decision-making. This should help promote a new social order
Aborigines will learn better from their fell ow Aborigines, eg heal th promotional and preventative behaviour and use of health services
Primary and tertiary health activities in curative, promotional preventative and rehabil- itative services. Functional literacy and numeracy training is an integral part of the curriculum.
2 levels -
Basic Skills - generally 1 - 2 years
Post Basic Skills - modules of at least 6 months duration
Completion of courses is open-ended in duration as there are no educational selection criteria for new students. Students proceed, by predominantly on-site training, at their own pace.
PRIORITIES IN CONCERNS
1.. Content
Does the clinical component of the curriculum adequately cover the problems a Heal th Worker faces in day-to-day work? Does i t meet the perceived needs of students and their communities? Is the literacy-numeracy component adequate?.
2. Teachers
How do they feel about their role? Do they find time to teach or is there role conflict? Do they feel adequately prepared and supported to teach? Are they deployed to ensure regular weekly teaching input to each student?
3. Methods
Are teaching methods and techniques effective? Are the supporting aids adequate?
4. Assessment
Are the right things being assessed? The right questions asked? Are they a measure of the Health Workers problem- solving competence and safety? Are assessment methods appropriate? Does ongoing assessment, even after
·graduation, have a place?
5. Programme Evaluation
Is the information-collecting system and its analysis adequate and functioning so as to enable management decisions to be made about the programme.
13
6. Students
How are they really selected?
What is their attitude and why?.
How can more males be involved?
How many Health Workers should there be per population? - on what grounds?
7. Location
Is the relationship training appropriate attitudes confront attachments?
between on-site and centralised in time and continuity? What the students during hospital
CAMPBELL MILLER
ABORIGINAL HEALTH WORKER TRAINING PROGRAMME IN THE NORTHERN TERRITORY, AUSTRALIA
TRAINEES:
270-300 Health Workers scattered over a vast area of coastal to desert country. Heal th Workers are selected by their communities, aged between 16 & 50, of very varying literacy level, women outnumbering men by 2 - 3: 1. Average literacy level consistent with a 9-10 year old in a European schooling system . . Trainees are employed by their communities, by missions, or health service organisations.
TEACHERS:
Most of the teachers are Nursing Sisters living on-site in scattered rural communities. They have little teaching orientation and so far, minimal teaching in-service training.
This teaching is supplemented by some on-site input by visiting doctors and health surveyors. In some cases senior heal th workers are teaching their juniors. A few tutors in the three training centres teach both in regional centres and on-site.
THE TRAINING PROGRAMME:
A two skill level course of Basic and Post Basic Skills training, both in the development stage. Training is predominantly on-site supplimented by short in-service courses or hospital attachments. Courses are based on mastery learning.
A Basic Skills check list supported by a Procedure Manual has been developed. Post Basic Skills modules are in the process of development, all in a task and problem orientated format.
15
The scope of the courses is curative, promotive, preventative and rehabilitative.
THE CONTEXT:
Small, scattered population (27,000) of. Aborigines living in small rural commmunities (20-1500). Population structure is that of "developing countries", with high morbidity and mortality related to infections, nutrition, violence,
alcoholism and delayed usage of health services.
The health service system:- A network of rural community heal th centres owned or funded by State or Federal Heal th Departments. Most centres are supervised by nurses but an
increasing number are run by Aboriginal Health Workers.
PROGRAMME PHILOSOPHY
To promote and strengthen the LIFE of Aboriginal people as a means of ensuring their survival and growth.
MY CONCERNS
That student learning experiences are adequate to realist- ically expect them to achieve the objectives of their courses.
The relevance of the clinical component of the curriculum to all Health Workers.
MY ROLE AS EVALUATOR Relationship:
With evaluation project - designer, public relations, advisor, some involvement in implementing.
With training programme curriculum co-ordinator, advisor, negotiator with professionals
With Dept. Management making.
RESOURCES AND CONSTRAINTS:
informant, involved in decision
No resources allocated specifically to evaluation. Any training programme evaluation is done by programme managers and implementers. The whole programme is operating within the constraints of the delivery of health services, eg the teachers are generally also deliverers or administrators of health services.
EVALUATION SCOPE AND AUDIENCE
To institute formative, built-in evaluation of the programme, for management purposes.
The audience is programme management and implementers, the curriculum co-ordinator, assistant-secretary of nursing.
CONCERN: That student learning experiences are adequate to realistically expect lhem to achieve the objectives· of their courses.
Q:.JESTION
Do students receive frequent regular t~aching input?
:s
the teaching effective?IN FORMAT ION Teacher/student Teaching/learning contact.
Teacher/student relationship Student progress and success
Teachers• prograrmne, methods and
techniques.
INDICATORS
Frequency and duration of learning experiences
Incidence of after hours social contact Incidence of bush and out station excursions Student attrition
Numbers completing courses or units of courses
Presence or absence of teacher plans
Extent to which curriculum is followed
Extent to which suggested methods and techniques are used.
CRITERIA METHODS
&
SOURCES Every student receivesleast 5 hours per week learning experience in addition to normal work duties
at Diaries and log books
Every student has
learning experiences at least 3 times per week.
At least weekly
after hours social contact At least weekly
bush or outside excursions
Less than average student attrition Student completion of courses or units within stated time frame
Existence of teacher plans
,
Teacher objectives of each unit met :>
At least one suggested metnod or technique used
in e ... '"h lesson.
of students and teachers . observation of teacher:
student contact.
Interview teachers and students
Interview teacher and students
Examination of students records and work books Analysis of employment records
Examination of teaching roles
Classroom and work observation
Teacher self assessment questionaire
.J
