An evaulation of a management training programme

Copyright is owned by the Author of the thesis. Permission is given for

a copy to be downloaded by an individual for the purpose of research and

private study only. The thesis may not be reproduced elsewhere without

the permission of the Author.

AI{ EVALUATICN OF A

I4ANAGEMENT TMINING PROGRAJ"IVIE

A

thesis

pnesented

in partial ful-filnent of the

nequinements

fon the

degnee

of

Docton

of

Philosophy

in

Psychology at

Massey Univensity

,Judith-Anne Bnook 1982

fcilo - Gr

I1

ABSTRACT

The

evaluation of

a management

tnaining

prognamme

for

nesearch

scientists

and

senior technicians

formed

the basis of this

applied nesearch

pnoject.

Befone, du::ing and

after

^{measures were used}

to

assess

the

knowledge,

skills, attitudes

and work behaviouns

of trainees

and

control

group membens

in order to

^compare

the

changes

which oceumed as

a result of the programme.

Both fonmative and summative evaluation were incorporated

into the

expenimental design and

careful attention

was

paid to internal

and extennal

validity

issues

by the

use

of

matched

contnol

^groups,

multiple

measures

of the

dependent and independent

variables, including both objective

and

subjective

methods.

self-neports

and neponts by tr"aineest

supeniors.

^{Repeated measu::es}

of

knowledge,

skills, attitudes

and wonk behaviour:s were taken

utilizing a longitudinal

design and some

neplication

was condueted

with other

groups

of

tnainees

at

other

times.

The

plan of

appnoach

involved

an

initial analysis of training

needs o examination

of the input to training including

course content and

training

teehniques, immediate and longen tenn

evaluation of

subjectsf reactions to the

^pnogramme and an assessment

of the

outcomes

of tnaining.

^Outcome

evaluation

compnised measunes

of

effectiveness

at both

immediate and

intermediate levels with

measurements taken immediately and

at thr:ee, six

and twelve months

following training.

Effectiveness was assessed

in relation to the

goals determined during

the

preliminany

analysis of tnaining

^needs,

the

focus

shifting

gnadually

during the

cour"se

of evaluation

fnom an emphasis on

learning effects to

changes

in

^wonk behavioun and penformance.

Formative

evaluation in the

form

of

feedback

to

course

controllers

was pnovided thnoughout whil-e

the

summative

evaluation

consisted

of

^a

final

neporting

of the effectiveness of the training

pr"ogr:amme oven a

twelve month peniod and

the

consisteney

of results

^fnom

a

subsequent sample

of tnainees.

A model was suggested

to

enable

the

fonmative and summative aspects

of evaluation to

^proceed simultaneously

within a single study. Finally,

by monitoring

the effects of

envinonmental

factons, the

wr"iten was

able to

begin

to

disti.ngrrish

the

modenator

vaniables openating

at various levels of evaluation

and move towands

construct validation of a theonetical

rnodel

of training

and evaluation.

1 l-1

ACI(NO,tLEDGEI4E,ITS

I

wish

to

thank Pnofesson Geonge Shouksmith

fo:r his

supenvision and

constructive cniticism

throughout

this

pnoject.

I

^would

also like to

acknowledge

the

suppont given by othen members

of the

Psychology Depantment at_Massey

Univensity, panticulanly

Mn Mike Smith whose

intenest in the p::oject

and advice

as

second supenvisor was rnuch

appniciated,

and Dn Dave Clanke who provided assistance and suggestions

in the initial stages.

Sincene thanks ane due

to

Ms Manganet-Many Oulaghan and Ms

Gail

Tyson who had the

difficult task of typing the final

manuscnipt, and membens

of

the Psychology Depantment

office staff, ineluding

Ms

Alison

Stewanto who

assisted with the

pnepanation.

Final1y, I

^would

like to

thank Dn Richand Br:ook

for his

assistance

with the analysis of data,

and othen membens

of

^my

family fon their.

tole::ance and patience.

Abstnact

Acknowledgements

Chapten 1.

Chapten 2.

Chaoter- 3.

TABLE OF CCNTENTS

The evaluation process

t.t fntroduction

t.2 Brief histony

and

definitions of

evaluation

1.3 Some expenimental design pnoblems

evaluation

neseanch

A

theonetical

model

of

evaluation

2.t

The Wontnan model

2.2

evaluation

3.3 Critenion

development

3.4

Tnaining

evaluation

methodology

1n

l't 1.r-a

13 13 Page

15 16

t7

18

20 2L 22 25 26 27

37 38 43 The

3.1 3.2

2.7.1

Formative and summative

evaluation

2.7.2 Validity

issues

2.7.3 Political

considenations

2.7.4 Ethics

and

confidentiality 2.1.5

Randonization and quasi-

expenimental design

2.1.6

Measu:rement

of

vaniables A

decision - theoretic

^appnoach

2.2.1

The

place of value

^judgements

2.2.2 Subjectivity

2.2.3 Panticipant -

obsenvation technology

of tnaining

evaluation

Introduction

A

pnactical

fnamewonk

fon

evaluation

of

management

tnaining

3.2.1,

Problem-centned

training 3.2.2

The

cycle of

evaluation

3.2.3

The discovery approach

to

29 29

31 33 33

Chapten 4.

Chapten 5.

Chapten 6.

5.3

Intnoduction

to the

pnesent research

4.1

Background

to the evaluation

study

4.2

Afuns

of the

^pnesent nesearch Preliminany studies

5.1

Intnoduction

5.2 Identifying tnaining

^needs

5.2.1

Subjects

5.2.2

Measu::ing instnument

5.2.3

^Procedune

5.2.4

^Results

5.2.5

Discussion

5.2.6 Inten-nater reliability 5.2.7

Conclusions

Constr:uction

of attitude

scaLe

5.3.1

Subjects

5.3.2 Attitude scale

development

5.3.3

Concluding comments

6.4

65 65 66 66 68 70 72 75 81 85 86 88 88 90 Page

50

.50

57

103 103

104 104 Overview

of

expenimental

design

⁹²

6

.1 Intr:oduction

⁹²

6.2 Subjects

⁹³

6.3 Methods

⁹⁷

6.3.1

Context

evaluation

⁹⁸

6.3.2 Pre-test measures

⁹⁸

6.3.3

^Reactions

evaluation

⁹⁹

6.3.4

^{Imnediate outcome}

evaluation

⁹⁹

6.3.5

Intenmediate outcome

evaluation

¹⁰¹

6.3.6 Ultimate

^outcome

evaluation

^tO2

6.3.7

Extennal

validity

and

netiability

of

scales Concluding comrnents

Evaluation

of

irnmediate outcomes

7.1

Intnoduction Chapter 7.

7.2

Reasons

fon attending

^counses

7.3

Expected aneas

of

onganizational and

pensonal change

7.4

Traineesr assessment

of

sessions

7.5

Evaluation

of

course

topics 7.6

Tnaining techniques prefer-ned

7.7

Changes

in attitudes

towands the

section

leaden nole

Fo11ow-up studies

8.1

Intnoduction

8.2

Evaluation

of

intenmediate outcomes

8.2.1 Effects of

work penformance

aften

thnee months

8.2.2 Areas of organizational

and

Page

105

to7

110

Lt2

722

725

130 130 130

130

Chapten 8.

Chapten 9.

8.2.3 8.2.4 8.2. ⁵

personal

change

!37

Allocation of time to topics

140

Indirect

measures

of effectiveness

142

Penceived understanding and

nelevance

of topics

8.2.6

Changes

in

behaviour

aften six

months

- traineest report 8.2.7

Changes

in

behavioun

aften six

months

-

supenionsf neport

8.2.8

Onganizational

clinate

8.2.9

Twelve month

follow-up

study

8.3

^Surnmarry

of findings

Case studies

9.1 Introduction 9.2

Aim

of

studies

9.3

^{The nepertony}

gnid

technique

9.4

^Method

9.5

Results and discussion

9.5.1

^{Case study A}

9.5.2

Case study B

145

150

160 165 173 t76

180 180 180 181

t82

18s

t87

191

9.5.3

^Case

9.5.4

^Case

9.5.5

^Case

9.5.6

^Case

9.6

Conclusions

Chapten

10. Replication studies

²¹⁰

10.1 Introduction

2tO

7O.2 Aim

2!3

10.3 Subjeets

213

10.4 Procedune

213

10.5 Results

^21,4

10.5.1

^Reasons

fon

attending

courses

214

10.5.2 Training

techniques

pnefenned

275

10.5.3 Attitude scale

216

10.5.4

Performance

aften three months

2Lg

10.5.5 Allocation of time to topics

2I9

10.5.6 Indirect

measunes

of effectiveness

220

10.5.7

Undenstanding

of topics

222

10.5.8

Onganization

climate

guestionnaine 225

10.5.9 Effects of

age, tenur:e and

leadenship

experience

²²⁷

10.6 Discussion

²²⁷

10.6.1

Reasons

fon attending courses

228

10.6.2

Tnaining techniques

pnefenned

228

10.6.3 Attitude scale

228

10.6.4

Perfor:mance

aften

thnee

months

229

10.6.5 Allocation of

tfune

to topics

²³⁰

10.6.6 Indirect

measures

of effectiveness

²³¹

10.6.7

Understanding

of topics

23t

10.6.8

Onganization

climate

questionnaine 232

10.6.9 Effects of

age, tenune and

leadenship expenience

tO.7

Conclusions

232 232

study ^C study ^D study E

siudy

F

Page

19s 198 201.

204 207

Chapten

11.

The

effectiveness of the tnaining

courses

tt.t

Intnoduction

tl.2

Context

of tnaining

11.3 Ultimate objectives of training 11.4

Tnaining input

11.5

hunediate outcones

of tnaining 11.6

Intermediate outcomes

of tnaining

!1.7

0nganizational envinonment

1L.8

Reactions

to training 11.9 Training

techniques 11.10 Conclusion

Chapter

12.

Conclusions concerning

evaluative

r:esearch

Page

23s 235 235 237 239 241, 242 248 25L 255 2s6

257

Figune

2.t

2.2 3.1 8.1

8.2

8.3

8.4

8.5

9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9 9.10

LIST OF FIGURES

Wontmanrs model

of evaluation

r"esearch Pnocess

evaluation

as

a

feedback loop Tnaining/evaluation

control

system Changes

in

understanding

of topics

at thnee

times, pre-tesa (tf),

immediate

post-test (t^)

and thnee nonths

(t^) 'zc

Avenage numben

of

^{behavioun changes (1)}

attempted and

(2)

implemented, by

tnainees and contnols

Number

of

behavioun changes made

in

each behavioural category expnessed as

a

propontion

of subjects in the

group Summary

profiles of

onganizational

climate fon

govennment and semi- government onganizat ions

Sumnary

profiles of

onganization climate

fon

leadens and non-leaders

Dendnognam

of

elements

fon

Case A

Dendnog:ram

of

constnucts

for

^{Case A}

Dendnogram

of

elements

for

Case B

Dendnognam

of

constnucts

fon

Case B

Dendnognam

of

elements

fon

Case C

Dendrognam

of

constnucts

fon

Case C

Dendrognam

of

elements

fon

Case D

Dendnognam

of

constnucts

for

Case ^D Dendnognam

of

elements

for:

^{Case E}

Dendrognam

of constructs fon

Case E

14 15 32 Page

149

153

159

169

169 787 189

t92

193 195 196 198 199 202 203

Figune ^Page

9.11

Dendnognam

of

elements

fon

Case

F

²⁰⁴

9.t2

Dendnogram

of constructs fon

Case

P

²⁰⁶

10.1

Changes

in

r:ndenstanding

of topics

at

thnee

times, pne-test (t1),

immediate

post-test (tr)

and

three

nonths

(tr)

223

1,0.2

^Summany

pr.ofiles of

onganizational

climate fon

government and semi-

government

onganizations

²²⁶

10.3

^Sunmany

pnofiles of

onganizational

clftnate fon

leadens and

non-leadens

²²⁶

tI.7

The

cycle of evaluation including

moderator

vaniables

²³⁶

!2.1

Genenal

cnitenion

model

of

managenial

effectiveness

²⁵⁹

t2.2

Decision-making model

of evaluation

²⁶⁶

LIST OF TABLES

Table Page

3.1

Tnaining

evaluation

^schemes

of

Hamblin and

Wanr,

Bird

and

Rackham

³⁴

4.1

The seven phases

of the

^{pnesent evaluation}

studv

^59-64

5.1

Subjects who

took pant in

precourse

interviews

⁶⁸

5.2

^Numben

of

responses

in ten

categories

of

management

by

four" sub-groups

of

respondents

5.3

^Numben

of

^nesponses

in five

categonies

of

pensonal and group

nelationships

^by foun sub-gltoups

of

nespondents

5.4

The fnequency

of incidents

placed

in

categonies by

the

two ^judges

6.1

Subjects

involved in

main evaluation study categonized

by

^{guouPs (nows)} and

types of

employing onganization ^(colunns)

6.2

Biognaphical chanactenistics

of

subjeets

in all

^groups

6.3

Average ages, tenure and leadenship expenience

of

tnainees and contnol

gr3oup

membens

⁹⁶

7.1 Pre-test

responses

to

^{question concenning}

reasons

fon attending counses

¹⁰⁶

7.2

Subjectsr nank ordening

of

aneas

of

onganizational change

expected

¹⁰⁹

7.3

Subjectsr nank ondening

of

aneas

of

pensonal change

expected

109

'7.4

Changes

in

penceived understanding and

nelevance

following tnaining

¹¹⁵

74

81

85

94

95

TabIe Page

7.5

Medians and chi-squares

of

nesults obtained on

the

perceived undenstanding

and nelevance

scales

¹¹⁸

7.6

Nunber:

of

subjects who

indicated

pre- ference

fon

each

of six types of

tnaining methods

⁷²³

7.7

^Means and standard

deviations of attitude scale

scores obtained

by

expenimental and

contnol

group

subjects

¹²⁷

7.8

Results obtained when

attitude

scale

scones

are

companed

fon different groups

128

8.1

^Numben

of

^{tnainees who}

indicated

^changes

in

^wonk

behaviour

732

8.2

Numben

of

behavioun changes

speeified

^by

individual

counse membens

after

thnee

months

¹³³

8.3

Subjectsr nank ondening

of

aneas

of

onganizational change

following tnaining

¹³⁸

8.4

Subjectsr nank ondening

of

aneas

of

^pen-

sonal change most

affected by

^the

tnaining cour.ses

¹³⁹

8.5

Subjectsr pnefenences

fon time allocation

to

counse

topics

¹⁴¹

8.6

Changes

in tnaineesr

per:ceptions

of

^unden-

standing and nelevance

of topics

studied

duning

tr:aining

¹⁴⁷

8.7

Number

of

subjeets who

indicated

behavioural change

following tnaining

courses

¹⁵³

8.8

Retnospective change behaviour reponted by

subjects six

months

aften training peniod

^15tr

TabIe Page

8.9

The

r:elation

between irnrnediate measures

of attitude

^change and

longer

tenm change

score

at six months

¹⁵⁵

8.10

Relationships between wonk behaviour and

age,

tenure and expenience

of

Group A and

B

subjects

¹⁵⁷

8.11

Conrelation between fnequency

of

change and

age,

tenune and

expenience

¹⁵⁸

8.12

Table showing number

of

^rrYesrr

,

rrNort and

trDontt _Knowrr responses made

by

four:teen

controlling officers

^about

thein

^sub-

ondinatesr wonk

behavioun

164

8.13

Medians and nanges

of natings

made

by

sub- groups

of subjects (tnaining

_{Gnoups A and}

B, cornbined) t7t

8.14

^{Gnoup means} and standard

deviations of

tnaining effectiveness

on thnee

scales

t75

10.1

Modal nankings

of

^neasons

fon

^attendance

given

by

tnaining

group

members

^2ts

tO.2 Attitude

^{scale means and standand}

deviations fon

gnoups

A,

B C and D

at

t:Lmes, befone

(tr)

and immediately aften

(t,, _z

) tnaining

^2t7

10.3

^Numben

of

^{tnainees who}

indicated change

2!8

10.4

Numben

of

changes

specified by individual

course mernbens

aften

thnee

months

?"n

10.5

Subjectsr pneference

fo:r

amount

of tine

allocated

to'counse

topics

^22O

10.6

Subjects who read books

nelated to

counse

topics

22!

Table Page

tO.7

Subjects who discussed counse

with

work

colleagues

22!

10.8

Subjects who maintained

contact

with

other

counse

mernbens

222

10.9

Numben

of positive

changes

in

undenstanding

of

course

topics fon

gnoups

A, B,

C and

D

224

10.10 T statistics

obtained using Fniedmanfs

test

on

subjectsr

nank ondening

of level of

undenstanding

of topics

224

10.11

Medians and ranges

of natings

made by Gnoup

C and D tnainees on

the climates within

thein onganizations

225

LO.I2 Relationship

between wonk behavioun and

age,

tenure and experience

of

Gnoup C and

D

subjects

227

Numben

of

nesponges, mean score and nange

on each

cnitenion

26g

Rank-ordening

of

^session assessments on

each

cnitenion

269

Ia

Ib

Appendix

III

VII

VIII II

LTST OF APPEI{DICES

Results

of

session assessment fonms

of section

7.4

Analysis of

nepentony

gnid

data

of

chapten ⁹

Six

rnonth

follow-up

questionnaine fon tnainees and

control

subjects

Six

rnonth

follow-up

guestionnaine fon contnol-Iing

officens

IX

X

Twelve nonth follow-up questionnair"e

Page

268

270

287

289

293 IV

V

Pnecounse intenviews and questionnaines 275

Session assessment fonm 280

Immediate

post

counse questionnaines 28!

Three month fol1ow-up questionnaine 284

VI

The

Kelly

nepentony gnid 294