Budgetary criteria in performance evaluation: a critical
appraisal using new evidence
David Otley
a,*, Raili M. Pollanen
baDepartment of Accounting and Finance, Lancaster University, Lancaster LA1 4YX, UK bFaculty of Administration, Regina University, Regina SK, S45 0A2, Canada
Abstract
Previous studies on the use of budgetary criteria in performance evaluation have used a wide range of variables, measures and models, but have concentrated upon extending rather than repeating previous work. This study includes replications of parts of ®ve previous studies and obtains results that dier in many respects from previous ®ndings. Because of the use of similar measures and models, it can be concluded that the dierent results stem primarily from the dierent sample of managers studied. More generally, it is suggested that control practices dier across organiza-tions, cultures and time; universal ®ndings should not be expected. Rather, a more coherent programme of work is required to explicate the varied ways in which budgetary techniques can be used in performance evaluation and man-agement.#2000 Elsevier Science Ltd. All rights reserved.
1. Introduction
The past 25 years have seen a small but con-tinuing stream of studies that have sought to explore the in¯uence of budgetary controls on managerial performance. The seminal work in this ®eld was the study by Hopwood (1972) which explored the role of accounting data in managerial performance evaluation. It was quickly followed by a study which, using similar measures, found con¯icting results (Otley, 1978). This apparent con¯ict stimulated a stream of work, much of it from Australian authors (e.g. Brownell, 1982, 1985; Dunk, 1989; Harrison, 1992), which intro-duced additional variables in order to provide contingent explanations of the observed dier-ences in managerial behaviour.
In one sense, this stream of work was clearly successful. A variety of exploratory studies pro-duced a range of ®ndings and statistically sig-ni®cant results. Indeed, Brownell and Dunk (1991) stated that ``the continuing stream of work devo-ted to this issue constitutes, in our view, the only organized critical mass of empirical work in man-agement accounting at present''. Even so, the work is characterised by a number of de®ciencies. Dierent authors have tackled dierent aspects of the issue; dierent studies use dierent subsets of variables; in many cases, the same variable is measured in importantly dierent ways. The sam-ples of managers studied have been drawn from dierent types of organization in dierent coun-tries and the sample size has been generally quite small. Some studies use samples of managers drawn from a single organization; others use managers from dierent organizations. Most
www.elsevier.com/locate/aos
0361-3682/00/$ - see front matter#2000 Elsevier Science Ltd. All rights reserved. P I I : S 0 3 6 1 - 3 6 8 2 ( 9 8 ) 0 0 0 3 1 - 2
surprisingly, no single study has been precisely replicated; each new piece of work has chosen to vary some aspect of the research design so that it is impossible to gauge the general validity of any of the reported work.
The motivation behind the study reported here is to repeat, in a dierent context, several of the major previous studies to assess the robustness of their ®ndings and to form the basis of further work that is truly cumulative in nature. Such an endeavour requires attention to be paid to the conceptualisation of relevant variables, their mea-surement and the type of sample drawn. However, this is a substantial exercise and the work reported here concentrates upon repeating some of the more important previous studies. This has led to each previous study being replicated using (as far as possible) exactly the same measures as origin-ally used, despite the known problems with some of the measurement instruments, especially eva-luative style. The issues involved in conceptualiz-ing evaluative style is the subject of another paper (Otley and Fakiolas, 1998). A reasonably large sample (n=ca. 125) for this body of literature was used, consisting of managers drawn from a single sector having common control practices in order to reduce the confounding eects of using a sam-ple of managers drawn from a variety of dierent enterprises and industries.
The results indicate a range of outcomes. Some results are con®rmed; some ®ndings are insignif-icant statistically; but other results are contrary to those previously reported. At the very least, this indicates the need for a more careful and bench-marked approach to research design in this ®eld. At present, we cannot judge the extent to which statistically signi®cant reported results are likely to be generalizable to other contexts. It is hoped that this paper represents one small step along the path towards the accumulation of more reliable results in this important area.
2. Hypotheses
A set of interaction studies were selected, all of which made use of Hopwood's original instrument to measure evaluative style or what has now
become known as Reliance on Accounting Per-formance Measures (RAPM). These were Brow-nell (1982), BrowBrow-nell and Hirst (1986), Dunk (1989), Brownell and Dunk (1991) and Harrison (1992). All of these studies included budgetary participation as an explanatory variable, and all considered managerial performance as the depen-dent variable, with the exception of Harrison (1992) where job-related tension (JRT) was used as a surrogate. However, as JRT was also used as an additional dependent variable in Brownell and Hirst, it seemed appropriate to include JRT as an additional dependent variable. Two studies also used job satisfaction as a dependent variable, and two used job satisfaction as a moderating variable, so this was also included. Finally, two studies identi®ed task uncertainty as a moderating vari-able, but each used dierent measures. We chose to include task uncertainty, but to use an alter-native measure better suited to our context. The above selection gave a reasonable spread of stu-dies and hypotheses to be replicated within the constraint of using a questionnaire of reasonable length that would enable a high response rate to be obtained.
The study therefore attempts to replicate some important parts of previous studies by testing the following hypotheses, stated in their null form:
1. Brownell (1982) H1: There is no interaction
between supervisory evaluative style and
budgetary participation aecting
perfor-mance or job satisfaction.
2. Brownell and Hirst (1986) H2: There is no
interaction between budget emphasis, bud-getary participation and task uncertainty aecting performance or JRT.
3. Dunk (1989) H3: There is no interaction
between budget emphasis and budgetary participation aecting performance.
4. Brownell and Dunk (1991) H4: There is no
interaction between budget emphasis, bud-getary participation and task uncertainty aecting performance.
5. Harrison (1992) H5: There is no interaction
3. Measurement of variables
3.1. Budget emphasis [RAPM]
Much of the preceding work, and all the studies being replicated, based their measurement of bud-get emphasis on Hopwood's (1972) instrument. The eight-item instrument developed by Hopwood was subsequently modi®ed and used by Otley (1978), Brownell (1982), Brownell and Hirst (1986) and others. The ten-item variant used by Brownell and Hirst was used here, but with two minor var-iations. First, the measure used here includes one attitude item instead of the two items into which Brownell and Hirst had split the original scale item. Second, an additional item regarding co-operation with peers was added. Neither of these modi®cations aected the two essential scale items, namely `my long-term concern with costs and revenues' and `my ability to meet budgets in the short-run' that Hopwood contrasted to develop his measure of evaluative style.
Both an importance rating and a ranking scale were used, as in Hopwood's original study. How-ever, in order to help ensure that the ranking question was properly answered following repor-ted problems in its use by Brownell (1982), the question was asked twice, once with rankings required and once for importance scores. This proved successful in obtaining complete responses. The information obtained from these two sets of scores enable all variants of the budget emphasis measure used in the major studies being replicated to be precisely derived.
3.2. Participation
The six-item measure developed by Milani (1975) was used as it had been used in almost identical form in all the previous studies. These items were incorporated into a larger, nine-item scale that included three extra items developed by Pope and Otley (1996). However, these extra items did not correlate well with the Milani items and were dropped from the further analysis. When the scale was factor analysed all six items loaded onto a single factor and the six-item Milani scale showed a Cronbach alpha of 83%. A seven-point
importance scale was used, which is identical to all the previous studies except Dunk (1989) who appears to have used a ®ve-point scale. The descriptive statistics are shown in Table 1.
3.3. Task uncertainty
This proved to be the most dicult variable to replicate as the two previous studies that incorpo-rated this variable used dierent measures. As each measure used multi-item scales, it was not felt desirable to include a variety of dierent measures into the questionnaire for reasons of length. The measure developed by Sathe (1974), which itself slightly modi®ed Duncan's (1972) measure, was selected because of its broad scope and likely applicability to the research site chosen. This measure incorporates three main dimensions of task uncertainty: lack of information about envir-onmental factors; inability to attach probabilities to environmental factors; and not knowing the eects of incorrect decisions. The broader scope of this measure made it more suitable for use in the public sector organizations being studied. The original scale had 12 items which yielded a Cron-bach alpha of 64%, but reliability analysis indi-cated that this would improve to 70% if two items were omitted. As neither previous study being replicated had used the same measure, these two items were removed. Factor analysis indicated three underlying factors, as expected, accounting for 50% of the total variance. A single measure comprising of the ten selected items was therefore used.
3.4. Job satisfaction
analysis indicated ®ve factors, accounting for 59% of the total variance, but with a predominant initial factor which accounted for 29% of the var-iance. It therefore seemed reasonable to use a sin-gle scale comprising of the 20 items which had a Cronbach alpha of 85%.
3.5. Job-related tension
Job-related tension [JRT] was measured using the 15-item instrument developed by the Institute for Social Research at the University of Michigan (Kahn, Wolfe, Quinn, Snoek, & Rosenthal, 1964). This has been widely used with consistently reli-able results. Two items on budget-related tension were added to the instrument for other purposes, but not used in the analyses reported here. A few minor changes in wording were made as with pre-vious scales. Factor analysis indicated four factors accounting for 58% of the total variance, but with one predominant factor accounting for 30%. Thus a single 15-item scale was used and an overall Cronbach alpha of 81% obtained. The average JRT score of 2.50 is comparable to that found in other studies.
3.6. Performance
The nine-item scale developed by Mahoney, Jerdee, and Carroll (1963) has been widely used. It consists of eight sub-dimensions and one overall question. However, in the literature being repli-cated, only the overall measure was ever used after it was demonstrated that the overall measure explained a substantial proportion of the variance in the other eight items. Thus, only the overall measure was selected for use in this study, to keep the questionnaire length within bounds.
4. Research site
In selecting a research site, two con¯icting requirements became apparent. Although drawing managers from a wide range of dierent organi-zations may appear to give representativeness and allow results to be generalized, the restricted range of control practices observed in a narrowly based questionnaire survey does not assist in controlling for the impact of unobserved practices. It may, therefore, be more meaningful to select managers
Table 1
Descriptive statistics of measures
Variables No. of
items
Sample size
Mean SD Theoretical range Actual range
Min Max Min Max
Role of accounting inf. (Brownell, 1985) 2 123 1.99 0.93 1 5 1 5
Relative role of accounting inf. (Harrison, 1992) 2/8 121 1.00 0.50 0.2 5 0.3 4.0
Budget participation (Milani, 1975) 6 121 2.25 0.95 1 7 1 6.2
Task uncertainty (Sathe, 1974) 10 121 2.49 0.33 1 5 1.8 3.8
Job satisfaction (Weiss, Davis, England, & Lofquist, 1967) 20 123 2.02 0.41 1 5 1.2 3.3
Job-related tension (Kahn et al., 1964) 15 127 2.50 0.45 1 5 1.2 3.7
Performance (Mahoney et al., 1963) 1 126 4.02 0.65 1 5 2.0 5.0
Distribution of evaluative styles
Evaluative style BC (%) BP (%) PC (%) NA (%)
This study 6 8 30 56
Hopwood (1972) 20 10 26 44
Otley (1978) 13 56 28 3
Brownell (1982) 21 24 24 31
from a single organization where such practices might both be observed and have a more restricted range of variation. However, the results would not necessarily be generalizable beyond the organiza-tion concerned, which would also have to be quite large in order to allow a sample of adequate size to be drawn. Previous studies have used both types of sample. In the event, it was decided to restrict the scope of the study to a single sector, but to sample from as complete a range of organizations as pos-sible within that sector. At the very least, the results obtained ought to be generalizable to that sector.
The chosen sector was the administration of universities and colleges in the province of Ontario, Canada. This allowed a relatively homo-geneous population of institutions large enough for budgetary control techniques to be an impor-tant control mechanism, and subject to common control procedures. Although the sector contains dierent types of organization, because of report-ing and control requirements determined by the Ontario Ministry of Education and Training they face a relatively uniform control environment, at least in terms of upward accountability. Access was negotiated through the Deputy Minister, by writing to the presidents of 35 publicly funded colleges and universities, of whom 16 agreed to participate. These presidents supplied lists of senior administrators having signi®cant budgetary responsibilities and 176 such people were selected for the ®nal mailing list (with titles such as vice-president for academic, administrative, ®nancial or service operations, or dean of an academic programme area). Such people typically reported to the president either directly or via just one level of management.
A substantial questionnaire, consisting of the questions already outlined plus an approximately equal number of additional items relating to other budgetary behaviours was sent to each of the 176 selected respondents. A total of 134 replies (76%) were obtained after some follow-up by telephone, of which 127 were usable (72% net response rate). These replies appeared to be representative of the institutions sampled, with no evidence of
non-response bias being apparent.1 A few
ques-tionnaires were not fully completed in all respects, so the sample size is not 127 for all analyses, but it
always exceeds 120 and is reported for each set of results. Individual responses were kept anon-ymous, but a report of the aggregate results was prepared which was sent back to the presidents of participating colleges. Summary statistics of the variables relevant to this paper are given in Table 1. These indicate a wide range of responses and considerable variation between respondents on the independent and dependent variables.
5. Results
The results will be presented as a series of com-parisons between the ®ndings of this study, and those found in the individual studies replicated, as one of the main purposes of this paper is to draw attention to the similarities and dierences that
occur.2 Similarities in ®ndings may indicate that
underlying results are persistent and stable fea-tures of behaviour across a range of organizations and situations; dierences, by contrast, indicate that the previous results should not be treated as universal, and that more attention needs to be paid to the circumstances under which greater generalisation is likely to be possible. It should be noted that the measures used are adjusted to match those used in the original studies wherever possible; this means that the reported results can-not be compared across studies, except where indi-cated, as dierent scales may be used in each case.
5.1. Brownell (1982)
This study examined the eect of budget emphasis and participation on job satisfaction and managerial performance, in particular hypothesis-ing an interaction between budget emphasis and
1For reasons of con®dentiality, it was not possible to deter-mine response rates by institution; however, descriptive data relating to respondents appeared to appropriately re¯ect the characteristics of the sector.
participation. The sample was 48 managers hold-ing cost centre responsibility in a US manufactur-ing company. Budget emphasis was measured using Hopwood's method of comparing the rank-ing of the two relevant items from the 10-item measure. Brownell then combined Hopwood's Budget-Constrained [BC] and Budget-Pro®t [BP] categories into a high budget emphasis category
(coded ÿ1), and his Pro®t Conscious [PC] and
Non-Accounting [NA] categories into a low bud-get emphasis category (coded +1). The descriptive statistics for each study are shown in Table 2, with this study's variables recoded to exactly match those of Brownell (1982).3
There are two main dierences. First, there was a considerably higher level of budgetary partici-pation in this study compared with Brownell's. Therefore analyses were re-run coding participa-tion as high or low using both this study's mean and Brownell's mean. Although this causes some dierences in the results, the main interaction eect is virtually unchanged. Second, the distribu-tion of budget emphasis scores is rather dierent, with only 14% [compared to 45% in Brownell (1982)] being categorised as high budget
empha-sis.4 Nevertheless this still represents some 17
respondents, the same absolute number as Brow-nell, and does not aect the comparisons by category.
The results of regressing the data in the interac-tion model are shown in Table 3, for both job satisfaction and managerial performance. These clearly indicate dierent results. The interaction term is on the borderlines of statistical signi®cance at the 5% level for job satisfaction in this study, but is not signi®cant for managerial performance. These results are thus quite dierent to those found by Brownell (1982). With regard to job satisfaction,
the supposed incompatible combinations of parti-cipation and budget emphasis (high±high and low± low) are associated with high job satisfaction and the compatible combinations (low±high and high± low) are associated with low job satisfaction. This result is con®rmed by means tests and graphical analysis, and by using two alternative measures of budget emphasis. With regard to performance, where Brownell found a signi®cant and positive result, this study found a negative, albeit insignif-icant, result. If an alternative measure of budget emphasis is used [combining all the accounting styles (BC+BP+PC) into one category and non-accounting into the other] the negative result becomes signi®cant at the 2% level.
In summary, the results of this study are not con-sistent with Brownell's ®ndings; the relationships found tend to be in the opposite direction, although not always achieving statistical signi®cance. This is unlikely to be an artefact of the dierent distribu-tion of budget emphasis that was observed, nor of the dierent absolute level of participation. Rather, it has to be laid at the door of the dierent context in which the study was conducted. What is clear is that the Brownell (1982) result is not uni-versal; the limits of its validity need further exploration.
5.2. Brownell and Hirst (1986)
Part of Brownell and Hirst's (1986) study repli-cated Brownell (1982) but using job-related ten-sion (JRT) as a dependent variable in place of performance and job satisfaction. This will be considered before the main part of their study that added task uncertainty as a further explanatory variable.
One relatively minor dierence between the two studies is that budget emphasis is not measured by the relative rankings of the top 3 items on the Hopwood scale, but uses the top 4 items. This
procedure is followed here for our own results.5A
direct comparison of the eect of budget emphasis and participation on JRT with Brownell and Hirst (1986) cannot be made as they do not present this restricted equation (although it can be compared with the 3-way interaction equation presented later). However, the 2-way interaction coecient
3In all cases, re-coding was performed to give exact compar-ibility between all the reported regression coecients. This goes some way towards countering Southwood's (1978) assertion that only the highest order interaction coecient is meaningful because other coecients can change if Likert-scaled variables are re-coded. We would contend that where measures of vari-ables are precisely replicated, and the same coding is used in two studies, then it is both valid and useful to compare all reported coecients.
in a two-variable regression model is positive with
a value of +0.271 (p=0. 0258), again indicating
that the so-called compatible combinations are associated with high JRT and the incompatible combinations with low JRT. This emphasises the
dierence between these results and Brownell (1982) and appears to be the opposite result to that found by Brownell and Hirst (1986), albeit their result was only signi®cant under conditions of low task uncertainty. However, the main thrust of the Brownell and Hirst (1986) study was to include a third explanatory variable, namely task uncertainty. As has been explained earlier, a dif-ferent measure of task uncertainty has been used in this study, but this should not, in principle, substantially aect the statistical signi®cance of the highest order interaction term in the regression equation, so the results are compared in Table 4. The 3-way interaction that was signi®cant in Brownell and Hirst is not signi®cant here, and has the opposite sign. Both models had similar expla-natory power (19%). Finally, Brownell and Hirst's ®nding that the interaction between budget emphasis and participation was signi®cant only under conditions of low task uncertainty was not replicated here. Not only was there no dierence in the interaction eect between groups based on task uncertainty, but the overall eect was in the opposite direction.
The overall results of this study are therefore not supportive of Brownell and Hirst's conclusions. Task uncertainty was relevant here only as a main eect. The same contradictory result was found
for the budget emphasis participation
interac-tion for JRT as had been found for job satisfac-tion and managerial performance. No 3-way
5It should be noted that if the top 4 criterion had been used in the previous comparisons, the results are similar, but both interaction coecients become statistically signi®cant (job satisfaction at the 2% level and performance at the 0.5% level). Table 2
Descriptive statistics for continuous variablesÐcomparison with Brownell (1982)
Variables No. of items Sample size Mean SD Theoretical range Actual range
Min Max Min Max
This study
Budget participation 6 121 34.48 5.69 6 42 11 42
Job satisfaction 20 123 79.59 8.18 20 100 54 97
Managerial performance 1 126 7.03 1.29 1 9 3 9
Brownell (1982)
Budget participation 6 38 25.79 7.10 6 42 13 39
Job satisfaction 20 38 76.50 7.25 20 100 58 93
Managerial performance 1 38 6.84 0.95 1 9 5 8
Note: Consistent with Brownell (1982), budget participation was reverse-coded and the total (rather than the average) score used. Similarly, job satisfaction was reverse-coded and converted to a 20±100 scale. Managerial performance was re-coded to a 9-point scale.
Table 3
Results of regressing job satisfaction and managerial perfor-mance on budget emphasisÐcomparison with Brownell (1982)
Variables This study Brownell (1982)
Value p Value p
Job satisfaction
Intercept 78.4 0.0000 76.5 <0.01 Budget participation ÿ0.41 0.0050 ÿ0.19 ns Budget emphasis 1.17 0.2448 ÿ0.79 ns Participation
budget emphasis
ÿ0.28 0.0515 0.05 ns
R2=0.19 R2=0.06 Managerial performance
Intercept 7.08 0.0000 6.71 <0.01 Budget participation 0.03 0.2302 ÿ0.09 <0.01 Budget emphasis ÿ0.06 0.7352 0.19 <0.10 Participation
budget emphasis
ÿ0.04 0.1533 0.04 <0.025
interaction was found. This indicates that the organization studied diered in its characteristics from the Australian manufacturing company stu-died by Brownell and Hirst (1986).
5.3. Dunk (1989)
Dunk's study appears to be an exact replication of Brownell (1982) as it is concerned with the
interac-tion between budget emphasis and participainterac-tion. However, there is a major dierence in the way in which budget emphasis is measured. Although based on a Hopwood-type scale, the ranking method is not used (partly because of diculties in obtaining rank information in prior studies). Rather, the absolute importance of the two budget based items (namely, `meeting the budget' and `concern with costs') was used. Further, whereas Hopwood contrasted the relative importance of these two items, Dunk (1989), following Brownell (1985), adds the absolute importance ratings toge-ther to produce a single score, which he argues represents the absolute emphasis placed on bud-getary criteria.
Descriptive statistics for the variables concerned as given in Table 5. These are reasonably similar to those reported by Dunk, except that there is a higher level of budget emphasis in the managers studied by Dunk (production managers from 26 ®rms in northern Britain). Most notably, the range of scores obtained by Dunk is con®ned to the higher end of the scale. The results of the regres-sion equation to estimate the 2-way interaction are given in Table 6. Although the sign of the interac-tion is the same in both cases, the statistically sig-ni®cant result found by Dunk is not found here. Indeed, the model is a very poor ®t for this study's data, with the regression model explaining less than 2% of the variation, compared to nearly 20% for Dunk's data. Dunk concluded that, with the results of his study, evidence for all three interac-tion eects (i.e. positive, negative and null) had been found. However, this study does not provide much support for Dunk. More importantly, the
Table 4
Results of regressing job related tension on budget emphasis, budget participation and task uncertaintyÐcomparison with Brownell and Hirst (1986)
Variables This study Brownell and Hirst (1986)
Value p Value p
Job-related tension
Intercept 37.51 0.0000 38.57 <0.01 Budget participation ÿ0.085 0.6400 ÿ0.3140 <0.01 Budget emphasis ÿ0.747 0.3296 ÿ1.465 ns Task uncertainty ÿ2.148 0.0059 0.1090 ns Participation
budget emphasis
0.353 0.0547 ÿ0.1193 ns
Participation task uncertainty
ÿ0.015 0.9354 0.0089 ns
Budget emphasis task uncertainty
ÿ0.717 0.3492 0.3400 ns
Participation budget emphasis task uncertainty
0.137 0.4523 ÿ0.2167 <0.07
R2=0.19 R2=0.19
Note: Because a dierent measure of task uncertainty was used in the two studies compared above, the only meaningful compar-ison that can be made in the above Table relates to the sign and statistical signi®cance of the 3-way interaction coecient.
Table 5
Descriptive statistics for continuous variablesÐcomparison with Dunk (1989)
Variables No. of items Sample size Mean SD Theoretical range Actual range
Min Max Min Max
This study
Budget participation 6 121 24.98 3.79 6 30 9.3 30
Budget emphasis 2 123 8.02 1.87 2 10 2 10
Managerial performance 1 126 4.02 0.65 1 5 2 5
Dunk (1989)
Budget participation 6 26 23.42 4.15 6 30 15 30
Budget emphasis 2 26 9.39 0.75 2 10 8 10
measure of budget emphasis used clearly aects the results. The validity of the additive measure used by Dunk (1989) clearly depends upon there being a positive intercorrelation between the two items being aggregated. Dunk did not report this for his sample, although the total scores were all high; for our data, the correlation coecient is +0.59.
Again, there is no clear result here. Despite having a considerably larger sample, we were unable to attain statistical signi®cance for the interaction coecient. But more importantly, the dierent measure used by Dunk does not allow his study to be used to perform the reconciliation between Brownell (1982) and Brownell and Hirst (1986) that it claims to attempt.
5.4. Brownell and Dunk (1991)
Brownell and Hirst (1986) found a signi®cant 3-way interaction between budget emphasis, partici-pation and task uncertainty when the dependent variable was JRT. However, the results were insigni®cant when the dependent variable was managerial performance. The Brownell and Dunk (1991) study is an attempt to ®nd such a 3-way interaction by using more careful measurement techniques, in particular, relating to task uncer-tainty where they use the Van de Ven and Delbecq (1974) measure. However, they also modi®ed the measure of budget emphasis, by moving away from the original Hopwood measure to the abso-lute importance measure originally reported in Brownell (1985) and used by Dunk (1989). The current study can be used to partially replicate Brownell and Dunk (1991), with all measures except task uncertainty being identical. Descrip-tive statistics are given in Table 7.
The data are reasonably comparable, given the dierence in scales used to measure task uncer-tainty, but even here re¯ect a similar range of variation. Brownell and Dunk's sample is slightly more budget-oriented, largely accounted for by the lack of those having low budget orientations, as can be seen from Table 7. The 3-way interaction regression equation was run exactly as reported by Brownell and Dunk (1991) giving the results reported in Table 8. Although the overall model explains just 18% of the variance compared with
Table 6
Results of regressing managerial performance on budget emphasis and budget participationÐcomparison with Dunk (1989)
Variables This study Dunk (1989)
Value p Value p
Managerial performance
Intercept 4.03 0.0000 3.92 <0.01 Budget participation 0.012 0.5120 0.007 ns Budget emphasis 0.00 0.9999 ÿ0.006 ns Participation
budget emphasis
ÿ0.021 0.2277 ÿ0.050 <0.05
R2=0.02 R2=0.20
Table 7
Descriptive statistics for continuous variablesÐcomparison with Brownell and Dunk (1991)
Variables No. of items Sample zsize Mean SD Theoretical range Actual range
Min Max Min Max
This study
Budget participation 6 121 34.48 5.69 6 42 11 42
Budget emphasis 2 123 11.04 2.80 2 14 2 14
Task uncertainty 10 125 32.40 5.01 10 70 22 52
Managerial performance 1 126 5.52 0.97 1 7 2.5 7
Brownell and Dunk (1991)
Budget participation 6 79 31.22 7.49 6 42 6 42
Budget emphasis 2 79 11.73 1.85 2 14 6 14
Task uncertainty 14 79 49.96 9.20 14 98 24 78
28% for Brownell and Dunk, the 3-way interaction coecient is statistically signi®cant in both cases.
Following Brownell and Dunk, separate regres-sions were computed for high task uncertainty and low task uncertainty conditions. Despite dierent measures being used, if they give only broadly similar rankings, this procedure should allow the two separate regression results for the two condi-tions to be accurately compared. Brownell and
Dunk (1991) found the participation budget
emphasis interaction to be signi®cant only under low task uncertainty; under high task uncertainty, it was very nearly signi®cant, but in the opposite direction. By contrast, neither of our separate regression equations showed signi®cant coe-cients under either condition. Furthermore, no signi®cant support was found for participation being related to high performance under high task uncertainty. Therefore, use of Brownell's absolute budget emphasis measure did not produce con-sistent results for managerial performance.
These results indicate the diculty in interpreting a 3-way interaction coecient. A similar, statisti-cally signi®cant, interaction coecient can be gen-erated by a variety of underlying mechanisms; although a model may predict a speci®c sign for a 3-way interaction coecient, ®nding such a result does not imply the underlying model is sound. A more detailed examination of the mechanisms that have generated it is required. In this case, although the interaction term is similar in sign, magnitude and statistical signi®cance, it is clear that the way in which it was generated was quite dierent to that observed by Brownell and Dunk (1991).
5.5. Harrison (1992)
Harrison's (1992) study was primarily designed to investigate the eect of national culture on the relationships previously studied. As the present study did not attempt to measure culture, and was designed to hold culture constant, no direct repli-cation is possible here. However, an additional aspect of Harrison's study was that he used a fur-ther variant of the budget emphasis measure. Arguing that Brownell's absolute measure (i.e. summing the importance scores on the two rele-vant Hopwood variables) does not accord with Hopwood's original concept, he develops a rela-tive measure by dividing (four times) the absolute score by the sum of the eight other scores from the ten-item scale. The four times multiplier is to standardise the score, which now has a score of unity if budgetary and non-budgetary criteria have equal average importance. It should be noted that this represents a potential improvement on the Brownell absolute measure, but is still con-ceptually distinct from the original Hopwood measure.
Using Harrison's measure produced no sig-ni®cant regression coecients for job satisfaction or managerial performance. For JRT, no coe-cients were signi®cant when the 3-way interaction term was included in the equation, but if it was deleted the results set out in Table 9 are produced. This shows a signi®cant interaction between bud-get emphasis and task uncertainty. To investigate this further a series of means tests were conducted, as shown in Table 10, with the data being split at
Table 8
Results of regressing managerial performance on budget emphasis, budget participation and task uncertaintyÐcompar-ison with Brownell and Dunk (1991)
Variables This study Brownell and Dunk (1991)
Value p Value p
Managerial performance
Intercept 37.74 0.0139 6.22 0.0000
Budget participation
ÿ0.832 0.0431 ÿ0.0649 0.276
Budget emphasis
ÿ3.00 0.0305 0.2526 0.411
Task uncertainty ÿ1.048 0.0208 ÿ0.0124 0.247 Participation
budget emphasis
0.083 0.0266 0.0961 0.006
Participation task uncertainty
0.028 0.0255 0.0017 0.144
Budget emphasis task uncertainty
0.099 0.0161 ÿ0.0032 0.587
Participation budget emphasis task uncertainty
ÿ0.003 0.0134 ÿ0.0016 0.018
R2=0.184 R2=0.276
the mean values of each variable. This clearly indicates that, despite the signi®cant interaction coecient, it is task uncertainty that is the major driver of the dierences in job-related tension. Thus, a statistically signi®cant interaction eect does not necessarily imply the existence of a major substantive connection. Prior work has con-centrated on statistical signi®cance alone; this example indicates the need also to pay attention to the size of the substantive eect.
5.6. Summary of results
The results are varied, but clearly many are inconsistent with previously reported work. Not only are the Brownell (1982) results not replicated,
but the signs of the 2-way interaction coecients reported are opposite, although not always statis-tically signi®cant. The same applies to the 3-way interaction eect found by Brownell and Hirst (1986). By contrast, it may be argued that the results reported here are not inconsistent with those reported by Dunk (1989), although our model explained only 2% of the variance in per-formance compared with Dunk's 20%. Further, the results are consistent with the negative 3-way interaction between budget emphasis, participa-tion and task uncertainty on managerial perfor-mance reported by Brownell and Dunk (1991). Finally, comparison with Harrison (1992) indi-cates that although a regression analysis found a statistically signi®cant 2-way interaction between budget emphasis and task uncertainty on JRT, more detailed analysis showed this to be sub-stantively explained by the sole eect of task uncertainty. This indicates the diculty in inter-preting interaction coecients in this context, where there is likely to be multi-colinearity between variables, and the need to report the size of substantive eects as well as levels of statistical signi®cance.
6. Discussion
Several complementary lines of argument can be developed from these results. First, it can validly be suggested that the main explanation for the dierent results lies in the dierent organization involved in this study. Indeed, this result was anticipated in the design of the study and is the reason why each study has been replicated as pre-cisely as possible using identical instruments. Hopwood's original work was conduced in cost centres in an integrated steel works in the USA. Otley's subsequent study was conducted at a site deliberately chosen to minimise the eect of a possible inappropriate match between accounting reports and other measures of performance; namely the nationally-owned coal industry in the UK. These dierences led Hirst (1981) to spec-ulate about the use of task uncertainty to explain the con¯icting ®ndings. The samples for each of the studies replicated here are set out below:
Table 9
Results of regressing job-related tension on budget participa-tion, budget emphasis and task uncertainty (3-way interaction not included)
Variables This study
Value p
Job-related tension
Intercept 31.61 0.4033
Budget participation ÿ1.032 0.2806
Budget emphasis 19.06 0.2798
Task uncertainty 0.726 0.4674
Participationbudget emphasis 0.265 0.4748 Participationtask uncertainty 0.016 0.5409 Budget emphasistask uncertainty ÿ0.838 0.0165
R2=0.20
Table 10
Comparison of job-related tension means by budget emphasis and task uncertainty
Job-related tension Task uncertainty
High Low
Budget emphasis
High 40.5 35.0
(7.1) (7.9) n=26 n=23
Low 39.4 35.4
(5.3) (5.5) n=41 n=29
Study N Sample
Brownell (1982) 38 Managers from US
manufacturing company
Brownell 76 Line managers from a
and Hirst (1986) variety of functions in a
large Australian manu-facturing company
Dunk (1989) 26 Production managers in
UK (Scottish)
manu-facturing company (con-sumer products)
Brownell 79 Managers from dierent
and Dunk (1991) functions in 46 dierent
Australian manufactur-ing ®rms
Harrison (1992) 115 Singaporean managers
96 Australian managers
Looked at from this standpoint, it is perhaps unsurprising that dierent studies have come up with dierent results. The samples are both small and diverse. Moreover, the variables used and the ways in which they were measured also diered signi®cantly, although the replications reported here have attempted to minimise this cause of dif-ference. To expect consistency over such a diverse range of managers over a 20 year period is clearly optimistic.
What is perhaps surprising is the lack of more precise replication and extension over such a per-iod. This is in marked contrast with the tradition in the `hard' sciences where replication is the cor-nerstone of scienti®c reliability. Given that so little work has been conducted, and given the apparent propensity of academic journals in management accounting to prefer novelty to precise replica-tion,6the situation outlined may not be surprising,
but neither does it give con®dence in the general-izability of any reported results. Clearly, more empirical work is needed in a ®eld that is
embryonic, and the contingency approaches
adopted do provide a simple way forward in
explaining dierences. However, rather than
developing ever increasing numbers of interac-tions, a more fruitful way forward may be in the more careful speci®cation of 2-way analyses con-ducted in a variety of settings.
A further issue lies in the choice of sampling frame, in particular whether to restrict a sample to a single organisation or to attempt to construct a representative sample across a range of organiza-tions. Given that accounting controls form only a small part of a wider set of organizational controls that dier signi®cantly across organizations, it would seem wise initially to restrict consideration to single organizations so as to control for this eect. Such an approach is likely to increase the chance of forming understandings of the manner in which variables interact in a speci®c setting before attempting more ambitious generalisations. A second line of argument concerns the research methods adopted. The above studies have gen-erally focused upon questionnaire-based methods of measurement, although it should be noted that these have been combined with varying degrees of observation of the sites selected. For example, Hopwood states that his question to elucidate budgetary emphasis was constructed using the terminology currently in use within the organiza-tion being studied. Otley, although using data gathered by questionnaire for much of his analy-sis, also interviewed all the managers sampled and collected a range of contextual information that was used to interpret his results. It can reasonably be argued that a major aim of current work should be the development of understandings about the operation of organizational control processes; if such an aim is accepted then a greater emphasis on case-based and longitudinal work would seem to be appropriate.
A third issue is the development of appropriate concepts and measures. Central to this ®eld of work is the concept of budget emphasis ®rst put forward by Hopwood. This clari®ed the distinc-tion between short-term devices intended to improve reported budgetary results and a longer-term emphasis on real economic performance. This is an important distinction and one which would bear further investigation, relating as it does to wider aspects of `short-termism' in the
operation of enterprises that have stock market quotations. However, it should also be recognised that some of the developments of Hopwood's measure do not preserve this distinction, measur-ing instead some overall emphasis (whether abso-lute or relative) on meeting budget targets. This may also be an important explanatory variable, but it should not be confused with the relative emphasis placed on short-term vs long-term indi-cators of performance. There is clearly work to be done in developing better measures to capture this distinction. It is clearly still important, as indi-cated by the interest shown by companies in adopting performance measurement frameworks,
such as the Harvard `balanced scorecard'
approach (Kaplan and Norton, 1996). Finally, it is also worth noting in passing that the social-psy-chological variables being used in accounting
stu-dies are looking dated, most having been
developed in the 1960s. Accounting researchers need to draw on more recent developments in the social-psychological literature to improve the quality of their own research.
Finally, there is evidence to suggest that the role of budgeting as a tool of managerial control is changing (see Bunce et al., 1995) and that other, often non-®nancial, performance measures are increasingly being used. In addition, organizational forms are also undergoing change, with large, divi-sionalized organizations being replaced by smaller, more focused organizations linked into value chains Otley (1994). Studies of the role of account-ing techniques in organizational control need to ensure that they are able to take account of such changes, and not to assume that control practices have remained constant over the last 25 years.
7. Conclusions
Budgetary control systems still form a major plank of the control apparatus of many organiza-tions, although there are also indications that their role is changing. The stream of work on the impact of budget use on managerial behaviour and performance that has emanated from Hop-wood's seminal (1972) study has made an impor-tant contribution to understanding the roles that
budgetary information can play in performance evaluation. However, it has important limitations that are not always clear from the summaries of subsequent studies, where the same words are sometimes used to describe quite distinct concepts. Despite attempts to derive coherent explanations of behaviour, the evidence suggests that control practices and their eects dier markedly across organizations and cultures.
There are at least three paths that appear to oer the potential for progress. Firstly, there is the continuation of the traditional contingency-based approach. This needs more careful speci®cation and measurement of variables, both from an accounting point of view, and in its use of wider,
social-psychological developments. Secondly,
there is considerable scope for more intensive studies of single organizations aimed at elucidat-ing the impact of dierent accountelucidat-ing control practices within their wider context. Such studies could usefully include a longitudinal dimension that traces the antecedents to control practice change and its consequences. Thirdly, the new developments in organizational control practices that have been re¯ected, somewhat belatedly, in changes in management accounting practice also require study. Even if it were possible to make universal prescriptions at a given point in time, it is unlikely that such prescriptions would survive the passage of time under conditions of signi®cant organizational change. The connection of such control practices with the more general cultural assumptions upon which they are based also pro-vides a fruitful focus for study.
The last decade has seen major new develop-ments in management accounting practice. How-ever, these have been driven by a predominantly technical imperative, and have neglected the wide variety of ways in which managers actually use the data provided. The major contribution made by the behavioural accounting literature in since its inception in the 1950s has been to draw attention to the impact of accounting systems on human behaviour, and vice versa. This is no less impor-tant today, and the new control mechanisms being adopted by organizations will provide a rich vari-ety of practice which accounting researchers can both learn from and contribute to.
References
Brownell, P. (1982). The role of accounting data in performance evaluation, budgetary participation and organizational eec-tiveness.Journal of Accounting Research(Spring), 12±27. Brownell, P. (1985). Budgetary systems and the control of
functionally dierentiated organizational activities.Journal of Accounting Research(Autumn), 502±512.
Brownell, P., & Dunk, A. (1991). Task uncertainty and its interaction with budgetary participation and budget empha-sis: some methodological issues and empirical investigation. Accounting, Organizations and Society, 693±703.
Brownell, P., & Hirst, M. (1986). Reliance on accounting information, budgetary participation and task uncertainty: tests of a three-way interaction. Journal of Accounting Research(Autumn), 241±249.
Bunce, P., Fraser, R., & Woodcock, L. (1995). Advanced bud-geting: a journey to advanced management systems. Man-agement Accounting Research, 253±265.
Duncan, R.B. (1972). Characteristics of organizational envir-onments and perceived environmental uncertainty. Adminis-trative Science Quarterly, 313±327.
Dunk, A.S. (1989). Budget emphasis, budgetary participation and managerial performance: a note.Accounting, Organiza-tions and Society, 321±324.
Harrison, G.L. (1992). The cross-cultural generalizability of the relation between participation, budget emphasis and job-related attitudes.Accounting, Organizations and Society, 1±15. Hirst, M.K. (1981). Accounting information and the evaluation of subordinate performance: a situational approach. The Accounting Review, 771±784.
Hopwood, A.G. (1972). An empirical study of the role of accounting data in performance evaluation. Journal of Accounting Research, (Supplement), 156±182.
Kahn, R.L., Wolfe, D.M., Quinn, R.P., Snoek, J., & Rosen-thal, R.A. (1964). Organizational Stress: Studies in Role
Con¯ict and Ambiguity. Toronto, Canada:John Wiley and Sons.
Kaplan, R.S., & Norton, D.P. (1996).The balanced scorecard: translating strategy into action. Boston, MA: Harvard Busi-ness School Press.
Mahoney, T.A., Jerdee, T.H., & Carroll, S.J. (1963). Develop-ment of Managerial Performance: A Research Approach. Cincinnati, OH: South-Western Publishing.
Milani, K. (1975). The relationship of participation in budget-setting to industrial supervisor performance and attitudes: a ®eld study.The Accounting Review, 274±284.
Otley, D.T. (1978). Budget use and managerial performance. Journal of Accounting Research(Spring), 122±149.
Otley, D.T. (1994). Management control in contemporary organizations: towards a wider framework. Management Accounting Research, 289±299.
Otley, D.T., & Fakiolas, A. (1998). Reliance on accounting performance measures: dead end or new beginning? Accounting, Organizations and Society,24, 219±232. Pope, P., & Otley, D.T. (1996). Budgetary control and
perfor-mance evaluation: an empirical analysis of bank branches. Working Paper, University of Lancaster.
Sathe, V. (1974). Structural adaptation to environment: study of insurance company departments and branch banks. Doc-toral Dissertation, Ohio State University.
Southwood, K.E. (1978). Substantive theory and statistical interaction: ®ve models. American Journal of Sociology (March), 1154±1203.
Van de Ven, A.H., & Delbecq, A.L. (1974). A task contingent model of work-unit structure.Administrative Science Quarterly, 183±197.
Weiss, D.J., Davis, R.V., England, G.W., & Lofquist, L.H. (1967). Manual for Minnesota Satisfaction Questionnaire. Minnesota Studies in Vocational Rehabilitation, Vol. 22, Minneapolis, MN, University of Minnesota Industrial Rela-tions Center.
