Chapter 5 Phase 1 (Quantitative data collection and analysis)
5.5 Data analysis and preliminary results
5.5.3 Further analysis of elements: comparing by group
Section 5.4.4.6 discussed the importance of data reduction and showed that composite scales are reliable and applicable to exploratory research; also, average scores are preferable. As shown in section 3.2.5, practice theory assumes there are differences in practice among practitioners, according to their characteristics. Thus, further analysis was conducted to compare participants’ perceptions on the factors above. The average scores served as substitutes for the factors and the tests were conducted at p < .05 and 95 percent confidence interval (CI).
The main proposition “There is no difference in perception among participants on EOEP” was drawn in section 3.4. The following were derived to organize the analysis and draw conclusions:
1. There are no differences in perception on Evaluation Framework, according to participants’ characteristics
2. There are no differences in perception on Use of results, according to participants’
characteristics
3. There are no differences in perception on Organizational capacity, according to participants’ characteristics
4. There are no differences in perception on Shared understanding, according to participants’ characteristics
5. There are no differences in perception on Stakeholder participation, Evaluation Framework according to participants’ characteristics
6. There are no differences in perception on Affective issues, according to participants’
characteristics
The characteristics considered were: position, background, experience (i.e., years of experience and knowledge of EOEP). Section 5.4 showed the nature of data determines the appropriate analysis techniques. In consideration of the smaller sample size (N=106) and non-normality, non-parametric techniques were adopted for further analysis. Mann-Whitney U and Kruskal- Wallis tests were conducted. The results are displayed in Tables 5.21 and 5.22. The corresponding decisions to support or reject the propositions above are summarized in Tables E3.1 and E3.2 (Appendix E3).
5.5.3.1 Analysis by position and experience
Both position and experience consisted of two categories, therefore, Mann-Whitney U test was an appropriate technique to explore differences in perceptions of EOEP. Table 5.21 shows the results (test statistics, 2-sided significance, and mean ranks of groups). Table E3.1 shows whether or not the results supported the proposition. The highlighted propositions were rejected.
There were significant differences by position across all factors, except Affective issues. Also, the mean ranks of those in management position (i.e., Yes) were higher across all the factors—
a suggestion that more participants in this group agreed or strongly agreed with the items which composed the factors.
Similarly, there were significant differences among participants across all factors according to experience, except on Affective issues. The mean ranks showed many participants with more than five years’ experience agreed.
5.5.3.2 Analysis by background and knowledge
Both professional background and knowledge consisted of three categories and, therefore, Kruskal-Wallis test was appropriate. Table 5.22 shows the results (Chi-Square, degree of freedom, and significance at p = .05 and CI = .95). Table E3.2 shows whether or not the results supported the propositions. The highlighted propositions were rejected.
The mean rank of participants was highest and lowest for Evaluation and Other backgrounds respectively. That is, more participants within Evaluation agreed or strongly agreed on the items which formed the factors while the opposite applied to Other. Overall, there were significant differences according to background, except for Evaluation framework. However, such observations may not be conclusive (Field, 2017); thus, further analyses (pairwise comparisons) were conducted.
Figure E3.1 (Appendix E3) shows the results. A yellow line indicates a significant difference between respective pairs of factors. There were significant differences among all the groups on Organizational capacity. This was the only factor on which Evaluation and IT differed (i.e., higher perception compared to the latter). On the other five factors, Evaluation had the highest means and while Other had the lowest.
On knowledge, Table 5.22 shows there were significant differences across all factors, except Affective issues. The mean ranks of participants with Above average knowledge were highest across all factors (i.e., agreed or agreed strongly). Figure E3.2 (Appendix E3) shows there were significant differences across all groups on Organizational capacity. On the other five factors, there were significant differences between Above average against Average and Minimum.
Table 5.21: Mann-Whitney test (Differences by management position and experience)
Elements
Framework Recommendations Capacity Understanding Participation Issues
Management Mann-Whitney U 618.00 958.50 955.00 965.50 1012.00 1202.00
Z -4.64 -2.44 -2.46 -2.42 -2.11 -.86
Asymp. Sig. (2-tailed) .00 .02 .01 .02 .04 .39
Mean rank Yes 70.93 62.62 62.71 62.45 61.32 56.68
No 42.51 47.75 47.69 47.85 48.57 51.49
Experience (Years) Mann-Whitney U 1884.50 1935.50 2065.00 1810.00 1967.50 1696.00
Z 3.13 3.46 4.28 2.70 3.71 1.96
Asymp. Sig. (2-tailed) .00 .00 .00 .01 .00 .05
Mean rank 1 to 5 45.01 44.13 41.90 46.29 43.58 48.26
> 5 63.76 64.82 67.52 62.21 65.49 59.83
Table 5.22: Kruskal-Wallis test (Differences by background and knowledge, df=2)
Elements
Framework Recommendations Capacity Understanding Participation Issues
Background Chi-Square 2.18 17.43 41.45 6.13 9.60 7.97
Asymp. Sig. .34 .00 .00 .05 .01 .02
Mean rank IT 49.78 55.33 55.79 51.93 56.83 57.34
Evaluation 60.76 70.72 80.52 65.30 64.24 62.39
Other 52.50 38.40 30.50 46.53 41.47 42.25
Knowledge Chi-Square 25.97 9.24 30.63 12.59 22.03 2.02
Asymp. Sig. .00 .01 .00 .00 .00 .36
Mean rank Minimum 36.82 50.41 28.75 41.41 31.64 46.93
Average 43.74 44.31 48.13 47.24 50.14 52.43
Above average 72.00 64.31 71.83 66.10 68.31 58.01
An Agree-Disagree Likert scale provides negative and positive directions (Dolnicar, 2013).
The following general conclusions may be drawn. There were differences among participants across all factors, except Affective issues. Managers from Evaluation and IT backgrounds with experience (i.e., more than five years of experience and above average knowledge) were likely to have a positive perception about EOEP (i.e., agree or strongly agree about items). On organizational capacity, there were differences across all categories by position, background, years of experience, and knowledge.
The structure which described participants’ perceptions of EOEP was further explored with cluster analysis to understand how the elements combined (i.e., patterns of performance) and the differences among groups. The results were compared to the foregoing conclusions.