This section elicited the opinions of academic staff from the IT department on the newly developed system. The purpose of this survey was to:
Elicit the responses from academic staff on whether the newly developed system was able to meet their requirements in terms of the systems capabilities and functionality;
Elicit the responses from the participants on User Interface Satisfaction (UIS) regarding the newly developed system; and
157
Determine the extent to which the new system was able to meet the objectives indicated in section 5.4.
The researcher chose the IT department for the survey because most academic staff from this department are experienced in areas such as programming, the design and development of software and databases, networking and IS. Their inputs were necessary in improving aspects relating to user interface as well as the functionality of the developed system.
Twenty-eight (28) staff members from the IT department participated in the survey. Each participant was given a questionnaire to complete (Refer to Annexure B for the questionnaire).
The participants were required to fill in the questionnaire during or after their interaction with the newly developed system. The results of the survey are indicated in Table 6-16. The results for questions (a) to (e) relate to User Interface Satisfaction (UIS). The results for questions (f) to (j) relate to the capabilities and functionality of the new system.
Statements
Results of the Study in terms of the number of responses for each category
Strongly Agree Agree Neutral Disagree Strongly Disagree
a) The positioning of messages on the screen is consistent.
15 (or 54%)
4 (or 14%)
6 (or 21%)
3 (or 11%)
Nil
b) The prompts for input is clear. 13 (or 46%)
7 (or 25%)
4 (or 14%)
3 (or 11%)
1 (4%) c) The system gives error messages
that clearly tell me how to fix problems.
6 (or 21%)
10 (36%)
4 (or 14%)
5 (or 18%)
3 (or 11%) d) The organisation of information are
clearly laid out and are visually appealing.
16 (or 58%)
6 (or 21%)
6 (or 21%)
Nil Nil
e) The terminology used is clear. 12 (or 43%)
6 (or 22%)
2 (or 7%)
6 (or 21%)
2 (or 7%) f) The system is capable of effectively
creating a computerised portfolio of an academic.
8 (or 29%)
13 (or 46%)
4 (or 14%)
2 (or 7%)
1 (or 4%)
g) The system is able to fairly rank and select candidates who are due for:
158
The overall results indicate that most of the respondents felt that their expectations in terms of user interface and functionality of the system have been met. This can be deduced when the number of responses for the columns “Strongly Agree” and “Agree” are examined. The number of responses in these columns are the largest for most of the statements. This indicates that the
An award.
A promotion.
5 (or 18%)
13 (or 46%)
12 (or 43%)
6 ( or 21%)
7 (or 25%)
6 ( or 21%)
4 (or 14%)
2 (or 7%)
Nil
1 (or 4%) h) The system is able to monitor and
process the performance of an academic in terms of the core strategic goals such as:
Teaching and Supervision.
Research and Innovation.
Administration.
Writing and Publication.
Consultancy.
External engagement.
10 (or 36%)
13 (or 46%)
11 (or 39%)
12 (or 43%)
8 (or 29%)
9 (or 32%)
13 (46%)
6 (or 21%)
7 (or 25%)
8 (or 29%)
13 (or 46%)
11 (or 39%)
3 (11%)
5 (or 18%)
6 (or 21%)
7 (or 25%)
5 (or 18%)
8 (or 29%)
1 (4%)
3 (or 11%)
4 (or 14%)
1 (or 4%)
1 (or 4%)
Nil
1 (4%)
1 (or 4%)
Nil
Nil
1 (or 4%)
Nil
i) The system is capable of easily identifying the:
Strengths and
Weaknesses of an academic.
18 (or 64%)
20 (or 71%)
6 (or 22%)
5 (or 18%)
4 (or 14%)
2 (or 7%)
Nil
1 (or 4%)
Nil
Nil
j) With the new system, it is easier to input the data using linguistic values such as ‘very weak’, ‘weak’,
‘average’, ‘good’ and ‘very good’
rather than using precise values.
19 (or 68%)
7 (or 25%)
2
(or 7%) Nil Nil
Table 6-16: Results of Usability Study
159
majority of respondents “Strongly Agree” or “Agree” with most of the statements, indicating a high level of acceptability for the newly developed system. However, the results of the study also indicated that there were some areas that could be improved upon.
Close to 30% of the respondents felt that the system was not very effective in informing the user on how to fix errors and close to 28% of the respondents felt that the terminologies indicated on the screen were not very clear. The researcher took note of this when improvements of the system was implemented. The respondents however indicated a high level of satisfaction on the other aspects relating to user interface such as positioning of messages on the screen, layout of information and prompts for data inputs.
The rest of the questions focused on the capability and functionality of the system. Around 70%
of respondents “Strongly Agreed” or “Agreed” that the system was effective in evaluating academic staff in terms of the core strategic goals of the university such as Teaching and Learning, Research and Innovation, Administration, Writing and Publication, Consultancy and External Engagement.
One of the objectives (or functionality) of the system is to identify the strengths and weaknesses of an academic. For this objective, the results indicated a very high level of satisfaction among respondents as 86% felt that the system was successful in correctly identifying the strengths and 89% felt that the system was capable of correctly identifying the weaknesses of respondents. A fairly large percentage of respondents were satisfied on how the system was able to select candidates for an award (61%) and a promotion (67%). However, selecting candidates for an award or a promotion ranks the lowest when compared to the other capabilities of the system.
Those respondents who were successful in attaining an award or a promotion using the conventional evaluation system therefore “Strongly Disagreed” or “Disagreed” with the new system of evaluation. This is confirmed by the results in Figure 7-5 which indicated that 29% of respondents were evaluated using the conventional evaluation system when they applied for a promotion. These respondents are therefore reluctant to be subjected to a new system of evaluation which they are not familiar with.
160
The most important result of the study indicated that 93% of respondents preferred the new system because it was easier to input the data using linguistic values (such as ‘very weak’,
‘weak’, ‘average’, ‘good’ and ‘very good’) instead of using precise values. This was also deduced when the results of the open-ended question (question 10 of the questionnaire in Annexure B) was analysed. The analysis of the open-ended question also concluded that it was the first time that respondents were able to input linguistic values which resulted in the monitoring and processing of an academic’s performance in terms of the core strategic goals of the university, the identification and strengths and weaknesses as well as the ranking and selection of candidates for an award or a promotion.
It can be concluded that the results of the usability study indicates a high level of satisfaction amongst respondents in terms of user interface and capabilities of the system. The results of the usability study also assisted in identifying some minor weaknesses in the system, which the researcher improved on.