This section will go over the approach that will be used to ensure that the proposed solution addresses the problem as outlined in Section 1.3.1. The developed BI system must address all the research objectives for this study.
Therefore, for each critical success factor, the validation approach will demonstrate that the BI system addressed the applicable success factor.
The steps that will be followed to validate the proposed solution are depicted in Figure 2-11.
Figure 2-11 Validation method
The first stage of the validation framework comprises of implementing a BI system using the proposed framework. As a result, a case study will be carried out for a specific organisation.
The proposed framework will be used to create a BI system in this case study.
Each phase of the implementation will be addressed, and the results of each step will be presented. Once the BI system has been created, it will be distributed to the organisation's users so that they can begin using it for report and dashboard creation.
The next step of the validation process is to collect data, which will be done using a variety of methods. First, data on the number of reports generated will be gathered.
Ideally, the data should contain information such as a report ID and name, the creation date, the latest update date, the report dataset size, the average report refresh time, and the average number of times the report succeeds or fails when data is automatically updated.
Finally, data on user satisfaction will be gathered through a survey. The survey will be based on Doll and Torkzadeh's End-User Computing Satisfaction (EUCS) instrument [60].
Figure 2-12 displays a 12-item EUCS model with 5 components.
Figure 2-12 End-User Computing Satisfaction Model
Implement the
system Collect Data Analyse Results
EUCS
Content Accuracy Format Ease of Use Timeliness
A framework for implementing a scalable business intelligence system 44 The instrument determines user satisfaction with respect to several system components such as data content, data accuracy, data format, usability, and system timeliness or speed. This is accomplished by creating a questionnaire with multiple questions, each of which is closely related to one of the five components [60]. As seen below, the questionnaire is constructed with a five-point Likert-type scale.
• 1 = Almost never
• 2 = Some of the time
• 3 = About half of the time
• 4 = Most of the time
• 5 = Almost always
The questionnaire also includes two additional questions that probe the user's overall impression and level of satisfaction with the system. The questionnaire is broken into several sections, the first of which focuses on general information about the user and the second on how the user interacts with the system. The final section is the user review based on the EUCS model.
The first section includes the following four questions about the user:
1. What is your gender?
2. What is your highest qualification obtained?
3. How long have you been working for the organisation?
4. How many people report to you?
These four questions provide information about the different respondents to demonstrate the range of users who utilise the BI system. The first question asks about the respondent’s gender. The second question determines the respondent's level of education. The third question reveals how long the respondent has been with the specific organisation, and the fourth question reveals the respondent's seniority.
The second section focuses on determining whether the user utilises the system and what they use it for. Two questions are featured in the second section:
1. Have you ever used the BI system?
2. If yes, what do you use the BI system for?
The last section of the questionnaire is grouped according to the five components of the EUCS instrument and are given below. The questions are based on the original questions found in the EUCS model [60].
Content
1. Does the BI system provide the precise information you need?
2. Does the data provided by the BI system meet your needs?
3. Does the BI system allow you to create reports that seem to be exactly what you need?
4. Does the BI system provide sufficient information?
A framework for implementing a scalable business intelligence system 45 Accuracy
5. Are you satisfied with the accuracy provided by the BI system?
Format
6. Do you think the data provided by the BI system is presented in a useful format?
Ease of Use
7. Is the BI system user friendly?
8. Is the BI system easy to use?
Timeliness
9. Does the BI system get the information you need in time?
10. Does the BI system provide up-to-date information?
Overall
11. Are you satisfied with the BI system?
12. Is the BI system successful?
Following the collection of data, an analysis is carried out to obtain specific statistics such as central tendency and dispersion. Thus, the mean and standard deviation will all be calculated and analysed. For system quality, a correlation and regression analysis will also be done to see the impact of specific system variables on the overall performance.
The mean is calculated to determine the data's central tendency.
As demonstrated in Equation 2-1, the mean is calculated by taking the sum of the data points and dividing it by the number of data points that were summed together.
𝑋̅ = ∑ 𝑥𝑖 𝑛
Equation 2-1 Formula for calculating the mean
The standard deviation and coefficient of variation (CV) are used to examine data dispersion and, more specifically, to determine how well the mean describes the data points.
Equation 2-2 is used to determine the standard deviation, where X represents the mean, xi
represents data point i and N indicates the number of data points.
𝑆𝐷 = √∑(𝑥𝑖 − 𝑋̅) 𝑁
Equation 2-2 Formula for standard deviation
A framework for implementing a scalable business intelligence system 46 The coefficient of variation (CV) is a measure of relative variability, defined as the standard deviation divided by the mean. Using the CV helps comparing the overall precision of two analytical methods easier. The CV is a more accurate comparison than the standard deviation because the standard deviation often increases with analyte concentration.
Furthermore, apart from system usage, the questionnaire is organised into areas that are closely related to a critical success criterion. Question 1 through 6 describe the quality of information. System quality is addressed in questions 7 through 10, while overall user satisfaction is described in questions 11 and 12.
As a result, a 3.5 target of favourable user ratings will be necessary to ensure that the BI system addresses the three important factors. A rating above 3 indicates a favourable user rating.
Finally, to ensure that the system is being used, the number of reports generated, and the amount of data used by the reports will be examined. The validation framework will demonstrate that the BI system addresses the aspect of system usage (which is another success component of the BI system) by analysing this data.