DeLone and McLean Model of Academic Information System Success

(1)

92

DeLone and McLean Model of Academic Information System Success

UMAROH Sofia¹, BARMAWI Mira Musrini¹

1Department of Information System, ITENAS, Institut Teknologi Nasional Bandung, Indonesia

Abstract

Information system (IS) is crucial in today’s educational institution. Academic information systems that provide reliable and well-presented information are essential for the university to improve strategic performance in management and decision making. To achieve educational-related goals, University makes investments in IS expecting positive impacts but often lacks post-implementation review. Thus, evaluation of IS “success” is an important aspect to ensure successful service delivery. This is indicated by the effective use of the system and the positive influence it has on their customers This is indicated by the effective use of the system and the positive impact on their customers. This study addresses factors that influence the successful implementation of the Academic Information System (AIS) using IS success models. The structural model has been tested using an empirical approach based on data collected from 187 active users of AIS. This study found compared to other quality dimensions, information quality has the greatest influence on user satisfaction. However, perceived system quality is not a significant predictor of system use and user satisfaction

Keywords: success model, net benefit, structural equation modelling.

Received: 13 December 2020 To cite this article:

Umaroh, Sofia. Barmawi, M. M., “DeLone and McLean Model of Academic Information System Success”, in Electrotehnica, Electronica, Automatica (EEA), 2021, vol. 69, no. 2, pp. 92-101, ISSN 1582-5175.

1. Introduction

IT (information technology) has become a significant influence on improving organizational performance and service delivery in various sectors, such as education, business, health, finance, and tourism [1], [2].

In the field of education, technological advances have changed higher education institutions (HEIs) in service delivery, especially in terms of learning, research, and administrative services [2].

This led the HEIs to improve the performance of the institution from both academic and managerial perspectives. Managerial performance of HEIs showed by how universities can satisfy their customers (students, lecturers, and other partners) by using IT in service delivery. consequently, IT institutional alignment on both academic and managerial performance is a major concern for HEIs.

This challenge leads HEIs to consider the effective IT integration of services within universities. HEIs must improve their decision-making process, optimize their resource by integrating and analyse all available information, and provide service quality [3].

One of the IT strategies is to develop computer- based information systems as communication, data, and information management. Information systems developed with IT support can be the main competitive differentiator of an organization. Effective information systems can direct individuals or organizations to strategic decision-making.

This encourages HEIs to consider developing appropriate information systems to help their academic, management, and administration to survive and compete in this global competition. Real-time information systems that provide reliable information can improve strategic performance in management in HEIs and its decision-making [3].

Therefore, the implementation of an effective Academic Information System (AIS) is a success factor of HEIs. Development and implementation of high-quality information systems require expensive IT investments.

However, a major investment in IT is expected to result in business value creation, but organizations do not always get the related-positive value as expected [2].

Besides, the business benefits of implementing information systems are difficult to identify at the beginning. This is because some of the business benefits of information systems are intangible which do not directly lead to an identifiable increase in performance.

Besides, investments in IS expecting positive impacts but often lack of post-implementation review.

Information systems often refer to the relationship between humans, algorithmic processes, data, and technology. It means the success of information systems does not only focus on the technology-related used but also the way users interact with the systems in supporting their business processes [4].

Information systems that do not meet the requirements can increase the level of user dissatisfaction with the system, this can lead to system

(2)

failure. In other words, the critical success factor can be seen from the background factor such as user satisfaction, user behaviour, user participation, and also conceptual factor includes quality, system, and management [5].

Therefore, institutions need to evaluate the implementation of their information systems so that the factors that influence the success and failure of the implementation can be identified.

In previous studies, many researchers have evaluated the successes and failures related to the implementation of information systems. There are several factors of information system failure includes user, use behaviour, user satisfaction, attitudes, and levels of expectations, organizational management, infrastructure facilities, and usage patterns that have been identified [5]. Thus, information system quality is a critical aspect of competitive advantages for the organization [6].

However, successful information systems measurement is not enough only to examine the factors, but it is necessary to analyse the relationship between these factors to obtain more comprehensive results [7].

IS success model proposed by DeLone and McLean is a popular model to evaluate the extent to which the implementation of information systems is successful by assessing the relationship of each influencing factor.

There are six measures of the IS success used in this model, including the quality dimension consisting of system quality, information and services, system use, and their impact on individuals and organizations [8].

D&M success models have been evaluated to measure the success of information systems both empirically and through meta-analysis [1], [7], [17]–

[19], [9]–[16].

The results suggest that information systems success can be evaluated by measuring the relationship between quality dimensions and user usage and satisfaction, and its relationship to individual and organizational benefits [20].

Previous research on evaluating the D&M success model has shown mixed results because many factors affect the effectiveness of the system, such as organization, environment, and the people involved.[8].

In addition, as a multidimensional concept, D&M success models can be evaluated at various levels of the organization. Therefore, it is become essential to understand what these change that can impact the results of IS Success evaluation. The main purpose of this research are:

(1) to measure the system success on AIS of HEIs in Institut Teknologi Nasional Bandung;

(2) to empirically evaluate a D&M success model with the context, environment, and specific characteristics of the system involved.

2. Materials and Methods 2.1. D&M Success Model

In this study, we adopted the updated D&M success model to evaluate the success of the information system. In 2003, DeLone and McLean proposed six components or variables that affect the success of information systems called the Information System

Success Model which includes system quality (SQ), information quality (IQ), service quality (SvQ), system use (SU), user satisfaction (US), and net benefit(NB) [21].

Figure 1 shows the D&M success model which explains that the Quality dimensions collectively or separately impact IS use and user satisfaction.

Figure 1. The updated version of the D&M Model (2003) Positive experiences with Usage will lead to greater user satisfaction as well as Increased User satisfaction will lead to increased intention to use and thus usage of the system. As a result of Usage and User satisfaction, certain Net Benefits will arise.

The updated D&M success model as depicted in Figure 1 does not only measure the relationship between variables independently but also as a whole affects each other. The relationship between each dimension in this model is shown by testing the hypothesis that has been proven in previous studies. A quality system supported by quality information will encourage users to use the system. When the user perceives the benefits of the system through its quality and the content presented in the information system, it will encourage intensive use based on the need to improve performance or complete work more quickly.

Intensive use is supported by satisfaction in using the system. Since the more users experience the benefits of the system directly, the more their performance increases. Increased individual performance can lead to improved organizational performance, which is known as net benefits.

2.4. Hypothesis

According to the D&M success model, this study was conducted to test the hypothesis as depicted in Figure 2 of this paper. 10 hypotheses represent a positive relationship between the dimensions of the model.

Figure 2. IS Success Model to be Tested

(3)

In this study, the causal relationship between variables as shown in Figure 2 was investigated.

Research related to the validation of the D&M model has shown mixed results. The results of evaluating the relationship between the dimensions that make up the success of IS show inconsistent findings. This is due to the application of the D&M model which is not uniform considering that the information system is multidimensional and has different characteristics [22].

The results of tests that have been carried out on the impact of the quality dimensions on system use and user satisfaction are proven to have a positive effect, but some are not. Several studies in the field found that system quality has a positive relationship with user satisfaction and system use [7], [14], [23], [24].

However, other results found the relationship between these dimensions does not indicate a positive impact [11], [13], [25], [26].

This is due to the determinants of users in using the system not only because of its simplicity but because it has become a necessity. Other findings indicate that users perceive the quality of the system less because they think the system has not been implemented optimally [11].

Meanwhile, the relationship between system quality and user satisfaction shows mostly consistent results with the model, where it is found the positive relationship between dimensions [9], [11], [16]–[18], [20]–[22], [27], [34].

A good quality system represented by the quality of its information, the system itself, and the services provided by the system can encourage user satisfaction and intensity of the system use [8]. I

n line with this study, it was found that the quality dimensions of the model are shown to significantly influence user satisfaction and system usage [13][9].

Thus, the following hypotheses:

— H1: System Quality (SQ) has a positive association with System Use (U);

— H2: System Quality (SQ) has a positive association with User Satisfaction (US);

— H3: Information Quality (IQ) has a positive association with System Use (U);

— H4: Information Quality (IQ) has a positive association with User Satisfaction (US);

— H5: Service Quality (SvQ) has a positive association with System Use (U);

— H6: Service Quality (SvQ) has a positive association with User Satisfaction (US).

User satisfaction with current usage can be measured by the user's dependence on the system, the frequency and duration of use, the number of applications and tasks, and the intention to use it [49].

User satisfaction increases when users use information systems with high credibility, information generated and good services have an indirect effect on increasing usage [10].

However, if a user experienced inappropriate while using the system, it will directly reduce user satisfaction.

User satisfaction as a significant predictor of actual usage and the opposite has been confirmed by many researchers [10], [13], [17], [18], [21], [27], [32].

Accordingly, we propose the following hypotheses:

— H7: System Use (U) has a positive association with User Satisfaction (US);

— H8: User Satisfaction (US) has a positive association with System Use (U).

Net benefits indicate the contribution of information systems through user satisfaction and positive system use to the success of individuals and organizations [8].

System use and user satisfaction dimension are closely related [21].

According to the D&M success model, positive experiences in using a system led to greater user satisfaction. It means certain net benefits will occur depend on usefulness and user satisfaction received. the relationship and correlation between system use and net profit has been extensively confirmed by previous studies [7], [10], [11], [13], [16], [30].

Related research found there is a positive effect of user satisfaction (US) and system use (U) on net benefit in the university through Academic Information System implementation [11].

It is relevant with other study obtained the same results that the user satisfaction variable has a positive effect on the net benefit variable, which indicates a change in the user satisfaction variable will affect the net benefit variable [7], [9], [12], [16], [25].

Therefore, the rest hypothesis is as follows.

— H9: System Use (U) positively influences the Net Benefit (NB);

— H10: User Satisfaction (US) positively influences the Net Benefit (NB).

2.5. Methods

This research is a case study research, which focuses on certain cases using individuals or groups as study material. In this study, the research subjects were private universities in Bandung. Data collection was carried out through online surveys of active users of Academic information systems in the university environment.

The target population consists of students, lecturers, and employees who are members of a system spread over 3 faculties. The variables in this study were divided into two categories, namely the independent variable and the dependent variable or the independent variable and the dependent variable. The independent variables include system quality (SQ), information quality (IQ,) and service quality (SvQ). While the dependent variable includes use (U), user satisfaction (US), and net benefits (NB). Each variable is measured by related indicators by adopting previous research which is shown in Table 1.

Table 1. Indicators of each Construct

Construct Indicators References

System Quality (SQ)

System Flexibility, Error Recovery, Ease of use, Language, Time to Respond

[8], [23], [31]

Information Quality (IQ)

Completeness, Precision, Accuracy, Reliability, Currency, Format of Output, Understandability

[8], [23], [31]–[33]

(4)

Construct Indicators References Service Quality

(SvQ) Assurance, empathy,

responsiveness [21]

Use (U) Frequency of Use,

Nature of Use [21], [23]

User Satisfaction (US)

Information satisfaction, overall

satisfaction [21]

Net Benefit (NB)

Usefulness, Productivity,

Effectiveness, Speed of Accomplishing Task

[14], [21], [23], [34], [35]

Table 1 describes that the measurement items for each construct (latent and manifest variables) were obtained following the literature review of the related research. The latent variables in this study represent the constructs of the D&M success model which consists of system quality (SQ), information quality (IQ), service quality (SvQ), usage (U), user satisfaction (US), and net benefits (NB). The construct of the model is measured using 41 indicators as the basis of the instrument items.

Likert scale is used for each measurement item in the questionnaire which consists of seven points ranging from strongly disagree (1) to strongly agree (7). The validity of each question item was tested using convergent and discriminant validity tests.

Online survey data were collected from students and lecturers as active users of the AIS at the university. This information system was selected to test in this research because SIKAD is an integrated academic system that provides several academic services for lecturers, students, and academic staff. A total of 187 responses were collected through an online survey. The validity test results 39 indicators meet the validity requirements of a total of 41 indicators. Reliability testing was carried out to measure the extent of internal consistency of the instruments in this study. The test results obtained that all constructs have good reliability with the average value of 0,955 of CA and 0,965 of CR. Furthermore, PLS- SEM was used to analyse and test the causal correlation between construct variables in the D&M success model.

3. Results and Discussion

In this study, the information system success model is formed by 6 constructs and 41 indicators spread over each construct. 10 relationships will be tested using PLS- SEM. There are two types of structural equation models, covariance and component-based SEM which is popularly known as Partial Least Square (PLS). PLS-SEM aims to predict the effect and explain the theoretical relationship between two variables [36]. Model evaluation with PLS-SEM is done by testing the measurement model to determine the extent to which the measured variable represents the construct, then the structural model is to measure how the constructs are related to each other.

At the measurement model stage, the construct validity and instrument reliability were tested. If the model meets the criteria at this stage, then testing the structural model is carried out. Structural models are used to evaluate the relationship between constructs or latent variables using the R2 test and the t-test.

SmartPLS version 3.3 is used to test the Measurement and Structural model.

3.1. Respondents Demographic

Online questionnaires distributed in the Faculty of Industrial Technology, covering 6 study programs. The results of the distribution of the questionnaires are presented in Table 2.

Table 2. Respondent Demographics

Sample Demographic Frequency Percent

Gender

Female 53 28%

Male 134 72%

Total 187 100%

Position

Student 171 91%

Lecturer 16 9%

Total 187 100%

Age

17 – 21 146 78%

22 - 26 26 14%

26 - 35 5 3%

36 - 45 5 3%

> 46 5 3%

Total 187 100%

Department

Electrical Engineering 18 10%

Mechanical Engineering 12 6%

Industrial Engineering 56 30%

Chemical Engineering 11 6%

Informatics 51 27%

Information System 39 21%

Total 187 100%

Table 2 shows 187 respondents who had filled the survey as active users of SIKAD, including students and lecturers. The distribution of respondents based on gender obtained 72 % men and 28 % women. Based on the study program, respondent distributed as follow:

Electrical Engineering 10 %; Mechanical Engineering 6 %;

Industrial Engineering 30 %; Chemical Engineering 6 %, from Informatics 27 % and Information Systems 21 %.

According to the results, it was found that more than 60 % of the respondents showed a positive response to the questionnaire items.

The response was spread over each construct consists of system quality 55.4 %, information quality 63.5 %, service quality 59.3 %, system using 56.4 %, user satisfaction 5,8 %, and net benefit 67.3 %. Meanwhile, the negative response was obtained by 20.9 % which is spread over each construct. The highest percentage of negative responses was service quality at 27 %, followed by information quality at 21.5 %, service quality, and user satisfaction at 21 %, system use at 20.5 %, and finally net benefits at 14 %. The neutral response of 19%

was obtained from 23 % of system use as the largest percentage, followed by user satisfaction at 21 %, service quality and benefits around 19 %, service quality at 17 %, and information quality at 15 %. Overall, the responses obtained from the distribution of questionnaire data in this study showed a rather positive response to each model construct.

(5)

3.2. Measurement Model (Outer Model)

According to Hair, there are 4 steps in the measurement model, includes [36]:

— Step 1: Indicator loadings, to examine the extent to which the construct explains the variance of its indicator, a good loading value must be greater than 50 %.

— Step 2: Construct Reliability, to determine the reliability of the internal consistency of each construct.

— Step 3: Convergent Validity, to measure the construct indicators converge, thus explaining the variance of items. It is assessed by evaluating the average of extracted variance (AVE) across all indicators related to a particular construction. The rule of thumb of acceptable AVE must be higher than or equal to 0.50. This level or higher indicates that the construct means 50 percent or more of the variance of the indicator.

— Step 4: Discriminant validity, to examine that one construction can be differentiated from another. It using the Fornell – Larcker criterion method and the correlation method Henseler et al. [37] hetero-trait-mono-trait ratio (HTMT).

3.2.1. Indicator Loading

The outer loading of each Indicator was analysed it confirms the measurement items (indicators) indicate acceptable item reliability. Based on Hair (2018), outer loading should be above 0.7. According to the result, we found that all the items have met the criteria which is all its values are above 0.7.

3.2.2. Construct Reliability

This assessment aims to determine each construct’s internal consistency reliability. The reliability exists pressed by the extent to which a variable is consistent about what it wants to measure. Cronbach's alpha is used in this measurement to examine the lower limit of the reliability value of a construct, while composite reliability measures the real reliability value of a construct. The value of alpha or composite reliability is greater than 0.7 although the value of 0.6 is considered reliable [37].

The reliability test results as shown in Table 3 indicated that the Cronbach alpha and the composite reliability value for each construct is above 0.70.

Table 3. Reliability test results Variables Composite

Reliability (CR ≥ 0.7)

Cronbach’s Alpha (α ≥

0.7) AVE (≥ 0.5)

SQ 0,962 0,956 0,717

IQ 0,966 0,959 0,803

SvQ 0,963 0,954 0,814

U 0,965 0,954 0,845

US 0,961 0,939 0,891

NB 0,971 0,966 0,807

This is clear that the set of indicators has well explained its latent construct and all constructs possess adequate reliability.

3.2.3. Convergent Validity

The next step is to examine convergent validity to determine which certain indicators have measured the construct appropriately. According to table 3, we can see that the AVE value of each construct is above 0.5.

Thus, The Items of construct measures are highly correlated.

3.2.4. Discriminant Validity

The discriminant validity test was carried out by examining the cross-loading value on each indicator.

When the construct correlation value with its indicator shows a greater value than other constructs, it means that the latent constructs have predicted the indicator of their block better than others. Based on the cross- loading value in the model, two items fail to meet the criteria and were deleted from the model. These are Information satisfaction (US11) from the information quality construct and usefulness from the net benefit (NB1) construct.

After re-estimate by eliminating the NB1 and US11 indicators, the construct correlation with the measuring indicator is greater than the other construct measures.

The re-estimation results show that each indicator has the highest load value on the constructs associated with it compared to other constructs. This shows that the latent constructs have predicted indicators in the related construct better than other constructs.

The next step is the Fornell–Larcker criterion method. Discriminant validity is considered good when the square root value of AVE for each construct is greater than the correlation value between constructs and other constructs in the model. As presented in Table4, the diagonal value (AVE) must be greater than the other values in the same construct.

Table 4. Fornell Larcker Criterium

Var SQ IQ SvQ U US NB

SQ 0,847 IQ 0,845 0,896 SvQ 0,742 0,741 0,902

U 0,715 0,743 0,758 0,919 US 0,746 0,785 0,762 0,8 0,944 NB 0,739 0,764 0,761 0,797 0,835 0,898 Referred to Table 4, the value with a bold number indicates the largest number in each construct relationship. This suggests that the latent construct has explained more variance in its item measure than with other constructs.

Discriminant validity also can be assess using Hetero- trait-Mono-trait Ratio (HTMT) by Hensler which is presented in Table 5.

(6)

Table 5. the HTMT Correlation Matrix

Var SQ IQ SvQ U US NB

SQ

IQ 0,882

SvQ 0,775 0,773

U 0,748 0,776 0,792 US 0,786 0,827 0,804 0,845 NB 0,768 0,793 0,791 0,830 0,876

The upper limit of the discriminant measurement ratio is 0.85, thus the distribution of the ratio value below 0.85 is considered a valid discriminant. However, values below 0.90 a construct also have good discriminant validity [37].

The HTMT results shown in Table 5 found two relationship constructs almost reach 0.9 but still acceptable. Based on the measurement model, the validity measurement results show that each indicator succeeds in explaining its construct well based on the cross-loading measurement and the comparison of the AVE value.

In addition, the results of the reliability measurement explain that the indicators and variables in the model can provide reliable measurement results as shown by the Cronbach Alpha and composite reliability results. Thus, the measurement of the construct in this model is valid and reliable to be tested in the structural model.

3.3. Structural Model (Inner Model)

At this stage, structural model analysis and hypothesis testing which represent the relationship between constructs are carried out. The structural model or often refer to inner model has been evaluated using three tests, includes R-square (R² test, the f- square (f²) test, and the path coefficient. Discriminant validity is considered good when the square root value of AVE for each construct is greater than the correlation value between constructs and other constructs in the model. The f² is used to assess the relative impact of a dependent variable on the independent variable. While the estimated path coefficients or t-values are used to examine the significant relationship between variables in the model. The R² scores show the extent to which a certain independent latent construct affects the dependent latent variable. The R² test result of 0.190 is considered weak; 0.333 has moderate, and 0.670 is considered to have a substantial effect [36]. Whereas the path coefficient shows the direct effect of the variables determined as causes on the variables determined as a result.

3.3.1. R² test

The R² test measures the effect of certain exogen construct on the endogen construct, whether it has a substantive effect.

Table 6 shows the R² value of endogenous constructs which consists of user satisfaction, system use, and net benefit.

Table 6. R-square Endogenous

Constructs

R-square R-square

Model 1 Model 1

U 0,652 0,652

US 0,742 0,742

NB 0,744 0,744

As we can see in Table 6, model 1 represented that the quality dimension of SIKAD is moderate to substantially explained system use variable of the SIKAD at 65 %. These quality dimension constructs together explained 74.2 % of the variance in user satisfaction which is considered substantial. A substantial effect of the variance in the net benefit of SIKAD was explained by user satisfaction and system use of SIKAD in both model 1 and model 2 at 74.4 %. In model 2, the quality dimensions of SIKAD are substantially explained 70.6 % of perceived use and 69 % of user satisfaction of SIKAD.

Based on the results of variance testing in both models, the exogenous variables in this study were proven to have explained moderately to substantially its endogenous construct.

3.3.2. F-square test

The f² test aims to assess the relative impact of an exogenous construct on the endogenous construct. F² test value has 3 criteria, consisting of a value between 0.020 and 0.150 considered small, moderate criteria in between 0.150 and 0.350, and if exceeds 0.350 considered large. F-square values are retrieved using SmartPLS software presented in figure 3.

Figure 3. F-square values

According to figure 3, the effect size of each relationship dimension is considered small to medium represented by the red bar in figure 3.

The path f information quality to system use and user satisfaction shows a small effect size. System

(7)

quality also shows a small effect size on system use and user satisfaction. The service quality path also shows a small effect size on user satisfaction. This suggests that the predictor variables of information quality and service quality have a small to medium effect on the measured variable (U, US, and NB). The system quality dimension found does not affect system use and user satisfaction, this indicates that the system quality is not a reason for the user to using the system and perceived user satisfaction of the system. However, surprisingly we found that the effect size of user satisfaction on the net benefit is large, this indicates that the expected net benefit will be achieved only if the user is satisfied with the system.

3.3.3. Path Coefficient (β)

At this stage, the estimated path coefficients identifying the power relationship between independent variables and the dependent variable. Path coefficient evaluation is obtained by a bootstrapping method using SmartPLS.

Figure 4 shows the result of the path coefficient between constructs.

Figure 4. Structural model path coefficients

As can be seen in figure 4, positive original sample values of all relationships indicate a positive relationship between constructs. The quality of information on user satisfaction indicates that the information quality has a positive direct effect of 31 % on the system user variable (β = 0.43). However, the direct effect on the user satisfaction variable was found to be smaller at 28 %. Service quality path coefficients result in a positive influence on the system use construct at 42 % directly, while the relationship with user satisfaction is considered smaller at 24 %. Surprisingly, we found that the direct effect of system quality on system use and user satisfaction is only 9 % to 10 %. This indicates the contribution of system quality towards user satisfaction system use is very small. This indicates that the user of SIKAD might ignore the system quality.

In other words, system quality is not a critical reason for users when using SIKAD.

Hypothesis testing was performed by examining the corresponding path coefficient values showing predictive and significant signs at the level of P <0.050 as a supported hypothesis. The T-statistic value shows whether a construct relationship has a significant effect on the criteria for a T-statistic value greater than the t- table. The output results with the bootstrapping method are presented in Table 7.

Table 7. Result of Path Analysis and Hypothesis Testing H Path β Std.

Dev T-Stat. Support F² Effect size H1 SQ -> U 0,13 0,1 ^*1,29 no 0,01 - H2 SQ -> US 0,09 0,09 ^*0,99 no 0,01 - H3 IQ -> U 0,31 0,09 ^**3,5 yes 0,07 small H4 IQ -> US 0,28 0,08 ^***3,33 yes 0,07 small H5 SvQ -> U 0,42 0,1 ^***4,35 yes 0,21 medium H6 SvQ -> US 0,21 0,08 ^***2,52 yes 0,06 small H7 U -> US 0,37 0,081 ^****4,537 yes 0,18 medium H8 US -> U 0,42 0,13 ^****3,36 yes 0,18 medium H9 U -> NB 0,36 0,07 ^****5,25 yes 0,18 medium H10 US -> NB 0,548 0,075 ^****7,309 yes 0,42 large Legend: ****significant at p<0.001; ***significant at p<0.010;

**significant at p<0.050; *not significant;

According to the result of path analysis and hypothesis testing presented in table 7, most of the hypotheses of this study are supported.

The relation of SQ -> U obtained a path coefficient of 1.29 which indicates that the quality of the system is not proven to have a significant effect on usage (t-statistic <t-table 1.97).

Likewise, SQ -> US was not proven to have a significant effect as indicated by the t-statistic value of 0.99 with a path coefficient of 0.09 (t-statistic

<t-table 1.97).

Therefore, according to the data results, the hypotheses H1 and H2 are rejected. There is mixed support for this relationship at the level of individual analysis in the literature where some studies show support and others were not [8].

Another study found that perceived system quality did not guarantee system use, this means system quality was not a major consideration for use or discontinuation which leads to system quality perhaps not a good predictor of use. Since SIKAD is a mandatory system, the evaluation of the e-Government success in previous research is relevant that system quality does not have a significant effect on system use. This can be caused by users in the era of the internet have higher computer literacy skills and internet experiences. In addition, citizens do not place the quality of the system or the ease of use of the e-Government system as important things in determining the use of the system [25].

(8)

Thus, users seem more concerned about the service provided by the IT department and the content quality than system quality. According to the results of the qualitative data collection in this study, most of the respondents complained about system quality perceived by the user. This quality dimension including response time, error recovery, and convenience in using and accessing the system.

However, the responses from users also indicate that SIKAD has met the user needs toward SIKAD, especially in managing grades, academic information, and managing academic schedules while they ignore the quality of the system itself. This is caused by the quality of the system is not a reason for users to use and get benefits from SIKAD. Another reason for that could be that SIKAD is a mandatory system to use and this is relevant to the finding that uses, either perceived or actual, is only relevant if such use is not mandatory [35].

However, good system quality despite mandatory use can lead to satisfaction and positive use. So that this can be a consideration for the university to improve its system quality. This findings are relevant with previous study [11], [13], [26].

Hypotheses H3 and H4 were supported. This implies that the quality of information of SIKAD is perceived by users as increasing system usage and user satisfaction with the system. This confirms that the quality of information is a determining factor for user-perceived satisfaction and positive use of the system. For instance, aspects of information quality such as clear information, constantly updated content, valid and easily understood information can contribute to user satisfaction in enjoying the information available.

Besides, the use of the system becomes more positive because users feel helped by the system, especially in completing certain tasks. These findings are relevant with related study [11], [13], [33].

For service quality, hypotheses H5 and H6 were supported. Service quality has a significant effect on both system use and user satisfaction. This indicates that the services provided by the IT department to SIKAD users have led to satisfaction for its users. For example, aspects of service quality such as quick response and empathy given by IT staff whenever a user complains can provide a feeling of comfort for the user in using the system. The better the services provided by IS personnel of SIKAD to users, especially when an error occurs, it can lead to positive use from the user perspective. These finding was relevant with previous studies [7], [11], [14], [16], [26].

Furthermore, positive use by users due to good service support can encourage users to get satisfaction while using the system. Related studies also confirmed this finding that the quality of services provided by IS personnel can positively affect perceived satisfaction [1], [7], [9]–[12], [14].

Furthermore, the use of the system is proven to have a significant effect on user satisfaction and perceived net benefits so that H7 and H9 are supported with a medium of the size effect. The path coefficient on the relationship between use and net benefits is 0.36 indicating that 36% of system use has a direct effect on net benefits. This finding is relevant to related research which indicates that time and frequency of system use

do not necessarily lead to higher task performance, but it is how the system is used that can determine task performance [38].

User satisfaction as a significant determinant of net benefits has a direct effect of 37 %.

These findings demonstrate the positive effects of system use on user satisfaction. This indicates that system use and user satisfaction dimension are important precedents of net benefit [15]. Hypotheses H8 and H10 were also supported. The results of hypothesis testing show that the evaluation of the D&M success model explains that positive use intention and user satisfaction are proven to produce the expected net benefits. Net benefits are defined based on the usefulness of the system, the ability of the system to increase user productivity, efficiency, and effectiveness in completing tasks. This is indicated by the response obtained that the net benefits have generally been obtained by users in the form of completed tasks, searching for the necessary information, and ease in the academic process. In other words, high user satisfaction will affect system use, and positive use toward the system will have an impact on net benefits so that the benefits received by users will be better. Related to the relationship between the dimensions of use and user satisfaction, the process of use must precede user satisfaction because the frequent use of a good quality system can encourage satisfaction, but the more users feel satisfied with the quality of the system, information and services can also encourage continued use.

continuously. Then these two dimensions have a reciprocal relationship.

5. Conclusion

Evaluation on academic information system success in a university using D&M Model the This research was conducted to test the success of information system implementation in the university using the updated D&M Model. In this study, we propose and validate a comprehensive and multidimensional success model of academic information systems, which considers six measures of success consist of quality dimension, user satisfaction, system use, and net benefits.

Overall, these findings support the model. This is indicated by the hypothesis that the relationship between the six construct variables is significantly supported by the data except for the relationship of system quality to usage and user satisfaction. The findings of this study provide some important implications for research and practice of educational institutions in which the quality of the system does not have a significant direct effect on user satisfaction (β = 0.09) and system use (β = 0.13). This indicates inconsistencies with most previous studies. SIKAD with a quality system is needed but not sufficient to provide benefits. The quality of the system only ensures that SIKAD is running normally. Compared to the system quality, the information quality has a greater influence on user satisfaction (β = 0.28) and system use (β = 0.31).

Users are starting to perceive IS as a part of their work life. Thus, the operation of the system is no longer a critical issue. however, users perceive the quality of the information provided by the system as one of the reasons for users to get satisfaction.

(9)

The results also indicate that user satisfaction and system use were found to be strong predictors of net benefit. The impact of the dimensions of user satisfaction is quite large on net benefits (R² = 0.74).

This indicates that the greater the user's satisfaction with SIKAD, the greater the net benefits that users get.

The results of this test show that the quality dimensions produced by SIKAD have an impact on the dimensions of information quality and service quality with a moderate effect on system use and user satisfaction. Meanwhile, the biggest influence of the predictor variable on the net benefit is user satisfaction.

Based on the results of this study, since the effect size of the quality dimension is still considered small, the results suggest that the University should focus on improving the quality dimension. Universities should pay attention to the information contained in the system, the services provided by the relevant departments, and the quality of the system especially in terms of improved response times, error handling, better user experience, and a more effective interface. Thus, it can lead to user satisfaction and positive use of the system so that net benefits can be achieved.

This paper has limitations. Firstly, this research is based on field studies of mandatory information systems in a particular organizational context, which is an educational institution. Thus, the findings of this study may be specific to this organizational context. Then in developing research instruments, it must be done more carefully, especially in determining the right indicators from the perspective of both users and organizations. So that the results of the research can have implications for both individuals and organizations level.

Furthermore, the sample should be collected representing all active users in the university to confirm, evaluate, or refine the model. In this study, the sample was taken from one faculty.

6. Bibliographic References

[1] D. Al-Fraihat, M. Joy, R. Masa’deh, and J. Sinclair,

“Evaluating E-learning systems success: An empirical study,” Comput. Human Behav., vol. 102, no. August 2019, pp. 67–86, 2020, doi:

10.1016/j.chb.2019.08.004.

[2] J. C. Byungura, Improving IT Integration for Higher Education Institutional Institutional Performance, no. 19. 2019.

[3] C. Guerrero and J. E. Sierra, “Impact of the Implementation of a New Information System in the Management of Higher Education Institutions,” Int.

J. Appl. Eng. Res., vol. 13, no. 5, pp. 2523–2532,

2018, [Online]. Available:

http://www.ripublication.com.

[4] D. M. Kroenke, Experiencing MIS, 8 edition. Prentice Hall, 2019.

[5] D. N. T. Gunawardhana, “Classification of Failure Factors in Information Systems,” Int. J. Innov. Educ.

Res., vol. 3, no. November, pp. 201–211, 2015, [Online]. Available: IS.

[6] D. Russo, P. Ciancarini, T. Falasconi, and M. Tomasi,

“A Meta-Model for Information Systems Quality,”

ACM Trans. Manag. Inf. Syst., vol. 9, no. 3, pp. 1–

38, 2018, doi: 10.1145/3230713.

[7] L. H. Trihandayani, I. Aknuranda, and Y. T. Mursityo,

“Penerapan Model Kesuksesan Delone dan Mclean

pada Website Fakultas Ilmu Komputer (FILKOM ) Universitas Brawijaya,” J. Pengemb. Teknol. Inf.

dan Ilmu Komput. Univ. Brawijaya, vol. 2, no. 12, 2018.

[8] S. Petter, W. DeLone, and E. McLean, “Measuring information systems success: Model, dimensions, measures, and interrelationships,” Eur. J. Inf. Syst., vol. 17, no. 3, pp. 236–263, 2008, doi:

10.1057/ejis.2008.15.

[9] H. Irawan and I. Syah, “Evaluation of implementation of enterprise resource planning information system with DeLone and McLean model approach,” 2017 5th Int. Conf. Inf. Commun. Technol. ICoIC7 2017, vol.

0, no. c, 2017, doi: 10.1109/ICoICT.2017.8074721.

[10] F. Sapty Rahayu, R. Apriliyanto, and Y. Sigit Purnomo Wuryo Putro, “Analisis Kesuksesan Sistem Informasi Kemahasiswaan (SIKMA) dengan Pendekatan Model DeLone dan McLean,” Indones. J.

Inf. Syst., vol. 1, no. 1, pp. 34–46, 2018, doi:

10.24002/ijis.v1i1.1704.

[11] S. Robo, D. B. Setyohadi, and A. J. Santoso, “An Identification of Success of Academic System Application Using Delone and McLean Design,” Int.

Conf. Inf. Commun. Technol., pp. 827–832, 2018.

[12] J. M. Hudin, Y. Farlina, R. A. Saputra, A. Gunawan, D. Pribadi, and D. Riana, “Measuring Quality of Information System Through Delone Mclean Model in Online Information System of New Student Registration (SISFO PPDB),” 2018 6th Int. Conf.

Cyber IT Serv. Manag. CITSM 2018, no. Citsm, pp. 1–

6, 2019, doi: 10.1109/CITSM.2018.8674310.

[13] Veronica and A. D. Suryawan, “User Satisfaction Survey of Performance Management Dashboard Using Delone McLean Method: A Case Study,” Proc.

2019 Int. Conf. Inf. Manag. Technol. ICIMTech 2019, vol. 1, no. August, pp. 542–547, 2019, doi:

10.1109/ICIMTech.2019.8843761.

[14] R. A. Zuama, J. M. Hudin, D. Puspitasari, E. H.

Hermaliani, and D. Riana, “Quality dimensions of Delone-McLean model to measure students’

accounting computer satisfaction: An empirical test on accounting system information,” 2017 5th Int.

Conf. Cyber IT Serv. Manag. CITSM 2017, 2017, doi:

10.1109/CITSM.2017.8089318.

[15] C. Tam and T. Oliveira, “Understanding the impact of m-banking on individual performance: DeLone &

McLean and TTF perspective,” Comput. Human Behav., vol. 61, pp. 233–244, 2016, doi:

10.1016/j.chb.2016.03.016.

[16] A. H. Aldholay, O. Isaac, Z. Abdullah, and T.

Ramayah, “The role of transformational leadership as a mediating variable in DeLone and McLean information system success model: The context of online learning usage in Yemen,” Telemat.

Informatics, vol. 35, no. 5, pp. 1421–1437, 2018, doi: 10.1016/j.tele.2018.03.012.

[17] E. Krasniqi, D. Hyseni, and B. Cico, “An Evaluation of e-Education System in Kosovo based on DeLone and McLean IS Success Model,” in 2019 8th Mediterranean Conference on Embedded Computing, MECO 2019 - Proceedings, 2019, no.

June, pp. 1–4, doi: 10.1109/MECO.2019.8760053.

[18] P. N. Sharma, M. Sarstedt, G. Shmueli, K. H. Kim, and K. O. Thiele, “PLS-based model selection: The role of alternative explanations in information systems research,” J. Assoc. Inf. Syst., vol. 20, no.

4, pp. 346–397, 2019, doi: 10.17005/1.jais.00538.

[19] B. Prasetyo, F. Adnan, and S. A. K. Wardhani, “A

(10)

measurement framework for analyse the influence of service quality and website quality on user satisfaction (Case study: An IT service in Jember University),” Int. Conf. Electr. Eng. Comput. Sci.

Informatics, vol. 2018-Octob, no. 2000, pp. 56–61, 2018, doi: 10.1109/EECSI.2018.8752845.

[20] S. Petter, W. Delone, and E. R. Mclean, “Journal of the Association for Information Systems The Past , Present , and Future of ‘ IS Success ,’” J. Assoc. Inf.

Syst., vol. 13, no. May 2012, pp. 341–362, 2012.

[21] W. H. DeLone and E. R. McLean, “The DeLone and McLean model of information systems success: A ten-year update,” J. Manag. Inf. Syst., vol. 19, no.

4, pp. 9–30, 2003, doi:

10.1080/07421222.2003.11045748.

[22] A. Jeyaraj, “DeLone & McLean models of information system success: Critical meta-review and research directions,” Int. J. Inf. Manage., vol.

54, no. April, p. 102139, 2020, doi:

10.1016/j.ijinfomgt.2020.102139.

[23] J. Iivari, “An Empirical Test of the DeLone-McLean Model of Information System Success,” Data Base Adv. Inf. Syst., vol. 36, no. 2, pp. 8–27, 2005, doi:

10.1145/1066149.1066152.

[24] O. Isaac, A. Aldholay, Z. Abdullah, and T. Ramayah,

“Online learning usage within Yemeni higher education: The role of compatibility and task- technology fit as mediating variables in the IS success model,” Comput. Educ., vol. 136, no.

February, pp. 113–129, 2019, doi:

10.1016/j.compedu.2019.02.012.

[25] Y. S. Wang and Y. W. Liao, “Assessing eGovernment systems success: A validation of the DeLone and McLean model of information systems success,”

Gov. Inf. Q., vol. 25, no. 4, pp. 717–733, 2008, doi:

10.1016/j.giq.2007.06.002.

[26] H. Roky and Y. Al Meriouh, “Evaluation by Users of an Industrial Information System (XPPS) Based on the DeLone and McLean Model for IS Success,”

Procedia Econ. Financ., vol. 26, no. 0, pp. 903–913, 2015, doi: 10.1016/s2212-5671(15)00903-x.

[27] H. Jin-yuan, W. Xin-yi, and X. Shu-huan, “Evaluation of E-government Service Quality Based on SERVQUAL,” 2012 Int. Conf. Manag. Sci. Eng., pp.

1973–1978, 2012.

[28] A. Rai, S. Lang, and R. Welker, “Assessing the Validity of IS Success Models: An Empirical Test and Theoretical Analysis,” Inf. Syst. Res., vol. 13, pp.

50–69, 2002, doi: 10.1287/isre.13.1.50.96.

[29] U. Kulkarni, S. Ravindran, and R. Freeze, “A Knowledge Management Success Model: Theoretical Development and Empirical Validation,” J. Manag.

Inf. Syst., vol. 23, no. 3, pp. 309–347, 2007, doi:

10.2753/nus0742-1222230311.

[30] J. H. Wu and Y. M. Wang, “Measuring KMS success:

A respecification of the DeLone and McLean’s model,” Inf. Manag., vol. 43, no. 6, pp. 728–739, 2006, doi: 10.1016/j.im.2006.05.002.

[31] J. E. Bailey and S. W. Pearson, “Development of a Tool for Measuring and Analysing Computer User Satisfaction,” Manage. Sci., vol. 29, no. 5, pp. 530–

545, 1983, [Online]. Available:

http://www.jstor.org/stable/2631354.

[32] D. Sedera and G. Gable, “A factor and structural equation analysis of the enterprise systems success measurement model,” Twenty-Fifth Int. Conf. Inf.

Syst., pp. 449–464, 2004, [Online]. Available:

http://eprints.qut.edu.au/4733/.

[33] H. Liang, Y. Xue, and Z. Zhang, “Understanding online health information use: The case of people with physical disabilities,” J. Assoc. Inf. Syst., vol.

18, no. 6, pp. 433–460, 2017, doi:

10.17705/1jais.00461.

[34] F. D. Davis, “Perceived usefulness, perceived ease of use, and user acceptance of information technology,” MIS Q. Manag. Inf. Syst., vol. 13, no.

3, pp. 319–339, 1989, doi: 10.2307/249008.

[35] W. H. DeLone and E. R. McLean, “Information systems success: The quest for the dependent variable,” Uncertain Supply Chain Manag., vol. 3, no. 2, pp. 181–188, 1992, doi:

10.5267/j.uscm.2014.12.002.

[36] J. F. Hair, W. C. Black, B. J. Babin, R. E. Anderson, W. C. Black, and R. E. Anderson, Multivariate Data Analysis. 2018.

[37] J. Henseler, C. M. Ringle, and M. Sarstedt, “A new criterion for assessing discriminant validity in variance-based structural equation modelling,” J.

Acad. Mark. Sci., vol. 43, no. 1, pp. 115–135, 2014, doi: 10.1007/s11747-014-0403-8.

[38] H. Sun, R. T. Wright, and J. B. Thatcher, “Revisiting the impact of system use on task performance: an exploitative-explorative system use framework,” J.

Assoc. Inf. Syst., vol. 20, no. 4, pp. 398–433, 2019, doi: 10.17705/1.jais.00539.

Funding Sources

This work was financially supported by LP2M Institut Teknologi Nasional Bandung of Indonesia, through the Grant scheme of Penelitian Dosen Pemula Itenas (PDPI) 2020.

Authors’ Biographies

SOFIA UMAROH was born in Bandung, West Java (Indonesia), on November 5, 1990.

She graduated from Universitas Pendidikan Indonesia (UPI) Indonesia of Computer Science Education in 2012.

She received a master's degree from Institut Teknologi Bandung (ITB) of Informatics in 2016. She is now a lecturer in the Department of Informatics, Faculty of Industrial Technology at Institut Teknologi Nasional Bandung, Indonesia

Mira Musrini Barmawi was born in Bandung (Indonesia), on June 25, 1968.

She graduated from Institut Teknologi Bandung (ITB) of Mathematics in 1990. She received a master's degree from Institut Teknologi Bandung (ITB) of Informatics in 2006.

She is now a lecturer in the Department of Informatics, Faculty of Industrial Technology at Institut Teknologi Nasional Bandung, Indonesia

(11)