Measuring user satisfaction with data warehouses: an exploratory study
Lei-da Chen
a, Khalid S. Soliman
b,*, En Mao
c, Mark N. Frolick
c aNorthern Michigan University, Michigan, USAbHofstra University, Hempstead, New York, USA cThe University of Memphis, Tennessee, USA
Received 31 May 1998; received in revised form 15 November 1998; accepted 25 April 1999
Abstract
Data warehouses are users driven; that is, they allow end-users to be in control of the data. As user satisfaction is commonly acknowledged as the most useful measurement of system success, we identify the underlying factors of end-user satisfaction with data warehouses and develop an instrument to measure these factors. The study demonstrates that most of the items in classic end-user satisfaction measure are still valid in the data warehouse environment, and that end-user satisfaction with data warehouses depends heavily on the roles and performance of organizational information centers.# 2000 Elsevier Science B.V. All rights reserved.
Keywords:Data warehousing; End-user computing; Information center; User satisfaction
1. Introduction
Over the past 17 years, many surveys have been conducted to determine the critical issues facing information systems (IS) executives. Making effective use of data resources was always in the top-ten list [5,9,10,36]. The results re¯ect the high value of data resources and the importance of managing them effec-tively. In recent years, businesses have captured a wealth of data and the volume continues to grow at a phenomenal rate [19,31]. The movement from cen-tralized to decencen-tralized data processing and end-user computing (EUC) in the 1980s has enormously increased the organization's overall productivity [8,39]. At the same time, EUC resulted in a large
number of databases throughout the organization; these are often, however, stored in incompatible for-mats [7]. As a result, there is a need to control the storage, access, and processing of data. This will make more effective use of data. Consequently, data ware-housing becomes a viable option for many organiza-tions. Mayers [34] indicates that the environments that should consider data warehouses as a solution are those that have low-quality data, heavy analysis, real time needs, and massive volumes of data.
William Inmon de®ned a data warehouse as ``a subject-oriented, integrated, nonvolatile, time-variant collection of data organized to support management needs'' [6]. Its purpose, as stated in the de®nition, is to facilitate corporate decision-making. Data warehous-ing provides end-users with decision-makwarehous-ing tools that create a competitive advantage. Moreover, it positions the organization to meet the demands of *Corresponding author. Fax:1-516-9377314.
E-mail address: ksoliman@hotmail.com (K.S. Soliman).
electronic commerce [45]. Data warehouses differ from operational databases and management informa-tion systems (MIS): the ®rst supports daily business transactions, while the second supplies information to managers, in the form of periodic reports. A data warehouse collects information from multiple systems and stores it in a fashion that allows end-users to have faster, easier, and more ¯exible access to key informa-tion [16,44].
Data warehousing environment is a discovery-dri-ven process of extracting meaningful information, and, hopefully, knowledge, from a large store of detailed data. It serves as the infrastructure that col-lects data centrally and then supplies cleaner, more accurate, and detailed data for analysis [17]. System input consists of raw data. The sources of raw data are corporate transactional systems, archive systems, and any other relevant systems.
Users interact with the data warehouse through user friendly data mining tools. The Gartner Group de®ned data mining as ``the process of discovering meaningful new correlation, patterns, and trends by sifting through large amounts of data stored in repositories and by using pattern recognition technologies as well as statistical and mathematical techniques'' [27]. Data warehousing and data mining are highly inter-depen-dent. While data warehouses are the bases for the analysis performed by data mining, they depend on data mining to enhance their value. By adding such components to a data warehouse, companies are achieving payback of 10±70 times their original investment [40].
Although structured queries are allowed in data warehouses, their true usefulness lies in their ability to answer vague queries. In traditional database sys-tems, users, rather than syssys-tems, do the discovery; structured queries on speci®c subjects must be con-structed prior to the search. In other words, a database system can answer only those questions that users know how to ask. Users of data warehouses are often managerial end-users, with little or no programming and statistical expertise [3]. Discovery-driven data warehousing automates the analysis of data and helps in answering the questions the users do not know how to ask.
Vague questions can be asked to obtain a wide range of possibilities from the data warehousing system. Therefore, data warehouses often uncover previously
unknown and unexpected, but ultimately comprehen-sible information [35], as trends and patterns within massive volumes of data are often invisible to the human eye. However, the use of powerful hardware and software tools can unveil these trends and pat-terns.
Inmon et al. [23] stated that a data warehouse is user driven. It allows end-users to be in control of the data and to create the reports that they need freely. There-fore, data warehouses provide users with much greater ¯exibility in using data than traditional IS. As orga-nizations move from centralized data warehousing and mining, which require the support of specialists in a mainframe-centered environment, to client/server data warehousing and mining, which allow end-users to operate directly from their desktop computers, even greater ¯exibility and more possibilities are demon-strated.
This article discusses end-users' satisfaction with data warehouses. Using a survey of 42 business man-agers, the researchers conducted an exploratory factor analysis to establish the instrument. The ®ndings and implications of the study are discussed. The article will conclude with some potential research questions, which can be used to strike the development of a more general measurement for end-users' satisfaction with data warehouses.
2. Objectives
Academicians and practitioners have considered data warehousing to be a valuable decision support tool; however, little systematic research has been undertaken to measure its success in organizations. Much of the literature to date discusses its popularity [1], rapid growth [11], and the critical issues [41], but without an accurate measure of success, organizations cannot evaluate the success of their costly data ware-house projects. Therefore, the development of such a measurement should be of top priority for people interested in the data warehousing phenomenon.
emer-ging technology is dif®cult, our research focuses only on user satisfaction.
User satisfaction has been acknowledged as the most useful measurement of system success [20]. DeLone and McLean identi®ed three reasons why user satisfaction has been widely used as the single measure of IS success: (1) high degree of face validity, (2) development of reliable tools for measure, and (3) conceptual weakness and unavailability of other mea-sures.
3. Research methodology
Since 1980, much research has concentrated on ®nding a valid measure for user satisfaction. Among this research, Bailey and Pearson [4] developed a semantic differential instrument, with 39 items mea-suring overall computer user satisfaction. This was later revised by Ives et. al. [24] to a 13-item instru-ment. Based on the instrument of Ives et al., Doll and Torkzadeh [13] developed a 12-item instrument, which consisted of ®ve factors of end-user computing satisfaction: information content, accuracy, format, ease of use, and timeliness. It was later con®rmed by them [14] to be valid and reliable as a standardized measure of user satisfaction with a speci®c applica-tion. This research uses their 12-item instrument for the following reasons: ®rst, the instrument has been well validated and used widely; second, the instrument was developed particularly for end-user computing applications, and the major users of data warehouses are non-IS professionals; and, third, we found the 12 items employed in the instrument are applicable.
Recent advancements in information technology have made users more conscious of the speed and connectivity of IS. Large volumes of data and com-putationally intensive analyses are putting enormous pressure on the choices of hardware and software for an organization's data warehousing applications [25]. The recent recovery of mainframe sales has been ascribed to data warehouses' need for large and powerful computing equipment [2]. Therefore, our research includes these aspects.
A large number of recent studies emphasize the effect of support groups on user satisfaction [43]. Information centers (ICs) that provide adequate train-ing and support, as well as develop policies for
con-trol, are factors consistently affecting the success of EUC [37]. Since data warehousing is still a relatively new concept, training and support from ICs are neces-sary for end-users to operate their system more ef®-ciently and effectively.
4. Research design
A literature review was conducted on related topics. It resulted in the identi®cation of 35 potential research items. These items were incorporated into a prelimin-ary questionnaire, which was sent out for review by a number of academicians and practitioners. The pre-test respondents were asked to rate the relevance of the items in terms of end-user satisfaction with data ware-houses.
5. Criterion-related validity of the questionnaire
A criterion-related validity shows how closely the measure relates to some relevant criterion behavior external to the measuring instrument. A global item of end-users' satisfaction with data warehousing was included in the questionnaire; thus, the extent to which each item correlates to the global item is indicative of its criterion-related validity. Correlation between the overall users' satisfaction and individual item scores were calculated using Pearson's correlation coef®-cients. Items were retained if the signi®cance level of correlation with the overall satisfaction was less than 0.05 (p< 0.05). The results showed that ®ve out of 21 items failed to meet the requirement. The eliminated items were up-to-date information, avail-ability, response time, ease of use, and accessibility.
The deletion of the question regarding up-to-date information, which was one of the original items in the Doll and Torkzadeh instrument, may result from users' understanding that a data warehouse is a collec-tion of, in most cases, historical business data [22]. Its primary purpose is to supply information to managers to discover trends and forecast the future. Therefore, the focus of the data collection of a data warehouse is the breadth and depth of the data.
The time requirement for business decision making is usually lower than that for transactional processing. While transactional systems require split-second response time, decision supporting data warehouses usually do not. Users often assume that it will take some time before their inquiries are answered. Given the nature of a data warehouse in supporting strategic decision making, the requirement on the response time is thus not as critical as quality of the results.
The deletion of the questions on availability and accessibility of data warehouses suggested that data warehouses support a part, not all, of managers' work, and the availability and accessibility are not required for everybody in the company. They are hardly useful in making routine decisions. Moreover, processing large amounts of data in data warehouses puts enor-mous pressure on the system's performance [29].
Ease of use was also one of the items in the Doll and Torkzadeh instrument. A possible explanation could be that the users of data warehouses are mainly business managers with a reasonable amount of busi-ness training and a higher educational background
than the majority of end-users; so, the ease of use of IS seemed to carry less weight. However, ease of use has always been a good indicator of user satisfaction with IS. Thus, the authors suspect that its deletion in our study may be due to the small sample size.
6. Factor analysis
Exploratory factor analysis was conducted both to assess the construct validity of the measure and to determine the underlying factors of end-user satisfac-tion with data warehouses. It is suggested that, for factor analysis, there should be four to ®ve times as many observations as there are items to be studied. However, Hair et al. [21] pointed out that factor analysis has been used in several studies when the ratio of number of observations to items was about 2. In this study, the ratio is 2. Nevertheless, it is suggested that the results be interpreted with caution. The sig-ni®cance level of the results tends to suffer from the small sample size as well [6]. In this study, Bartlett's
test of sphericity (p0.00) indicates that correlation among items exists, and the Kaiser±Meyer±Olkin measure (0.73) showed middling sampling adequacy, according to Dziuban and Shirkey [15].
The factor analysis was conducted on the 16 retained items using principal components analysis as the extraction technique and varimax as the method of rotation. Without specifying the number of factors, three factors with eigenvalues greater than 1 emerged. The factor matrix is presented in Fig. 1. All of the primary factor loadings are greater than 0.5, which demonstrates a good match between the factor and the items.
The three factors and items within the factors are presented in Fig. 2. Most of the items from the Doll
and Torkzadeh instrument were loaded on factors 2 and 3. This suggests that these items are still valid in measuring user satisfaction with data warehousing. Factor 1 (Support provided to end-users) consists of six items regarding the support provided by ICs to end-users of data warehouses, which were not in Doll and Torkzadeh's instrument.
7. Discussion of findings
Our ®ndings suggest that end-users' satisfaction is dependent on the support provided by ICs, which is considered to span four stages: initiation, expansion, formalization, and maturity [33].
Considering the fact that data warehousing is fairly new to many organizations, it is safe to assume that their ICs, when dealing with the data warehous-ing project, are in the initialization stage. Thus, the end-users' performance and satisfaction is highly dependent on the training and support provided by the ICs.
However, the success of the data warehouse project depends on both the IC and the end-users. On the ICs' side, its success depends on IC's ability to deliver adequate training and support to the end-users. On the end-users' side, the end-users have to understand the meaning of the data included in the data ware-house [26]. Accordingly, it is the ICs' responsibility to increase the awareness among the end-users regarding the data in the warehouses. The lesson for ICs' managers here is that the successful manage-ment of end-users requires extensive and careful planning [28].
ICs know that there is more to excellent service than near-perfect uptime [42]. Speci®cally, ICs must under-stand the differences among the end-users: their abil-ity, functions, and needs. In their study, Ford et al. [18] discovered that demographic factors, prior computer training, and experience had signi®cant impacts on the participants' use of computers. Rivard [38] found that support provided to end-usershad the highest correla-tion with user satisfaccorrela-tion among the six factors that contribute to user satisfaction, and experiences have shown that higher user satisfaction can be achieved when ICs provided the kind of support preferred by end-users. Therefore, the support offered by the ICs needs to assess those differences in order to provide adequate service [32].
8. Conclusion
In this paper, we have discussed the important role played by the ICs in enhancing end-users' satisfaction in dealing with a data warehousing environment. Data warehousing applications usually take time to develop and perfect. IC's interaction with end-users during the development and improvement phases will increase satisfaction with the system. Communication with end-users is the most important success factor for ICs [30]. This study has found that the support pro-vided by ICs should be valued as a crucial part of
evaluating end-users' satisfaction with data ware-houses.
Acknowledgements
The authors would like to thank Laura Edwards and Karen Reddin for their assistance.
Appendix A. Twenty-one items used in the survey
1. Do you think that data warehouses provide the precise information you need?
2. Do you think that the information content of data warehouse meet your needs?
3. Do you think that data warehouse provides reports that seem to be just about exactly what you need?
4. Do you think that data warehouse provides sufficient information for your decision making? 5. Do you think that data in data warehouse is
accurate?
6. Do you think that you are satisfied with the accuracy of data in the data warehouse? 7. Do you think that the output of data warehouse is
presented in a useful format?
8. Do you think that the information extracted from data warehouse is clear?
9. Do you think that data warehouse is user friendly?
10. Do you think that data warehouse is easy to use?a 11. Do you think that you get the information you
need in time from data warehouse?
12. Do you think that data warehouse provides up-to-date information (the time lag between data warehouses and operation is minimal)?a
13. Do you think that data warehouse will be available to you if it exists in your organization?a 14. Do you think that data warehouse has an
acceptable response time?a
15. Do you think that data warehouse is accessible whenever you want to use them and wherever you are?a
16. Do you think that IS department provides satisfactory support to users using data ware-house?
enhancement of data warehouses will be re-sponded by IS department cooperatively? 18. Do you think that data warehouse applications
provides proper level of on-line assistance and explanation?
19. Do you think that data warehouse applications provides users with adequate error-control facil-ities, including error prevention, error detection, error correction and error recovery.
20. Do you think that IS department provides users with adequate level of training on using data warehouses?
21. Do you think that data warehouse developers interact with users extensively during the devel-opment of data warehouse?
(Items having superscript `a' were eliminated due to insigni®cant correlation with user satisfaction.)
References
[1] R. Adhikari, Migrating legacy data, Software Magazine 16(1), 1996, pp. 75±80.
[2] E.L. Appleton, Sales surge as mainframes find a role in client/ server, Datamation 41(10), 1995, pp. 48.
[3] C.W. Axelrod, Cashing in on data mining, Wall Street & Technology 14(12), 1996, pp. 60±62.
[4] J.E. Bailey, S.W. Pearson, Development of a tool for measuring and analyzing computer user satisfaction, Manage-ment Science, 1983, 530±545.
[5] L. Ball, R. Harris, SMIS members: a membership analysis, MIS Quarterly 6(1), 1982, pp. 19±38.
[6] J.J. Baroudi, W.J. Orlikowski, The problem of statistical power in MIS research, MIS Quarterly 13(1), 1989, pp. 87± 106.
[7] F. Bergeron, C. Berube, End users talk computer policy, Journal of System Management, 1990, 14±16.
[8] F. Bergeron, S. Rivard, L.D. Serre, Investigating the support role of the information center, MIS Quarterly 14(3), 1990, pp. 247±260.
[9] J.C. Brancheau, B.D. Janz, J.C. Wetherbe, Key issues in information systems: 1994±95 SIM Delphi results, MIS Quarterly, 1996, 225±241.
[10] J.C. Brancheau, J.C. Wetherbe, Key issues in information systems management, MIS Quarterly 11(1), 1987, pp. 23±45. [11] K. Bull, The ideal file cabinet, Informationweek 510, 1995,
pp. 42±48.
[12] W.H. DeLone, E.R. McLean, Information systems success: the quest for the dependent variable, Information Systems Research 3(1), 1992, pp. 60±95.
[13] W.J. Doll, G. Torkzadeh, The measurement of end-user com-puting satisfaction, MIS Quarterly 12(2), 1988, pp. 259±274.
[14] W.J. Doll, W. Xia, G. Torkzadeh, A confirmatory factor analysis of the end-user computing satisfaction instrument, MIS Quarterly 18(4), 1994, pp. 453±461.
[15] D.D. Dziuban, E.C. Shirkey, When is a correlation matrix appropriate for factor analysis? Some decision rules, Psychological Bulletin 81, 1974, pp. 358±361.
[16] J. Edwards, A clear view, CIO 8(4), 1994, pp. 74±80. [17] K. Fogarty, Data mining can help to extract jewels of data,
Network World 11(23), 1994, pp. 40±43.
[18] F.N. Ford, W.N. Ledbetter, T.L. Roberts, The impact of decision support training on computer use: the effect of prior training, age, and gender, Journal of End-user Computing 8(3), 1996, pp. 15±23.
[19] J. Gessaroli, Data mining: A powerful technology for database marketing, Telemarketing 13(11), 1995, pp. 64±68. [20] T. Guimaraes, Y. Gupta, Measuring top management satisfaction with the MIS department, OMEGA 16(1), 1988, pp. 17±24.
[21] J.F. Hair, Jr., R.E. Anderson, R.L. Tatham, B.J. Grablowsky, Multivariate Data Analysis, Macmillan Publishing Co., New York, 1984.
[22] W.H. Inmon, WHEREFORE Warehouse, Byte 23(1), 1998, pp. 88±90.
[23] W.H. Inmon, J.D. Welch, K.L. Glassey, Managing the Data Warehouse, Wiley, New York, 1997.
[24] B. Ives, M. Olson, J. Baroudi, The measurement of user information satisfaction, Communications of the ACM 26(10), 1983, pp. 785±793.
[25] B. Jeffery, Building bigger, better platforms, Midrange Systems 8(4), 1995, pp. 24±25.
[26] J. King, Don't skimp on the training, Computerworld 3(26), 1996, pp. DW7.
[27] C.D. Krivda, Unearthing underground data, LAN 11(5), 1996, pp. 42±48.
[28] A.L. Lederer, V. Sethi, Planning for end-user computing: top-down or bottom-up? Information Management Review 2(2), 1986, pp. 25±34.
[29] S. Levine, A to-do over what to do with the data: what are carriers keeping in their data warehouse and what are they doing with everything else? America's Network 21(101), 1997, pp. 36±38.
[30] S.R. Magal, H.H. Carr, H.J. Watson, Critical success factors for information center managers, MIS Quarterly 12(3), 1988, pp. 413±425.
[31] B. Mason, Data mining: exploring the unknown, Discount Merchandiser 35(10), 1995, pp. T58±T60.
[32] R. Mirani, W.R. King, The development of measure for end-user computing support, Decision Sciences 25(4), 1994, pp. 481±498.
[33] A.R. Montazemi, D.A. Cameron, K.M. Gupta, An empirical study of factors affecting software package selection, Journal of Management Information Systems 13(1), 1996, pp. 89±105. [34] M. Mayers, Do you really need a data warehouse, Network
World 12(51), 1995, pp. 26.
[35] R. Newing, Data mining, Management Accounting ± London 74(9), 1996, pp. 34±36.
systems management issues in the 1990s, MIS Quarterly 15(4), 1991, pp. 474±499.
[37] L.E. Rittenberg, A. Senn, End-user computing, Internal Auditor 50(1), 1993, pp. 35±39.
[38] S. Rivard, Successful implementation of end-user computing, Interfaces 17(3), 1987, pp. 25±33.
[39] J.F. Rockart, L.S. Flannery, The management of end user computing, Communications of the ACM 26(10), 1983, pp. 776±784.
[40] J.R. Ross, Data mining: digging deeper for information treasures, Stores 78(8), 1996, pp. 71±72.
[41] T. Sakaguchi, M. Frolick, A review of the data warehousing literature, Journal of Data Warehouse 2(1), 1997, pp. 34±54. [42] M. Santosus, Perception is reality, CIO 9(2), 1995, pp. 38±48. [43] M.H.A. Tafti, A three-dimensional model of user satisfaction with information systems, International Journal of Informa-tion Resource Management 3(1), 1992, pp. 4±10.
[44] J. Teresko, Data Warehouses: Building them for decision-making power, Industry Week 245(6), 1996, pp. 43±46. [45] J. Van den Hoven, Data warehousing: new name for the
accessibility challenge, Information System Management 14(1), 1997, pp. 70±72.
Lei-da Chen is a faculty member of Walker L. Cisler College of Business at Northern Michigan University. His re-search interests include electronic com-merce, data warehousing and mining, and end-user computing. His works have been published in several academic journals, includingJournal of Manage-ment SystemsandJournal of Information Systems Management, as well as a number of major conference proceedings.
Khalid S. Solimanis a faculty member of Frank G. Zarb School of Business at Hofstra University. He has more than 12 years of management and consulting experience. His research interests in-clude electronic commerce, data ware-housing and data mining, and software reuse. His research efforts have been published in several academic journals, includingJournal of Management Sys-tems and a number of national and international conference proceedings.
En Maois a Ph.D. student of Manage-ment Information Systems at the Uni-versity of Memphis. She received her BBA from Southern Arkansas Univer-sity, and MBA from Arkansas State University. Her research interests in-clude global information systems, inter-national business, technology transfers, information systems development in developing countries, and electronic commerce.
