Citation classics from the information systems literature
Kent A. Walstrom
a,*, Lori N.K. Leonard
b,1aApplied Computer Science Department, Campus Box 5150, Illinois State University, Normal, IL 61790-5150, USA,
bSchool of Accounting, University of Tulsa, 600 S. College Ave., Tulsa, OK 74104-3189, USA
Received 14 September 1999; accepted 3 January 2000
Abstract
An analysis of 118,364 references from 3752 articles published in top IS journals during the 10 years from 1986 to 1995 is
presented. Ninety-one `citation classics' were identi®ed by combining the results of this analysis with previous ®ndings.
Within this group of 91 publications, 13 `super classic' publications were identi®ed. These classics and super classics can be
categorized along any of several dimensions to gain a better understanding of the IS ®eld. These ®ndings can be used, among
other things, for developing reading lists and identifying the characteristics that make a publication `classic'.
#
2000 Elsevier
Science B.V. All rights reserved.
Keywords:Diffusion of IS research; IS literature; Citation analysis
ISRL: IB0301; IB0301.01
1. Introduction
Whether we are attempting to understand the broad
foundations of a research discipline or the speci®c
implications of a narrow stream of research within a
discipline, we need to review what has already been
discovered so we can explore that which is yet
unknown. As the information systems (IS) ®eld grows
and matures there is an increase in the amount of new
literature published each year. Scanning the total
volume thus becomes a daunting task and doctoral
students interested in the cross-disciplinary study of
IS, ask `can you suggest reading material for me in the
IS ®eld?' Moreover, practitioners may want to read the
`best' that the academic ®eld has to offer. The purpose
of this study was to identify the most cited IS
pub-lications from the past 10 years and use them to
identify the `classic' publications in the ®eld.
2. Literature review
The `must read' publications list of 10 experts in the
IS ®eld would certainly not be the same, either in
content or order. The differences in these lists may be a
result of several factors, including: the large number of
articles (several thousand to choose from), the
diver-sity of backgrounds of the experts, and the research
and teaching interests currently being pursued by
them. Examining several Ph.D. seminar reading lists,
revealed marked differences in the material required
of IS Ph.D. students in an introductory course.
Select-ing just one of these might result in important works
Information & Management 38 (2000) 59±72
*Corresponding author. Tel.:1-309-438-8338; fax:1-309-438-5113.
E-mail address: [email protected] (K.A. Walstrom).
1Tel.:1-918-631-2787.
being missed, though they are considered important by
a greater number of experts. The continual demand for
IS professionals and knowledge workers in the
mar-ketplace makes the demand for IS faculty members
either constant or increasing. Government and
uni-versity pressure provides a demand for IS literate
researchers, since IS technology is applicable to all
®elds of inquiry.
To gain an understanding of any large body of
knowledge requires considerable time and effort.
Reading the best the ®eld has to offer maximizes
the use of time resources. People now realize that it
is no longer possible to read all the available literature
and ®ltering strategies have, therefore, emerged. Over
the years, literacy experts have published their
®lter-ing strategies in the form of `lists of great books' in an
attempt to help mankind identify which works merit
investigation [1,2,19,20,24]. The thoughts of these
experts are summed in the following quote:
``No single person has a right to say which are
the hundred or thousand best books.
. . .
But
numerous lists of great books have been drawn
up by various persons at one time or other, and
they provide what is on the whole the best guide
through the bewildering shelves of any large
library.''
Van Doren in [25].
A similar filtering strategy can be used in
understand-ing academic literature. In the academic world, one of
these lists is provided at the end of every refereed
journal article or conference proceedings produced.
This reference list can be used to identify sources on
the topic of interest in the article. Collected together
over time and over a variety of publication outlets,
these lists provide a composite list of the best sources
available in the field of IS. The Proceedings of the
First ICIS meeting in 1980 reveals an attempt to
identify the literature which had to be included in a
doctoral level program of study [10].
2.1. The early years
Dickson [17] identi®ed ®ve early IS books that
included readings: Sanders [29]; Coleman and Riley
[6]; Sanders [28] Davis and Everest [11]; and Dock
et al. [18]. A review of these books provides a
parsi-monious look at articles that were then considered by
these authors to be important. Table 1 shows the
articles that appeared in more than one of these books.
One article, Ackoff 1967, appeared in all the ®ve
books. Two articles, Dearden 1972 and Kriebel
1972, appeared in three of the ®ve books. Another
20 articles appeared in two of the books.
2.1.1. Citations
Classics can be identi®ed subjectively by asking
individuals, but the views of several experts yield a
more representative selection of publications.
Accord-ing to de Solla Price [16], a `classic' is any publication
cited four or more times in a year. Citation analysis has
also been identi®ed as another means of determining
`classic' publications [5,21,27]. Citation analysis
allows for the collection of perceptions of hundreds,
or even thousands, of `experts', simultaneously and
over time. `Citation classics' are extraordinary,
because of the large number of citations they receive.
While, 30 years ago considerably fewer articles were
written and printed than today, this number still
pro-vides a good starting place. Especially since we are not
interested in all available literature, only that small
portion regarded as IS literature.
3. The middle years: 1972±1985
Researchers produced citation ®ndings about the IS
literature for the period from 1972 to 1985. Culnan, in
two separate studies [7,8], reported her ®ndings of a
co-citation analysis of the IS discipline. The appendix
from these studies is helpful in identifying works that
met the `classic' criteria in the IS discipline from 1972
until mid-1985. Table 2 summarizes the ®ndings from
Culnan's ®rst co-citation analysis from 1972 to
mid-1982. The number of citations per year was
deter-mined by dividing the total number of citations
identi®ed from the study by 10.5. During this ten
and a half year period, seven publications were found
to meet the criteria of four or more citations a year:
1. Churchman, 1971.
2. Mason and Mitroff, 1973.
3. Churchman, 1968.
4. Mason, 1969.
5. Ackoff, 1967.
6. Lucas, 1975.
Table 1
Articles appearing in more than one book of readings (listed by frequency)
Article Sanders,
1970 [29]
Coleman and Riley, 1973 [6]
Sanders, 1974 [28]
Davis and Everest, 1976 [11]
Dock,etal., 1977 [18]
Ackoff, R.L., 1967, `Management Misinfornation Systems,'Management Science, December, pp. B-147-B-156.
X X X X X
Dearden, 1972, `MIS is a Mirage,'Harvard Business Review, January±February, pp. 90±99. X X X Kriebel, 1972, `The Future of MIS,'Business Automation, June, pp. 18±19. X X X Adams, W., 1972, `New Role for Top Management in Computer Applications,'Financial Executive,
April, pp. 54±56.
X X
Allen, B.R., 1972, `Computer Security: Part I.,'Data Management, January, pp. 18±24. X X Allen, B.R., 1972, `Computer Security: Part II.,'Data Management, February, pp. 25±30. X X Andrus, 1971, `Approaches to Information Evaluation,'MSU Business Topics, Summer, pp. 40±45. X X
Argyris, C., 1971, `MIS: The Challenge to Rationality,'Management Science, February. X X Beishon, R.J., 1970, `Information Flow and Managers' Decisions,'Managerial Accounting, June. X X Deckard, N.S., 1970, `Capturing the Development Costs of an Integrated MIS?'Journal of Systems
Management, January.
X X
Dickson, G.W., 1968, `Management Information±Decision Systems,'Business Horizons, December. X X Dickson, G.W. and Simmons, J.K., 1970, `The Behavioral Side of MIS: Some Aspects,'
Business Horizons, August.
X X
Ernst, M., 1971, `Management, The Computer, and Society,'Computers and Automation, September, pp. 8±11.
X X
Fredricks, W.A., 1971, `A Manager's Perspective of Management Information Systems,'
MSU Business Topics, Spring.
X X
Gibson, C.F. and Nolan, R.L., 1974, `Managing the Four Stages of EDP Growth,'Harvard
Business Review, January±February.
X X
Holmes, R.W., 1970, `Twelve Areas to Investigate for Better MIS,'Financial Executive, July. X X Hoos, I.R., 1971, `Information Systems and Public Planning,'Management Science, June,
pp. B658±B671.
X X
Kanter, J., 1972, `Impact of Computers on the Business Organization,'Data Management, April, pp. 20±23.
X X
Nanus, B., 1972, `Managing the Fifth Information Revolution,'Business Horizons, April. X X Nichols, G.E., `On The Nature of Management Information,'Management Accounting, April 1969,
pp. 9±13.
X X
Nicholson, C.H., 1970, `MIS In Perspective,'Chemical Engineering Progress, January. X X Tolliver, E.M., 1970, `Myths of Automated Management Systems,'
Journal of Systems Management, August.
X X
Zani, 1970, `Blueprint for MIS,'Harvard Business Review, November±December. X X
K.A.
W
alstr
om,
L.N.K.
Leonar
d
/Information
&
Manageme
nt
38
(2000)
59±72
Table 2
Culnan's co-citation analysis findings for 1972 to mid-1982 (listed by frequency)
Publication # Citations Citations
per year
Churchman, C.W., 1971,The Design of Inquiring Systems, Basic Books: New York 103 9.8 Mason, R.O. and Mitroff, I.I., 1973, `A Program for Research on Management Information Systems,'Management Science, January (19:5),
pp. 475±487.
63 6
Churchman, C.W., 1968,The Systems Approach, Delacorte Press: New York 63 6
Mason, R.O., 1969, `A Dialectical Approach to Strategic Planning,'Management Science, April (15:8), pp. B403±B414. 53 5 Ackoff, R.L., 1967, `Management Misinformation Systems,'Management Science, (14:4), pp. 147±156. 52 5 Lucas, H.C., 1975,Why Information Systems Fail, Columbia University Press: New York. 50 4.8 Churchman, C.W. and Schainblatt, A.H., 1965, `The Researcher and the Manager: A Dialectic of Implementation,'Management Science, (11:4),
pp. B69±B87.
47 4.5
Anthony, R.N., 1965,Planning and Control Systems: A Framework for Analysis, Harvard University, Graduate School of Business Administration, Division of Research: Boston, MA.
41 3.9
Davis, G.B., 1974,Management Information Systems: Conceptual Foundations, Structure and Development, McGraw-Hill: New York. 35 3.3 Gorry, G.A. and Scott-Morton, M., 1971, `A Framework for Management Information Systems,'Sloan Management Review, (13:1), pp. 55±57. 34 3.2 Swanson, E.B., 1974, `Management Information Systems: Appreciation and Involvement,'Management Science, October (2:2), pp. 178±183. 34 3.2 Ackoff, R.L., 1970,A Concept of Corporate Planning, Wiley±Interscience: New York 34 3.2 Ackoff, R.L., 1974,Redesigning the Future: A Systems Approach to Social Problems, Wiley: New York. 34 3.2 Ackoff, R.L., 1962,Scientific Method: Optimising Applied Research Decisions, Wiley: New York. 32 3 King, W.R. and Cleland, D.I., 1975, `The Design of Management Information Systems: An Information Analysis Approach,'Management Science,
November (22:3), pp. 286±297.
25 2.4
Mitroff, I.I., 1971, `A Communication Model of Dialectical Inquiring Systems: A Strategy for Strategic Planning,'Management Science, June (17:10), pp. B634±B648.
24 2.3
K.A.
W
alstr
om,
L.N.K.
Leonar
d
/
Informat
ion
&
Management
38
(2000)
Mitroff, I.I., 1979,The Subjective Side of Science: A Philosophical Inquiry into the Psychology of Apollo Moon Scientists, Elsevier Scientific Publishing Company: New York.
24 2.3
Dearden, J., 1972, `MIS is a Mirage,'Harvard Business Review, January±February, pp. 90±99. 22 2.1 McKenny, J.L. and Keen, P.G.W., 1974, `How Manager's Minds Work,'Harvard Business Review, May±June (52:3), pp. 79±90. 21 2 Nolan, R.W., 1973, `Managing the Computer Resource: A Stage Hypothesis,'Communications of the ACM, (16:7), pp. 399±405. 20 1.9 Hiltz, S.R. and Turoff, M., 1978The Network Nation: Human Communication via Computer. 17 1.6 Keen, P.G.W. and Scott-Morton, M.S., 1978,Decision Support Systems: An Organizational Perspective, Addison-Wesley Publishing Company:
Reading, MA.
16 1.5
Ackoff, R.L., 1960, `Unsuccessful Case Studies and Why,'Operations Research, March±April (8:2), pp. 259±262. 15 1.4 Chervany, N.L. and Dickson, G.W., 1974, `An Experimental Evaluation of Information Overload in a Production Environment,'Management
Science, June (20:10), pp. 1335±1344.
13 1.2
Nolan, R.W., 1973, `Computer Data Bases: The Future is Now,'Harvard Business Review, September±October (51:5), pp. 98±114. 13 1.2 Mitroff, I.I. and Featheringham, T.R., 1974, `On Systematic Problem Solving and the Error of the Third Kind,'Behavioral Science, (19),
pp. 383±393.
13 1.2
Mitroff, I.I., Betz, F., Pondy, L.R. and Sagasti, F., 1974, `On Managing Science in the Systems Age: Two Schemas for the Study of Science as a Whole Systems Phenomenon,'Interfaces, (4:3), pp. 46±58.
12 1.1
Bennett, J.L., 1972, `The User Interface in Interactive Systems,'Annual Review of Information Science and Technology, (7), pp. 159±196. 12 1.1 Whisler, T.L., 1970,The Impact of the Computer on Organizations, Praeger: New York. 11 1 Danzinger, J.N., 1977, `Computers, Local Governments, and the Litany of EDP,'Public Administration Review, (37:1), pp. 28±37. 11 1 Lucas, H.C., 1975, `Performance and the Use of an Information System,'Management Science, April (21:8), pp. 908±919. 10 1 Mitroff, I.I. and Emshoff, J.R., 1979, `On Strategic Assumption-Making: A Dialectical Approach to Policy and Planning,'Academy of
Management Journal, January (4:1), pp. 1±12.
10 1
King, W.R. and Cleland, D.I., 1971, `Manager-Analyst Teamwork in MIS,'Business Horizons, April, pp. 59±68. 10 1 King, W.R., 1964, `Performance Evaluation in Marketing Systems,'Management Science, July (10:4), pp. 659±666. 10 1
K.A.
W
alstr
om,
L.N.K.
Leonar
d
/Information
&
Manageme
nt
38
(2000)
59±72
Table 3
Culnan's co-citation analysis findings for 1980 to mid-1985 (listed by frequency)
Publication # Citations Citations
per year
Keen, P.G.W. and Scott Morton, M.S., 1978,Decision Support Systems: An Organizational Perspective, Addison-Wesley: Reading, MA 81 14.7 Mason, R.O. and Mitroff, I.I., 1973, `A Program for Research on Management Information Systems,'Management Science, January (19:5),
pp. 475±487.
68 12.4
Lucas, H.C., 1975,Why Information Systems Fail, Columbia University Press: New York 60 10.9 Zmud, R.W., 1979, `Individual Differences and MIS Success: A Review of the Empirical Literature,'Managment Science, February (25:2),
pp. 966±979.
42 7.6
Anthony, R.N., 1965,Planning and Control Systems: A Framework for AnalysisHarvard University Press: Boston, MA 38 6.9 Dickson, G.W., Senn, J.A., and Chervany, N.L., 1977, `Research in Management Information Systems; The Minnesota Experiments,'
Management Science, May (23:9), pp. 913±923.
36 6.5
Davis, G.B., 1974,Management Information Systems: Conceptual Foundations, Structure and Development, McGraw-Hill: New York 30 5.5 Benbasat, I. and Schroeder, R.G., 1978, `An Experimental Investigation of Some MIS Design Variables,MIS Quarterly, June (2:2), pp. 43±54. 27 4.9 Alter, S., 1980,Decision Support Systems: Current Practice and Continuing Challenges, Addison-Wesley: Reading, MA 27 4.9 Zmud, R.W., 1978, `An Empirical Investigation of the Dimesionality of the Concept of Information,'Decision Sciences, April (9:2), pp. 187±196. 27 4.9 Benbasat, I.W. and Dexter, A.S., 1979, `Value and Events Approaches to Accounting: An Experimental Evaluation,'Accounting Review,
October (54:4), pp. 735±749
24 4.4
Gorry, G.A. and Scott Morton, M.S., 1971, `A Framework for Management Information Systems,'Sloan Management Review, Fall (12:1), pp. 55±70.
23 4.2
Ackoff, R.L., 1967, `Management Misinformation Systems,'Management Science, December (14:4), pp. B147±156 23 4.2 McKenney, J.L. and Keen, P.G.W., 1974, `How Managers' Minds Work,'Harvard Business Review, May±June (52:3), pp.79±90. 20 3.6 Kling, R., 1980, `Social Analyses in Computing,'Computing Surveys, March (12:1), pp 61±108. 20 3.6 Sprague, R.H. and Carlson, E.D., 1982,Building Effective Decision Support Systems, Prentice Hall: Englewood Cliffs: NJ 19 3.5 Ginzberg, M.J., 1978, `Steps Toward More Effective Implementation of MS and MIS,'Interfaces, May (8:3), pp. 57±63. 16 2.9 King, W.R., 1978, `Strategic Planning for Management Information Systems,'MIS Quarterly, March (2:1), pp. 27±37 16 2.9 Sprague, R.H., 1980, `A Framework for the Development of Decision Support Systems,'MIS Quarterly, December (4:4), pp. 1±26. 16 2.9 Swanson, E.B., 1974, `Management Information Systems: Appreciation and Involvement,'Management Science, October (20:2), pp. 178±188. 16 2.9 Ein Dor, P. and Segev, E., 1978, `Organizational Context and the Success of Management Information Systems,'Management Science,
June (24:10), pp. 1067±1077.
13 2.4
Rockart, J.F., 1979, `Chief Executives Define their Own Data Needs,'Harvard Business Review, March±April (57:2), pp. 81±91. 12 2.2 Mason, R.O. and Mitroff, I.I., 1981,Challenging Strategic Planning Assumptions, John Wiley: New York 11 2 Nolan, R.L., 1979, `Managing the Crises in Data Processing,'Harvard Business Review, March±April (57-2), pp. 115±126. 11 2 Keen, P.G.W., 1983, `Information Systems and Organizational Change,'Communications of the ACM, January (24:1), pp. 24±33. 11 2 Ginzberg, M.J., 1975,A Process Approach to Management Science Implementation, Unpublished Ph.D. dissertation, Massachusetts
Institute of Technology, Cambridge, MA
10 1.8
K.A.
W
alstr
om,
L.N.K.
Leonar
d
/
Informat
ion
&
Management
38
(2000)
Table 3 summarizes the ®ndings from Culnan's
second co-citation analysis of the IS ®eld from
1980 to mid-1985. The number of citations per year
was determined by dividing the total number of
cita-tions identi®ed from the study by 5.5. During this ®ve
and a half year period, 13 publications were found to
meet the criteria of four or more citations a year:
1. Keen and Scott Morton, 1978.
2. Mason and Mitroff, 1973.
3. Lucas, 1975.
4. Zmud, 1979.
5. Anthony, 1965.
6. Dickson, Senn, and Chervany, 1977.
7. Davis, 1974.
8. Benbasat and Schroeder, 1978.
9. Alter, 1980.
10. Zmud, 1978.
11. Benbasat and Dexter, 1979.
12. Gorry and Scott Morton, 1971.
13. Ackoff, 1967.
From the two previous lists of 20 total publications,
three publications appear on both:
1. Ackoff, R.L., 1967.
2. Lucas, H.C., 1975.
3. Mason, R.O. and Mitroff, I.I., 1973.
These three publications demonstrate their ability to
remain classics over a fourteen and a half year time
span. (In fact, one publication, Ackoff, 1967, also
appeared in all readings of the ®ve books). Since these
two studies do not provide citation data for each year,
but only a summary total, some information may be
lost between the ®ndings of the two studies. In an
attempt to compensate for this loss, the number of
citations for both studies were added together and
divided by 16. While this method is not precise, the
information gained from the analysis is worth the risk
of introducing distortion. Using this, three more
pub-lications meet the criteria for being classic:
1. Anthony, R.N., 1965.
2. Davis, G.B., 1974.
3. Keen, P.G.W. and Scott Morton, M.S., 1978.
4. Methodology
Over 10 years have elapsed since the time frame
covered by the Culnan studies. The current study picks
up where this study ended: the 10 year period from
1986 to 1995. With the plethora of journals available,
the options were numerous. The top IS journals were
selected for analysis. Herein lies a dif®culty. In some
®elds, such as Accounting and Economics, there
appears to be consensus. In other ®elds (often younger
disciplines), there will be a variety of answers from
different people. There may be some agreement about
the top one or two. However, because of the number of
journals considered IS related there are likely to be 25
or 30 major journals. Over the years, empirical
evi-dence has been collected to provide a broader picture
of the journal forums: Table 4 shows a summary of the
top journals.
Based on these studies nine journals were selected
for this study. Citation data was collected from each
article published in each of these journals for the time
period under investigation, 1986±1995.
Table 4
Top information systems journals rankings
Journal Hardgrave and
Walstrom, 1997 [23]
Walstrom, et al., 1995 [30]
Gillenson and Stutz, 1991 [22]
MIS Quarterly 1 1 2
Information Systems Research 2 3 ±
Management Science 3 4 1
Communications of the ACM 4 2 3
Journal of MIS 5 7 5
Decision Sciences 6 8 4
Decision Support Systems 10 11 ±
Information & Management 20 12 12
Organization Science 8 ± ±
The Social Science Citation Index (SSCI) was used
to collect as much of the data as possible. The SSCI
contained all the citations from
Communications of
the ACM
,
Decision Sciences
,
MIS Quarterly
, and
Management Science
. It also contained citation
infor-mation for some years for
Decision Support Systems
,
Information & Management
,
Information Systems
Research
, and
Organization Science
. The remaining
citations from these journals and all the citations from
the
Journal of Management Information Systems
were
collected manually by the authors. Once all the
cita-tions were collected, they were ordered to determine
which publications appeared most often.
5. Results
Table 5 shows that 3752 articles were found in the
nine journals during the years from 1986 to 1995
inclusive. In addition, 118,364 references were
attached to the 3752 articles. These 118,364
refer-ences were used for further analysis. An increase in
the number of articles published from 323 in 1986 to
378 in 1995 is noted, with a high of 433 in 1993. The
number of references more than doubled from 7406 in
1986 to 15,762 in 1995, with a high of 16,955 in 1992.
The number of references per article nearly doubled
from 23 in 1986 to 42 in 1995, with the latter being the
high.
The SAS FREQ procedure was used to identify any
publication among the 118,364 citations which
appeared four or more times in any one year from
1986 to 1995. This list of publications was then added
to the previous list of publications identi®ed in both
the Culnan studies. A total of 278 publications were
identi®ed for further investigation. A total citation
count for each of these 278 publications was
per-formed for the time period 1986±1995. Any
publica-tion which averaged at least four citapublica-tions per year
over the 10 year period, or at least, 40 total citations
were identi®ed as a highly visible work in the IS ®eld
of study. Table 6 shows that from the 278 publications,
61 were cited an average of four or more times per year
from 1986 to 1995. Seven publications were cited over
100 times: ®ve books and two journal articles.
Not all the articles have an equal probability of
meeting the cutoff of 40 citations. Works published
after 1985 have less likelihood of meeting the cutoff
criteria, because they have fewer years in which to
reach 40 citations. This makes the achievement of
articles such as DeSanctis and Gallupe (1987) and
Brancheau and Wetherbe [3] even more remarkable,
because they were available for citation 3 years less. In
fact, even without considering the lag between
manu-script acceptance and actual publication (which is
often well over a year) a work published one year
is unlikely to be cited until at least the next year. Ten of
the 61 works in Table 6 were published in 1986 or
later.
To compensate for this inequity, an additional
ana-lysis was run to identify publications which met the
average cutoff of four or more citations per year, but
not the total cutoff of 40 citations. Table 7 shows an
additional 15 publications (published after 1985)
which averaged four or more citations since the year
of their publication.
Combining the articles from Tables 6 and 7 results
in 76 total publications for further consideration: 29
books, 47 journal articles. The books appear more
often towards the top of the list and the journal articles
appear more frequently towards the bottom. Table 8
shows the journals which published these 47 articles.
Nearly half of the articles were published in either
MIS
Quarterly
or
Management Science
. Nearly 25 percent
more were published in
Communications of the ACM
and
Harvard Business Review
. Among the nine other
journals each containing one article were two newer
journals,
Information Systems Research
and
Organi-zation Science
.
Table 5
Table 6
Publications averaging four or more citations per year from 1986 to 1995 (listed by frequency)
Publication Total
86±95
Citations per year
Porter, M.E., 1980,Competitive Strategy: Techniques for Analyzing Industries and Competitors, Free Press, New York. 141 14.1 Sprague, R.H. and Carlson, E.D., 1982,Building Effective Decision Support Systems, Prentice-Hall, Englewood Cliffs, NJ. 134 13.4 Thompson, J.D., 1967,Organizations in Action: Social Science Bases of Administrative Theory, McGraw-Hill: New York. 124 12.4 Keen, P.G.W. and Scott Morton, M.S., 1978,Decision Support Systems: an Organizational Perspective, Addison-Wesley Pub. Co., Reading, MA. 119 11.9 DeSanctis, G. and Gallupe, R.B., 1987, `A Foundation for the Study of Group Decision Support Systems,'Management Science, May (33:5), pp. 589±609. 116 11.6 Nunnally, J.C., 1967, 1978,Psychometric Theory, McGraw-Hill, New York.a 108 10.8 Ives, B., Olson, M.H. and Baroudi, J.J., 1983, `The Measurement of User Information Satisfaction,'Communications of the ACM, October (26:10), pp. 785±793. 104 10.4 Ives, B. and Olson, M.H., 1984, `User Involvement and MIS Success: A Review of Research,'Management Science, May (30:5), pp. 586±603. 98 9.8 Weick, K.E., 1969, 1979,The Social Psychology of Organizing, Addison-Wesley Publishing Company: Reading, MA.a 95 9.5 Keeney, R.L. and Raiffa, H., 1976,Decisions with Multiple Objectives: Preferences and Value Tradeoffs, Wiley: New York. 91 9.1 Williamson, O.E., 1975,Markets and Hierarchies, Analysis and Antitrust Implications: A Study in Economics of Internal Organizations, Free Press: New York. 88 8.8 McFarlan, F.W., 1984, `Information Technology Changes the Way You Compete,'Harvard Business Review, May±June (62:3), pp. 98±103. 86 8.6 Porter, M.E. and Millar, V.E., 1985, `How Information Gives You Competitive Advantage,'Harvard Business Review, July±August (63), pp. 149±160. 86 8.6 Porter, M.E., 1985,Competitive Advantage: Creating and Sustaining Superior Performance, Free Press: New York. 84 8.4 Cyert, R.M. and March, J.G., 1963,A Behavioral Theory of the Firm, Prentice-Hall: Englewood Cliffs, NJ. 81 8.1
March, J.G. and Simon, H.A., 1958,Organizations, Wiley: New York. 79 7.9
Davis, G.B., 1974, 1985,Management Information Systems: Conceptual Foundations, Structure and Development, McGraw-Hill: New York.a 77 7.7 Pfeffer, J., 1978,The External Control of Organizations: A Resource Dependence Perspective, Harper & Row: New York. 76 7.6 Huber, G.P., 1984, `Issues in the Design of Group Decision Support Systems,'MIS Quarterly, September (8:3), pp. 195±204. 71 7.1 Ives, B. and Learmonth, G. P., 1984, `The Information System as a Competitive Weapon,'Communications of the ACM, December (27:12), pp. 1193±1201. 70 7.0 Rogers, E.M., 1962, 1983,Diffusion of Innovations, Free Press: New York.a 68 6.8 Brancheau, J.C. and Wetherbe, J.C., 1987, `Key Issues in Information Systems Management,'MIS Quarterly, March (11:1), pp. 23±45. 67 6.7 King, W.R., 1978, `Strategic Planning for Management Information Systems,'MIS Quarterly, March (2:1), pp. 27±37. 63 6.3 Bailey, J.E. and Pearson, S.W., 1983, `Development of a Tool for Measuring and Analyzing Computer User Satisfaction,'Management Science,
May (29:5), pp. 530±545.
62 6.2
Mintzberg, H., Raisinghani, D. and Theoret, A., 1976, `The Structure of `Unstructured' Decision Processes,'Administrative Science Quarterly, June (21:2), pp. 246±275.
62 6.2
Galbraith, J.R., 1973,Designing Complex Organizations, Addison-Wesley Publishing Company: Reading, MA. 61 6.1 Markus, M.L., 1983, `Power, Politics, and MIS Implementation,'Communications of the ACM, June (26:6), pp. 430±444. 61 6.1 Zmud, R.W., 1979, `Individual Differences and MIS Success: A Review of the Empirical Literature,'Management Science, February (25:2), pp. 966±979. 61 6.1 Gorry, G.A. and Scott Morton, M.S., 1971, `A Framework for Management Information Systems,'Sloan Management Review, (13:1), pp. 55±70. 60 6.0 Lawrence, P.R. and Lorsch, J.W., 1967,Organization and Environment: Managing Differentiation and Integration, Division of Research,
Graduate School of Business Administration, Harvard University: Boston, MA.
59 5.9
Newell, A. and Simon, H.A., 1972,Human Problem Solving, Prentice-Hall: Englewood Cliffs, NJ. 59 5.9 Daft, R.L. and Lengel, R.H., 1986, `Organizational Information Requirements, Media Richness and Structural Design,'Management Science,
May (32:5), pp. 554±571.
56 5.6
Cash, Jr., J.I. and Konsynski, B., 1985, `IS Redraws Competitive Boundaries,'Harvard Business Review, March±April (64:2), pp. 134±142. 55 5.5 Simon, H. A., 1960,The New Science of Management Decision, Harper, New York. 55 5.5 Swanson, E.B., 1974, `Management Information Systems: Appreciation and Involvement,'Management Science, October (20:2), pp. 178±188. 54 5.4 Williamson, O.E., 1985,The Economic Institutions of Capitalism: Firms, Markets, Relational Contracting, Free Press: New York. 53 5.3
K.A.
W
alstr
om,
L.N.K.
Leonar
d
/Information
&
Manageme
nt
38
(2000)
59±72
Table 6 (Continued)
Publication Total
86±95
Citations per year
Burns, T. and Stalker, G.M., 1961, 1966,The Management of Innovation,Tavistock Publications: London.a 52 5.2 Dickson, G.W., Leitheiser, R.L. and Wetherbe, J.C., 1984, `Key Information Systems Issues for the 1980's,'MIS Quarterly, September (8:3), pp. 135±159. 52 5.2 Chandler, A.D., 1962,Strategy and Structure: Chapters in the History of the Industrial Enterprise, M.I.T. Press: Cambridge, MA. 51 5.1 Dennis, A.R., George, J.F., Jessup, L.M., Nunamaker, J.F. and Vogel, D.R., 1988, `Information Technology to Support Electronic Meetings,'
MIS Quarterly, December (12:4), pp. 591±624.
51 5.1
Huber, G.P., 1984, `The Nature and Design of Post-Industrial Organizations,'Management Science, August (30:8), pp. 928±951. 49 4.9 McGrath, J.E., 1984,Groups: Interaction and Performance, Prentice-Hall, Englewood Cliffs, NJ. 49 4.9 Nelson, R.R., 1982,An Evolutionary Theory of Economic Change, Belknap Press of Harvard University Press: Cambridge, MA. 49 4.9 Parsons, 1983, `Information Technology: A New Competitive Weapon,'Sloan Management Review, Fall (25:1), pp. 3±14. 49 4.9 Sprague, R.H., 1980, `A Framework for the Development of Decision Support Systems,'MIS Quarterly, December (4:4), pp. 1±26. 49 4.9 Galbraith, J.R., 1977,Organization Design, Addison-Wesley Publishing Company: Reading, MA. 48 4.8 Mintzberg, H., 1979,The Structuring of Organizations: A Synthesis of the Research, Prentice-Hall: Englewood Cliffs, NJ. 48 4.8 Bonczek, R.H., Holsapple, C.W. and Whinston, A.B., 1981,Foundations of Decision Support Systems, Academic Press: New York. 47 4.7 Rockart, J.F., 1979, `Chief Executives Define their Own Data Needs,'Harvard Business Review, March±April (57:2), pp. 81±91. 47 4.7 Rockart, J.F. and Flannery, L.S., 1983, `The Management of End User Computing,'Communications of the ACM, October (26:10), pp. 776±784. 46 4.6 Argyris, C. and Schon, D.A., 1978,Organizational Learning, Addison-Wesley: Reading, MA. 45 4.5 Benson, D.H., 1983, `A Field Study of End User Computing: Findings and Issues,'MIS Quarterly, December (7:4), pp. 35±45. 45 4.5 Johnston, H.R. and Carrico, S.R., 1988, `Developing Capabilities to Use Information Strategically,'MIS Quarterly, March (12:1), pp. 37±48. 45 4.5 Davis, F.D., 1989, `Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,'MIS Quarterly, September (13:3),
pp. 319±340.
44 4.4
Kraemer, K.L. and King, J.L., 1988, `Computer-Based Systems for Cooperative Work and Group Decision Making,'ACM Computing Surveys, June (20:2), pp. 115±146.
44 4.4
Nunamaker, J.F., Applegate, L.M. and Konsynski, B.R., 1987, `Facilitating Group Creativity: Experience with a Group Decision Support System,'
Journal of Management Information Systems, Spring (3:4), pp. 5±19.
44 4.4
Markus, M.L. and Robey, D., 1988, `Information Technology and Organizational Change: Causal Structure in Theory and Reseach,'Management Science, May (34:5), pp. 583±598.
43 4.3
Kahneman, 1979, `Prospect Theory: An Analysis of Decision Making Under Risk,'Econometric, March (47:2), pp. 263±291. 42 4.2 Tversky, 1974, `Judgement Under Uncertainty: Heuristics and Biases,'Science, (185), pp. 1124±1131. 42 4.2 Eisenhardt, K.M., 1989, `Building Theories from Case Study Research,'Academy of Management Review, (14:4), pp. 532±550. 41 4.1 Nolan, R.L., 1979, `Managing the Crises in Data Processing,'Harvard Business Review, March±April (57:2), pp. 115±126. 41 4.1
aTwo dates listed indicating both editions of the same book were counted together.
K.A.
W
alstr
om,
L.N.K.
Leonar
d
/
Informat
ion
&
Management
38
(2000)
Table 7
Additional publications averaging four or more citations per year since publication (listed by frequency)
Publication 1986±1995 Citations
per year
DeLone, W. H. and McLean, E. R., 1992, `Information Systems Success: The Quest for the Dependent Variable,'Information Systems Research, March (3:1), pp. 60±95.
24 8
Huber, G.P., 1990, `A Theory of the Effects of Advanced Information Technologies on Organizational Design, Intelligence, and Decision Making,'
Academy of Management Review, (15:1), pp. 47±71.
30 6
Nunamaker, J.F., Dennis, A.R., Valacich, J.S., Vogel, D.R. and George, J.F., 1991, `Electronic Meeting Systems to Support Group Work,'
Communications of the ACM, July (34:7), pp. 40±61.
24 6
Davis, F. D., Bagozzi, R. P. and Warshaw, P. R., 1989, `User Acceptance of Computer Technology: A Comparison of Two Theoretical Models,'
Management Science, August (35:8), pp. 982±1003.
35 5.8
Davenport, T.H., 1993,Process Innovation: Reengineering Work Through Information Technology, Harvard Business School Press: Boston. 11 5.5 Doll, W. J. and Torkzadeh, G., 1988, `The Measurement of End-User Computing Satisfaction,'MIS Quarterly, June (12:2), pp. 259±274. 35 5 Jarvenpaa, S.L., 1989, `The Effect of Task Demands and Graphical Format on Information Processing Strategies,'Management Science,
March (35:3), pp. 285±303.
30 5
DeSanctis, G. and Poole, M.S., 1994, `Capturing the Complexity in Advanced Technology Use: Adaptive Structuration Theory,'
Organization Science, May (5:2), pp. 121±147.
5 5
Hartwick, J. and Barki, H., 1994, `Explaining the Role of User Participation in Information System Use,'Management Science, April (40:4), pp. 440±465. 5 5 Gallupe, R. B. and DeSanctis, G., 1988, `Computer-Based Support for Group Problem-Finding: An Experimental Investigation,'MIS Quarterly,
June (2:2), pp. 277±296.
33 4.7
Zuboff, S., 1988,In the Age of the Smart Machine: The Future of Work and Power, Basic Books: New York. 32 4.6 Straub, D. W., 1989, `Validating Instruments in MIS Research,'MIS Quarterly, June (13:2), pp. 147±169. 26 4.3 Malone, T.W., Grant, K.R., Turbak, F.A., Brobst, S.A. and Cohen, M.D., 1987, `Intelligent Information-Sharing Systems,'Communications
of the ACM, May (30:5), pp. 390±402.
34 4.25
Siegel, J., 1986, `Group Processes in Computer-Mediated Communication,'Organizational Behavior and Human Decision Process, (37:2), pp. 157±187. 37 4.1 Nunamaker, J.F., 1989, `Experience at IBM,'Decision Support Systems, (5:2), pp. 183±196. 24 4
K.A.
W
alstr
om,
L.N.K.
Leonar
d
/Information
&
Manageme
nt
38
(2000)
59±72
As shown in Table 9, 97 authors were involved in
writing the 76 publications. One author, Nunamaker,
was involved in four of the publications. Five authors,
DeSanctis, Huber, Ives, Porter, and Simon were
involved with three of the publications. Sixteen
authors were involved with two of the publications
and 75 authors were involved with one of the
pub-lications.
Of the 76 publications, 17 were published between
1955 and 1975 and 14 more were published between
1976 and 1980. As might be expected, over half were
published in the 1980s: 20 were published between
1981 and 1985 and 20 more were published between
1986 and 1990. The last ®ve were published between
1991 and 1995.
6. Discussion
6.1. The traditional classics
From the early books of readings and Culnan's
studies, 19 individual publications were identi®ed
as classic. This list includes:
1. Ackoff, 1967.
2. Alter, 1980.
3. Anthony, 1965.
4. Benbasat and Dexter, 1979.
5. Benbasat and Schroeder, 1978.
6. Churchman, 1968.
7. Churchman, 1971.
8. Churchman and Schainblatt, 1965.
9. Dearden, 1972.
10. Dickson, Senn, and Chervany, 1977
11. Davis, G.B., 1974.
12. Gorry and Scott Morton, 1971.
13. Keen and Scott Morton, 1978
14. Kriebel, 1972.
15. Lucas, 1975.
16. Mason, 1969.
17. Mason and Mitroff, 1973.
18. Zmud, 1978.
19. Zmud, 1979.
7. The new classics
From 1986 to 1995, a new-breed of classic IS
publication was identi®ed. Sixty-one publications
meet the classic criteria. The other 15 works, if they
continue to be cited at the same rate they have been
since their publication, will be new classics. Of
parti-cular interest was the ratio of 47 journal articles and 29
books. Books typically consist of many more pages
than articles.
The list of traditional classics combined with the list
of new classics represents 91 individual publications.
Identi®ed over about a 25 year period, approximately
four works per year were represented.
Table 8
Journal distribution of the most cited articles, 1986 to 1995
Journal Number of articles
MIS Quarterly 12
Management Science 11
Communications of the ACM 6 Harvard Business Review 5 Academy of Management Review 2 Sloan Management Review 2 Nine other journals Once each
Table 9
Authors of the top cited articles, 1986±1995
Author Number of articles
Nunamaker 4
DeSanctis 3
Huber 3
Ives 3
Porter 3
Simon 3
Davis, F 2
Dennis 2
Galbraith 2
Gallupe 2
George 2
Konsynski 2
March 2
Markus 2
Mintzberg 2
Olson 2
Rockart 2
Scott Morton 2
Sprague 2
Vogel 2
Wetherbe 2
Williamson 2
7.1. Super classics
From the classic publications, some can be
identi-®ed as `super classic'. de Solla Price [14] refers to the
concept but fails to provide a de®nition. The inference
was that a super classic had to remain classic over time
versus the `well-known phenomenon of papers being
considered obsolescent after a decade'. Some articles
meet this criteria of super classics. The citing of
Ackoff, 1967 has fallen off over the years (averaging
two citations per year from 1986 to 1995), however, its
commanding presence in the early years and from
1972 to 1985 makes it the leading candidate for the
®rst super classic in the IS ®eld.
Four publications were present as classic in the
®ndings from Culnan's studies and from the analysis
of citations from 1986 to 1995: Davis, 1974 (and
Olson, M.H., 1985); Gorry and Scott Morton, 1971;
Keen and Scott Morton, 1978; and Zmud 1979.
There-fore, a total of ®ve publications appear to meet the
criteria of super classic because of their sustained
citation visibility over time.
Other publications should perhaps be considered
super classic because of the number of citations over a
shorter period of time. This is also termed the
`imme-diacy factor' or `bunching' discussed by de Solla
Price. For these publications, if the total been spread
out over the 24 year period from 1972 to 1995, they
would have averaged four citations per year, or a total
of 96 citations. These eight publications were:
1. Churchman, 1971.
2. DeSanctis and Gallupe, 1987.
3. Ives, Olson, and Baroudi, 1983.
4. Ives and Olson, 1984.
5. Nunnally, 1967, 1978.
6. Porter, 1980.
7. Sprague and Carlson, 1982.
8. Thompson, 1967.
A review of Table 7 reveals several publications, such
as DeLone and McLean (1992), that may sustain
citation visibility over a longer period of time. They
are worthy of note for future development.
7.2. Uses for the findings
Findings from this study can be used to develop
reading lists for doctoral students and by IS
research-ers to identify gaps in their knowledge. It can also be
used as a surrogate for the entire IS ®eld, allowing
researchers to pre-test their theories on a small sample
of the most visible works before examining a greater
quantity of literature. For instance, a follow up of the
points of work and reference studies conducted by
Culnan and Swanson [9] and Cheon et al. [4] could be
conducted using the ®nding of this study.
Third, these ®ndings can be used to update past
studies which have identi®ed content patterns in the IS
®eld or methodological evolution of the ®eld. These
type of studies could be follow ups to the intellectual
structure mapping of MIS and intellectual
develop-ment studies conducted by Culnan. The underlying
disciplines impacting the IS ®eld can be analyzed by
examining the base discipline of these publications.
Invisible colleges (corresponding groups of active and
knowledgeable people in any particular research ®eld
corresponding [15]), can sometimes be identi®ed by
performing co-citation analysis using the 97 authors
who were involved with the 76 publications.
It also provides a means to examine how IS research
matches needs identi®ed by practitioners during the
same time period, in studies such as Neiderman et al.
[26] and Brancheau and Wetherbe [3]. Finally, further
study of these classic articles could reveal the
char-acteristics that make an article or book, a `classic'
(similar to [12,13]).
8. Conclusions
With all the new material being published, perhaps
some of the older works are being pushed aside and
not included in the curriculum of Ph.D. students and
young faculty. Identifying these `classics' allows the
neophyte to ®nd and ®ll the gaps in his or her
knowl-edge base. A greater understanding and appreciation
for the classic literature results in greater
understand-ing and comprehension of the current literature.
A citation analysis was performed on the 118,364
references attached to the 3752 articles published in
nine top IS journals during the time period from 1986
to 1995. While citation analysis is not a perfect means
of identifying classic publications, it provides an
extensive method of determining highly visible works,
or citation classics. Ninety-one were identi®ed by
combining the results of this analysis with previous
®ndings and 13 super classic publications were
iden-ti®ed.
References
[1] M.J. Adler, C. Van Doren, How to Read a Book: The Classic Guide to Intellectual Reading, Simon & Schuster, New York, 1972.
[2] M.J. Adler, P. Wolff, A General Introduction to the Great Books and to a Liberal Education, Encyclopedia Britannica, Chicago, 1959.
[3] J.C. Brancheau, J.C. Wetherbe, Key issues in informa-tion systems management, MIS Quarterly 11 (1), 1987, pp. 23±45.
[4] M.J. Cheon, C.C. Lee, V. Grover, Research in MIS Ð points of work and reference: a replication and extension of the Culnan and Swanson study. DATA BASE (1992) 21±29. [5] D.E. Chubin, A.L. Porter, F.A. Rossini, `Citation classics'
analysis: an approach to characterizing interdisciplinary research, Journal of the American Society for Information Science 35 (6), 1984, pp. 156±172.
[6] R.J. Coleman, M.J. Riley, MIS: Management Dimensions, Holden-Day, San Francisco, CA, 1973.
[7] M.J. Culnan, The intellectual development of management information systems, 1972±1982: a co-citation analysis, Management Science 32 (2), 1986, pp. 156±172.
[8] M.J. Culnan, Mapping the intellectual structure of MIS, 1980±1985: a co-citation analysis, MIS Quarterly 11 (3), 1987, pp. 341±353.
[9] M.J. Culnan, E.B. Swanson, Research in management information systems, 1980±1984: points of work and reference, MIS Quarterly 10 (3), 1986, pp. 289±301. [10] G.B. Davis, The knowledge and skill requirements for the
doctorate in IS, in: Proceedings of the First International Conference on Information Systems, 1980, pp. 174±186. [11] G.B. Davis, G.C. Everest, Readings in Management
Informa-tion Systems, McGraw-Hill, New York, 1976.
[12] M.S. Davis, `That's classic!' the phenomenology and rhetoric of successful social theories, Philosophy of Social Science (1986) 285±301.
[13] M.S. Davis, `That's interesting!' towards a phenomenology of sociology and a sociology of phenomenology, Philosophy of Social Science (1971) 309±344.
[14] D.J. de Solla Price, Little Science, Big Science, Columbia University Press, New York, 1963.
[15] D.J. de Solla Price, Ethics of scientific publication, Science, 147 (1964) 655±657.
[16] D.J. de Solla Price, Networks of scientific papers, Science, 149 (1965) 510±515.
[17] G.W. Dickson, Management information systems: evolution and status, Advances In Computers 20, 1981, pp. 1±37. [18] V.T. Dock, V.P. Luchsinger, W.R. Cornette, MIS: A
Manage-rial Perspective, Science Research Associates, Chicago, 1977. [19] J. Erskine, The Delight of Great Books, The Bobbs-Merrill
Company, Indianapolis, IN, 1928.
[20] C. Fadiman, The Lifetime Reading Plan, 3rd Edition, Harper & Row, New York, 1988.
[21] E. Garfield, Mapping the world of science: is citation analysis a legitimate evaluation tool? In: D.N. Jackson, J.P. Rushton (Eds.), Scientific Excellence: Origins and Assessment, Sage, Newbury Park, CA, 1987.
[22] M.L. Gillenson, J.D. Stutz, Academic issues in IS: journals and books, MIS Quarterly 15 (4), 1991, pp. 447±452. [23] B.C. Hardgrave, K.A. Walstrom, Forums for MIS scholars,
Communications of the ACM 40 (11), 1997, pp. 119±124. [24] C.L. Hind, One Hundred Best Books? G.P. Putnam's Sons,
New York, 1928.
[25] J. Lee, Have You Read 100 Great Books? Jasper Lee Company, New York, 1946.
[26] F. Niederman, J.C. Brancheau, J.C. Wetherbe, Information systems management issues for the 1990s, MIS Quarterly 15 (4), 1991, pp. 475±495.
[27] C. Oppenheim, S.P. Renn, Highly cited old papers and the reasons why they continue to be cited, Journal of the American Society for Information Science 29, 1978, pp. 225±231. [28] D.H. Sanders, Computers and Management: In a Changing
Society, 2nd Edition, McGraw-Hill, New York, 1974. [29] D.H. Sanders, Computers and Management, McGraw-Hill,
New York, 1970.
[30] K.A. Walstrom, B.C. Hardgrave, R.L. Wilson, Forums for management information systems scholars, Communications of the ACM 38 (3), 1995, pp. 93±107.
Kent A. Walstrom is an Associate Professor of Information Systems at Illinois State University. He received his PhD from Oklahoma State Univer-sity. Dr. Walstrom's research interests include executive information systems, the history of the information systems field, electronic commerce, and the use of information systems to support leadership. Previous publications include articles inCommunication of the ACM, Computers and Operations Research, Information & Management,
Journal of Computer Information Systems, among others.
Lori N.K. Leonard is an Assistant Professor of Management Information Systems at the University of Tulsa. Dr. Leonard received her PhD from the University of Arkansas and is an active member of the Decision Sciences In-stitute. Her research interests include electronic commerce, electronic data interchange, ethics in computing, simu-lation, and data warehousing. Her pub-lications have appeared in Journal of
Computer Information Systems, Proceedings of the National
Decision Sciences Institute,Proceedings of the Decision Sciences
Institute Southwest Region, and Proceedings of Americas
