In the process of interaction with online databases, users’ knowledge structure affects their online searching. Domain knowledge has an impact on users’ search strategy and tactics. Shute and Smith (1993) examined domain knowledge’s impact on on- line searching. An expert intermediary made extensive use of domain knowledge to generate suggestions for refining a topic; the intermediary also frequently applied
Interactve IR n Onlne Database Envronments
knowledge-based search tactics in each search. In the expert’s knowledge-based sug- gestions, 80.3% were generated spontaneously, which is more than another nonexpert intermediary’s 47.8% suggestions. The results of the study help researchers to model cognitive processes of searchers, and further offer implications for computerized intermediary systems that suggest topic refinement for information seekers. Shiri and Revie’s (2003) findings are partly consistent with Shute and Smith’s (1993) results that topics identified as moderately or very familiar were connected with more cognitive and physical moves than topics identified as unfamiliar.
Domain knowledge has different effects on users with different levels of information retrieval knowledge. Hsieh-Yee (1993) further investigated the effects of subject knowledge on search tactics of novice as well as experienced users. After analyz- ing data collected by protocols, transaction logs, and observation, she found that subject knowledge affected experienced searchers’ search tactics but not novice users’ search tactics. In other words, subject knowledge does not have an effect on searching until only after users become experienced users. At the same time, searchers’ experience affected their search tactics differently depending on whether they searched questions in their subject area or not. She suggested a new interface facilitating novice searchers’ search style and promoting system features, for ex- ample, prompting features to guide novice users using system features. She also posed a question about system design: “Should an interface be designed to such a way that little adjustment would be required of its users? Or should it be designed to change their behavior as painlessly as possible?” (p. 170). However, not every study showed the effects of domain knowledge on online searching. For example, Wildemuth, de Blieck, Friedman, and File (1995) found that personal domain knowledge has little relationship to search proficiency, such as search results, term selection, efficiency, and so forth.
Information retrieval knowledge is also essential in online searching. The experi- ence of online searchers also determines their behaviors and performance. Howard (1982) found that the most experienced group performed the most cost-effective searches and achieved the highest precision ratio. Siegfried, Bates, and Wilde (1993) discovered that scholars had a high level of competence in searching Dialog after one day of training. Sutcliffe, Ennis and Watkinson (2000) discovered the marked differences between novices’ and experts’ search behavior, especially in query construction. After monitoring a group of law students searching QUICKLAW, Yuan (1997) found that searching experience affected end user behavior, such as the increase in participants’ set of commands and features used, increase in search speeds, change of learning approaches, and so forth. It is an effective approach for enhancing online IR systems by incorporating expert knowledge into systems.
Fidel (1991) explored the process of search-key selection based on actual searches performed by professional online searchers. She developed the selection routine, which is a decision tree that searchers intuitively use when they select search keys.
The selection routine was determined by two criteria: 1) whether a term is a common
term or a single-meaning term, and 2) whether a term can be mapped to a descriptor.
She suggested the incorporation of the selection routine into the knowledge base of intermediary expert systems.
One type of knowledge is not enough for users to effectively interact with online databases. Both domain knowledge and information retrieval knowledge are needed for online searching. Marchionini, Dwiggins, Katz, and Lin (1993) analyzed the roles of domain and search expertise in online information-seeking. A series of studies was conducted in searching hypertext or full-text CD-ROM and involving professional search intermediaries and domain exerts from computer science, busi- ness/economics and law. These studies demonstrated that information-seeking is a problem-solving process. It requires both domain and search knowledge. While domain knowledge helps experts quickly understand the problem and have clear expectation about possible answers, search knowledge helps professional search- ers develop a high level of expertise both conceptually and procedurally, enabling them to effectively retrieve information. The major contribution of these studies is that the findings unveil different roles domain knowledge and search knowledge play in users’ information retrieval process. More research needs to explore when and how users need different types of knowledge and the interplay among different types of knowledge.
Searcher.Characteristics/Cognitive.Styles/Search.Styles
There is no agreement as to whether user characteristics affect their behavior and search performance. Harter (1984) pointed out there were wide differences in terms of online searchers’ attitudes as well as behaviors. In early research, mathematical ability was found to be correlated with the ability to search interactively or with better search performance (Davis, 1977; Vigil, 1983). The reason might be that at that time, the design of online databases was more for expert intermediaries instead of end users. Bellardo (1985) investigated attributes of online searchers and their relationships to search outcome. The results indicated that verbal and quantitative GRE scores are predictors of searching skill, but only to a small extent. She raised doubts about whether searching performance can be predicted or determined by users’ cognitive or personality traits.
Among all the personal characteristics, cognitive styles and search styles were the characteristics that had most impact on searching. Cognitive styles affect users’ interac- tions with IR systems; to be more specific, information-seeking behavior and search performance. Cognitive styles are defined as “tendencies displayed by individuals consistently to adopt a particular type of information processing strategy” (Ford et al., 2002, p. 728). After correlating cognitive style measures with 111 postdoctoral
Interactve IR n Onlne Database Envronments
researchers’ perceptions of their problem-solving and information-seeking behavior and with those of the search intermediary who performed searches for them, Ford et al. (2002) found that field-independent users took a more analytic and active ap- proach in retrieving information than field-dependent ones. Simultaneously, holists exhibited more exploratory and serendipitous behavior than serialists, who might prefer a step-by-step approach in seeking information. The results of the study help the development of models of interactive IR and design of interactive IR systems to facilitate users with different types of cognitive styles.
The findings of this study are consistent with a previous study of searching CD ROM databases, in which Ford, Wood, and Walsh (1994) also found that cognitive styles (global/analytic) were highly related to search behavior; specifically, global users employed more broad search strategies than analytic users. Cognitive styles influence search behavior as well as perceived search performance. For example, in a study of undergraduate students’ online searches of CD ROM databases, us- ers’ cognitive styles (global/analytic) were found to be associated with levels of satisfaction with search results and perceived search success (Wood, Ford, Miller, Sobczyk, & Duffin, 1996).
According to Bellardo (1985), “interactive” and “fast batch” are the two types of searching styles that are the subject of investigation in early research. Many of the searchers were fast batch searchers who made little use of the interactive capabili- ties of online systems. They did not reformulate queries, nor did they browse titles of retrieved documents for relevancy (Fenichel, 1981; Oldroyd & Citroen, 1977).
These studies explored the search styles of users, but they did not further analyze characteristics of search styles. After analyzing 47 professionals performing job-re- lated searches, Fidel (1991) found that search styles, especially three characteristics of searching styles, have impact on searching behavior: the level of interaction dur- ing a search; the preference for types of moves, operational or conceptual; and the preference of type of search key, textwords, or descriptors. In particular, interactive searchers make more moves than less interactive searchers, but the level of interac- tion does not represent quality. Compared to conceptualist searchers, operationalist searchers use textwords more frequently, consult a thesaurus less, and make fewer recall moves. Textwords searchers are operationalist searchers, and do not use a thesaurus.
Cognitive styles and search styles are interrelated. Cognitive styles influence users’
search styles. The existing research has explored cognitive styles and search styles and their impact on search behavior and search performance. Few researchers have investigated the relationships between cognitive styles and search styles. In addition, each style has its value and problems. Thus, the question is whether the design of online IR systems should guide users to different styles or introduce different styles to users so that they can integrate them together.
Ease.of.Use.vs..User.Control.
Lancaster (1979) pointed out that ease-of-use is an important criterion for the selec- tion of an information retrieval system. Krichmar (1981) compared Dialog’s and ORBIT’s command language in terms of their ease of use from users’ attitudes and perceptions. His study is based on the following factors that define ease of use: the difficulty of recalling a command, the effort and frustration involved in entering a given command, the need to remember the sequence of argument values following a command, not completely understand the meaning of a command. The results showed that frustration with one or more important features of a system could have a negative impact on the perception of an entire system. Researchers have proposed measurable elements for ease of use, such as learnability, speed of user task per- formance, user error rates, and subjective user satisfaction (Hix & Hartson, 1993;
Shneiderman & Plaisant, 2004). However, research on the standard measures for ease-of-use is ongoing. Furthermore, ease-of-use is a complicated concept involving different tradeoffs (Thimbleby, 1990).
Not every user prefers ease-of -use. Different users have different requirements for what they need IR systems to do for them. Ease-of -use vs. user control becomes an issue more for online databases because these IR systems were traditionally designed for information professionals and only recently started being designed for end users. These systems have to take into account needs of both novice and expert users. Bates (1990), in her influential article, asked a reflective question about online systems: “What capabilities should we design for the system, and what capabilities should we enable the searcher to exercise?” (p. 576).
Influenced by this idea, Xie (2003) studied users’ evaluation of features of a variety of online databases in terms of ease-of-use and user control based on questionnaires, diaries, logs, and open-ended reports. The results showed that users considered both ease-of-use and user control as important for effective information retrieval. Us- ers’ requirements for ease-of-use and user control did change in the course of their interactions with the system and in the course of learning different systems. They needed more control after they had more understanding of IR systems and acquired more retrieval skills. The results also indicated that experienced users preferred more user control over novice users. While ease-of-use can mostly be achieved by system design, user control can only be accomplished by the collaboration between system design and user involvement. According to Vickery and Vickery (1993), user involvement is the decision that has to be made for interface design. Some interfaces only ask users for information statements, while others require users to be actively involved in the process of formulating search queries by providing guidance for users. More research is needed to define ease-of-use and user control from users’
perspectives, in particular from different types of user groups to examine whether users have same perceptions of ease-of-use and user control.
Interactve IR n Onlne Database Envronments
Evaluation.Criteria.for.Interactive.Online.IR.Systems
Relevance is a traditional measurement for IR system evaluation, and it is also a crucial measurement for interactive IR systems. However there are issues that need to be dealt with for relevance judgment during user-system interactions. First, it is difficult to control the situational dynamism of user-centered relevance estimation during the interaction between users and systems. In studying subjects’ engaging the LISA ondisc, Bruce (1994) identified a method to allow users to articulate the cognitive schema for estimating relevance at each phase of the IR interaction:
problem state, system interaction, and document interaction. This methodology pro- vides a mechanism for monitoring the impact of the IR interaction on user-centered relevance judgment. Second, it is difficult for users to have dichotomous choices for relevance judgment for interactive online systems. Researchers have defined the middle range of relevance to cover partially relevant and partially not relevant in addition to relevant and not relevant based mainly on what is missing and what is present by users (Greisdorf & Spink, 2001; Spink & Greisdorf, 2001; Spink, Greisdorf, & Bateman, 1998). After analyzing 32 users’ searching and evaluating results derived from Dialog, Greisdor (2003) suggested that the relevance judgment process is a problem-solving and decision-making exercise involving cognitive ac- tivities. According to Greisdor (2003), users went through a multiple-stage process of relevance evaluation during IR system interaction, and considered the topicality, pertinence, and then utility of a retrieved item in relevance judgment. Not on topic, not pertinent, not useful, and useful can be associated to not relevant, partially not relevant, partially relevant, and relevant, respectively.
IR system evaluation is a crucial component of IR research. The key question is what the unique criteria for evaluating interactive IR systems are. Su (1992, 1994) conducted a study to identify appropriate measures for evaluating interactive in- formation retrieval. After analyzing the data from 40 users’ interactions with six professional intermediaries searching large online systems, she tried to identify the best evaluation measures for interactive IR performance. The results revealed that value of search results is the best single measure for IR performance. Users’
satisfaction with search results and users’ satisfaction with precision of the search were strongly correlated with value of search results. However, precision is not sig- nificantly correlated with success. To users, recall is more important than precision.
There are several reasons for this: first, high precision does not mean high quality, and users’ satisfaction with precision is a better indicator of IR performance. Second, users’ tasks that lead them to look for information also affect whether recall is more important to them. The high percentage of users in this study that require complete information to accomplish their tasks (e.g., dissertation/thesis, grant application, etc.) also influences the result. Users’ satisfaction with the completeness of the search results, users’ confidence in the completeness of the search results, and users’ satis- faction with the precision of the search may serve as good measures of interactive
search performance. Both interaction and effectiveness factors are important in IR evaluation, and interaction factors are more important than effectiveness factors.
In addition, time is a significant factor of success.
Su’s findings demonstrate that relevance is not the only measurement for IR system evaluation. Her identified measurements were partly verified by other studies. Hersh, Pentecost, and Hickam (1996) compared two commercial MEDLINE systems by applying a task-oriented approach to IR system evaluation, including measuring success at answering questions, user certainty in answering questions, time to answer questions, ability to find relevant articles, and satisfaction with the user interface.
They concluded that the task-oriented approach was an effective evaluation method for assessing IR systems in terms of whether these systems can be used to solve real information problems. In their large-scale study, Saracevic and Kantor (1988b, 1988c) discussed the five utility measures (worth scale, user’s time, dollar value assigned, problem resolution scale, and satisfaction scale) as effectiveness measures for IR systems in addition to precision and recall, especially their relationships with relevance odds. They found that when relevance and precision odds increased, users considered the results to be worth more time, to have high dollar values, to make a high contribution to the problem solution, and to provide a high level of satisfaction..
One utility measure is related to recall odds. When recall odds increased, less time was taken for users to evaluate results. The major contribution of this study is the identification of the utility measures and their relationship to relevance odds. Although researchers used different terms to name evaluation criteria, they identified similar key evaluation criteria. However, the identified evaluation criteria mainly focused on the search performance of online systems, they failed to assess the user-system interaction process in online searching.
Summary
One unique phenomenon in online database environments is that intermediary studies have accounted for a large portion of the interactive studies mainly be- cause professional intermediaries were the main searchers of online databases before the emergence of the Web. The cost and complexity of command language have contributed to the problem. At the same time, intermediary studies can shed some lights on how users interact with searchers, online systems, and documents.
In online environments, intermediary studies have contributed to the research on domain knowledge’s impact on online searching by Shute and Smith (1993); types of interactive feedback by Spink (1997) and Spink and Saracevic (1998); cognitive styles affecting information seeking behavior by Ford et al. (2002); shifts in search problems/stages/focus by Robins (1997, 2000), Spink and Wilson (1999) and Olah (2005); intermediaries’ elicitation styles by Wu and Liu (2003); and evaluation criteria
Interactve IR n Onlne Database Envronments
for interactive IR systems by Su (1992, 1994). Many of these studies also suggest how to incorporate their findings into system design, specifically to implement the role of the intermediary into the design of online IR systems. Table 3.1 presents a summary of interaction studies in online database environments.
Task studies enable researchers to understand the impetus for information retrieval and to further develop theories of task-based IR process. The remaining question is: What is the relationship between tasks and user goals? User goals are also con- sidered the driving force of information retrieval, as discussed in chapter 2. Are tasks a part of user goals? How can tasks and user goals be defined? In addition, how can the complexity of tasks be defined? Are levels of task complexity different for different users, or is there a standard way to define them? What are the other dimensions of tasks that influence online searching? These questions need to be investigated further.
Levels of search strategies are the center of attention in interaction studies of online databases. Compared with OPAC studies, researchers have conducted more in- depth studies on search strategies, and have identified different types of micro- and macro-levels of strategies. However, the strategy studies are still on the level of the identification of the types of search strategies; they do not go further to explore what lead to the users’ application of different search strategies. In addition, researchers need to further examine the relationships among tactics, moves, and strategies. Are tactics and moves a part of strategies, and if so, how are strategies constituted by them? Identification of shifts in search strategies, stages, and foci is just the first step in understanding users’ information-seeking behavior during their interactions with intermediaries, IR systems, and information. In order to design IR systems to facilitate those shifts, we need to further identify the patterns between the shifts and the factors that lead to the shifts.
In general, researchers agree that domain knowledge and information retrieval knowledge affect users’ information-seeking behavior and search performance.
Expert users can make better use of domain knowledge than novice users. While providing term selection is a popular tool for assisting domain knowledge, offer- ing different interfaces for expert users as well as novice users is a suggestion for offering retrieval knowledge help. However, research on knowledge structure has not been incorporated into the design of Help systems for online databases. That is why online Help is inadequate in existing online systems (Trenner, 1989; Xie &
Cool, 2000). Further research needs to look into when and how users need differ- ent types of knowledge, and the interactions among different types of knowledge and their impact.
Although there is a disagreement about whether searcher characteristics affect search performance, researchers do agree that searcher characteristics, especially their cognitive styles/search styles, do influence searchers’ behavior. Interactive IR systems need to be designed to help users with different cognitive styles/search