on three sessions. The first session discussed the three major components on the Internet: search engine ratings and features, information covered on the Internet, and the growth of users. The second session covered the tools for Web retrieval, which consisted of both traditional retrieval tools and new generation tools. The third session pointed out the future directions of Web retrieval. According to Kobayashi and Takeda, intelligent and adaptive Web services are the future direction.
Evaluation of Web search engines is the essential component of research on search engines. Oppenheim, Morris, McKnight, and Lowley (2000) reviewed the litera- ture of the evaluation of Web search engines, mainly emphasizing methodologies for evaluation and the actual evaluation criteria. The problem for evaluation is there are no standard tools developed for the evaluation of Web search engines.
Su (2003a) reviewed relevant literature from 1995-2000 for the development of a model of user evaluation of Web search engines. The proposed model focuses on performance measures associated with both users and systems and nonperformance characteristics related to users. She found there was a lack of evaluation from the end-user perspective.
Interaction.Studies
Interaction studies in Web search engine environments can be classified into the following categories: (1) levels of user goals/tasks, (2) usage pattern: patterns of query formulation and reformulation, (3) patterns of multimedia IR, (4).information search behaviors/strategies of different user groups, (5) the impact of knowledge structure, (6) criteria for the evaluation of Web search engines, and (7) comparison with other online IR systems.
Interactve IR n Web Search Engne Envronments
of simple classifications. In the hierarchy, informational searches were refined to have a series of subgoals: directed, undirected, to get advice, to locate information, and to obtain a list. Transactional searches were renamed “resource searches,” as the underlying user goal is to obtain a resource, such as to download a file. The richness of user goals in retrieving information requires Web search engines to have corresponding interfaces. However, these studies limited user goals only to the current search goal level.
Users not only have diverse goals for their current searches, but they also hold levels of goals in the Web environment. In her Web searching at-home study, Rieh (2004) validated Xie’s (2000) four levels of user goals in the Web environment: long-term goals (e.g., gain knowledge, professional achievement, etc.), leading search goals (prepare for an event, prepare for an online class, plan for a vacation, etc.), current search goals (look for papers, products, hotels, etc.), and interactive intentions (locate, find, read, etc.). The levels of user goals also impose a goal structure in that higher levels of user goals have an impact on lower levels of user goals. Furthermore, the findings of this study indicated that people in a Web-searching environment engaged in all four levels of goals, and they had more diverse tasks in the Web-searching environment than in the work places identified by Algon (1997) and goals in the libraries discussed by Xie (2000). In this environment, users sometimes looked for information just for curiosity or for entertainment purposes.
Researchers have also examined the impact of levels of goals and other factors on Web searching. Based on observation and interviews with 31 participants’ Internet and Web online catalogue searching, Slone (2003) examined how three levels of goals—broad or situational, specific, and format—plus age differences influenced search approaches. Broad goals represent the situations that lead users to search, such as educational, recreational, personal, and so forth, and they have an impact on other goals. Specific goals are related to what users search for, such as a specific subject, known organization, and so forth. Format goals are associated with the types of information users want, such as full-text articles, images, e-mail, and so forth. The findings showed that children and adults older than 45 presented similar search approaches. One possible reason is that recreational goals were identified more by children while personal goals were highly related to older adults, and both of these goals were found less motivating than educational or job-related goals.
Another significant finding is that the homogeneity of user goals is affected by age group. Children (recreational goals) and adults older than 45 (personal goals) have homogeneous user goals, but the age groups of 18 to 25 years, 26 to 35 years, and 36 to 45 years all have multiple goals within a group.
Task, another term related to user goals, is an important variable that affects users’
behaviors and outcomes. Bilal (2002) compared children’s behavior and success on three tasks: assigned fact-finding tasks, assigned research-oriented tasks, and self- generated tasks. Fifty percent of the children succeeded on the fact-finding tasks, 69% partially succeeded on the research-oriented tasks, and 73% succeeded on the
fully self-generated tasks. The results indicated that children were more successful on the fully self-generated tasks than the other two types of tasks. Their success on the fully self-generated tasks was attributed to the simplicity of the topics, their ability to modify the topics as they needed to, and their motivation in pursuing topics of interest. Children also exhibited different behaviors for different types of tasks.
They performed the highest analytic searches on fact-based and self-generated tasks and the lowest analytic searches on research-based tasks. They used more natural language queries on fact-based tasks, less on research-based tasks, and none on the fully self-generated tasks. They browsed more and made more moves on the fully self-generated tasks than other two tasks. They looped and backtracked more searches on the fact-based tasks than other tasks. To sum up, tasks influence users’
search behaviors and performance.
Schacter, Chung, and Dorr (1998) found a similar difference in their study between ill-defined tasks and well-defined tasks, which are comparable to research-based tasks and fact-based tasks. Children performed better on ill-defined tasks than well- defined tasks, because ill-defined tasks require fewer analytical strategies. Children employed more analytic behaviors in achieving the well-defined tasks than in fulfill- ing the ill-defined tasks. The only difference is that Schacter et al. discovered that children overwhelmingly used browsing strategies regardless of their tasks. Ford, Miller, and Moss (2002) examined the relationships between tasks and system per- formance. Even though the selected two tasks all fell into the category of fact-based tasks, they represented tasks with different levels of difficulty. The results showed that simpler tasks correlated significantly with higher relevance scores. The findings of this study echoes the results of Bilal and Schacter et al.’s studies that retrieval performance is affected by task differences.
Not only tasks but also the interactions between tasks and other variables have impact on Web search activities. Kim and Allen (2002) explored the cognitive and task influences on Web search activities and outcomes based on two experiments.
The results showed that tasks had a significant effect on search outcomes as well as search activities. Relatively high precision and recall were related to known-item tasks, which is comparable to the results of previous studies. The interactions amoung task effects, cognitive abilities and problem-solving styles influenced the number of searches completed, sites viewed, keywords searched, and bookmarks made. The interaction effect indicated that compared with other IR system environments, the Web is more flexible for users to choose different search tools for different tasks.
Navarro-Prieto, Scaife, and Rogers (1999) associated tasks, search conditions, and levels of experience with users’ search strategies. They found that users’ cognitive strategies were affected by types of task (fact-finding and exploratory), search con- ditions (whether the information they looked for was in Web-dispersed structure or category structure), and levels of users’ search experience. The type of task had a strong influence on the experienced users’ search strategies. For example, in the Web-dispersed structure, experienced users took a bottom-up strategy or chose a
Interactve IR n Web Search Engne Envronments
mixed strategy at the beginning, and selected a bottom-up strategy later for the specific fact-finding task. Simultaneously, they chose a top-down strategy for the exploratory task. The interactions among multiple variables make it difficult for researchers to uncover the relationships between tasks and searching behaviors.
Further research is needed to reveal direct relationships between tasks and search behaviors/strategies.