It is the objective of universal text retrieval to enable users to effectively obtain the information they need without considering the language of the texts (Davis &
Ogden, 2000). Multilingual/cross-language retrieval becomes an important issue for IR in a variety of digital environments. However, the application of multilin- gual/cross-language in commercialized systems is rare (Gey, Kando, & Peters, 2005). The majority of research is still in its preliminary stages. In this section, the author discusses relevant works on interactive multilingual/cross-language retrieval research, mainly in the interactive track of Cross-Language Evaluation Forum (iCLEF). In iCLEF, researchers explore how to facilitate users to retrieve information in unknown languages, in particular in European languages. Users are at the center of the information retrieval process, especially when they have to search for information in languages that they are not familiar with. They need help in formulating and reformulating queries as well as identifying relevant documents from the retrieved results (Peters, 2005).
Query formulation and reformulation is one of the main research areas for interactive CLIR. It seems there are multiple approaches for researchers to deal with multilin- gual/cross-language retrieval issues on query formulation. Assisted translation is a major research area for interactive cross-language retrieval, in particular the query formulation process in which users are involved in the process. One approach is to offer online bilingual resources for improving query modifications. Davis and Ogden (2000) conducted a preliminary experiment on an interactive, cross-language text retrieval system that provides a browser-based interface for entering English
queries. A subject with no knowledge of German formed an English query based on TREC topics, and modified the German query by evaluating documents and using online bilingual resources. The results of the study showed that a bilingual dictionary was the main resource for query modifications. In this study, the subject interacted with online bilingual resources for query modifications. However, it was difficult to generate compound terms in German.
Lopez-Ostenero, Gonzalo, and Verdejo (2005) reported their results on using noun phrase for query formulation, translation, and modification based on their iCLEF 2002 experiment. The results revealed that phrase-based summaries performed better in assisting users to formulate and refine their queries than interactive word- by-word-assisted translation. Corroborating the quantitative results, the observation data revealed that users were unwilling to select translations for words that had different choices for the translation from the assisted translation system. While Lopez-Ostenero, Gonzalo, and Verdejo (2005) focused on assisted translation from interacting users to obtain a phrase-based query, Dorr et al. (2004) examined the assisted translation by selecting individual query terms based on three resources:
the document-language term, possible synonyms, and example of usage. After comparing users’ query reformulations under the automatic and manual conditions, Dorr et al. (2004) found that user-assisted translation selection for query terms was useful because it achieved the same search effectiveness with fewer query iterations compared with the automatic condition. One limitation is it did not have the same effect in query reformulation. This study also identified different search behaviors under automatic and manual conditions. Under the former condition, users’ tactics were similar to monolingual tactics. Under the latter, their tactics were varied and complicated.
Similar to Dorr et al.’s (2004) study, Petrelli, Demetriou, Herring, Beaulieu, and Sanderson (2003) examined two different levels of control over the query transla- tion mechanism with four subjects: delegation and supervision. When users input queries, the system translated queries. No user interventions were involved in the delegation condition, but users verify and modify queries in the supervision con- dition. Even though users found more relevant documents when they had greater control over the translation, the results also found differences among users, topics, and tasks. Petrelli, Levin, Beaulieu and Sanderson (2006) enhanced their previous research with 16 subjects involving four different languages pairs (Finnish to English, English to Finnish, Swedish to English, and English to Swedish) to further explore which interaction model should be used in cross-language retrieval. Interestingly, the performance data and user feedback did not correspond to each other. The results of this study showed that supervised mode performed better than delegated mode in both precision and recall, although the difference was small. At the same time, user feedback revealed that users preferred the delegated mode, but the difference was not big either. That echoed their previous research that users favored the simplest interaction, if they were happy about their retrieved documents with their initial
TREC and Interactve Track Envronments
queries (Petrelli et al., 2004; Petrelli, Hansen, Beaulieu, & Sanderson, 2002). Us- ers had different opinions toward the delegated and supervised modes. While some participants favored the delegated mode because it offered speed and less effort, the other participants liked the supervised mode for their ability to check and update the query translation process as well as to get inspiration for query reformulations.
To balance users’ preferences, the solution is to take the delegated mode as the default, but also provide query translation on top of the result list, enabling users to supervise the translation.
Document selection is another important area for CLIR research. Researchers have conducted a series of experiments to compare different techniques for facilitating interactive relevance judgment. At three sites with about 20 subjects participating in the main experiments, Oard, Gonzalo, Sanderson, Lopez-Ostenero, and Wang (2004) compared three techniques for document selection: full machine translation, rapid term-by-term translation, and focused phrase translation. The results showed that machine translation performed better in supporting relevance judgment tasks than term-by-term translation, while focused phrase translation enhanced recall. The subjects of the study reported that it was easy to accomplish relevance judgment tasks with the machine translation system, and phase translation required more user interpretation for the same task. Lopez-Ostenero, Gonzalo, and Verdejo (2005) compared the standard Systran translations with phrase-based translations in sup- porting document selections. The quantitative data and user feedback indicated that noun phrase translation summary was a valuable feature in supporting relevance judgments. Moreover, it was cheaper to generate noun phrase translations than the full machine translation. In addition to phrase translation, thumbnails were also used to assist making relevance judgments.
In Davis and Ogden (2000)’s study discussed above, the subject examined the re- trieved documents in thumbnails and German equivalents, and then submitted them to be translated into English. The top 10 documents were judged as relevant or not- relevant. The results of the study showed a low percentage of error. Dorr et al. (2004) explored how the two approaches supported users in recognizing relevant documents:
one extracts the first 40 translated words in each news story, and another one uses an automated parse-and-trim approach to generate headlines. Overall, it was easier for users to make relevance judgments in using the first 40 words approach than the headline approach, and therefore they were more confident in making relevance judgments. However, the researchers did point out that the headlines generated in this study could not represent the informative summary in general. In addition, the headlines generated in this study were shorter than 40 words.
Tasks play important roles in interactive IR as well as in interactive multilingual IR. Zhang, Plettenberg, Klavans, Oard, and Soergel (2007) explored subjects’
task-based interaction with an integrated multilingual and multimedia IR system.
Eight participants were involved in different types of search tasks, and multiple methods were applied to collect data. The results of the study demonstrated that
tasks did have an impact on the performance of users’ multilingual retrieval. In general, users were able to obtain answers for factual questions. However, they had great difficulty in searching for high-level questions related to opinions and reactions because they could not develop search strategies that worked well with the multilingual IR system. This study also yielded some unique characteristics of users’ information-searching behavior for multilingual retrieval. For example, users broadened their searches instead of applying specific query terms because of their ineffectiveness in multilingual IR. Working on the same types of tasks, He, Wang, Luo, and Oard (2005) compared two types of summarizations for answering factual questions: Keyword-In-Context (KWIC) summary and passage summary for CLIR.
The results showed there was little difference between the two types of summaries for this type of task in an experiment with eight subjects. However, the difficulty of the task did affect CLIR. To be specific, the time spent on the task and the number of query iterations was correlated with question difficulty.
Summary:.Impact.and.Limitation.of.TREC...
Interactive.Track.Studies
The TREC environment provides a platform for researchers to compare results and their experience. The TREC Interactive Track has made significant contributions to research on interactive information retrieval:
1. The major contribution of the TREC Interactive Track is the development of a general framework for the investigation of interactive information retrieval, and for the evaluation and comparison of the performance of interactive IR systems (Dumais & Belkin, 2005). This framework includes the applied methodologies, the experimental designs, and the techniques for reporting the evaluation and comparison results.
2. The interactive track encourages researchers, and, more importantly, offers an opportunity for researchers to share common tasks, topics, document collec- tions, evaluation methods, and their experience in interactive IR research.
3. In this environment, different aspects of an interactive retrieval process can be controlled to a certain degree in order for researchers to understand how user-system interactions affect the retrieval outcome (Yang, Maglaughlin, &
Newby, 2001). The controlled environment enables researchers to analyze the key relationships in interactive IR.
4. In the Interactive Track, each team conducted a series of studies along the way. More important, their latter studies are built on their prior results and