TEXT REPRESENTATIONS AND THEIR USE
3. REPRESENTATIONS THAT CHARACTERIZE THE CONTENT OF TEXT
3.1 Set of Natural Language Index Terms
Indexing often consists of drawing natural language index terms directly from the document text (Lancaster & Warner, 1993, p. 80 ff.). This process is called extraction indexing. The index terms extracted are content terms in the form of single words or phrases (Harter, 1986, p. 42). The number of
terms extracted varies from a few ones to a large number, depending upon the need to more or less in detail represent the text's content (Salton, 1975b, p. 17). The index terms can have a weight indicating their importance in representing content (Sparck Jones, 1973). Full-text search (see chapter 1) is the simplest form of extraction indexing: Each word in the text can act as an index term.
Indexing with natural language terms has advantages and disadvantages (Blair & Maron, 1985; Harter, 1986, p. 51 ff.; Lancaster, 1986, p. 161 ff.;
Furnas, Landauer, Gomez, & Dumais, 1987; Salton, 1989, p. 276; Krovetz &
Croft, 1992). It has the advantage of being very expressive and flexible, of representing a variety of access points and perspectives of a text, and of easily representing new and complex concepts. The indexing vocabulary is less tightly controlled than controlled language index terms and a great variety of index descriptors is normally identifiable. Because of the lack of fixed index terms, the natural language index terms make a textual database portable and compatible across different document collections. There are however disadvantages. The words of the text have the property of being potentially ambiguous (e.g., homonyms). Index phrases are usually less ambiguous because each content word in the phrase provides the context for the others. Moreover, words and phrases of the text are often too specific in representing text content preventing generic searches of information in texts.
There is the difficulty of capturing the underlying concepts.
When extracting words and phrases from texts, a total lack of vocabulary control is rare, because different morphological variants of one term or different synonyms of one term are often replaced by one standard form (Lancaster & Warner, 1993, p. 84.). For instance, the least particular morphological variant (usually noun) of the terms selected is used.
3.2 Set of Controlled Language Index Terms
Assignment indexing is attributing terms to a document text from a source other than the document itself. The terms could be drawn from the indexer's head. More commonly, assignment indexing involves assigning terms or labels drawn from some form of controlled vocabulary (Lancaster, 1986;
Salton, 1989, p. 230; Lancaster, 1991, p. 13 ff.; Meadow, 1992, p. 68 ff.;
Lancaster & Warner, 1993, p. 80 ff.). The assigned terms are also called descriptors.
A controlled vocabulary is basically a predefined list of index terms constructed by some authority regarding the management of the document collection. The index terms of the list are single words or complete phrases.
Usually, the vocabulary is more than a mere list. It will generally incorporate
some form of semantic structure. Two types of relationships between index terms are commonly identified: the hierarchical and associative relationships. The set of controlled language index terms is called a classification system(Beghtol, 1986).
Indexing with controlled language index terms assumes a predefined long-term set of users’ interests (Belkin & Croft, 1992). Usually, the classification system provides a valid, often structured vocabulary for the subject content of a document collection. But, for a given document base many classification systems can be employed, possibly reflecting other aspects of the content than topicality. The classification system can vary in time and content. It always reflects a structure that for a given task hopefully over a long time is useful.
Common examples of classification systems are subject thesauri, broad subject headings, and classification schemes (Harter, 1986, p. 40 ff.). A thesauruscontains a variety of concepts, their equivalents, and related terms.
It contains the various surface forms of concepts in texts. Thesauri are usually derived from existing and growing collections of documents in one subject discipline. The vocabulary in a thesaurus is meant to address problems of synonymy and semantic ambiguity in these collections. Subject headings represent the structure of the topics of heterogeneous document collections. Another type of artificial language for document representation is the very broad classification scheme. An example hereof, is the Dewey Decimal Classification (DDC) used in the U.S. to classify books, which is an a priori representation of all human knowledge in a great hierarchy.
Indexing with controlled language index terms has advantages and disadvantages (Harter, 1986, p. 51 ff.; Lancaster, 1986, p. 161 ff.;
Svenonius, 1986).
The advantages especially regard the generality, the property of being unambiguous, and the preciseness of the terms. Regarding generality and the property of being unambiguous, the controlled language index terms control the variation of surface features for identical or similar concepts and thus deal with synonymy and other term relations and with semantic ambiguity (Blair & Maron, 1985; Furnas et al., 1987; Krovetz & Croft, 1992; Riloff &
Lehnert, 1994). Because they are unambiguous in meaning, they are readily translated in other languages for use in applications that retrieve texts across languages. Moreover, because the terms represent general access points to text classes, they are easily employed in generic searches (Harter, 1986, p.
41 ff.), in document routing and filtering according to general classes (Belkin & Croft, 1992), in linking texts (Agosti, 1996), or in constructing topic maps of texts (Zizi, 1996). An initial classification of texts often precedes an information extraction task, so that the correct set of class-
specific natural language processing techniques can be used (DeJong, 1982;
Young & Hayes, 1985; Liddy & Paik, 1993). Regarding preciseness, the controlled language phrases often function as precoordinated index terms that indicate and standardize specific relations between the content words of the phrases (Salton & McGill, 1983, p. 58; Soergel, 1994). For instance, a text can be indexed with the precise phrase "solvents, effects on color spectra of dyes". Controlled language index terms are useful when the texts can be represented by accurate and unambiguous concepts, irrespective of their being general or specific.
Controlled language index terms also have disadvantages. They permit only a few access points to a text or to represent a few perspectives.
Moreover, they are rather inflexible to adapt to the needs of the users of the texts. So, the vocabulary must be regularly updated to account for changes in interest and search concepts, and changing document collections. When the vocabularies are not interchangeable, retrieval systems based on controlled language index terms are less portable and less compatible across different collections. Controlled language index terms can complement the natural language index terms in a text representation (Hearst, 1994).
Relation with text classification and categorization
Indexing with a controlled language vocabulary is related to text classification (Lancaster, 1991, p. 14 ff.). The term “classification” refers to the process of grouping entities. Text classification refers to the formation of text classes that are conceptually closely related. The classes often contain texts that treat the same subject. The term “ textcategorization” is used for the classification of textual documents with respect to a set of one or more pre-existing category labels or controlled language index terms by which the classes are identified.
Class assignment is binary (a text is or is not a member of the class) or graded (a text has a degree of class membership) (Sparck Jones, 1973;
Cleveland & Cleveland, 1990, p. 112). The latter corresponds to the assignment of weights to the controlled language index terms.
Exactlyone descriptor ormultiple terms are assigned to one text. But, the number of terms assigned is usually restricted (Salton, 1975b, p. 17). When multiple terms are used, a text can belong to different related and unrelated classes. The index terms that are keys to these classes are dependently or independently assigned. The former is the case when, for instance, index terms of a hierarchical classification system are assigned: The assignment of one term involves the assignment of terms that are higher in the hierarchy.
The classes – even the ones of a same level in a hierarchical classification
system – are usually not mutually exclusive (Harter, 1986, p. 56). The division of the real world into genus, species, subspecies, etc. does not always result in distinct classes. Often a better indexing is obtained by independently assigning each index term, especially the ones of the same level of a hierarchy.
3.3 Abstract
Another important form of a text representation is the abstract or summary. A summary is a condensed derivative of the source text. A summary is concerned about content information and its expression. There are many different forms of summaries (Sparck Jones, 1993; Rowley, 1988, p. 11 ff.). Usually, abstract and summary are considered as synonyms and will be used likewise throughout this book. However, sometimes a slight distinction is made. Then, an abstract rather refers to a stand-alone document surrogate (e.g., abstracts in technical journal literature), while the summary is an inherent part of the document text, which stresses its salient findings.
A minimal function that a summary should provide is being indicative of the text's content. An indicative abstract helps a reader to decide whether consulting the complete document will be worthwhile. An informative abstract reports on the actual content of the text and presents as much as possible the information contained in it. Such an abstract can act as a stand- alone text surrogate. An extractis composed of pieces of text extracted from the original and may have an indicative as well as an informative function.
Sections or fragments of the text represent its content and/or its flavor, or highlight significant information. The latter type of abstract is called a highlight abstract. An abstract consisting of keywords serves as a crude indicator of the subject scope. The content of a text can be summarized in a profile. A profile is a frame-like representation containing distinct slots that each has a well-defined semantic meaning. The slots are filled with information from the text. A critical abstract not only describes a text’s content, but also evaluates its content and its presentation. A comparative abstractevaluates a text’s content and presentation with those of other texts, or represents the summary of multiple document texts.
The information content of an abstract is usually expressed in coherent text. As seen above, some abstract types present the information in other forms ranging from an extract and profile to a list of index terms.
An abstract is highly valued as a condensed and comprehensible representation of a text’s content. It is especially appreciated by human readers for assessing the relevancy of the original text (Rowley, 1988, p. 12 ff.).
Relation with text indexing
Text indexing and abstracting are closelyrelated(Lancaster, 1991, p. 5 ff.; Sparck Jones, 1993; Sparck Jones & Galliers, 1996, p. 28). The abstractor writes a narrative description of the content of a document text, while the indexer describes its content by using one or several index terms.
But, the many forms of abstracts make this distinction more and more blurred. A brief summary may serve as a complex structured index description, which provides access to the text collection, while a list of key terms may serve as a simple form of abstract. Many forms of text representations are intermediate forms of indexing descriptions and abstracts, Abstracts are supposed to be more exhaustive in representing content than an indexing description (Cleveland & Cleveland, 1990, p. 105;
Stadnyk & Kass, 1992).