Attribute Content and Values
4.3.4 Transformation of Graphics
We can search text for the occurrence of words, codes, or phrases, or can determine subject matter from word occurrences. But with today’s technology, we are not that far along in determining the content of graphic images. Pictures can be digitized in the sense that color or gray scale at specific positions within the picture can be represented digitally. Thus, we have a digital set of coordi- nates, x,y, and the color or gray scale represented as a number from 0 to n. But what do these pixels represent?
There are two ways of processing text that are far ahead of what can be done with graphics. In one, a searcher provides words and the search system converts these to related words by stemming, looking up in a thesaurus, or find- ing others that frequently co-occur. In the second, the system can form an abstract of the contents of the text. In graphics, we can do some of the first but not the second, except in very limited domains. In general, we must first describe content by use of text, and then search the text. Since different indexers may see quite different images, especially in non-representational art, the limitations are severe. A review of indexing methods for images is found in Rasmussen (1997).
In a few specialized applications, however, successful storage, search, and retrieval of graphic images is possible. The problem is to prepare a generalized rep- resentation of some object, then search a graphic file for occurrences of this pat- tern. For example, we could describe a cow graphically, then search for a match.
But, no standardized example of a cow exists to match every artist’s vision of one.
Fingerprinting is probably the outstanding example of successful graphic IR. Fingerprints are made up of a series of ridge lines in that portion of the skin called friction skin, occurring on the hands and feet. These ridge lines tend to form into general patterns known as arches, loops, and whorls as in Fig. 4.3a. In addi- tion to the general pattern, there are many fine details, called minutiae, that can be recorded, digitized, and used as the basis for IR, i.e., for finding a matching print in a file, given a print to use as the basis for a search. The print used for
88
4 Attribute Content and ValuesCh004.qxd 11/20/2006 9:54 AM Page 88
searching might come from a recently arrested person charged with a crime or from an accident victim for whom identification is sought. It also might come from a latent print found at the scene of a crime.
The minutiae recorded consist of such features as ridge endings, bifurcations (splitting of a ridge into two), islands, or lakes, illustrated in Fig. 4.3b. (Science of Fingerprints, 1979, p. 29). Islands are ridges of very short length, and lakes are formed when a ridge bifurcates, then rejoins. The original fingerprint filing and search systems concentrated on the overall patterns. The more modern systems also use minutiae. By plotting the location and type of the minutiae of a print, enough information can be recorded to distinguish it from almost all the other prints. It is not necessary that the mechanical search retrieve a unique match. It is good enough, even preferable, for the search to retrieve a modest number of possibly matching prints, so that a human expert can make the final decision about whether or not a match has occurred, and testify to it in court. We do about the same in text searching. We do not expect a retrieval system to get exactly the one item we are searching for; we are normally content to retrieve a small set from which we select the best.
Minutiae will vary, even on prints of the same person, taken at different times and under different circumstances. The skill of the technician in inking and impressing the hand, or scratches and the like on the fingers, can cause minor variations. Thus, the algorithm used by a computer for matching prints must be one that looks for a degree of match, not an exact match, between prints.
Essentially, it looks for a fuzzy set of matching prints.
This type of system works because, in spite of the complexity of the image, users are able to identify exactly what aspects of the image they are interested in,
4.3 Transformations of Values
89
Figure 4.3
Characteristics of fingerprints: some of the common general forms at a are: tented arch, loop, and whorl. Minutiae within a pattern are shown in b in which can be seen exam- ples of (1) a lake (a small white space surrounded by ridges), (2) a bifurcation, (3) an island (a short ridge, surrounded by white space), and (4) a ridge ending. (Photos courtesy of U.S.
Federal Bureau of Investigation.)
and then to devise a means of measuring and recording those aspects, namely minutiae and ridge patterns. With something like news photographs we can, at best, describe in words the principal object in view, or the place and date of the photograph. We cannot identify everything of possible future interest such as an interesting (as determined later) background object.
There are other examples of the need for searching and matching of graphic images. Again in police work, it would help to be able to match descriptions of a face with stored photographs. While police artists are sometimes successful in depicting a face based on eyewitness accounts, such descriptions are often unreli- able. Is there a method of automatically scanning a photograph to record features objectively that can later be used to search for a match, from a new photo or a description? Work is being done, but no solution is readily at hand at this time.
In trademark law it would help to be able to find any trademark matching an existing or potential one. When a company wishes to create and register a new trademark, it must conduct a search of previously registered marks to assure that no duplicate or near duplicate exists. Further, it is the responsibility of trade- mark owners to protect their rights from infringement by others. There are no trademark police impartially looking for violations; hence, many companies, in addition to verifying the lack of a match for a proposed new mark, routinely search for near matches to their existing marks, looking for possible violations.
But, again, there is no way at this time for a computer to match the graphics.
The searcher must be content with a search of a verbal or encoded description of a trademark, with the final matching being done by the searcher.
Web site search engine indexing concentrates on text in HTML format.
While Web pages may be highly dependent upon graphics, Java programs and mul- timedia presentations, these will not normally be indexed except by manual assign- ment of textual terms by some engines. However, work is being done in this area.
“Engineers at Purdue University are developing a system that will enable people to search huge industry databases by sketching a part from memory, penciling in mod- ifications to an existing part or selecting a part that has a similar shape.” (“Purdue engineers design ‘shape-search’ for industry databases,” Innovations Report http://www.innovations-report.de/html/berichte/informationstechnologie/
bericht-27629.html). Funkhouser at Princeton is at work on a 3D search engine that will support queries in sketch and 3D model forms. (Princeton Shape Retrieval and Analysis Group, n.d.)
There are other examples of the need for searching and matching of graphic images. Again in police work, it would help to be able to match a pic- ture of a face with a stored collection of photographs. Is there a method of auto- matically scanning a photograph to record features objectively that can later be used to search for a match, from a new photo or a description? Some success has been achieved, but no complete solution is at hand at this time.
Faces are not as well structured as fingerprints. The one point that is fairly easy to find is the pupil of an eye. From there, a program can search for the other eye, eye brows, a nose or nostrils, and a mouth. But these are never as well defined as fingerprint minutiae. The photographs are recorded as a matrix of pixels,
90
4 Attribute Content and ValuesCh004.qxd 11/20/2006 9:54 AM Page 90
usually each element representing the gray-scale value (0 = black, 255 = white).
One technique, called global, is to use a photograph as the query, comparing it with stored ones in a database by matching all the pixels without regard for what part of a face is being compared at any pixel. Potentially better is a method called local. It makes use of the facial features that have been detected and the distance and angles between them, somewhat as in fingerprint matching. But people change, shave or do not shave, dye their hair, turn grey, scowl or smile, and they are photographed under different lighting and posture conditions. The problem is not fully solved. (Bruce,1988; Chellappa et al. 1995, Eigenface Tutorial; 2006;
Harmon, 1973; Hjelmas and Low, 2001.)