Chapter 2 LITERATURE REVIEW
2.3 Knowledge Representation with OntologIES
2.3.5 Ontology Evaluation
There are several existing methods for evaluating ontologies, some of which are part of the overall methods of building an ontology. A systematic review has been done on the existing methods in an attempt to consolidate them for the development of an appropriate ontology evaluation framework [58], [59]. A comprehensive analysis of existing methods was carried out by [58] in order to develop a theoretical framework that will include the advantages of the existing approaches. Also, [59] made an improved attempt to develop a framework from the assessment of existing approaches, including the framework by[58]. Pak and Zou [59] posit that their proposed framework will provide better theoretical understanding of ontology evaluation, and serve as guidance for ontology evaluation. A recent effort in developing a comprehensive approach to evaluation is provided in [55] through the introduction of a process for evaluating the syntactic, semantic and pragmatic quality of an ontology.
It is important to note that the knowledge of a domain expert is required for ontology evaluation.
Pak and Zhou [59] De Nicola et al [55] and Gangemi et al [58] recognise that the involvement of experts in evaluation processes is highly significant. The knowledge of an expert in specifying the concepts is useful for assessing the coverage and correctness of the ontology in representing the application domain. Even in automated approaches, experts are required to provide contexts and meanings for terminologies. Domain experts and knowledge engineers’ input are vital in evaluating some quality attributes, such as clarity, navigability and expandability of an ontology, which can be difficult to evaluate through quantification methods.
2.3.5.1 OQual Ontology Evaluation Methodology
A comprehensive effort for the development of a theoretical framework for ontology evaluation was first made by [58]. Consideration was given to some existing qualitative and quantitative measures of ontologies from previous works before their integration into a single framework. The
framework was developed into a formal model that consists of a meta-ontology O2 and an evaluation and validation method oQual. O2 characterizes ontologies as semiotic objects while oQual complements the meta-ontology evaluation and validation and is implemented as a diagnostic task over ontology elements, processes, and attributes; it also evolves. Based on the models, three main types of measures for ontology evaluation are identified in the framework:
structural, functional and usability-related measures. The structural measure is related with the topological and logical properties of an ontology, which may be measured by means of a metric measure. Functional measures relate to the intended use of a given ontology and of its components and the usability measure relates to the level of annotation of a given ontology, the ease of identifying the properties and its suitability.
2.3.5.2 Evaluation with the UPON Methodology
The UPON methodology provides an evaluation process in order to verify and validate the completeness and correctness of an ontology in the application domain. The process is classified into four quality assurance measures: syntactic, semantic, pragmatic and social quality assurance measures [55]. Syntactic quality measures assure the quality of an ontology in respect of the formal style in which it is written. This is verified in the implementation workflow through the choice of quality software for the ontology development. The semantic quality is concerned with the presence of contradictions of concepts. The pragmatic quality relates to the measurement of the quality of an ontology’s content and its usefulness for users, irrespective of its syntax and semantics. Lastly, social quality assures the general criteria of an ontology’s acceptance, access and usage. This is more applicable during the publication of an ontology for use and can be measured through the assessment of access and usage of the ontology in real time.
Semantic and pragmatic quality are explicitly assessed in the test workflow package in UPON methodology. While semantic quality can be mainly verified by checking the consistency of the ontology using a reasoner from the tool employed in ontology implementation. Pragmatic quality
is related to fidelity, relevance, and completeness of the content of the ontology considering its requirements and goals [55].
In the test workflow package, the ontology is to be checked against the requirement and need for developing the ontology in order to ascertain its competence. This is based on the settings of CQs as criteria for evaluating the initialised goals of the ontology at the requirement workflow. The testing of these questions is carried out at the test workflow at which the use cases are executed and queries are built for extracting knowledge from the ontology. The output of the querying process is used as a validation for the CQs and decides if the ontology is fit for the purpose for which it was designed. One other process included in ensuring the competence of the ontology is by involving the DE in evaluating the extent to which the ontology has answered the CQs.
2.3.5.3 Other Approaches to Ontology Evaluation
A mixed method for evaluating ontologies was suggested by Bilgin et al [60] as most appropriate because most existing ontology evaluation methods focus on functionality-related issues rather than structural ones; very few focus on the structure of the ontology. This may be due to the importance of the ontology usage as, no matter how good the structure of an ontology is, its significance in the domain is highly dependent on its functions, which are closely related to its usability. Furthermore, the functionality of an ontology is mostly measured by evaluating its appropriateness as the semantic backbone of either decision-support or information systems that operate in the domain represented by the ontology [60]. Therefore, the evaluation of an ontology in describing domain concepts should consider multiple approaches that focus on the functionality of the ontology.
Likewise, Pak and Zhou [59] reviewed several ontology evaluation approaches to propose a new ontology evaluation framework. They identified five dimensions for classification and assessment of existing evaluation approaches. These dimensions are scope layer, lifecycle, quality principles and methods. The dimensions are further utilised as guidelines for recommending suitable
approaches for evaluating an ontology. The ontology scope approach is useful for confirming appropriate implementation of the specification and design aspect of ontology engineering. The layer based approach to ontology evaluation provides a means of stratifying an ontology into several layers which can be examined within its context. These include lexicon/vocabulary, structure/architecture, representation/semantic and context/application. The ontology lifecycle approach proposes an evaluation that runs through the entire lifecycle of an ontology, from specification through conceptualisation to integration of an ontology into other ontologies.
Various processes of evaluation are applicable to different stages of the ontology’s lifecycle. The ontology quality principles approach focuses on the quality elements of an ontology; these are the consistency, completeness, conciseness and reusability of an ontology. Lastly, Pak and Zhou’s ontology evaluation methodology is concerned with the process of evaluating an ontology only, whether it be verification or validation that has been used to evaluate the ontology, irrespective of the method employed for the ontology construction.