Glossary of Terms
Chapter 3: Methods Review
3.2 Going from qualitative data to computer models
67
• Providing a common language to emerge from the interactions between diverse stakeholders and this may allow for further cross-boundary discussion and discourse;
• Allowing issues to emerge instead of being prescribed, as is the case when using a questionnaire.
Okoli and Pawlowski (2004) also describe how the Delphi method is related to a word in French, bricolage, which means ‘to use whatever resources and repertoire one has to perform whatever task one faces’. The Delphi method achieves this because of its flexible nature, and is hence a good starting point for finding the information and understanding that already exists.
There are a number of different methods for undertaking this type of group communication:
conference telephone call, committee meeting, formal conference or seminar, workshop, email or internet (Linstone and Turoff 2002; Rixon et al. 2007b).
68 to develop the computational model. It can be argued that by using the second approach, much of the richness of the qualitative information about stakeholders’ perceptions can be captured (Becu et al. 2003). Additionally, in situations where limited computational and cognitive capacities cause individuals to apply ‘bounded rationality’ using ‘fast and frugal heuristics’
(Gigerenzer and Todd 1999), the second method would appear superior in modelling human decision making. However, an advantage of the first approach is that models can be directly linked to theories in fields such as sociology, economics and psychology. The first approach also does not require as much effort in terms of knowledge engineering.
Knowledge engineering is mainly about the second type of model development described in Figure 3-1 and in this case, once qualitative information has been collected, it feeds into the task of creating conceptual models either formally, such as by using grounded theory, or informally by a combination of deduction and induction by the model builder (Siau and Tan 2005); and it may also be done with only personal agreement on the conceptual model, or with common (i.e.
group) agreement on the conceptual model (Siau and Tan 2005).
Figure 3-1: Second approach for eliciting and using qualitative data to develop models
In other words, with a typically low transaction cost, while a modeller may create a model independently using only his/her own intuition, deduction and induction; at the other end of the spectrum, a conceptual model may be generated based on a formal and documented process where a group voice is elicited and where it is assumed that the group is more likely to agree on
69 the language, and underlying assumptions and structures; allowing it to be used for collective decision making, as per the Companion Modelling framework (Barreteau 2003a).
3.2.1 Mental models
A key part of translating qualitative information to models is the representation of cognitive processes and human decision making. A common view in the modelling of human decision making is that each person has a different mental model, a systems representation, which is used to guide decisions and behaviour. Dray and colleagues describe what seems to be a common posture of this area:
We recognise the epistemic construction of knowledge and believe that the nature of individual representations is socially constructed through people's interactions with their physical and social environment. We agree on the fact that these adaptive mental models can be partly elicited through Knowledge Engineering-based techniques and translated into conceptual models [and], we assume that social groups carry collective representations (collective frames) of their environment and that these mental models can be partly elicited from wisely selected representatives (Group Voices). We argue that individuals belonging to the same group share the same representation, but their behaviour is driven by personal motivations and tacit knowledge. Thus, they can temporarily dismiss part of the shared representation (Dray et al. 2006b: 2).
Carley describe the ‘mental models’ as including definitions, procedures, examples and so on, and that they are ‘internal symbolic representations of the world or aspects of the world [and individuals] use these mental models to negotiate their lives, determine which actions to take and construct the social world’ (Carley 1997: 535). She also makes the point that individuals adapt their mental models over time, and that they can also have multiple mental models utilised in various contexts (Carley 1997). It is also argued that individuals are not fully aware of their own mental models, and that they are in fact unobservable as they exist only in the mind employing tacit and unarticulated knowledge (Carley 1997). This seems to be in line with the popular science theories laid out by Gladwell (2005) in regards to how individuals are largely
70 unaware of much of the processes and representations which people use for making decisions, in particular in relation to what is often referred to as intuition. In relation to the use of mental models in complex decision making situations, Gladwell also argues that too much information, rather than only the most significant information, can often interfere with the accuracy of judgments.
Groups and teams may also have shared mental maps, although this appears to be more contentious and ambiguous in terms of the exact definition (Carley 1997). Definitions concern how information is shared, the degree of sharing and the awareness of sharing. Searle (1995) describes the intricate and complicated processes of how social reality and social facts like money and sports, in essence group mental models (social facts), are constructed and reinforced.
This is largely outside the scope of this literature review. This area of philosophy is often referred to as social constructionism.
3.2.2 Conceptual models
An important intermediate step between mental models and computational models are the conceptual models. Carley describes a concept as follows:
A concept is a single ideational category. A concept can be a single word such as
‘organisation’ or ‘process’, a composite word such as ‘information system’, or a more complex phrase such as ‘democratic and equal participation’ (1997: 538).
There are different types of conceptual models, such as Causal maps, Semantic maps and Concept maps (Siau and Tan 2005) while Carley (1997) uses the term Cognitive maps. The term Cognitive maps, is used for the purposes of this study.
71 Table 3-1: Cognition and quality in conceptual modelling
Quality type Description How human cognition is related?
Semantic quality Valid and complete
correspondence between the conceptual model and the domain
Relates to the scope of the domain. Discrepancies between individual interpretation and participant knowledge can be used to establish approximations of semantic quality
Syntactic quality Is the language appropriate? Limitation in human cognition capacity demands for simple models and languages
Pragmatic quality Is the model interpreted as intended?
Good pragmatic quality typically means interpreting a single meaning with the lowest possible cognitive effort
Physical quality Does the model bring understanding?
Refers to two processes: externalisation and internalisation. Externalisation brings knowledge of some social actors that is not widely available into the model that then becomes widely available.
Internalisation brings increased knowledge on the basis of making sense of the model.
Empirical quality Is the model accurate? Heavy cognition loads may cause frequent errors when conceptual models are being written or read
Social quality Do individuals agree on interpretations?
Difference in the frames of reference is the root reason for errors in social quality
Source: Siau and Tan 2005: 352
The quality of cognitive maps represents the validity of these, and as such the types of quality described in Table 3-1 are critical. Siau and Tan (2005) base their assessment of quality on a framework for conceptual modelling which includes correspondence between:
• Domain: the context of the modelling effort;
• Model: the conceptual model;
• Interpretation: the way that the audience perceives the model;
• Language: the way that the model is expressed.
Some of the qualities of a conceptual model refer to the relationship between the above elements:
• Semantic quality: correspondence between domain and model;
• Syntactic quality: correspondence between language and model;
• Pragmatic quality: correspondence between interpretation and model;
72
• Language-domain appropriateness: appropriateness of language in the domain;
• Audience-domain appropriateness: the appropriateness of the audience in the domain;
• Audience-language appropriateness: the appropriateness of language for the audience.
Furthermore, the social quality refers to the social value of the model; i.e. the correspondence between different individuals’ interpretations and the outcomes that they can generate on the basis of this. The empirical quality refers to the likelihood of error occurring when a model is read or written by different users, and typically relates to computer-human interaction. Physical quality refers to the extent of increased understanding that individual social actors gain from making sense of the model.
Cognitive maps, as defined by Carley (1997), are developed by eliciting and coding the elicited mental models of individuals or groups. This is described as an emergent structure which in other words is not pre-defined. As such the term cognitive map is a more generic term and is not bound by a framework in the same manner as causal maps, semantic maps or concept maps. The process that Carley describes for developing cognitive maps is via different types of textual analysis, mainly of statements. The elements of the cognitive map are concepts and relationships. There are several formal processes that are available for generating cognitive maps from a combination of qualitative and quantitative data. These are further described below.
3.2.3 Coding conceptual models
The task of Coding text into conceptual models has so far not been described. In order to code text into conceptual models, the research must make a large number of coding choices in particular in terms of filtering which parts of a text to use and which concepts to code (Carley 1997).
Polhill and Ziervogel (2006) describe a process for using textual analysis for developing a conceptual model. The type of conceptual model that they develop is an OWL ontology which can be described as a concept map. This concept map was developed in a case study in South
73 Africa (Polhill and Ziervogel 2006) based on data from the field work as well as three scientific papers. This was done in a number of steps which were repeated for each data source, including assembling material, classification, analysis and finally development of a knowledge representation; followed by analysis of logical consistency and refinement.
Another example of a coding process is given by Dray and colleagues which started with an initial knowledge elicitation process as follows:
Our methodology includes two sets of interviews. The first one, called Global Targeted Appraisal (GTA), is meant for eliciting Group Voices representations of the key components and processes at stake. The second one, called Individual Activities Survey (IAS), is conducted with individuals selected randomly among the different groups. It is used to validate and eventually quantify the interactions that unfold during the GTA interviews (Dray et al. 2006b: 2).
The above mentioned Global Targeted Appraisal (GTA) uses semi-structured interviews with selected individuals based on three exercises, semi-structured interviews using photos and card games as well as cognitive mapping based on individual’s home island. The exercise also involved a more structured and quantitative questionnaire for validation; as well as analysing transcripts and to analyse this using transcript analysis in a qualitative analysis software, Qualrus (Idea Works Inc. 2002).
Whilst these efforts are rigorous, the effort required is considerable and Dray and colleagues (2006b) report that approximately 60 person-days in the field work and approximately 50 person-days in qualitative analysis is required. For a PhD student, this would mean approximately 4 months of labour alone, including at least two months of field work. While the benefits are a more rigorous description, there is an overwhelming cost and effort required for undertaking this type of task. For a PhD study where the key focus is not on this aspect, this type of activity is not feasible and therefore more informal approaches would be sought.
74