Phase II: Evaluation
3.8. Validity and Reliability
The validity and reliability of the study in both phases were achieved through various mechanisms.Although some qualitative researchers have argued that the term validity does not apply to qualitative research, the need for qualifying checks or measures has been recognised (Golfashani, 2003). Validity and reliability are conceptualised as rigour, trustworthiness, and quality in the qualitative paradigm (Bashir et al., 2008). Within the context of analysis, verification strategies that confirm both validity and reliability of data are activities such as confirming methodological consistency, and sampling validity and are supported with theoretical underpinnings (Morse, Barrett, Mayan, Olson, & Spiers, 2002).
For the qualitative data, validity can be achieved via four testing criteria – credibility, dependability, transferability, and confirmability. Lincoln and Guba (1986) provided the groundwork for these techniques as a substitute to achieving validity and reliability in qualitative studies. These techniques are comparative to the measures used by positivists. These criteria were used for Phase I of the study while the fifth criterion of ‘authenticity’ (Lincoln, 1995) was additionally applied in Phase II. Lincoln (1995) cautions that some criteria may be more applicable at some stages than at other stages of the research. It was assumed that validating the conceptual model in Phase II required an additional criterion to ensure that the model is considered useful.
3.8.1. Validity and Reliability – Phase I
The instrument was piloted among information security experts initially before the actual data collection was started. Only the most salient questions were tested. The pilot test involved six experts to verify the validity of the semi-structured interview guide. Table 3-5 describes how validity and reliability were achieved for Phase I of the study, where the criteria was applied by systematically considering the associated techniques. This framing was also used by previous studies (Persad & Padayachee, 2015).
61
Table 3-5: Validity and Reliability Criteria for Phase I (adapted from Bradley (1993))
Criteria Technique
proposed to improve Validity and Reliability
Evidence of compliance
Credibility (“Adequate representation of the constructions of the social world under study and can be assessed both in terms of the process used in eliciting those representations”. (Bradley, 1993, p. 436).
-Protracted engagement -Peer debriefing -Member checking
-The engagement included semi- structured interviews.
-Peer debriefing is attained through data submission, tools, and analysis to the secondary researcher for cross checking (in this case the supervisor of the study).
-Member checking with the participants regarding their input.
Dependability “(i.e. consistency of the methods applied)” (Bradley, 1993).
-Maintaining an audit trail
-Triangulation -Systematic association
-Dependability is attained by keeping a catalogue of data records, (paper and digital format).
-The study depends on multiple sources of evidenced data to enhance validity (i.e., document analysis and interviews).
-The dependability of the research instruments involved systematically associating the items on the questionnaire with commonly used standards and the literature for standardisation.
Transferability “(i.e. the level in which the ‘working hypothesis’ could be transferred to another context)”
(Bradley, 1993, p. 436).
-Thick descriptions -Document analysis
-Transferability is attained via
‘thick description’ by acquiring richer inferences of the context through document analysis and background information on policies.
Confirmability “(i.e. the extent to which the characteristics of the data, as posited by the researcher, can be confirmed by others)” (Bradley, 1993, p. 437). It also implies achieving objectivity and impartiality.
-Data verification by a third party -Confirmation from participants
-Distribute interview transcripts to participants for confirmation.
-The impartiality of the study was ensured through deployment of research assistants and data collectors to reduce bias.
3.8.2. Validity and Reliability – Phase II
The effectiveness of an artefact must be validated by the application of a standard heuristic evaluation process (Jaferian et al., 2014). These items on the evaluation survey were adapted from previously validated instruments. A similar survey instrument developed by Purao and Storey (2008) provided guidelines on evaluating the adoption potential of DSR efforts. This study used these guidelines and reviewed existing relevant literature to develop the items. The validity of the instrument was achieved by leveraging relevant existing knowledge and using
62
the framing of the TAM which has been well validated by previous studies. The use of previously validated instruments provided a basis for face validity. To ensure content validity, a statistician and a language expert reviewed the instrument. The instrument was revised several times to ensure that the questions were clear and easy to understand. Cronbach’s alpha α which is one of the most widely used measures of reliability in the social and organisational sciences was employed to measure internal consistency.
Table 3-6 describes how validity and reliability were ensured for Phase II of the study.
Table 3-6: Validity and Reliability Criteria for Phase II (adapted from Sikolia, Biros, Mason & Weiser ( 2013))
Criteria Techniques proposed
to improve Trustworthiness
Evidence of Compliance
Credibility - “how much the data collected correctly represents the multiple realities of the phenomenon.” which relates to internal validity.
(Sikolia et al., 2013, p. 2).
-Protracted
engagement with participants
-Data Triangulation -Thick descriptions -Participant guidance of the inquiry
-Use of real participant words in the emerging theory
-Peer debriefs
-Negative Case analysis
(extracted from (Sikolia et al., 2013)
-Two iterations of data gathering
-Data triangulation using multi-methods -Provision of ‘thick descriptions’ in the analysis
-Participants are directed with the outcomes of the study where the experts will assist in improving the artefacts.
-Quotations from the participants is incorporated in the analysis
-The statistician conducted peer analysis -Any descriptions that were inconsistent with the expectations of the researcher was considered
Transferability - “the application of one set of findings to another context which relates to external validity” (Sikolia et al., 2013, p. 2).
- ‘Thick descriptions’
of the research methods
(extracted from (Sikolia et al., 2013)
Transferability is achieved by presenting clear accounts of the methodology to ensure repeatability. Provision of contextual information about how the data was collected, the organisational setting of the selected entities and the respondent’s setting.
The researcher provided existing situational analysis about the data collected such as organisational issues, information security culture of the organisation and the context of Ethiopia and the respondent’s level.
Dependability - “the validation that the data represents the changing conditions of the
-Analysis of a detailed audit by an observer (extracted from (Sikolia et al., 2013)
An experienced statistician conducted an inquiry audit (external audit) on the integrity of the result outputs to maintain dependability.
63