Chapter 3. Methodology: Contexts, Concepts, and Methodological Processes 3.1 Introduction and Overview of 1 Introduction and Overview of Chapter
3.7 Quality in Mixed Methods Research
Although a variety of different approaches for evaluating the quality of mixed methods research have been advanced,50 the quality of this study was contextualised holistically. This implies therefore, that quality criteria developed specifically for mixed methods research were used in this study, and are prioritised in this section. In particular, the quality in this study was
underpinned by Tashakkori and Teddlie’s (2008) mixed methods quality framework, and guided by O’Cathain’s (2010) adaptation of this framework.
O’Cathain’s (2010) framework for evaluating mixed methods research is centred on eight quality domains, each domain incorporating a variety of research quality principles and practices. Five intersecting quality domains pertaining to planning quality, design quality, data quality,
interpretive rigour, and inference transferability were prioritised in this study, and I included a sixth affective domain specifically suited to this study. The first quality domain refers to
elements of planning in a mixed methods study. Here, the quality of the study is perceived to be
50 Some evaluative frameworks for mixed methods studies advocate an individual components approach, where individual quantitative and qualitative components of the study are evaluated independently (Sale
& Brazil, 2004).
enhanced via the thoroughness of the literature review and the relevance of the guiding
conceptual and paradigmatic frameworks (Leech, Dellinger, Brannagan, & Tanaka, 2010). The rationale for the deployment of a mixed methods approach should also be adequately developed and transparent in an attempt to enhance the planning quality of the study.
The concern around transparency of the rationale leads into a second quality domain, that
pertaining to design quality and the transparency with which this is explicated. The transparency of design can be facilitated through the use and/or adaptation of the variety of available mixed method typologies (e.g., fully mixed sequential dominant status), reference to the dimensions of the design (i.e., level of mixing, sequencing, and status), and a visual representation of the design (see Figure 3-1) (Leech & Onwuegbuzie, 2009). In this way, the quality of the design can be independently evaluated by readers of the research. In addition, the design quality can be enhanced further by the breadth and depth of the design (Caracelli & Riggin, 1994). “Breadth”
may refer to the extent of the quantitative data (e.g., five years of UKZN graduation data), while
“depth” may refer to the density of the qualitative data (e.g., rich accounts of individual students in context). Although breadth and depth are not exclusive to quantitative and qualitative methods respectively, they do provide a measure against which the quality of the design (and data) can be evaluated.
The concern around breadth and depth of the design quality leads naturally into the third quality domain, that pertaining to data quality. In addition to the adequacy of sampling in the mixed method design, the quality of the data could be assessed through the transparency with which it is described and analysed. Within this domain of quality, Teddlie and Tashakkori (2009)
highlight the principle of analytic adequacy, and emphasise the selection of appropriate statistical tests for the quantitative phases of mixed method studies. For this study, several statistical tests were considered for Phase 1 of the study (see subsection 3.5.3.2), however the methodology of logistic regression was applied given the type of data available from UKZN’s DMI. O’Cathain (2010) also refers to the notion of integrative analytic rigour in data analysis, emphasising the way in which data analysis is integrated across quantitative and qualitative modes. For this study, several integrative data analytic steps were undertaken, these including the transformation of
quantitative data into qualitative data, the use of findings from one phase of the study to guide another, and the placement of data in a matrix for “within-case and across-case analysis”
(O'Cathain, 2010, p. 546). Figure 3-3 is a representation of the integrative analytic rigour exercised within this study.
A fourth domain of quality in mixed method research can be identified as interpretive rigour, which incorporates principles of inference, inference quality, and credibility and trustworthiness.
Tashakkori and Teddlie (2003) define inference as “a researcher’s construction of the relationships among people, events, and variables as well as his or her construction of respondents’ perceptions, behaviours, and feelings, and how these relate to each other in a coherent and systematic manner” (p. 692). It is the quality of these inferences that is important to consider in mixed methods research, this also taking into account the researcher’s capacity to show which inferences come from which data sources, and the theoretical consistency of the inferences. In addition, interpretive rigour and inference quality could also be assessed by the extent to which the articulation of the inferences is found to be plausible, credible, and trustworthy by others. The notion of meta-inferences is also important in mixed methods research, this referring to the way in which inferences across quantitative and qualitative modes are integrated. Overall, interpretive rigour takes into account the quality of the links between the findings, analysis, interpretations and conclusions (O'Cathain, 2010). For this study, the notion of descriptive validity (Maxwell, 2002) was related to the third and fourth domains of data quality and interpretive rigour. When discussing the mixed method design (see subsection 3.5.2), it was described how the research participants were involved in checking the descriptive accounts that arose from the data production process. Here, participants were asked to read and comment on the accuracy with which I had described their auto-photographical accounts. This emerged as a back-and-forth process between researcher and research participant, involving appropriate modification, addition, and removal of certain aspects of the descriptive accounts, and resulting in the descriptions contained in Chapter 5.
The fifth domain in O’Cathain’s (2010) framework (inference transferability) incorporates notions of external validity from quantitative research, and transferability from qualitative
research. For mixed methods research, “inference transferability” refers to the extent to which the conclusions are plausible for other contexts. The transferability of these inferences across contexts is usually assessed by a reader on the basis of thick descriptions and a thorough account of the research context (Dawson, 2009). Extracts from transcriptions are used in the analysis chapters of the thesis, and Appendix 15 contains a list of transcription symbols used. These symbols were adapted from Silverman’s (2011) recommendations, conventions from the Publication Manual of the American Psychological Association 6th Edition (APA, 2010), and everyday written grammar conventions.
The sixth or affective domain incorporates quality principles that are specific for this study and which resonate with contemporary activity theory concerns. As discussed earlier, the theorisation of emotion within activity theory is a contemporary notion of interest (see section 2.3.4 in
Chapter 2). As a result, it was relevant to incorporate this contemporary notion into the quality framework presented in this thesis. In light of this, I drew on the guiding principle that research should “communicate the emotional elements of how the participants and the researcher engaged in the research study” (Northcote, 2012, April, p. 107). The affective principle conveyed the emotional involvement of the participants in the study, and could therefore be conceptualised as a quality enhancement strategy.