There are no universally accepted criteria for judging the soundness and goodness of qualitative research. All discussions on the matter draw from the criteria for mainstream (i.e., quantitative)
research that includes internal and external validity, reliability, and objectivity. Positions on the matter vary. Some suggest establishing practical guidelines for using existing criteria with qualita- tive research. Others assert the existing criteria should be amended and modified for qualitative research. Still others argue the criteria should be abandoned and new criteria be formulated. The criteria of judgment used reflect the epistemological paradigms of the researchers—from a prag- matic approach of complementing the accepted ‘‘positivistic’’ criteria to denying them at all.
Generally, the criteria proposed specifically for qualitative research are articulated within nonposi- tivistic paradigms.
Huberman and Miles (1994) propose that a good qualitative manuscript should report the following contents: (1) sampling decisions made, both within and across cases; (2) instrumentation and data-collection operations; (3) database summary and size, as well as the method by which it was produced; (4) software used, if any; (5) overview of analytic strategies followed; and (6) inclusion of key data-displays supporting main conclusions. Strauss and Corbin (1990) argue that judgments of research quality should center around three issues: (1) validity, reliability, and credibility of the data; (2) adequacy of the research process; and (3) empirical grounding of research findings. They offer a set of criteria against which a grounded theory study can be judged in terms of the theory-generation aspects of the research: (1) Are concepts generated? (2) Are concepts systematically related? (3) Are there many conceptual linkages and are the categories well devel- oped? (4) Is much variation built into the theory?
Lincoln and Guba (Guba and Lincoln, 2005; Lincoln and Guba, 1985) offer two sets of criteria for qualitative research: trustworthiness (credibility, transferability, dependability, and confirmabil- ity) and authenticity (fairness, enrichment, education, stimulation to action, and empowerment).
Trustworthiness parallels the criteria in the positivistic-quantitative paradigm. Specifically, credibil- ity is the counterpart of internal validity, which can be achieved by carefully identifying the setting of the research, the population, and the underlying theoretical framework. Second, transferability parallels external validity and denotes the applicability of one set offindings to another context. This is usually problematic in qualitative research, but it can be improved by two strategies: explicitly stating theoretical parameters of the research (so that other researchers can decide upon generalizing the approach to their settings) and triangulation of the research methodologies. Third, dependability is similar to reliability and concerned with the consistency of the research—i.e., how the researcher accounts for changing conditions in the phenomena and changes in design. However, dependability is different from the positivist understanding of replicability because the social world is always constructed and thus replicability is a problem. Finally, conformability relates to objectivity or neutrality of the research. The criterion is‘‘Do the data help to confirm the generalfindings and lead to the implications?’’(Marshall and Rossman, 1995, p. 145). The researcher can never eliminate his or her bias, but should adopt strategies to provide a balance for bias in interpretation, such as playing devil’s advocate for one’s research partner, a constant search for negative instances, etc. The second set of criteria Guba and Lincoln present is about authenticity, including fairness, ontological authenticity (enlarges personal constructions, educative authenticity (leads to improved understand- ing of constructions of others), catalytic authenticity (stimulates to action), and tactical authenticity (empowers action).
King, Keohane, and Verba (1994) argue that both quantitative and qualitative research have the same underlying logic of inference. While valuing the role of interpretation in clarifying and defining concepts as well as idea and hypothesis generation, they insist that for evaluation of these hypotheses the logic of scientific inference applies. Therefore, qualitative researchers should also ask questions such as: How many observations are enough for valid inferences? How do you measure and improve data when you have already established the concepts? How valid is the generalization? Or perhaps, the question can be formulated as: How do you build an empirically sound theory on the basis of small number of often unique observations? They insist on situating every research inquiry in the framework of a broader theory to test for generalizability—i.e., bring the issues of internal and external validity to qualitative research. A single case study, for example,
can contribute to theory greatly if it disproves (falsifies) a predominant theory. More generally, King and his collaborators argue the single case was useful because it was a part of research program and it was compared against other observations (perhaps gathered by other researchers).
Critics argue that King, Keohane, and Verba (1994) maintain the ‘‘positivist’’ criteria for goodness of research (internal and external validity, reliability, and objectivity) and do not mention alternative criteria proposed for qualitative research (e.g., credibility; transferability; dependability and confirmability; and empathy proposed for constructivist paradigm). Although they leave room for qualitative research (hypotheses generation and concept definition and clarification), their attempt still may be seen as emphasizing‘‘the third part of scientific inquiry, the rigorous testing of hypotheses, almost to the exclusion of thefirst two—the elaboration of precise models and the deduction of their (ideally, many) logical implications—and thus point us to a pure, but needlessly inefficient, path of social scientific inquiry’’(Rogowski, 1995, p. 467; see also Brady and Collier, 2004; George and Bennett, 2005). Some researchers argue that even such a proposed division of labor is not very accurate, because‘‘the evaluation of theory with qualitative data is not inherently antithetical to qualitative research.’’But to do so,‘‘qualitative projects must be designed with the goal of theory testing in order to achieve this important objective’’(Ragin et al., 2004, p. 16). While others complain about unsubstantiated association of standards and rigor only with confirmatory and hypothesis-driven research (Ryan, 2005).
There are two other issues to which qualitative researchers should pay attention—the questions of ethics and‘‘the art and politics of interpretation.’’The ethical issues include informed consent, deception, privacy, harm, identification, and confidentiality, among others. Researchers should be aware of all the issues in the context of a research project, and make sure that they follow the established codes of conduct. Discussing the issue of interpretation, Denzin (1994) holds that‘‘the age of putative value-free social science is over,’’and‘‘any discussion of this process must become political, personal and experiential.’’Whether subscribing to this view or not, one must be aware of the tendency that Denzin predicts: a proliferation of ‘‘race-, ethnicity-, and gender-specific’’
interpretive communities and epistemologies. An equally important issue is how researchfindings are communicated to scholars, governments, communities, and individuals. Especially in action- oriented interventionist research, it is very important to tell stories that‘‘subjects’’or partners may be willing to listen to.
Marshall and Rossman (1995, pp. 146–148) present a practical checklist of 20 questions to help judge the quality of qualitative research. Although not necessarily applicable to all research situations, these guidelines give a good understanding of the criteria employed to judge qualitative research. With some abridgment, they are as follows:
1. Method is explicated in detail so that a judgment can be made about the method’s adequacy.
2. Assumptions and biases are expressed.
3. Research guides against value judgments in data collection and analysis.
4. There is evidence from raw data demonstrating the connection between thefindings and the real world; and it is done in an accessible and readable manner.
5. Research questions are stated and answered, and answers generate new questions.
6. Relationship with previous research is explicit, and the phenomena are defined clearly.
7. Study is accessible to other researchers, practitioners, and policymakers.
8. Evidence is presented that the researcher was tolerant to ambiguity and strove to balance his or her biases.
9. Report recognizes limitations of generalizability and helps the readers to find transfer- ability.
10. It should be a study of exploration and not reasserting theories from literature.
11. Observations are made of a full range of activities over a full cycle of activities.
12. Data is preserved and available for reanalysis.
13. Methods are devised for checking data quality (e.g., informants’knowledgeability, ulterior motives) and for guarding against ethnocentric explanation.
14. In-field work analysis is documented.
15. Meaning is elicited from cross-cultural perspective.
16. Ethical standards are followed.
17. People in the research setting benefit some way.
18. Data-collection strategies are the most adequate and efficient available. The researcher is careful to be reflexive and recognize when he or she is‘‘going native.’’
19. Study is tied into ‘‘the big picture.’’ The researcher looks holistically at the setting to understand the linkages among systems.
20. Researcher traces the historical context to understand how institutions and roles have evolved (Marshall and Rossman, 1995, pp. 146–148; reprinted by permission of Sage Publications, Inc.)