Another word of caution is that we developed these guidelines based on our years of experience as research methodologists. They do not represent a formal, rig- orously developed set of questions that can be used for a systematic review. They should, however, facilitate beginning efforts to critically appraise nursing studies.
We also note that there are questions in these guidelines for which there are no totally objective answers. Even experts sometimes disagree about methodologic strategies. You should not be afraid to “stick out your neck” to express an evaluative opinion—but your comments should have some basis in methodologic principles dis- cussed in this book. The critiquing guidelines are available in the Toolkit section of the accompanying CD-ROM ; the question list can be adapted, as appropriate.
Quantitative researchers use several criteria to assess the quality of a study, sometimes referred to as its scientific merit.Two especially important criteria are reliability and validity. Reliabilityrefers to the accuracy and consistency of infor- mation obtained in a study. The term is most often associated with the methods used to measure research variables. For example, if a thermometer measured Alan’s temperature as 98.1°F one minute and as 102.5°F the next minute, the reliability of the thermometer would be highly suspect. The concept of reliability is also important in interpreting the results of statistical analyses. Statistical reliability refers to the probability that the same results would be obtained with a completely new sample of subjects—that is, that the results are an accurate reflection of a wider group than just the particular people who participated in the study.
Validityis a more complex concept that broadly concerns the soundnessof the study’s evidence and the degree of inferential support the evidence yields. As with reliability, validity is an important criterion for evaluating methods to measure variables. In this context, the validity question is whether there is evidence to support the inference that the methods are really measuring the abstract concepts that they purport to measure. Is a paper-and-pencil measure of depression really measuring depression? Or, is it measuring something else, such as loneliness or stress? Researchers strive for solid conceptual definitions of research variables and valid methods to operationalize them.
Another aspect of validity concerns the quality of the researcher’s evidence regarding the link between the independent variable and the dependent variable.
Did a nursing intervention reallybring about improvements in patients’ outcomes—
or were other factors responsible for patients’ progress? Researchers make numer- ous methodologic decisions that can influence this type of study validity.
Qualitative researchers use somewhat different criteria (and different terminol- ogy) in evaluating a study’s quality. In general, qualitative researchers discuss meth- ods of enhancing the trustworthiness of the study’s data and findings (Lincoln &
Guba, 1985). Trustworthinessencompasses several different dimensions—credibil- ity, transferability, confirmability, dependability, and authenticity. These and other criteria for evaluating qualitative studies are described in Chapter 18.
Credibility is an especially important aspect of trustworthiness. Credibility is achieved to the extent that the research methods engender confidence in the truth of the data and in the researchers’ interpretations of (and inferences from) the data.
Credibility in a qualitative study can be enhanced through various approaches, but one strategy in particular merits early discussion because it has implications for the design of all studies, including quantitative ones. Triangulationis the use of mul- tiple sources or referents to draw conclusions about what constitutes the truth. In a quantitative study, this might mean having multiple operational definitions of a dependent variable to determine if predicted effects are consistent. In a qualitative study, triangulation might involve trying to understand the full complexity of a poorly understood phenomenon by using multiple means of data collection to con- verge on the truth (e.g., having in-depth discussions with study participants, as well as watching their behavior in natural settings). Or, it might involve triangulating the ideas and interpretations of multiple researchers working together as a team. Nurse researchers are also beginning to triangulate across paradigms—that is, to integrate
both qualitative and quantitative data in a single study to offset the shortcomings of each approach and enhance the validity of the conclusions.
Example of triangulation:
Casey (2007) explored how nurses encourage health-promoting practices in acute settings. Casey triangulated information from observations of nurse–patient interactions with data from in-depth interviews with both the observed nurses and, separately, the observed patients.
Nurse researchers need to design their studies in such a way that threats to the reliability, validity, and trustworthiness of their studies are minimized, and users of research must evaluate the extent to which they were successful.
TIP
In reading and critiquing research reports, it is appropriate to assume a “show me” attitude—that is, to expect researchers to build and present a solid case for the merit of their inferences. They do this by presenting evidence that the findings are reliable and valid or trustworthy.
Bias
Bias is a major concern in research because it can threaten the study’s validity and trustworthiness. In general, a biasis an influence that produces an error in an esti- mate or an inference. Biases can affect the quality of evidence in both qualitative and quantitative studies. Bias can result from a number of factors, including study participants’ lack of candor or desire to please, researchers’ preconceptions, or faulty methods of collecting data.
To some extent, bias can never be avoided totally because the potential for its occurrence is so pervasive. Some bias is haphazard and affects only small segments of the data. As an example of such random bias, a handful of study participants might fail to provide accurate information because they were tired at the time of data collection. Systematic biasresults when the bias is consistent or uniform. For example, if a spring scale consistently measured people’s weight as being 2 pounds heavier than their true weight, there would be systematic bias in the data on weight.
Rigorous research methods aim to eliminate or minimize bias—or, at least, to detect its presence so it can be taken into account in interpreting the findings.
Researchers adopt a variety of strategies to address bias. Triangulation is one such approach, the idea being that multiple sources of information or points of view help to counterbalance biases and offer avenues to identify them. In quantita- tive research, methods to combat bias often entail research control.
Research Control
One of the central features of quantitative studies is that they usually involve efforts to tightly control various aspects of the research. Research controlmost typically involves holding constant influences on the dependent variable so that the true rela- tionship between the independent and dependent variables can be understood. In other words, research control attempts to eliminate contaminating factors that might cloud the relationship between the variables that are of central interest.
The issue of contaminating factors—confounding (or extraneous)variables, as they are called—can best be illustrated with an example. Suppose we were inter- ested in studying whether urinary incontinence (UI) is a risk factor for depression.
There is some prior evidence that this is the case, but previous studies have failed to clarify whether it is UI per se or other factors that contribute to the risk of depression. The question is whether UI itself (the independent variable) contributes to higher levels of depression, or whether there are other factors that can account for the relationship between UI and depression. We need to design a study to con- trol other determinants of the dependent variable—determinants that are also related to the independent variable, UI.
One confounding variable in this example is age. Levels of depression tend to be higher in older people; at the same time, people with UI tend to be older than those without this problem. In other words, perhaps age is the real cause of higher depression in people with UI. If age is not controlled, then any observed relation- ship between UI and depression could be caused by UI, or by age.
Three possible explanations might be portrayed schematically as follows:
1.UI→depression 2.Age→UI→depression 3.Age
↓ depression UI
The arrows here symbolize a causal mechanism or an influence. In model 1, UI directly affects depression, independently of any other factors. In model 2, UI is a mediating variable—that is, the effect of age on depression is mediatedby UI. In this model, age affects depression through its effect on UI. In model 3, both age and UI have effects on depression, but age also increases the risk of UI. Some research is specifically designed to test paths of mediation and multiple causation, but in the present example age is extraneous to the research question. Our task is to design a study so that the first explanation can be tested. Age must be controlled if our goal is to learn the validity of model 1, which posits that, no matter what a person’s age, having UI makes a person more vulnerable to depression.
How can we impose such control? There are a number of ways, as we will dis- cussed in Chapter 9, but the general principle underlying each alternative is that the confounding variables must be held constant. The confounding variable must somehow be handled so that, in the context of the study, it is not related to the inde- pendent or dependent variable. As an example, let us say we wanted to compare the average scores on a depression scale for those with and without UI. We must then design a study in such a way that the ages of the two groups are comparable, even though, in general, the two groups are not comparable in terms of age.
By exercising control over age in this example, we would be taking a step toward explaining the relationship between variables. The world is complex, and many variables are interrelated in complicated ways. When studying a particular problem within the positivist paradigm, it is difficult to study this complexity directly;
researchers must usually analyze a couple of relationships at a time and put pieces
↓↓
together like a jigsaw puzzle. That is why even modest studies can contribute to knowledge. The extent of the contribution in a quantitative study, however, is often directly related to how well researchers control confounding influences.
Research rooted in the naturalistic paradigm does not impose controls. With their emphasis on holism and the individuality of human experience, qualitative researchers typically adhere to the view that to impose controls on a research set- ting is to remove irrevocably some of the meaning of reality.
Randomness
For quantitative researchers, a powerful tool for eliminating bias involves the con- cept of randomness—having certain features of the study established by chance rather than by design or researcher preference. When people are selected at random to participate in the study, for example, each person in the initial pool has an equal probability of being selected. This in turn means that there are no systematic biases in the make-up of the study group. Men and women have an equal chance of being selected, for example. Similarly, if study participants are allocated at random to groups that will be compared (e.g., an intervention and “usual care” group), then there can be no systematic biases in the composition of the groups. Randomness is a compelling method of controlling confounding variables and reducing bias.
Qualitative researchers do not consider randomness a desirable tool for under- standing phenomena. Qualitative researchers tend to use information obtained early in the study in a purposive (nonrandom) fashion to guide their inquiry and to pursue information-rich sources that can help them expand or refine their concep- tualizations. Researchers’ judgments are viewed as indispensable vehicles for uncovering the complexities of the phenomena of interest.
Example of randomness:
Mok and colleagues (2007) evaluated the effectiveness and safety of three flush solutions for maintaining peripheral intravenous locks in children. A total of 123 children were randomly assigned to receive either normal saline, 1 unit/mL of heparin saline, or 10 units/mL of heparin saline.
Masking or Blinding
A rather charming (but problematic) quality of people is that they usually want things to turn out well. Researchers want their ideas to work and their hypotheses to be supported. Study participants want to be cooperative and helpful, and they also want to present themselves in the best light. These tendencies can affect what par- ticipants do and say (and what researchers ask and perceive) and can lead to biases.
A procedure known as masking is used in many quantitative studies to prevent biases stemming from awareness. Masking involves concealing information from participants, data collectors, care providers, or data analysts to enhance objectivity.
For example, if study participants are not aware of whether they are getting an experimental drug or a sham drug (a placebo), then their outcomes cannot be influenced by their expectations of its efficacy. Masking involves disguising or with- holding information about participants’ status in the study (e.g., whether they are in a certain group) or about the study hypotheses.
The term blinding is widely used in lieu of masking to describe concealment strategies. This term has fallen into some disfavor, however, because of possible pejorative connotations. Medical researchers, however, appear to prefer blinding unless the people in the study have vision impairments.
When it proves to be unfeasible or undesirable to use masking, the study is sometimes called an open study, in contrast to a closed study that results from masking. When masking is used with only some of the people involved in the study (e.g., the study participants), it is often called a single-blind study, but when it is possible to mask with two groups (e.g., those delivering an intervention and those receiving it), it is called a double-blind study. The previously described study about children exposed to three different flush solutions (Mok et al., 2007) was double blind: neither the children nor the staff nurses (who collecting data about study outcomes such as intravenous complications) were aware of which solution had been used.
Reflexivity
Qualitative researchers do not use methods such as research control, randomness, or masking, but they are nevertheless as interested as quantitative researchers at discovering the true state of human experience. Qualitative researchers often rely on reflexivity to guard against personal bias in making judgments. Reflexivity is the process of reflecting critically on the self, and of analyzing and making note of per- sonal values that could affect data collection and interpretation. Qualitative researchers are trained to explore these issues, to be reflexive about all decisions made during the inquiry, and to record their reflexive thoughts in personal diaries and memos.
Example of reflexivity:
Hordern and Street (2007) conducted an in-depth study focused on issues of intimacy and sexuality in the face of a diagnosis of cancer. The researchers were reflexive throughout the process of data collec- tion and data analysis, making a “deliberate and systematic use” (p. E14) of their own responses to the evolving data analysis.
TIP
Reflexivity can be a useful tool in quantitative as well as qualitative research—self awareness and introspection can enhance the quality of any study.
Generalizability and Transferability
Nurses increasingly rely on evidence from disciplined research as a guide in their clinical practice. Evidence-based practice is based on the assumption that study findings are not unique to the people, places, or circumstances of the original research.
As noted in Chapter 1, generalizability is the criterion used in quantitative studies to assess the extent to which the findings can be applied to other groups and settings. How do researchers enhance the generalizability of a study? First and
foremost, they must design studies strong in reliability and validity. There is little point in wondering whether results are generalizable if they are not accurate or valid. In selecting participants, researchers must also give thought to the types of people to whom results might be generalized—and then select subjects in such a way that a representative sample is obtained. If a study is intended to have impli- cations for male and female patients, then men and women should be included as participants. If an intervention is intended to benefit poor and affluent patients, then perhaps a multisite study is warranted. Chapter 9 discusses generalizability at greater length.
Qualitative researchers do not specifically seek to make their findings general- izable. Nevertheless, qualitative researchers often seek information that might prove useful in other situations. Lincoln and Guba (1985), in their highly influen- tial book on naturalistic inquiry, discuss the concept of transferability, the extent to which qualitative findings can be transferred to other settings, as another aspect of a study’s trustworthiness. An important mechanism for promoting transferabil- ity is the amount of information qualitative researchers provide about the contexts of their studies. Thick description, a widely used term among qualitative researchers, refers to a rich and thorough description of the research setting and of observed transactions and processes. Quantitative researchers, like qualitative researchers, need to describe their study participants and their research settings thoroughly so that the utility of the evidence for others can be assessed.
Abstracts for a quantitative and a qualitative nursing study are presented below, followed by some questions to guide critical thinking. A review of the full journal articles would prove useful for learning more about the organization of the report and about study methods and findings.
EXAMPLE 1 ■ Quantitative Research Study
Prevalence and variation of physical restraint use in acute care settings in the United States (Minnick et al., 2007).
Abstract
Purpose:To describe physical restraint (PR) rates and contexts in U. S. hospitals.
Design:This 2003–2005 descriptive study was done to measure PR prevalence and contexts (census, gender, age, ventilation status, PR type, and rationale) at 40 randomly selected acute care hospitals in six U.S. metropolitan areas. All units except psychiatric, emergency,
operative, obstetric, and long-term care were included.
Methods:On 18 randomly selected days between 0500 and 0700 (5:00 am and 7:00 am), data collectors determined PR use and contexts via observation and nurse report.
R E S E A R C H E X A M P L E S A N D C R I T I C A L T H I N K I N G A C T I V I T I E S
Findings:PR prevalence was 50 per 1,000 patient days (based on 155,412 patient days).
Preventing disruption of therapy was the chief reason cited. PR rates varied by unit type, with adult intensive care unit (ICU) rates the highest obtained. Intra- and interinstitutional variation was as high as 10-fold. Ventilator use was strongly associated with PR use. Elderly patients were over-represented among the physically restrained on some units (e.g., medical), but on many unit types (including most ICUs) their PR use was consistent with those of other adults.
Conclusions:Wide rate variation indicates the need to examine administratively mediated vari- ables and the promotion of unit-based improvement efforts. Anesthetic and sedation practices have contributed to high variation in ICU PR rates. Determining the types of units to target to achieve improvements in care of older adults requires study of PR sequelae rate by unit type.
CRITICAL THINKING SUGGESTIONS*:
*See the Student Resource CD-ROM for a discussion of these questions.
1. “Translate” the abstract into a summary that is more consumer friendly. Underline any technical terms and look them up in the glossary. The Toolkit on the accompanying CD- ROM provides a worksheet that will allow you to enter your translation electronically.
2. Also, consider the following targeted questions, which may assist you in assessing aspects of the study’s merit:
a.Was this abstract an example of a traditional-style abstract?
b.In which part of the full paper would the information relating to the study purpose be found? How about the information on the design? Where would the researcher’s conclu- sions be located?
c. What were the independent and dependent variables in this study?
d.Is this study experimental or nonexperimental?
e. How, if at all, was randomness used in this study?
f. Comment on the possible generalizability of the study findings.
3. If the results of this study are valid and generalizable, what might be some of the uses to which the findings could be put in clinical practice?
EXAMPLE 2 ■ Qualitative Research Study
Globalization and the cultural safety of an immigrant Muslim community (Baker, 2007).
Abstract
Aim:This paper reports a study the aim of which was to further understanding of cultural safety by focusing on the social health of a small immigrant community of Muslims in a relatively homogeneous region of Canada following the terrorist attacks on 11 September 2001 (9/11).
Background:The aftermath of 9/11 negatively affected Muslims living in many centers of western Europe and North America. Little is known about the social health of Muslims in smaller areas with little cultural diversity. Developed by Maori nurses, the cultural safety con- cept captures the negative health effects of inequities experienced by the indigenous people of New Zealand. Nurses in Canada have used the concept to understand the health of Aboriginal peoples. It has also been used to investigate the nursing care of immigrants in a Canadian metropolitan center. Findings indicated, however, that the dichotomy between culturally safe and unsafe groups was blurred.