Cognitive interviewing encompasses a variety of approaches for eliciting qualitative data on how partic- ipants interpret and respond to a wide variety of situa- tions. Cognitive interviewing increasingly is used in the evaluation of technology interfaces such as websites and tools for informatics. It is used in education to understand how students think about content and respond to test items and in marketing to understand how to evaluate products better. This entry focuses on an especially salient application of cognitive interview- ing for researchers in varied disciplines—the develop- ment of structured questionnaire and interview items.
Beginning in the late 1970s, cognitive interviewing developed through the interdisciplinary efforts of cog- nitive psychologists and survey design methodologists.
The intent of these interdisciplinary collaborations was to study the cognitive processes that shape partici- pants’ responses to questions and to use cognitive the- ory to improve survey design. Cognitive interviewing increasingly is viewed as an essential aspect of devel- oping valid and reliable standardized measures.
Through cognitive interviews with members of the target population for a new measure, researchers are
able to identify problems with question interpretation as well as understand the kinds of information partic- ipants use in formulating their responses. Cognitive interviewing also provides insights into participants’
decisions to respond to questionnaire items in a partic- ular way. As a distinct form of qualitative interview- ing, cognitive interviewing makes important contributions to instrument development and survey design.
Problems Addressed by Cognitive Interviewing
Cognitive interviews are used to identify a variety of potential problems with items of structured instru- ments. Gordon Willis and colleagues identified key cognitive processes involved in responding to ques- tionnaires that were potentially problematic: compre- hension, retrieval, judgment, and response. Problems of comprehension are related to respondents’ under- standing of the item and whether or not that under- standing is in keeping with the investigator’s intent.
Sophisticated or overly technical language and lengthy complex questions can pose a threat to respondents’ ability to understand items. In other instances, respondents may understand the item but in an unintended way. For example, the item may ask about respondents’ beliefs about health promotion with the intent of eliciting information on what sorts of things are linked to a healthy lifestyle, but respon- dents may interpret the item as asking how religious beliefs contribute to health. Problems ofretrievalare related to respondents being able to recall certain information or experiences. For example, respondents may understand the question but have no memories of their experiences or opinions on the issue. Issues of judgment relate to respondents’ decisions about what information to provide and how to frame their answers. For example, if the question relates to a sen- sitive topic, participants may respond in a socially desirable way that does not reflect their actual experi- ences or opinions.Responseproblems have to do with how respondents fit their experiences and opinions into the response format of the questionnaire. As noted by Chris McQuiston and colleagues, members of certain cultural and ethnic groups have consider- able difficulty in translating their experiences and opinions into a Likert scale response set. Cognitive interviewing is useful in helping the investigator to uncover the nature and extent of all these measure- ment problems.
Cognitive Interview———89
Forms of Cognitive Interviewing Cognitive interviewing takes multiple forms that address different problematic aspects of standardized measures. Verbal probing and think-aloud techniques are the two main approaches to cognitive interviewing.
Verbal probing usually occurs after respondents have completed the questionnaire, with the researcher asking additional questions about how items were interpreted and the basis for participants’ responses.
Respondents may be asked to give their interpretations of specific words, their assessments of the accuracy of their responses, or their evaluations of the appropriate- ness of the questions. Verbal probing can be used to identify problems of comprehension, retrieval, judg- ment, and response. Respondents may be asked to give their interpretations of specific words or phrases, pro- viding input on the extent to which the items are understandable and being interpreted as intended. For example, using the previous example of the use of the wordbelief,verbal probing would reveal the extent to which respondents had different interpretations of the term. Verbal probing is useful for determining how participants weigh issues of social desirability in res- ponding to a question and for assessing the quality and extent of information that participants consider in for- mulating responses. Respondents can provide feed- back on whether or not they think future respondents will answer a question truthfully. Kathleen Knafl and colleagues used cognitive interviewing in developing a new measure of family management of childhood chronic conditions and found that parents strongly objected to the wordburden in some of the items and stated that future respondents would not respond truth- fully to any items using that word. If the investigator has concerns about the response format of the ques- tionnaire, verbal probing also can be used to judge its appropriateness for certain groups. Verbal probing interviews may be based on a standardized guide, with all participants being asked the same questions about items, or they may be individualized based on each participant’s responses. For example, special attention might be given to eliciting data on items that respon- dents failed to answer or that were answered in the extreme categories of the response set.
Closely related to verbal probing is the technique of paraphrasing, which asks respondents to restate each item in their words. Paraphrasing is especially useful in identifying problems of comprehension and the extent to which items evoke similar interpretations
across respondents. Multiple interpretations of an item provide compelling evidence that the item needs to be either revised or deleted from a measure that is under development. Paraphrasing, when used in con- junction with verbal probing, is useful in identifying respondents’ overall understanding of the item as well as pinpointing problematic aspects related to ambigu- ous or offensive wording.
Think-aloud interviewing is the second major cog- nitive interviewing approach. In the think-aloud inter- view, the respondent is asked to say what he or she is thinking while answering an item. Think-aloud inter- views provide information on retrieval and judgment issues. Through think-aloud interviews, the investiga- tor gains insight into what the respondent remembers about an event or the memories that inform a particu- lar opinion. This kind of information can be useful for determining an appropriate time frame for a question (e.g., number of times the respondent has attempted to lose weight or stop smoking) and for understanding what the respondent identifies as relevant and irrele- vant information when formulating a response to a question. Such information contributes to the revision of questions so that the respondent is retrieving the kind of information the investigator is seeking. Think- aloud interviewing typically takes place simultane- ously with instrument completion.
Study Design
Although there are few guidelines for sample size and selection when using cognitive interviews for instru- ment development, investigators typically report inter- viewing 20 to 30 respondents who represent the target population. As noted in the description of the different forms of cognitive interviewing, respondents may be asked to recall information retrospectively about their formulation of responses, or they may be asked to think- aloud the basis for their responses at the time when they are answering the questionnaire. Researchers using the verbal probing approach to cognitive interviewing vary in the extent to which they use a structured interview guide or rely on general open-ended questions such as,
“How did you arrive at your response to the question?”
In either case, cognitive interviews generate qualitative data that are meant to inform further instrument devel- opment. The analysis of these data focuses on the item as the unit of analysis, with the investigator review- ing all input on a given item to reach decisions about 90———Cognitive Interview
retaining, deleting, or revising the item. In some cases the analysis is based on a predetermined list of problem- atic aspects of items (e.g., lexical, temporal, logical), and in some cases problem identification is based on a content analysis of the cognitive data. Regardless of design or analytic approach, investigators consistently report that cognitive interviewing leads to substantial improvements in instrument quality.
Kathleen Knafl
See alsoInterviewing; Structured Interview; Unstructured Interview
Further Readings
Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods.Quality of Life Research, 12,229–238.
George, C. (2005). Usability testing and design of a library website: An iterative approach.OCLC Systems &
Services, 21,167–180.
Jobe, J., & Mingay, D. (1991). Cognition and survey measurement: History and overview.Applied Cognitive Psychology, 5,175–192.
Johnson, C., & Turley, J. (2006). The significance of cognitive modeling in building healthcare interfaces.International Journal of Medical Informatics, 75,163–172.
Knafl, K., Deatrick, J., Gallo, A., Holcombe, G., Bakitas, M., Dixon, J., & Grey, M. (2007). The analysis and
interpretation of cognitive interviews for instrument development.Research in Nursing and Health, 30, 224–230.
McQuiston, C., Larson, K., Parrado, E., & Flaskerud, J.
(2002). AIDS knowledge and measurement considerations with unacculturated Latinos.Western Journal of Nursing Research, 24,354–372.
Miller, K. (2003). Conducting cognitive interviews to understand question–response limitations.American Journal of Health Behavior, 27(Suppl. 3), 264–272.
Skelly, A., Samuel-Hodge, C., Elasy, T., Ammerman, A., Headen, S., & Keyserling, T. (2000). Development and testing of culturally sensitive instruments for African- American women with type 2 diabetes.The Diabetes Educator, 26,769–777.
Willis, G., Royston, P., & Bercini, D. (1991). The use of verbal report methods in the development and testing of survey questionnaires.Applied Cognitive Psychology, 5, 251–267.
Wu, H., & McSweeney, M. (2004). Assessing fatigue in persons with cancer.Cancer, 101,1685–1695.
C
OLLABORATIVER
ESEARCHCollaborative research is research “with” rather than research “on.” It is research that arises out of the expressed needs, interests, and questions of the stake- holders who are most invested in the research and its findings, and it is research conducted in relationship with them. Collaborative research reflects a move in the social sciences away from a hierarchical under- standing of research as informing practice to an understanding of the reciprocal and interdependent nature of research and practice. This entry focuses on the epistemological, ideological, and ethical beliefs that underlie collaborative research and the ways in which those beliefs are translated in the design, imple- mentation, and dissemination of such research.
Collaborative research enables the voices of researchers and those in the field—practitioners, poli- cymakers, and other stakeholders—to be positioned alongside one another in a shared inquiry of mutual interest and benefit. Collaborative research can be situated within the specificity of a discipline (e.g., researchers, teachers, administrators, and parents inquiring into ways to enhance parent engagement in schooling), or it can be situated across disciplines where diverse voices and perspectives come together in multidisciplinary or intersectoral teams to pursue a research question in more comprehensive, holistic, or integrated ways (e.g., researchers, practitioners, and policymakers from health, education, justice, and social services, alongside parents and community members, researching reforms to the provision and integration of human services delivery to enhance the well-being and academic success of children and families living in an inner-city neighborhood).
Collaborative research can also be situated in a university–community relationship where an issue the community is puzzling over is researched with the support and facility of university academics (e.g., a concern posed to academic researchers in a depart- ment of nutrition by a community-based nonprofit food security organization regarding how to work with school lunch providers to enhance the nutritional value of lunches provided in school-administered programs). Although it can involve many types of alliances, common goals and mutuality are integral to collaborative research—a sense that each partner has much to learn from the other and that the results of the Collaborative Research———91
research will be richer through collaboration than any one partner could achieve without the other.
Beliefs From Which Collaborative Research Arises Collaborative research reflects a belief in experience as education and in all individuals as holders and con- structors of knowledge. It steps away from a com- monly held notion that theory is generated through research and by researchers and then is transmitted to the field where it is taken up by and acted on by poli- cymakers and practitioners. Rather than perpetuating such a top-down model, it promotes a side-by-side positioning in which the differing experiences and resulting knowledge that each individual brings to the research, whether as researcher, policymaker, practi- tioner, or stakeholder, is seen as valuable and valued.
It reflects a belief that regardless of one’s role, all of one’s actions in that role are simultaneously acts of theorizing and acts of practice. Collaborative research invites rich dialogue between and among individuals and the multiple perspectives they represent.
Collaborative research is a vehicle for democratic participation in processes of inquiry, problem solving, and social change. It reflects a belief that because social science research can affect the well-being of a social group, members of that group have a place in all aspects of the research. It is based on a shared com- mitment to furthering both the knowledge of the indi- vidual and the knowledge of the collective.
Collaborative research is relational research. It reflects a commitment to a special kind of care and attention in the provision of continuous opportunities for engagement, voice, and response for all research partners (or co-researchers, as they are frequently called). It works to level power differences by creating research structures that do not privilege the researcher’s voice over any other (e.g., research team conversation circles, where every voice is equal and heard, rather than research team meetings with set agendas or led by the principal investigator; opportu- nities for every co-researcher to speak back or write back to field text and research text). Collaborative research reflects an interactivity among team mem- bers based on living the research in caring and res- pectful ways. It calls for a wakefulness to each voice at all stages of the research process and a responsive- ness to shaping and reshaping research roles as the research unfolds.
Characteristics of Collaborative Research
Collaborative research is based on a relationship of equity among co-researchers working together to achieve mutually determined and mutually beneficial goals. It reflects a shared belief in both the means and the ends of the research. It reflects the engagement of co-researchers in all aspects of the research, from shaping the research question or puzzle and the para- meters in which the research is framed, to designing and engaging in the inquiry process, to discussing field texts, to making and communicating meaning through research texts, papers/articles, presentations, performances, and so on.
It is important to note that the mutuality of the research is based on equity, not equality. Although authentic and genuine participation of all co-researchers is critical to collaborative research, this will not look the same for everyone because of the variations in the individual’s role, the time the individ- ual can commit, and/or the skills, interest, or confi- dence the individual has in different aspects of the research. Some aspects of the research may be done together by co-researchers, whereas other aspects are done as divided labor; some aspects may be done syn- chronously, whereas other aspects are done asynchro- nously; and some aspects may be done to a greater extent or more frequently by some co-researchers, whereas other aspects are done to a lesser extent or less frequently by others. There may be a symmetry to the actions of co-researchers or perhaps a complementar- ity to the roles they play. Returning to the earlier exam- ple of an inquiry into parent engagement in schooling, co-researcher participation could unfold as follows:
All co-researchers would be involved in regularly scheduled and recorded research team conversations to share stories of their experiences with parent engage- ment, to respond to each other’s stories, to make mean- ing together of experiences captured in field text, and to monitor and adjust unfolding research plans. Some on-site co-researchers may keep field journals and/or reflective journals of their observations or experiences relating to parent engagement. Other on-site co- researchers may record research moments through photographs or the collection of artifacts. While uni- versity co-researchers are engaged in recorded conver- sations with participants (other parents, educators, community members, and staff members who are not on the research team but who can add to the inquiry), 92———Collaborative Research
other co-researchers may also be scheduling and facil- itating recorded conversations with other participants to add to the field text. Co-researchers may be reading from a diverse range of literature, from parenting mag- azines to scholarly books and articles, to infuse new and different ideas into research team conversations.
Co-researchers, alone or in small groups, may be visit- ing other school sites to learn about their beliefs and practices in relation to parent engagement. Co- researchers, together as a team or in small groups, may be sharing research in progress or research findings at the school or community level or within local, regional, or national research communities. Flexibility in individually and collectively determining roles, and in negotiating and renegotiating those roles as the research unfolds, enables each co-researcher to partic- ipate in ways that are comfortable yet maintain the inclusivity of multiple perspectives and enhance the richness and outcomes of the research program. What remains important is the sense of interdependence among co-researchers, a sense of shared responsibility for the whole of the research, and a sense of mutual respect for each individual and the contribution each individual makes to the research process and results.
Challenges of Collaborative Research Although the relational nature of collaborative research is one of its greatest richnesses, it is also one of its challenges. Building trust and establishing rap- port among co-researchers is central to all research activities yet takes time, contact, space, and support.
Spending time getting to know one another at the out- set of the research process is an important investment in the research and needs to be considered when the timeline for the research is being established. Buil- ding relationships in social and informal settings, per- haps over food and conversation where bonds can develop between individuals and where individuals can begin to know one another personally as well as in relation to research interests, enables the growing of a sense of trust that is essential to collaborative research. Relationships cannot be assumed or taken for granted; they require nurturing and facilitation.
To get to a place where co-researchers can risk being vulnerable with one another, where they can respectfully challenge one another’s thinking or inter- pretations, and where tension can be viewed as a con- structive part of the meaning-making process, there needs to be a move beyond the surface congeniality of
relationships to a deeper collegiality based on a moral and ethical commitment to one another and the research purpose. Explicitly establishing research team norms or strategies for conflict resolution or negotia- tion early in the research process can make having dif- ficult conversations much easier for any team member later on if that becomes necessary.
Other research team issues that may be beneficial to discuss at the outset of the research include leader- ship and coordination of the team, workload, division of labor, ownership of data, rights of publication, and career and status issues. Attention paid early on to each individual’s agenda and interests can avert any later conflict over issues such as power or knowledge differentials, a sense of appropriation of research data or findings, and recognition or status received from the research.
Advantages of Collaborative Research In response to the debate about how to bridge the gap between research and practice, collaborative research provides a possibility. Because collaborative research is typically conducted in a field setting with practi- tioners, the knowledge developed is already integral to practice rather than separated from it. Having co-researchers who are authentically inside the experience—co-researchers who have explored it and understand it from the inside—voids this concern with the research–practice divide.
Creating a collaborative research team, a team that represents multiple viewpoints and voices as well as differing positions in relation to the research puzzle, makes the research richer and more complex and pushes the inquiry deeper. The knowledge of the research context and the particularities, specificities, and subtleties of that research context that on-site co-researchers bring to the research add dimensions to the inquiry that outsiders might not discover, perceive, or understand because of their positioning. Working with co-researchers who are inside the experience invites a different problematizing of practice, a reflec- tion gained from participation in both the practice and the research of the practice that engagement in just one or the other could not produce.
Working as co-researchers rather than in the more typical researcher and participant relationship, researchers reduce the chances that they will “other”
the participants. When everyone’s viewpoints are laid alongside one another, discussed deeply, searched, Collaborative Research———93