Observer bias occurs when the observer’s background, expectations, or per-sonal perceptions influence the observation, making it inaccurate. If you are ob-serving students asking questions in a seventh-grade science classroom, your own beliefs about the interests and abilities of males and females in science might in-fluence what you record. You might overlook or categorize questions asked by girls differently from those asked by boys. A similar problem with accuracy in observa-tions is contamination, which occurs when the observer’s knowledge of the study affects his or her observations. If the observer knows that the researcher expects boys to ask higher order questions, then she or he might be more likely to notice and record this type of question for boys. A well-developed checklist should help avoid errors by clearly describing the different types of questions to be recorded.
Other ways to control observer bias are by training observers in the use of the checklist before the study (or by practicing the use of it yourself). Your accuracy in recording could be checked by videotaping the observations and asking a second observer to also rate the behaviors. Contamination is often controlled by keeping observers “blind” (or in the dark) about the expected outcomes of the study.
A final problem that can occur with observational measures is known as the halo effect. The halo effect occurs when an initial impression influences all sub-sequent observations, making them less accurate. If in your first set of observa-tions, Eager Edgar impresses you with his incisive questions and Cautious Caren only makes some simple and tentative comments, your later observations might reflect these first impressions. There might be a tendency to record more high-level questions for Edgar and fewer for Caren, even if Edgar’s question asking de-clines and Caren’s increases in sophistication.
Developing Measurement Procedures
Use of Observations
Although observation is a large part of how we learn (Bandura, Grusec, &
Menlove, 1966), observation as a tool of research requires systematic and careful examination of the phenomena being studied. Specifically, researchers who choose to use observation must conduct their observations in a way that results in accu-rate, unbiased, and richly detailed information. We casually observe people’s fash-ions, the way they wear their hair, and how they act, but this does not constitute observation as a research tool. Observation as a research tool requires training in both what to observe and how to record the observations.
Degrees of Researcher Participation. When conducting observations, the primary goal is to gather data that are accurate and naturalistic and, to the greatest extent possible, that reflect the reality of the situation as the participants see it. This ne-cessitates that you as the researcher become familiar with the setting and that the participants are comfortable with your presence. Before the start of the observation, you must decide to what degree you will allow yourself to be involved in the setting.
There are certainly varying degrees of involvement that you can choose. Accord-ing to Gold (1958), whose classification of observation is often used to describe the degree of participation of the observer, you could be one of the following:
• Complete participant. This means that you are a member of the group, and no one in the group is aware of the fact that you are also an observer. While this might allow a true “insider’s” view, it raises ethical concerns because, in essence, you are deceiving the participants.
• Participant as observer. In this situation, you are an active member of the group and actively participate in the group’s activities and interactions, but each mem-ber of the group knows that you are also serving a research role. In essence, a col-laborative relationship is developed between the observer and the participants.
Although this removes the ethical concerns presented by being a complete ob-server, you may compromise the natural interaction of the group.
• Observer as participant. Choosing to be an observer as participant removes you a bit from group membership. Although you certainly still have a connection to the group, you will not likely participate in the group’s activities.
• Complete observer. Here you might conduct your observations from behind a one-way mirror or in a public setting. You are not a member of the group and do not participate in the group’s activities.
When selecting a role as an observer, you must decide the extent to which you want your activities as a researcher to be overt or covert. This decision may
Self-Developed Measures and Qualitative Measurement 117
influence how the participants behave, respond, and react. As a qualitative re-searcher, you need to recognize the influence this might have on the reality of the phenomena being investigated.
Key Features of Careful Observation. According to Goetz and LeCompte (1984) and many other qualitative researchers, careful observation should include at least the following key features:
• An explanation of the physical setting. This would include an overall physical de-scription of the space. For example, in a classroom, this dede-scription would include the number of desks, the teacher’s work station, the number of students, whether or not there were computers and, if so, how many, and any other unique features the researcher feels should be noted.
• A description of the participants in the setting. Careful explanation of the partici-pants would include not only who is in the setting but also why they might be there and a description of their roles. In addition, any relevant demographic informa-tion should be included.
• Individual and group activities and group interactions. The researcher should ob-serve the activities the participants are engaging in. In other words, what is going on in the setting? Are there rules that are being followed? Special note should be made of the particular activities that will help to answer the foreshadowed questions.
• Participant conversation and nonverbal communication. Because qualitative data often include direct quotes, conversations should be observed in such a way as to note not only what is being said but also how it is being said.
• Researcher behavior. Because the researcher is part of the setting, careful at-tention must be paid to the influence the observer has on the behavior of the par-ticipants. Does the researcher’s presence in any way influence what is occurring in the setting?
Observational Protocols
The previous categories are still too broad to be much help in focusing one’s ob-servations. Therefore, many qualitative researchers develop an observational protocol specific to the topic of the study.
Protocols are generally designed to gather data that cannot be predicted in advance (e.g., conversations, observed natural behaviors, answers to open-ended questions). Even when collecting qualitative data, it is important to have a proto-col that will help guide the proto-collection of data in a systematic and focused manner.
Observational protocols may take different forms. In ethnographic or case study research, observational protocols are typically unstructured. Most include
brief phrases or questions identifying the types of actions, features of the setting, or interactions to focus on in the observation. In a proposal one might include the foreshadowed questions for the study to identify categories or areas that will be observed although these may change after the researcher enters the field based on the initial observations.
Some observational protocols include a recording sheet to be used in the ob-servation. The recording sheet may simply provide an organized space for record-ing the details of what happens in a particular settrecord-ing. Exhibit 5.4 presents an observational protocol and recording sheet for a qualitative program evaluation of an after-school poetry club.
Conducting and Recording Observations. Conducting good observational re-search takes time and practice. Before beginning, you should carefully prepare for your observations by reviewing the foreshadowed questions of your study to give you a clear idea of what behaviors and activities you will be looking for: they will focus your attention. Using your observational protocol and, if appropriate, a recording sheet, keep the following in mind the following:
• Keep your observations short. As your skills improve, you can increase the length of time that you are observing.
• Be alert to the behavior, conversations, and activities of the participants. You will want to remember as much from your observations as possible. Making a mental note or jotting down actual notes will be helpful.
• Concentrate on specifics. Avoid being global in your observations. Look for ex-amples of specific behaviors.
The observations or “raw data” produced by your observations will form the basis for the results and conclusions you draw in the study. As such, your recording of this information must be detailed, precise, and accurate. Most qualitative re-searchers write down their observations in the form of field notes, which are writ-ten descriptions of what the research observes in the field. To control for observer bias, qualitative researchers usually record both descriptive field notes and re-flective field notes. Descriptive field notes include the following information:
• Time, date, location, and length of observations
• List of participants
• Detailed descriptions of persons, interactions, activities, and settings observed
• Verbatim conversations and direct quotes
Reflective field notes include descriptions of the observers’ feelings and thoughts about what he or she is observing. These are often recorded as observer
Self-Developed Measures and Qualitative Measurement 119
EXHIBIT 5.4 SAMPLE OBSERVATIONAL PROTOCOL AND RECORDING SHEET.
Observational Protocol and Recording Sheet Foreshadowed Questions:
• What types of topics are addressed in student poems?
• What types of activities do club instructors use to encourage students to write?
• How do instructors provide feedback on student poems?
• How do students in the club interact with each other and with the instructors?
Date of observation:
Time of observation:
Setting:
Participants:
Observer:
Person(s): Comments Actions
Observer’s Reflections:
comments at the bottom of the recording sheet or as separate entries in a field notes log after the observations. Reflective field notes allow the researcher to re-flect on their own feelings, values, and thoughts in order to increase their aware-ness of how these might be influencing their observations. Exhibit 5.5 presents an example of descriptive field notes with reflective field notes recorded as an ob-server comment.
Use of Interviews
Most qualitative research includes interviews. The interview might be the major data collection tool of the study (particularly when the behavior of interest cannot be easily observed) or may be used to corroborate or verify observations.
An interview is basically a purposive conversation with a person or a group of persons.
Group Versus Individual Interviews. Many qualitative interviews are conducted one-on-one when the interviewer attempts to determine the participant’s feelings, interpretation, or reaction to an event (often referred to as a “critical-incident in-terview”) or a set of circumstances or life experiences (also known as “life histo-ries”). In one-on-one interviews, the researcher lets participants express their thoughts in their own words. For example, one-on-one interviews might be con-ducted with students who lived through the terrorist attacks on September 11, 2001, to capture their individual perspectives. In this same situation, the students could be interviewed as a group, also known as a focus group interview. With focus group interviews, the researcher is able to collect data from multiple partic-ipants and also to observe and record the interactions and group dynamics that unfold.
Interview Protocols
An important component of conducting a good interview is the construction of the interview protocol. An interview protocol should include a brief script for explaining the purpose of the study to the interviewee, places to record the date and background information on the interviewee, and the preliminary questions to be used in the interview. Because the procedures for conducting qualitative in-terviews are flexible, the questions serve as a starting point. A good interviewer will use the questions to begin the discussion and will then ask additional ques-tions based on the person’s responses. The actual interview may look more like a conversation than an interview with set questions and responses. Box 5.1 presents sample questions for students and teachers from the study of playtime by Perkins, Ayala, Fine, and Erwin (n.d., pp. 17–18).
Self-Developed Measures and Qualitative Measurement 121
EXHIBIT 5.5 SAMPLE FIELD NOTES.
Setting: 9:00 AM; XXX Preschool; City, State, 2-hour observation conducted by XXX.
Participants: primary focus 4-year-old white male (Max) and 50-year-old lead teacher (Ms. Smith)
Activities Observed: entrance into school, opening activity, structured play time, and play-ground activities
Observations:
Max entered room at 8:50 AM accompanied by his mom.
After brief goodbye, mom leaves and Max begins to play with toys set out by Ms. Smith.
Ms. Smith welcomes Max. “Hi Max.”
Max responds without looking up, “Hi.”
Ms. Smith calls all students to opening activity.
Max ignores instruction.
Ms. Smith instructs Max to join activity.
Max joins, after several reminders by Ms. Smith.
Opening song.
All students participate, except for Max. He sits quietly.
Students dismissed to interest center.
Max goes to blocks.
Other children are playing.
Max tries to join and one child says “We don’t want you to play. You will just mess things up.”
Max growls and continues to try to gain entry into the game.
One child says, “OKAY, you can play but don’t knock down our building.”
Max cooperates, but sings to himself and does not interact with the children.
Children struggle for the blocks.
Max growls and becomes angry.
Max knocks down the blocks and screams at the children.
They yell back and run to the teacher.
Teacher disciplines Max and puts him into a time-out.
Max growls.
OC: Max clearly has difficulty entering group activity. Even when he does there is almost no interaction with classmates. He does not verbalize his needs, but simply growls indicating his anger and frustration with the situation.
Box 5.1 Sample Interview Questions from Playtime Study
Student Interview Questions
1. Tell me about your science activity today.
2. What would you like to learn more about?
3. Why is it important to know (activity)?
4. What did you learn today that you didn’t know already?
5. What observations did you make?
6. What did you enjoy most about the science lesson?
7. Did you learn any new science words today? What were they?
8. If you were teaching a class about this activity, what would you tell the kids?
9. What would you like to tell the people who are running the science program about what did not work, what was hard for you, or what you didn’t like (about the activity)?
Teacher Interview Questions
1. What would you say is the level of (name of student)’s engagement in his/her work?
Can you give an example?
2. How do you accommodate (student)’s individual learning needs?
What kind of support do you typically provide?
3. What kind of questions does (student) ask you about his/her work?
Is he/she comfortable asking for help?
How frequent are his/her questions?
When does he/she ask questions?
4. Can you describe how (student) learns?
Can you give an anecdote that illustrates how?
Does (student) share his/her work with other students?
5. Does (student) observe and learn from what others are doing?
Can you give an example?
6. Can you give examples of how (student) displays knowledge?
How does (student) let you know that he/she has understood the lesson?
What other ways do you know (student) has learned the material?
7. What are (student)’s experiences in science?
Types of Interview Questions. Once you have decided to conduct an interview, you must decide the amount of structure you want in the interview. That is, will you conduct a structured, semistructured, or nonstructured interview? Structured interview is one in which the researcher comes to the interview with a set of
Self-Developed Measures and Qualitative Measurement 123
questions, does not deviate from those questions, and asks the same questions of all the participants. In a semistructured interview researchers usually pre-pare a list of the questions to be asked but allow themselves the opportunity to probe beyond the protocol. Nonstructured interviews are more conversation-like and allow for the greatest flexibility. The researchers may simply jot down a list of topics that they want to cover in the interview. Generally, in qualitative re-search, the researcher will conduct a semistructured or nonstructured interview.
Conducting a Good Interview. As with observational research, conducting in-terview research requires certain skills and practice with those skills. Often, one of the most critical steps in the process of conducting an interview is determin-ing the right person or people to interview. You will likely need to spend some time in the field observing the group you plan to study to determine which participants can contribute the most to your study.
As noted previously, an important component of conducting a good inter-view is the construction and use of the interinter-view protocol. Even when collecting semistructured interviews, it is important to have a protocol that will help guide the collection of data in a systematic and focused manner. Box 5.2 provides an example of the questions from an interview protocol. This protocol was devel-oped by Spaulding and Lodico (2004) to study community interest in a nontu-ition-based alternative school. Imagine the impossible task of trying to collect interview data from thirty community members without such a protocol. How would you remember all the questions? How could you be sure that you asked each person the same question in the same way? Varying a question by changing a few words can sometimes dramatically shift the intent of the question. Notice how the protocol helps to ensure a certain degree of standardization during the data collection process. In qualitative research, the researcher might also ask other questions during the interview to probe unexpected issues that emerge.
Box 5.2 Sample Interview Protocol for Feasibility Study
Community Member Questions
1. If you were to describe your community to someone, how would you describe it?
2. How has the community you just described changed over time? Probe: How has it changed?
Demographically?
Economically?
Religious affiliations?
School districts or school buildings?
3. Describe the relationship between your community and your local school district or building.
Strengths Barriers
Areas in need of improvement
Relationship with parents and parental involvement
4. What kind of response from the community do you think an alternative middle school that is nontuition-based would have? (Describe the program model.) 5. Do you believe an alternative school such as this would serve the community in
areas where the public school currently does not? Please explain.
After creating the protocol, you are ready to select and interview participants.
However, it is important to note here that when you conduct an interview, there are certain general procedures you should follow.
• Begin the interview by reintroducing yourself. Because you are doing qualitative re-search, you will likely have had some contact with the participant before the in-terviews. You may also want to introduce the general topic that you will be discussing.
• Remind the participant of the confidentiality of his or her responses. At this stage, the participant or guardian will have been told about confidentiality issues and will have agreed to participate in the study. However, it is important to review that in-formation at the beginning of the actual interview.
• Obtain general descriptive information. This type of descriptive information could include information about the participant or the issue or phenomenon being studied.
• Strive for neutrality. To maximize what participants tell you, it is particularly important that you are a good listener and nonjudgmental in your reactions. Be sensitive, and never act shocked or upset by what you hear. Being judgmental is likely to limit interaction and may cause the participant to question his or her level of trust in you. However, you must recognize that because you are conducting a qualitative interview, your values and personal biases, as well as those of the re-spondents, are a factor related to the kind of information you gather. As such, an-ticipate and document these to the best of your ability and consider the possible influence these might have on the data collected.
• Use effective probes. If you are conducting a semistructured or nonstructured interview, you will want to follow up on comments made by the participants with probes. A probe is a follow-up question that is asked to get clarification about a response. This will help the interviewer get more detailed information and give the respondent a chance to clarify responses. For example, say that you are inter-viewing a child like Max (discussed in Exhibit 5.5), who has difficulty interacting
Self-Developed Measures and Qualitative Measurement 125