Steps for Doing Descriptive-Survey Research
Step 1: Designing and Developing a Survey or Questionnaire
information that would appear on such a cover letter as part of an introductory statement. If a cover letter is not used, a paragraph that appears at the top of the survey can contain the essential information needed. Box 7.1 is a sample of what Alysia might write for her directions.
Box 7.1 Student Perception Survey
Note: Please Return by July 30
You have received this survey as part of a research study. The purpose of this study is to investigate student perception of school uniforms. This survey was designed to gather your beliefs about uniforms and how you believe uniforms will influence your school. Your responses are confidential and will not be shared with anyone in any way that identifies you as an individual. Only aggregated data will be presented in the final report. Your participation in this survey is completely voluntary and will not affect your grades. Your time and cooperation are greatly appreciated. If you have any questions regarding this survey or the study in general, please contact: Alysia Johnson, Admin-istration Intern, Baker High School, 123 Main Street, Los Angeles California.
Whether you use a cover letter or an introductory statement, the following information must be present in both:
• Purpose. You must provide the participants with information about the pur-pose of both the survey and the research project as a whole. If your survey is ask-ing participants about prior experiences, such as a professional development training or a new intervention-strategy program, you must clearly indicate that the purpose of the study is to collect information, opinions, or both, related to those specific prior experience(s). As the researcher, you must help to place the participants in the proper context for answering the questions. Do not assume that participants will make connections to those specific past experiences without being specifically instructed to do so. Such assumptions may result in invalid data that do not reflect the participants’ true beliefs or understandings about the experi-ence or phenomena you are studying.
• Confidentiality statement. As you likely remember from our ethics discussion, one of the most important duties of a researcher is maintaining confidential-ity. To do this, the researcher must ensure that what a participant says or reports will not be shared with anyone in any way (this also includes the write-up of the study) that would result in an individual being identified. Although this sounds
simple enough, survey researchers are often faced with the dilemma of not being able to report certain findings because individuals could be identified and what they said directly attributed to them. This can be a considerable problem when surveying in a small school district or other setting where there may be only one or two individuals in a particular position. Take, for example, a one-building K–12 school district in a rural community (yes, they do exist). In these districts, typically there is only one teacher per grade level. What if the findings indicated that the gym teacher, the English teacher, or the first-grade elementary teacher was dis-appointed with the administration for its inability to address issues surrounding students with special needs? The researcher would not be able to reveal this spe-cific information for fear of directly violating confidentiality. In larger settings, such concerns pose less of a problem. Take, for example, a large urban school dis-trict with many buildings at the elementary, middle, and secondary levels. A re-searcher could comfortably report that 50% of the first-grade teachers in the district believed that the administration is not meeting the needs of all students.
With a large number of teachers in the district, it would be virtually impossible to trace this finding back to the specific teachers with such concerns. Researchers should always carefully consider the issues that surround confidentiality. See Box 7.2 for a research example.
Box 7.2 Ethical Issue in Descriptive-Survey Research
Surveys in educational settings are often used to assess the incidence of self-reported problems in youth. For example, the Youth Risk Behavior Survey (YRBS), which is ad-ministered in hundreds of high schools each year, asks youth to report on how fre-quently they use drugs, engage in sexual activities, smoke, feel depressed, or engage in binge eating or vomiting to control their weight. The YRBS was designed by re-searchers at the University of Michigan to monitor changes over time in these risky behaviors. Surveys are also used in more local studies to determine the extent of prob-lem behaviors in a school or community or to monitor changes in probprob-lems when a prevention program is introduced. These surveys have the potential to reveal prob-lems that were previously unknown. However, one ethical dilemma that confronts sur-vey researchers is that students revealing the problems cannot be helped unless confidentiality is violated. Remember that confidentiality means that student responses cannot be shared with anyone, including parents. If a survey revealed that several stu-dents at a school were seriously depressed to the point of thinking about suicide, what might the researcher’s ethical responsibility be?
Legally, researchers are not required to report this type of problem. However, there are a variety of ways in which they might address the ethical concerns about doing nothing to help a student who might hurt himself or herself. Some researchers
Descriptive-Survey Research 161
might provide a general report to the school indicating that problems exist in certain areas and recommending that the school promote specific counseling services to ad-dress these. In fact, some researchers might work with the school in advance to make sure services are available in case the survey reveals problems. If the students’ answers indicated a need for more immediate attention, the researcher might arrange to talk with the students in private, expressing concern and making services available to them individually.
Often graduate students confuse confidentiality and anonymity. Unlike confi-dentiality, anonymity means that the survey does not require the participants or respondents to provide their name or any information that identifies them in any way (e.g., grade level currently teaching, number of years in current teaching position, name of school building). An anonymous survey is one that contains no demographic information that could be used to identify an individual. Survey de-velopment theory would support the use of an anonymous survey to increase the number of completed surveys that the researcher receives back. Participants are more likely to provide valid information if they believe there is no way that what they report will be traced back to them. Although these two benefits are certainly good reasons for using anonymous surveys, they make it impossible for a re-searcher to follow up with participants to collect any further in-depth information or validate what they have said.
• Statement of voluntary nature of study. In the introductory statement or cover letter, the researcher must inform participants that participation in the study is en-tirely voluntary. Participants must be informed that their lack of participation will not result in negative consequences. As you can see, Alysia made this clear to the students in her study. She told them that failing to participate would not influence their grades. For survey studies where the participant has no real “buy in” (a stake in the issues being explored by the survey), the decision to participate or not in the study would likely have little if any consequence. However, if a researcher was surveying people receiving social services or mental health services, respondents might fear that participation or nonparticipation could result in loss of their ser-vices. In this type of situation, a researcher must include a sentence that clearly states that participation or lack of it will not affect receipt of services. In addition, the researcher must inform participants that they have the right at any time to withdraw from the study or may choose not to answer any question.
• Contact information. The cover letter or introductory statement must provide contact information for use by participants in the event that they have a question or need further clarification on an item. The name of the principal investigator (also called the “P.I.”) should be listed somewhere, along with the name of the
in-stitution, group, or agency sponsoring the study (if there is one), a physical mail-ing address, phone number, and e-mail address (if available).
The cover letter or introductory statement must also include instructions on how to return the survey (e.g., there is a self-addressed-stamped envelope [SASE]
enclosed) and a deadline date for the return of the survey (on average, two weeks following the receipt of a survey is adequate; anything shorter or longer can lower response rates). Finally, the researcher can include any additional information spe-cific to the study that he or she believes to be important and should thank the par-ticipants for their time and cooperation.
Demographics. The next major section of a survey is the demographics section.
Demographics are descriptors that provide detailed information about partic-ipants in the study. The specific demographics you gather depend on the study you are conducting. If your participants are teachers, you might want to know what grade levels they teach, how long they have been teaching, their area of ex-pertise, and so forth. These are the typical demographics of interest to educational researchers. Presented in Exhibit 7.1 is an example of demographics that Alysia might gather for her study on students.
When deciding on the type of demographics you are going to include in your study, you might want to consider the following guidelines:
• Demographics should be derived from the literature review. When reviewing past stud-ies on your topic, be sure to note what demographics have been used in the past, as well as demographics that “emerge” from the literature. For example, you are interested in the topic technology integration and teacher pedagogy. On exami-nation of the literature, you discover that there is a consensus in the education field that the use of technology by teachers is somehow related to the length of time one has been a teacher. In other words, teachers who have been teaching more years are less likely to integrate technology into their teaching than new teachers. In creating demographics for the survey, you would certainly want to
Descriptive-Survey Research 163
EXHIBIT 7.1 SAMPLE DEMOGRAPHICS SECTION.
Sex: ____ male ____ female Age: ____ 12–13 ____ 14–15 ____ 16–17 ____ 18–19 ____ other Grade level: ____ 9th ____ 10th ____ 11th ____ 12th
Years in public school (fill in): ____ years
Any prior experience wearing uniforms? ____ yes ____ no
make sure that you have items gathering information on the number of years one has been teaching. Otherwise, you will have hundreds of surveys from teachers but no way to determine whether there was a difference between those teachers who have been teaching longer and those new to the field.
• Demographics should be used for specific purposes and not as a “shotgun” approach to gathering all possible information on participants. Many researchers pack as many de-mographics into a survey as they possibly can. Caution should be taken when se-lecting demographics. Too many demographic items may frustrate participants and negatively affect their desire to fill out the survey. Some participants may have concerns about how demographic information will be used, especially informa-tion on race and ethnicity.
• Demographics can go at the beginning or end of the survey, depending on their purpose.
Although demographics traditionally appear at the beginning of a survey, in some situations it may be more appropriate to end the survey with demographic items.
The decision to put the demographics later rather than sooner depends on how important these are to the researcher and to the study. If demographics are im-perative to the study, then they should go at the beginning of the survey when the participants are fresh and ready. If demographics are not imperative, they should go last, allowing the participants to exert most of their energies (and brain power) on answering the survey items.
Body of the Survey. The development of individual survey items is often confus-ing to those who have not created a survey. Students often mistakenly think that the subquestions are the survey questions. As we noted in the section on writing subquestions, the subquestions are written at the end of the literature and repre-sent the main sections or categories of the survey. The researcher then uses these categories to generate survey items.
For example, in Alysia’s study on student perceptions of school uniforms, one of her subquestions involved determining how students believed school uniforms would influence their school community. As in the example provided in Exhibit 7.2, the theme of “School Community” becomes the main header for that sec-tion of the survey. Notice how all the quessec-tions under that secsec-tion pertain to school community and solicit a quantitative response. The number of individual survey items in each section depends on the topic and can vary among the different cat-egories or themes of the survey. There is no “definitive” number of items that make up a survey. In general, for the purposes of the research proposal in this course, you should aim for 8 to 12 items for each category. In examining the five items presented, can you think of any other possible items that could be included for this section as they relate to outcomes?
Directions for Each Section. To ensure accuracy of the data, it is essential that, as the researcher, you provide instructions that clearly tell participants what to do (and think about) when answering each item of the survey. Although directions are touched on in a general sense in either the cover letter or in the beginning of the survey, each section or “theme” of the survey should include specific tions immediately preceding the detailed survey items. (Note the italicized direc-tions in Exhibit 7.2). This is important because in some situadirec-tions, each section of the survey may be intended to gather different types of information, and although the questions all look to be the same (let us say that they all refer in some way to school community), in fact, the researcher may want the participants to reflect and think about the sections quite differently. Putting certain parts of the direc-tions in ALL CAPS, italics, underlined, or bold (OR ALL FOUR) are effective ways of getting the participants’ attention.
Last, when writing directions for each survey section, clearly define any po-tentially ambiguous terms. This will help to increase the accuracy of the data. For example, if you are surveying elementary teachers regarding their technology use, using the word technology in the directions could mean different things to different people. You, the researcher, might be operating under the definition that “tech-nology” means computers—and nothing else. However, some of the teachers fill-ing out the survey might have a broader definition of the term, which includes
Descriptive-Survey Research 165
EXHIBIT 7.2 STUDENT PERCEPTIONS OF THE
IMPACT OF SCHOOL UNIFORMS ON THE SCHOOL COMMUNITY.
Read each question below and respond by circling the number that represents how you feel.
1 = strongly disagree; 2 = disagree; 3 = slightly disagree;
4 = slightly agree; 5 = agree; 6 = strongly agree
I think we will see positive changes in the 1 2 3 4 5 6 behavior of students in school cafeteria
I believe there will be less competition 1 2 3 4 5 6 among students.
I feel that students will get along better. 1 2 3 4 5 6
I am very positive about how school 1 2 3 4 5 6
uniforms will influence my school.
I believe my school will be a better place. 1 2 3 4 5 6
overhead projectors, VCRs, cable in the classroom, and the like, in addition to computers. (One of us actually encountered a group of teachers that believed, and reported, that Scotch tape is technology.) In these examples, participants’
broader definitions of the term “technology” would lead the researcher to over-report the use of technology in the study.
Criteria for Writing Good Survey Items. As discussed earlier, descriptive-survey researchers usually develop their own instruments. Sometimes they may borrow or modify items from surveys developed by other researchers. One should always ask for permission to use or adapt items from other researchers. Regardless of whether items are developed by the researcher or borrowed, all items should meet certain criteria for good survey items.
• Be clear and concise in the language used. Define all terms that could be inter-preted differently from what you intend. For example, if you are interested in studying teachers’ use of constructivist instructional approaches, it would be better to reword this term or define it. If you are ultimately interested in learning about teachers’ use of hands-on instruction, then you should modify the wording so that it is more specific and clear and use the words “hands-on learning.”
• Make sure that each survey item gathers data on one central idea or question. Although it is tempting to ask participants two questions in one item to save time and space, this will result in confusion, and it will be impossible to interpret responses. For example, consider an item asking participants to respond on a scale of “strongly agree” to “strongly disagree” to the following statement: “I believe that school uniforms will make the cafeteria and the halls more friendly.” If a participant agrees strongly with this statement with regard to the cafeteria but disagrees with regard to the halls, how should the participant respond to this item? Strongly agree? Strongly disagree? Not applicable? To fix these problems, the researcher must break this question down into two separate questions, each containing one of the essential ideas.
• Avoid using double negatives. For example: “It is not good practice for teachers to not meet the needs of all their students.” As you read this item, you probably felt yourself stop and think about it for a second before answering it, right? It is difficult to determine what a response of “strongly agree” or “strongly disagree”
would mean in this case.
• Make sure that items (particularly demographic items) have response sets that do not over-lap. This is a common mistake, even in professional surveys. Consider a survey item that asks participants to indicate annual income levels. If you have selections that overlap, as in the example below, it will be virtually impossible for you to de-cipher who has selected the correct response for their income level.
Indicate your income level by checking one of the categories below.
__ $0–$10,000 __ $10,000–$30,000 __ $30,000–$50,000 __ $50,000–$80,000 __ $80,000–$100,000 __ $100,000 or more
If you were trying to fill out this demographic item and you made $50,000 an-nually, which one would you select, $30,000-$50,000 or $50,000-$80,000? What if, like you, 99% of the sample made $50,000, and they all selected the first choice but you selected the second? The finding that 99% of the sample made
$30,000-$50,000 and that only 1% made $30,000-$50,000-$80,000 would be incorrect. In fact, in this example, the entire sample made $50,000 that year.
• Include all possible responses. For example, if you are asking participants how many times that week they used technology in their classrooms and you provide a frequency checklist like the one below, it is important that you provide a “zero”
or “none” category.
How many times did you integrate technology into your lessons last week?
__ 0 __ 1–3 __ 4–6 __ 7–9 __ 10–12 __13 or more
If you did not provide a zero as a possible answer, many of your participants may not be able to provide an accurate answer (assuming someone never used technol-ogy), and the data will overestimate the degree of technology integration. If the zero response is not provided, and 100% of teachers filling out the survey had not used technology in their teaching the week before, they either leave the question blank or might select the next item that most closely represents their situation: 1–3 times. If 60% chose “1–3,” the researcher would report that 60% of respondents used tech-nology one to three times in the week before being surveyed when really none had.
Including all possible responses also includes the use of the category “Other.”
The benefit of a researcher providing this category is that it opens the response setup so that the participant can provide the most accurate response possible. If you use the category “Other,” be sure to provide a place for the participant to actually write in the detailed response. However, if an overwhelming number of respondents select the “Other” category on many different items, this could mean that the researcher
Descriptive-Survey Research 167