The domain-sampling method requires the development of a pool of items. The process of creating a large pool of items is time-consuming and sometimes impracti- cal when a researcher plans to use just one knowledge test in a study. Even when not using the test pool, the researcher can develop more items than are required for the test. The researcher can then use the item analysis techniques presented later in this chapter to select the items that perform best for the population of interest.
Behavioral scientists often include all possible knowledge items in a given content domain to create a comprehensive test. For example, they may include all possible routes of HIV transmission on an HIV knowledge test. However, if the test is con- ceptualized within the domain-sampling model, scores on a test composed of an ad- equate representation of items should correlate with the scores on a test composed of a comprehensive set of items.
Multiple-Choice Items
1. Measure relevant information. One of the first recommendations is the assess- ment of relevant information for a given population. If our researcher wants to de- velop a test to measure the HIV knowledge of college students, the information should be relevant to college students. A specialist in HIV may be able to answer an item related to the number of infants infected with HIV. However, because this informa- tion is unlikely to be of interest to or known by college students, a better item may ask instead about HIV transmission, which college students need to know about.
Low-relevance stem: Overall, how many infants have been infected with HIV from their mothers since the beginning of the AIDS epidemic?
High-relevance stem: HIV can be transmitted though which of the following routes?
2. State a single, clear problem in the stem. In the poor stem that follows, the in- troduction of both beliefs and myths can be confusing for respondents. The better stem is simple and to the point.
Poor stem: People have many beliefs about how HIV is transmitted between persons. Some of these beliefs are correct and others are myths. Which of the following do you think is not a myth?
Better stem: Which of the following events can lead to the transmission of HIV?
3. State the stem in simple, clear language. The following poor stem is convo- luted and difficult to read, whereas the better stem is clearly stated.
Poor stem: A reduction in HIV can be achieved by which of the following strate- gies designed to reduce transmission?
Better stem: Which of the following can reduce the transmission of HIV?
4. Put as much of the necessary wording as possible in the stem (don’t repeat it in the answers).
Poor stem: Which of the following statements is not true?
A. Birth control pills will not prevent HIV transmission.
B. Urination after intercourse will not prevent HIV transmission.
C. Condoms will not prevent HIV transmission.
D. Cleaning needles with soap and water will not prevent HIV transmission.
Better stem: Which of the following can prevent transmission of HIV?
A. Birth control pills
B. Urination after intercourse C. Condoms
D. Cleaning needles with soap and water 5. State the item in positive form.
Poor stem: Which of the following is not an effective HIV prevention strategy?
Better stem: Which of the following is an effective HIV prevention strategy?
6. Emphasize a negative word if it appears in the stem. Sometimes it is im- possible to write a stem without using a negative word. In these cases, highlight the negative word to draw attention to it. Doing so will reduce errors due to misreading the stem.
Poor stem: HIV cannot be transmitted by . . . Better stem: HIV cannot be transmitted by . . .
7. Make certain that the intended answer is the correct answer or the best answer. In the example of poor answer choices that follows, both cleaning needles and using condoms can be correct answers. In the item with better answer choices, only con- doms is the correct answer.
Poor answer choices: Which of the following will prevent the transmission of HIV?
A. Birth control pills
B. Urination after intercourse C. Condoms
D. Cleaning needles
Better answer choices: Which of the following will prevent the transmission of HIV?
A. Birth control pills
B. Urination after intercourse C. Condoms
D. Cleaning needles with water
8. Make all answers grammatically consistent with the stem.
Poor answer choices: HIV is . A. a virus
B. a bacterium
C. has characteristics of both a virus and a bacterium D. no one really knows
Better answer choices: HIV is . A. a virus
B. a bacterium
C. a combination of a virus and a bacterium D. an unknown organism
9. Avoid verbal clues that lead to the correct answers or help eliminate in- correct answers. The poor stem that follows provides a clue that HIV is a virus.
Poor stem: Human immunodeficiency virus (HIV) is . A. a virus
B. a bacterium
C. a combination of a virus and a bacterium D. an unknown organism
Better stem: HIV is .
10. Make distracters plausible and appealing. In the following example of poor answer choices, one of the distracters (incorrect responses) is reading about HIV. Most college students would immediately identify this choice as incorrect or implausible.
Likewise, wearing a copper bracelet has not been associated with HIV prevention and may not function as a plausible distracter. In the better answer choices, the distracters of washing one’s hands, using birth control pills, and cleaning needles with water are likely to be more plausible because both washing of hands and clean- ing of needles are activities generally engaged in to prevent infection. Some students believe that birth control pills will prevent sexually transmitted diseases as well as pregnancy.
Poor answer choices: Sexually active people can reduce their risk of contracting HIV by
A. wearing a copper bracelet B. using birth control pills C. using condoms D. reading about HIV
Better answer choices: Sexually active people can reduce their risk of contracting HIV by
A. washing their hands B. using birth control pills C. using condoms
D. cleaning needles with water
11. Use “All of the above” and “None of the above” as responses sparingly.
Poor answer choices: Which of the following is a way to transmit HIV?
A. Unprotected sex B. Unclean needles
C. HIV-infected mother to infant D. All of the above
Better answer choices: Which of the following lists the three primary ways of transmitting HIV?
A. Unprotected sex, unclean needles, HIV-infected mother to infant B. Unprotected sex, unclean kitchen utensils, mosquitoes
C. Unprotected sex, unclean needles, mosquitoes
D. Unprotected sex, unclean kitchen utensils, HIV-infected mother to infant
12. Use the two-layered response option sparingly. The two-layered item con- sists of two sets of options. The first set consists of all correct, all incorrect, or a combination of correct and incorrect answer choices. The second set of options contains combinations of the answer choices presented in the first set of options. The task for the respondent is to select the correct response from the second set of options.
Because it requires two judgments, the two-layered item is more complex than the typ- ical multiple-choice item. As shown here, presenting just the second set of options with an appropriate stem simplifies the item.
Poor answer choices: Which of the following is a way to transmit HIV?
1. Unprotected sex 2. Kissing
3. HIV-infected mother to infant 4. Mosquitoes
A. 1 and 2 only B. 1 and 3 only C. 1, 2, and 3 D. 1, 3, and 4
Better answer choices: Which of the following lists the three primary ways of transmitting HIV?
A. Unprotected sex, unclean needles, HIV-infected mother to infant B. Unprotected sex, unclean kitchen utensils, mosquitoes
C. Unprotected sex, unclean needles, mosquitoes
D. Unprotected sex, unclean kitchen utensils, HIV-infected mother to infant 13. Make each item independent. If the answers to two items are interdependent, the respondent who selects the wrong answer for the first item has a high probability of selecting the wrong answer for the second item. A fairer test is produced by mak- ing the items independent.
1. Which of the following lists the three primary ways of transmitting HIV?
A. Unprotected sex, unclean needles, HIV-infected mother to infant B. Unprotected sex, unclean kitchen utensils, mosquitoes
C. Unprotected sex, unclean needles, mosquitoes
D. Unprotected sex, unclean kitchen utensils, HIV-infected mother to infant 2. For which of the three primary modes of transmission mentioned above can med-
ications be used to reduce the likelihood of transmission?
A. Unprotected sex B. Mosquitoes C. Needles or utensils
D. HIV-infected mother to infant
The correct answer for the first question is A, and the correct answer for the second question is D (HIV-infected mother to infant). If a respondent selected B or C to answer question 1, there is a possibility that he or she would select B, the wrong an- swer, for question 2.
One way to unlink the two questions would be to change the stem for question 2 as follows: “For which of the following modes of transmission can drugs be used to reduce the likelihood of transmission?”
14. Vary the position of the correct answer in a random fashion. After de- velopers complete the set of items, they review the positions of the correct answers.
If the correct answer often falls in a certain position (for example, frequently answer C is correct), respondents may use this placement as a clue to select the cor- rect answer. After test developers determine the order of the items, they should po- sition the correct answers by random selection or a similar process to vary their positions.
15. Vary the length of the correct answer to eliminate length as a clue. Ideally, answer choices are of similar length. (Of course, this parity is sometimes impossible to achieve.) When the correct answer is always the longest or the shortest in a set of items, respondents will use this clue to select the correct answer. Thus, when reviewing items, make certain that the correct answer varies in length and is consistently neither the shortest nor the longest choice.
True/False Items
The following are recommendations for writing true/false and yes/no items.
1. Include only one central significant idea in each statement.
Poor: A person can get HIV from mosquitoes and toilet seats.
Better: A person can get HIV from mosquitoes.
2. Word the statement so precisely that respondents can judge it as unequiv- ocally true or false. The poor item shown here does not contain the qualification that the partner has HIV. Thus, a respondent could reason that the statement is true only under certain conditions.
Poor: Can a person get HIV through having sexual intercourse?
Better: Can a person get HIV by having sexual intercourse with someone who is infected with HIV?
3. Keep the statements short and keep the language structure simple.
Poor: Human immunodeficiency virus is the same thing as acquired immuno- deficiency syndrome.
Better: HIV is the same as AIDS.
4. Use negative statements sparingly. Items containing negative words such as not and prefixes such as un- and non- are frequently misread by respondents. If a nega- tive term is used, emphasize it.
Poor: A person cannot get HIV from mosquitoes.
Better: A person cannot get HIV from mosquitoes.
An even better approach is to eliminate the word not (this will also change the cor- rect answer).
A person can get HIV from mosquitoes.
5. Avoid double negatives.
Poor: A person cannot reduce his or her chances of getting HIV by not shaking hands with someone who has HIV.
Better: A person can get HIV by shaking hands with someone who has HIV.
6. Attribute statements of opinion to some source.
Poor: People should use condoms to reduce their chances of contracting HIV.
Better: According to the CDC, sexually active people should use condoms to re- duce their chances of contracting HIV.
7. Avoid extraneous clues to the answer.
Poor: Condoms help prevent HIV; therefore, people can reduce their chance of getting HIV by using a condom when they have sexual intercourse.
Better: People can reduce their chance of getting HIV by using a condom when they have sexual intercourse.
8. Randomly vary the sequence of true statements and false statements (or yes statements and no statements). After completing the set of items, review it for per- centages of true statements and of false statements. Ideally, half of the items will be true and half will be false. Developers should order the items randomly to prevent a pattern of true and false statements that could serve as a clue for respondents.