ESSENTIALS OF RELIABILITY - of Psychological Testing

TRUTH AND ERROR IN PSYCHOLOGICAL MEASUREMENT One of the most enduring approaches to the topic of reliability is the classical test theory notion of the true score(see, e.g., Gulliksen, 1950). In a way, this notion could be said to represent the object of the quest, or Holy Grail, of the psycho- metric enterprise. Although true scores do not really exist, it is nevertheless possible to imagine their existence: True scores are the hypothetical entities that would result from error-free measurement. Methods for estimating the reliability of scores provide a way of estimating true scores, or at least the boundaries within which true scores might lie. The concepts of reliability and error in test scores—

which must obviously enter into consideration with regard to any score—are applied in parallel yet somewhat different ways when dealing with one or more scores of a single individual than when dealing with the scores of groups.

The Concept of the True Score in Individual Data

In classical test theory, an individual’s true scoreis conceptualized as the average score in a hypothetical distribution of scores that would be obtained if the individual took the same test an inﬁnite number of times. In practice, it is obviously impossible to obtain such a score for even one individual, let alone for many. In- stead of true scores, what one derives from tests are observed scores(i.e., the scores that individuals actually obtain).

With regard to a single score, the ideas presented so far can be stated succinctly by means of the following equation:

X_o= X_true+ X_error (4.1)

which expresses the concept that any observed score ( X_o) is made up of two components: a true score component ( X_true) and an error score component ( X_error).

From a realistic point of view, the magnitudes of both of these components will always remain unknown in any given instance. Nevertheless, in theory, the true score component is construed to be that portion of the observed score which re- ﬂects whatever ability, trait, or characteristic the test assesses. Conversely, the error component, which is deﬁned as the difference between the observed score and the true score, represents any other factors that may enter into the observed score as a consequence of the measurement process.

True Scores in Group Data

The singular importance of interindividual variability was already discussed in Chapter 2, where it was pointed out that the usefulness of psychological testing 118 ESSENTIALS OF PSYCHOLOGICAL TESTING

hinges on obtaining some variability across individuals. Without score variability, tests could not help us to make comparative decisions about people. It may also be recalled that, in the same chapter, the sample variance (s²) was deﬁned as the average amount of variability in a group of scores. Based on this information, Formula (4.1)—which pertains to a single test score—can be extrapolated and applied to the distribution of test scores obtained from a sample, or a population, in the following fashion:

Sample variance = s²=s_t²⫹s_e² (4.2) or

Population variance = σ²=σt 2⫹σe

2 (4.3)

Both of these formulas express the same idea, namely, that the variance in a set of observed scores for a sample (s²) or a population (σ²) consists of a portion that is true variance (s_t²orσt

2) and a portion that is error variance (s_e²orσe

2). True variance consists of those differences among the scores of individuals within a group that reﬂect their standing or position in whatever characteristic the test assesses.

Error variance is made up of the differences among test scores that reflect factors irrelevant to what the test assesses. Formulas (4.2) and (4.3) also imply that the reliability of scores increases as the error component decreases. In fact, a reliability coefficient(r_xx)—about which more will be said later in this chapter—may be de- fined as the ratio of true score variance (s_t²) to total test score variance (s²), or,

r_xx=ᎏs s

t 2

ᎏ2 (4.4)

In other words, if all the test score variance were true variance, score reliability would be perfect (1.00). A reliability coefﬁcient may be viewed as a number that estimates the proportion of the variance in a group of test scores that is accounted for by error stemming from one or more sources. From this perspective, the eval- uation of score reliability involves a two-step process that consists of (a) deter- mining what possible sources of error may enter into test scores and ( b) estimating the magnitude of those errors.

THE RELATIVITY OF RELIABILITY

Although the practice of describing testsas reliable is common, the fact is that the quality of reliability is one that, if present, belongs not to tests but to test scores.

This distinction is emphasized consistently by most of the contributors to the contemporary literature on reliability (see, e.g., Rapid Reference 4.1). Even

though it may seem subtle at first glance, the distinction is fundamental to an understanding of the implica- tions of the concept of reliability with regard to the use of tests and the in- terpretation of test scores. If a test is described as reliable, the implication is that its reliability has been established permanently, in all respects, for all uses, and with all users. This would be akin to saying that a fine piano that is well tuned will always be in tune and will sound just as good regardless of the kind of music that is played on it or of who plays it. In fact, the quality of the sound a piano makes is a function not only of the instrument itself, but also of variables related to the music, the piano player, and the setting (e.g., the acoustics of the room) where the piano is played. Simi- larly, although reliability in testing hinges to a significant extent on characteristics of the test itself, the reliability of test scores—which is what results from the use of the instrument and, like the music a piano makes, is what really matters—can also be affected by many other variables.

Even as applied to test scores, moreover, the quality of reliability is relative.

The score a person obtains on a test is not reliable or unreliable in any absolute sense. Rather, an obtained score may be more or less reliable due to factors uniquely pertinent to the test taker (e.g., fatigue, lack of motivation, the inﬂuence of drugs, etc.) or to the conditions of the testing situation (e.g., the presence of distracting noises, the personality of the examiner, the strictness with which time limits are enforced, etc.). All of these factors may singly or jointly affect the obtained score to a greater or lesser extent, including up to the point where the score becomes so unreliable that it should be discarded. Even though they are unrelated to the test itself, all such matters need to be taken into account in the assessment process.

In contrast, when reliability (r_xx) is considered from the point of view of test score data obtained from a large sample, under standardized conditions,the measurement errors that may impinge on the individual scores of members of the sample, while still present, are assumed to be distributed at random. Since random errors are equally likely to inﬂuence scores in a positive or a negative direction, the errors can also be assumed to cancel each other out. Even then, however, reliability estimates will vary from sample to sample depending on their composition and the circumstances in which the testing occurs. For instance, if a test is aimed at adults ranging in age from 18 to 90 years, the estimated reliability of its scores 120 ESSENTIALS OF PSYCHOLOGICAL TESTING

An excellent collection of readings on reliability, with more detailed and ex- tensive explanations of many topics covered in this chapter, can be found inScore Reliability: Contemporary Think- ing on Reliability Issues,edited by Bruce Thompson (2003b).

Rapid Reference 4.1

will be susceptible to the inﬂuence of different factors depending on whether the estimate is based on data obtained from the older age groups, from the younger age groups, or from a group that is representative of the entire age range for which the test is intended.

A Note About Truth and Error The need to identify and investigate the true components and the error components of scores is considered in more detail in the next section. At

this point, however, it is important to emphasize that these judgments always have to be made in relation to what a test attempts to assess and to the circumstances under which it is administered. For example, test scores that denote the speed with which individuals can place 100 pegs in a pegboard, if given under standardized conditions, would by and large reflect the test takers’ typical levels of manual dexterity quite reliably. If the same test were given (a) under conditions meant to distract the test takers or ( b) under conditions that unintentionally dis- tracted the test takers, both sets of scores would reflect levels of manual dexterity under distracting conditions. However, the influence of the distractions would be seen as a source of error variance, or as reducing the reliability of what the scores are intended to indicate only in the second instance.

SOURCES OF ERROR IN PSYCHOLOGICAL TESTING

As we have seen, error can enter into the scores of psychological tests due to an enormous number of reasons, many of which are outside the purview of psycho- metric estimates of reliability. Generally speaking, however, the errors that enter into test scores may be categorized as stemming from one or more of the following three sources: (a) the context in which the testing takes place (including factors related to the test administrator, the test scorer, and the test environment, as well as the reasons the testing is undertaken), ( b) the test taker, and (c) the test itself. Some of the errors stemming from these sources can be minimized or eliminated provided that proper testing practices are followed by the parties who are involved in the process of developing, selecting, administering, and scoring tests.

DON’T FORGET

• Reliability is a characteristic of test scores,rather than of tests themselves.

• The reliability of any measurement, and of psychological test scores in particular, is neither absolute nor immutable.Thus, the possible sources of error in measurement, and the extent to which they enter into any speciﬁc use of a test, must be taken into account, estimated, and reported every time test scores are employed (AERA, APA, NCME, 1999).

Still others, such as test takers’ carelessness or attempts at impression manage- ment in responding to test items, cannot be eliminated but may be detected by various types of checks built into the test. At any rate, practices related to the appropriate use of tests—practices most of which are aimed at reducing the error in scores—are discussed at greater length in Chapter 7, which deals with issues relevant to test selection, administration, and scoring, among others.

For the purpose of this discussion of reliability, it will be assumed that test users, test administrators, and test scorers carefully select the appropriate instru- ments, prepare suitable test environments, establish good rapport with the test takers, and both administer and score tests in accordance with well established standardized procedures. Furthermore, it will also be assumed that test takers are properly prepared and well motivated to take tests. Whether these assumptions are or are not justified in specific instances, the fact remains that the behaviors that they entail are subject to the control of one or more of the individuals involved in the testing process and are not pertinent to the tests themselves in a di- rect way. To the extent that these assumptions are justified, the error in test scores that stems from sources unrelated to the tests can obviously be eliminated or at least minimized.

In considering reliability for the remainder of this chapter, the sources of error to be discussed pertain primarily to factors outside the conscious control of the parties in the testing process, namely, random or chance factors. Before we proceed, however, it should be noted that measurement error can be systematic and consistent as well as random. Just as a balance may be off by a few pounds, a test may have an intrinsic characteristic of some sort that affects all test takers.

Traditional estimates of reliability may fail to detect this sort of consistent error, depending on its source, because they are based on methods meant to detect in- consistencies in the results of a test. Systematic and consistent errors in measurement affect not only the reliability but also the validity of test results. In order to detect them, one must be able to compare the results of one instrument with those of other tools that assess the same construct but do not share the fac- tor that causes the consistent error. To detect the error in the case of the balance that is off by a few pounds, for instance, one would have to weigh the same person or object on one or more additional and well calibrated scales.

Rapid Reference 4.2 lists some of the possible sources of error that may ren- der test scores inconsistent. This list categorizes the sources of error assessed by traditional estimates of reliability, along with the types of tests to which those error sources pertain most directly, and the reliability coefﬁcients typically used to estimate them. A conceptual explanation of each of the sources of error and reliability estimates is presented next, in the same order as they are listed in Rapid 122 ESSENTIALS OF PSYCHOLOGICAL TESTING

Reference 4.2. Considerations regarding when and how these concepts and procedures are applied in the process of test use are discussed in a subsequent section of this chapter.

Interscorer Differences

Interscorer (orinterrater)differencesis the label assigned to the errors that may enter into scores whenever the element of subjectivity plays a part in scoring a test. It is assumed that different judges will not always assign the same exact scores or ratings to a given test performance even if (a) the scoring directions speciﬁed in the test manual are explicit and detailed and ( b) the scorers are conscientious in applying those directions. In other words, score variability that is due to inter-

Sources of Measurement Error With Typical Reliability Coefﬁcients Used to Estimate Them

Appropriate Type of Tests Prone Measures Used to Source of Error to Each Error Source Estimate Error Interscorer differences Tests scored with a de- Scorer reliability

gree of subjectivity

Time sampling error Tests of relatively stable Test-retest reliability (r_tt) traits or behaviors a.k.a. stability coefﬁcient Content sampling error Tests for which consis- Alternate-form reliability

tency of results, as a (r_1I) or split-half reliability whole, is desired

Interitem inconsistency Tests that require inter- Split-half reliability or item consistency more stringent internal

consistency measures, such as Kuder-

Richardson 20 (K-R 20) or coefﬁcient alpha (α) Interitem inconsistency Tests that require inter- Internal consistency and content heterogeneity item consistency and measuresandadditional

combined homogeneity evidence of homogene-

ity

Time and content Tests that require stability Delayed alternate-form sampling error combined andconsistency of re- reliability

sults, as a whole

Rapid Reference 4.2

scorer differences does not imply carelessness either in preparing the directions for scoring or in the actual scoring of a test. It refers to variations in scores that stem from differences in the subjective judgment of the scorers.

Scorer Reliability

The basic method for estimating error due to interscorer differences consists of having at least two different individuals score the same set of tests, so that for each test taker’s performance two or more independent scores are generated. The correlations between the sets of scores generated in this fashion are indexes of scorer reliability.Very high and positive correlations, in the order of .90 or higher, suggest that the proportion of error that is accounted for by interscorer differences is 10% or less, since 1 – (≥.90) = ≤.10.

Time Sampling Error

Time sampling errorrefers to the variability inherent in test scores as a function of the fact that they are obtained at one point in time rather than at another. This concept hinges on two related notions, namely, (a) that whatever construct or be- havior a test evaluates is liable to fluctuate in time, and ( b) that some of the constructs and behaviors assessed through tests are either less subject to change, or change at a much slower pace, than others. For example, psychological constructs related to abilities, such as verbal comprehension or mechanical aptitude, usually are seen as being less prone to fluctuation over time than constructs related to personality, such as agreeableness or warmth. Within the realm of personality and emotional functioning the difference between more and less stable constructs has been codified in the traditional distinction of traits—which are construed to be relatively enduring characteristics—versus states,which are by definition tem- porary conditions. To some extent, this distinction may also be applied to cogni- tive characteristics. Verbal ability, for instance, is assumed to be far more stable within an individual than attention and memory capacities, both of which are more susceptible to the influence of transient conditions or emotional states. At any rate, it is clear that whereas a certain amount of time sampling error is assumed to enter into all test scores, as a rule, one should expect less of it in the scores of tests that assess relatively stable traits.

Test-Retest Reliability

To generate estimates of the amount of time sampling error liable to affect the scores of a given test, it is customary to administer the same test on two different occasions, separated by a certain time interval, to one or more groups of individuals. The correlation between the scores obtained from the two administrations 124 ESSENTIALS OF PSYCHOLOGICAL TESTING

is a test-retest reliability(orstability)coefﬁcient(r_tt) and may be viewed as an index of the extent to which scores are likely to ﬂuctuate as a result of time sampling error.

When this procedure is used, the time interval between the two test administrations always has to be speciﬁed, as it will obviously affect the stability of the scores. Realistically speaking, however, there are many factors that can differen- tially affect the test scores derived from a group of people across two occasions.

Because of this, there is no fixed time interval that can be recommended for all tests. If the interval is very short, for instance, test takers may remember their re- sponses from the first occasion and this could affect their scores on the second one. On the other hand, if the interval is very long, there is always the possibility that intervening experiences—including steps that the test takers may have taken in reaction to the first administration—may affect the scores on the second occasion. In addition to these considerations, test users must also evaluate stability coefficients from the point of view of theoretical expectations pertinent to the traits and behaviors assessed by the test. One example would be the differences in the rate of change that may be expected as a function of the age of the test takers. Reading comprehension, for instance, can change fairly rapidly in young chil- dren but should remain stable through adulthood, unless some unusual circum- stance—such as special training or brain injury—affects it.

Content Sampling Error

Content sampling erroris the term used to label the trait-irrelevant variability that can enter into test scores as a result of fortuitous factors related to the content of the speciﬁc items included in a test. A simple example of how content sampling error might enter into test scores is presented in Rapid Reference 4.3. This illus- tration of content sampling error is rather contrived because it pertains to error introduced into test scores as a result of faulty test construction, and thus could have been avoided easily. In the example, the teacher’s selection of items results in inadequate coverage of the content knowledge the test is supposed to evaluate.

Consequently, a good portion of score variability is unrelated to the students’

level of mastery of the speciﬁed material, rendering their scores not only less reliable but also less valid than they would be otherwise. A more typical example is posed by cases in which—for reasons outside of the control of the test devel- oper—the speciﬁc content of a test either favors or disadvantages some of the test takers, based on their different experiential histories. For instance, a test de- signed to evaluate reading comprehension may fortuitously include several passages that are familiar to some test takers and not familiar to others. Obviously, those test takers who are familiar with the passages would be able to answer ques-

Dalam dokumen of Psychological Testing - EPDF.MX (Halaman 130-164)