The NAEP estimation procedures start with the assumption that the proficiency of a student in an assessment area can be estimated from a student’s responses to the assessment items that the student received. The psychometric model is a latent regression consisting of four types of variables:
• Student proficiency
• Student item responses
• Conditioning variables
• Error variables
The true proficiency of a student is unobservable and thus unknown. The student item responses are known, since they are collected in an assessment. Also known are the conditioning variables that are collected for reporting (e.g., demographics) or may be otherwise considered related to student proficiency. The error variable is the difference between the actual student proficiency and its estimate from the psy- chometric model and is thus unknown.
The purpose of this appendix is to present the many ways in which ETS research- ers have addressed the estimation problem and continue to look for more precise and efficient ways of using the model. Estimating the parameters of the model requires three steps:
1. Scaling 2. Conditioning 3. Variance estimation
Scaling processes the item-response statistics to develop estimates of student proficiency. Conditioning adjusts the proficiency estimates in order to improve their accuracy and reduce possible biases. Conditioning is an iterative process using the estimation–maximization (EM) algorithm (Dempster et al. 1977) that leads to maxi- mum likelihood estimates. Variance estimation is the process by which the error in
the parameter estimates is itself estimated. Both sampling and measurement error are examined.
The next section presents some background on the original application of this model. This is followed by separate sections on advances in scaling, conditioning, and variance estimation. Finally, a number of alternate models proposed by others are evaluated and discussed.
The presentation here is not intended to be highly technical. A thorough discus- sion of these topics is available in a section of the Handbook of Statistics titled
“Marginal Estimation of Population Characteristics: Recent Developments and Future Directions” (von Davier et al. 2006).
The Early NAEP Estimation Process
NAEP procedures proposed by ETS were conceptually straightforward: the item responses are used to estimate student proficiency, and then the student estimates are summarized by gender, racial/ethnic groupings, and other factors of educational importance. The accuracy of the group statistics would be estimated using sampling weights and the jackknife method which would take into account the complex NAEP sample. The 3PL IRT model was to be used as described in Lord and Novick (1968).
This approach was first used in the 1983–1984 NAEP assessment of reading and writing proficiency. The proposed IRT methodology of that time was quite limited:
it handled only multiple-choice items that could be scored either right or wrong. It also could not make any finite estimates for students who answered all items cor- rectly or scored below the chance level. Since the writing assessment had graded- response questions, the standard IRT programs did not work, so the ARM was developed by Beaton and Johnson (1990). The ARM was later replaced by the PARSCALE program (Muraki and Bock 1997).
However, the straightforward approach to reading quickly ran into difficulties.
The decision had been made to BIB spiral the reading and writing items, with the result that many students were assigned too few items to produce an acceptable estimate of their reading proficiency. Moreover, different racial/ethnic groupings had substantially different patterns of inestimable proficiencies, which would bias any results. Standard statistical methods did not offer any solution.
Fortunately, Mislevy had the insight that NAEP did not need individual student proficiency estimates; it needed only estimates of select populations and subpopula- tions. This led to the use of marginal maximum likelihood methods through the BILOG program (Mislevy and Bock 1982). The BILOG program could estimate group performance directly, but an alternative approach was taken in order to make the NAEP database useful to secondary researchers. BILOG did not develop accept- able individual proficiency estimates but did produce a posterior distribution for each student that indicated the likelihood of possible estimates. From these distribu- tions, five plausible values were randomly selected. Using these plausible values
made data analysis more cumbersome but produced a data set that could be used in most available statistical systems.
The adaptation and application of this latent regression model was used to pro- duce the NAEP 1983–1984 Reading Report Card, which has served as a model for many subsequent reports. More details on the first application of the NAEP estima- tion procedures were described by Beaton (1987) and Mislevy et al. (1992).
Scaling
IRT is the basic component of NAEP scaling. As mentioned above, the IRT pro- grams of the day were limited and needed to be generalized to address NAEP’s future needs. There were a number of new applications, even in the early NAEP analyses:
• Vertical scales that linked students aged 9, 13, and 17.
• Across-year scaling to link the NAEP reading scales to the comparable assess- ments in the past.
• In 1986, subscales were introduced for the different subject areas. NAEP pro- duced five subscales in mathematics. Overall mathematics proficiency was esti- mated using a composite of the subscales.
• In 1992, the generalized partial credit model was introduced to account for graded responses (polytomous items) such as those in the writing assessments (Muraki 1992; Muraki and Bock 1997).
Yamamoto and Mazzeo (1992) presented an overview of establishing the IRT- based common scale metric and illustrated the procedures used to perform these analyses for the 1990 NAEP mathematics assessment. Muraki et al. (2000) provided an overview of linking methods used in performance assessments, and discussed major issues and developments in linking performance assessments.
Conditioning
As mentioned, the NAEP reporting is focused on group scores. NAEP collected a large amount of demographic data, including student background information and school and teacher questionnaire data, which can be used to supplement the nonre- sponse due to BIB design and to improve the accuracy of group scores.
Mislevy (1984, 1985) has shown that maximum likelihood estimates of the parameters in the model can be obtained when the actual proficiencies are unknown using an EM algorithm.
The NAEP conditioning model employs both cognitive data and demographic data to construct a latent regression model. The implementation of the EM algo- rithm that is used in the estimation of the conditioning model leaves room for
possible improvements in accuracy and efficiency. In particular, there is a complex multidimensional integral that must be calculated, and there are many ways in which this can be done, each method embodied by a computer program which has been carefully investigated for advantages and disadvantages. These programs have been generically labeled as GROUP programs. The programs that have been used or are currently in use are as follows:
• BGROUP (Sinharay and von Davier 2005). This program is a modification of BILOG (Mislevy and Bock 1982) and uses numerical quadrature and direct inte- gration. This is typically used when there are one or two scales being analyzed
• MGROUP (Mislevy and Sheehan 1987) uses a Monte Carlo method to draw random normal estimates from posterior distributions as input to each estimation step.
• NGROUP (Allen et al. 1996; Mislevy 1985) uses Bayesian normal theory. The requirement of the assumption of a normal distribution results in little use of this method.
• CGROUP (Thomas 1993) uses a Laplace approximation for the posterior means and variance. This method is used when more than two scales are analyzed.
• DGROUP (Rogers et al. 2006) is the current operational program that brings together the BGROUP and CGROUP methods on a single platform. This plat- form is designed to allow inclusion of other methods as they are developed and tested.
To make these programs available in a single package, ETS researchers Ted Blew, Andreas Oranje, Matthias von Davier, and Alfred Rogers developed a single program called DESI that allows a user to try the different latent regression programs.
The end result of these programs is a set of plausible values for each student.
These are random draws from each student’s posterior distribution, which gives the likelihood of a student having a particular proficiency score. The plausible value methodology was developed by Mislevy (1991) based on the ideas of Little and Rubin (1987, 2002) on multiple imputation. These plausible values are not appro- priate for individual proficiency scores or decision making. In their 2009 paper,
“What Are Plausible Values and Why Are They Useful?,” von Davier et al. described how plausible values are applied to ensure that the uncertainty associated with mea- sures of skills in large scale surveys is properly taken into account. In 1988, NCME gave its Award for Technical Contribution to Educational Measurement to ETS researchers Robert Mislevy, Albert Beaton, Eugene Johnson, and Kathleen Sheehan for the development of plausible values methodology in the NAEP.
The student plausible values are merged with their sampling weights to compute population and subpopulation statistical estimates, such as the average student pro- ficiency of a subpopulation.
It should be noted that the AM method (Cohen 1998) estimates population parameters directly and is a viable alternative to the plausible-value method that ETS has chosen. The AM approach has been studied in depth by Donoghue et al.
(2006a).
These methods were subsequently evaluated for application in future large-scale assessments (Li and Oranje 2006; Sinharay et al. 2010; Sinharay and von Davier 2005; von Davier and Sinharay 2007, 2010). Their analysis of a real NAEP data set provided some evidence of a misfit of the NAEP model. However, the magnitude of the misfit was small, which means that the misfit probably had no practical signifi- cance. Research into alternative approaches and emerging methods is continuing.
Variance Estimation
Error variance has two components: sampling error and measurement error. These components are considered to be independent and are summed to estimate total error variance.
Sampling Error
The NAEP samples are obtained through a multistage probability sampling design.
Because of the similarity of students within schools and of the effects of nonre- sponse, observations made of different students cannot be assumed to be indepen- dent of each other. To account for the unequal probabilities of selection and to allow for adjustments for nonresponse, each student is assigned separate sampling weights. If these weights are not applied in the computation of the statistics of inter- est, the resulting estimates can be biased. Because of the effects of a complex sam- ple design, the true sampling variability is usually larger than a simple random sampling. More detailed information is available in reports by Johnson and Rust (1992, 1993), Johnson and King (1987), and Hsieh et al. (2009).
The sampling error is estimated by the jackknife method (Quenouille 1956;
Tukey 1958). The basic idea is to divide a national or state population, such as in- school eighth graders, into primary sampling units (PSUs) that are reasonably simi- lar in composition. Two schools are selected at random from each PSU. The sampling error is estimated by computing as many error estimates as there are PSUs.
Each of these replicates consists of all PSU data except for one, in which one school is randomly removed from the estimate and the other is weighted doubly. The meth- odology for NAEP was described, for example, by E. G. Johnson and Rust (1992), and von Davier et al. (2006), and a possible extension was discussed by Hsieh et al.
(2009).
The sampling design has evolved as NAEP’s needs have increased. Certain eth- nic groups are oversampled to ensure that reasonably accurate estimations and sam- pling weights are developed to ensure appropriately estimated national and state samples.
Also, a number of studies have been conducted about the estimation of standard errors for NAEP statistics. Particularly, an application of the Binder methodology (see also Cohen and Jiang 2001) was evaluated (Li and Oranje 2007) and a
comparison with other methods was conducted (Oranje et al. 2009) showing that the Binder method under various conditions underperformed compared to sampling-based methods.
Finally, smaller studies were conducted on (a) the use of the coefficient of varia- tion in NAEP (Oranje 2006b), which was discontinued as a result; (b) confidence intervals for NAEP (Oranje 2006a), which are now available in the NDE as a result;
and (c) disclosure risk prevention (Oranje et al. 2007), which is currently a standard practice for NAEP.
Measurement Error
Measurement error is the difference between the estimated results and the “true”
results that are not usually available. The plausible values represent the posterior distribution and can be used for estimating the amount of measurement error in statistical estimates such as a population mean or percentile. Five plausible values are computed for each student, and each is an estimate of the student’s proficiency.
If the five plausible values are close together, then the student is well measured; if the values differ substantially, the student is poorly measured. The variance of the plausible values over an entire population and subpopulation can be used to esti- mate the error variance. The general methodology was described by von Davier et al. (2009).
Researchers continue to explore alternative approaches to variance estimation for NAEP data. For example, Hsieh et al. (2009) explored a resampling-based approach to variance estimation that makes ability inferences based on replicate samples of the jackknife without using plausible values.
Alternative Psychometric Approaches
A number of modifications of the current NAEP methodology have been suggested in the literature. These evolved out of criticisms of (a) the complex nature of the NAEP model and (b) the approximations made at different stages of the NAEP estimation process. Several such suggestions are listed below:
• Apply a group-specific variance term. Thomas (2000) developed a version of the CGROUP program that allowed for a group-specific residual variance term instead of assuming a uniform term across all groups.
• Apply seemingly unrelated regressions (SUR; Greene 2002; Zellner 1962).
Researchers von Davier and Yu (2003) explored this suggestion using a program called YGROUP and found that it generated slightly different results from CGROUP. Since YGROUP is faster, it may be used to produce better starting values for the CGROUP program.
• Apply a stochastic EM method. Researchers von Davier and Sinharay (2007) approximated the posterior expectation and variance of the examinees’ proficien- cies using importance sampling (e.g., Gelman et al. 2004). Their conclusion was that this method is a viable alternative to the MGROUP system but does not pres- ent any compelling reason for change.
• Apply stochastic approximation. A promising approach for estimation in the presence of high dimensional latent variables is stochastic approximation.
Researchers von Davier and Sinharay (2010) applied this approach to the estima- tion of conditioning models and showed that the procedure can improve estima- tion in some cases.
• Apply multilevel IRT using Markov chain Monte Carlo methods (MCMC). M. S.
Johnson and Jenkins (2004) suggested an MCMC estimation method (e.g., Gelman et al. 2004; Gilks et al. 1996) that can be adapted to combine the three steps (scaling, conditioning, and variance estimation) of the MGROUP program.
This idea is similar to that proposed by Raudenbush and Bryk (2002). A maxi- mum likelihood application of this model was implemented by Li et al. (2009) and extended to dealing with testlets by Wang et al. (2002).
• Estimation using generalized least squares (GLS). Researchers von Davier and Yon (2004) applied GLS methods to the conditioning model used in NAEP’s MGROUP, employing an individual variance term derived from the IRT mea- surement model. This method eliminates some basic limitations of classical approaches to regression model estimation.
• Other modifications. Other important works on modification of the current NAEP methodology include those by Bock (2002) and Thomas (2002).
Possible Future Innovations
Random Effects Model
ETS developed and evaluated a random effects model for population characteristics estimation. This approach explicitly models between-school variability as a random effect to determine whether it is better aligned with the observed structure of NAEP data. It was determined that relatively small gains in estimation using this approach in NAEP were not sufficient to override the increase in computational complexity.
However, this approach does appear to have potential for use in international assess- ments such as PISA and PIRLS.
Adaptive Numerical Quadrature
Use of adaptive numerical quadrature can improve estimation accuracy over using approximation methods in high-dimensional proficiency estimation. ETS research- ers performed analytic studies (Antal and Oranje 2007; Haberman 2006) using
adaptive quadrature to study the benefit of increased precision through numerical integration for multiple dimensions. Algorithmic development and resulting evalu- ation of gains in precision are ongoing, as are feasibility studies for possible opera- tional deployment in large-scale assessment estimation processes.
Antal and Oranje (2007) posited that the Gauss-Hermite rule enhanced with Cholesky decomposition and normal approximation of the response likelihood is a fast, precise, and reliable alternative for the numerical integration in NAEP and in IRT in general.
Using Hierarchical Models
In addition, several studies have been conducted about the use of hierarchical mod- els to estimate latent regression effects that ultimately lead to proficiency estimates for many student groups of interest. Early work based on MCMC (Johnson and Jenkins 2004) was extended into an MLE environment, and various studies were conducted to evaluate applications of this model to NAEP (Li et al. 2009).
The NAEP latent regression model has been studied to understand better some boundary conditions under which the model performs well or not so well (Moran and Dresher 2007). Research into different approaches to model selection has been initiated (e.g., Gladkova and Oranje 2007). This is an ongoing project.