II.1 Literature Review
II.1.4 Estimating Principal Effects
Compared to the robust literature on teacher quality and estimating teacher value-added, there are few studies that investigate principal value-added. To be specific, I distinguish between principal value-added and a much larger body of work that examines correlational relationships (most often correlational) between principals and student test scores. Within the principal effects literature, there are two general types of studies. The first group of studies aim to estimate directly the variance of principal effects by exploiting year-to-year
changes in achievement between successive cohorts of students (Branch et al., 2012; Coelli and Green, 2012). The intuition of this approach is that to the extent that principal quality matters for student outcomes, year-to-year changes in achievement will be greater in mag- nitude in cases where there was a principal transition between year t and year t+ 1. The second group of studies produce estimates of effectiveness for individual principals using value-added models (Grissom et al., 2015a; Dhuey and Smith, 2014, 2018; Chiang et al., 2016).
Branch et al. (2012) use a large sample of principals from Texas to estimate the variance of principal effects on student achievement. Specifically, they extend a methodology proposed by Rivkin et al. (2005) that leverages teacher transitions to estimate the variance of teacher effects. They find that the lower bound of the variance of principal effects is 0.05 student-level s.d. The authors note that while this direct estimation tends to produce variance estimates substantially smaller than the variance of principal value-added estimates, even 0.05 s.d.
constitutes a meanginful impact given that principals affect all of the students in a school.
Coelli and Green (2012) use data from British Columbia to estimate the variance of principal effects on high school graduation rates and 12th grade exam scores. A strength of the study is that the B.C. education system explictly sought to rotate principals through different schools, which helps to provide the critical variation needed to separate principal and school effects. In addition to estimating a model that assumes principal effects are immediate and constant (i.e., principals do not vary in their effectivness over time and their effects manifest as soon as they enter the school), they allow for the possibility that principal effects are increasing as a function of tenure in the school. Put simply, a new principal may not have much capacity to affect student outcomes in their first years in the school, which would imply that the variance of principal effects would be close to zero for those years.
As principals gain tenure, they have more influence over school factors that affect student outcomes (e.g., hiring and retention of high-quality teachers, creation of a positive school climate), and thus the impact of a high-quality principal is larger in magnitude.
Coelli and Green (2012) estimate that, using a model that treats principal effects as constant, a 1 s.d. increase in principal quality increases graduation rates by 1.8 percentage points (compared to a baseline mean of 82%). When accounting for the dynamic nature of principal effects, they find that principals become more consequential—a 1 s.d. increase in principal quality raises graduation rates by 2.6 percentage points. Due to the small sample of principals, however, their variance estimates are imprecise. The authors also note that, because the average observed tenure of principals in their data is three years, estimators that treat principal effects as constant will understate the importance of principals who remain in schools for a long period of time. They also find that, in terms of the variance attributable to principals, effects on English exam score are much larger than those for graduation rates. According to their calculations, if all students were in schools with principals in their sixth year at that school, differences in principal quality would explain roughly half of the student-level variation in English scores (compared to only 8% of the variation in graduation rates). This figure is almost certainly overstated, given repeated findings that teacher quality accounts for less than 20% of the student-level variation in achievement (e.g., Rivkin et al., 2005; Chetty et al., 2014).
Dhuey and Smith (2014) also use data from British Columbia to estimate principal value- added. Specifically, they examine student test score gains between grade 4 and grade 7, rather than gains between adjacent years. Due to data limitations (i.e., missing test scores, students changing schools), they drop more than half of their student sample. Additionally, because 5th and 6th grade achievement scores are unavailable, they must account for the effect of principals in these intermediary grades. Their approach is to define a modified school fixed effect, which is a unique combination of schood, grade 5 principal, and grade 6 principal. Their value-added model includes both principal and the modified school fixed effects. While Dhuey and Smith (2014) do not report network sizes, the high dimensionality of this modified school effect is concerning. They find that variation in principal quality has substantial effects on students’ achievement. In their two-way FE model, a principal 1 s.d.
above the mean increases math achievement by 0.41 s.d. and reading achievement by 0.30 s.d.
Grissom et al. (2015a) use data from Miami-Dade schools to explore different approaches to estimating principal effects. Specifically, they outline three general approaches: school ef- fectiveness, relative within-school effectiveness, and school improvement. School effectiveness is essentially a school value-added model, and does not attempt to separate the contribution of the principal from the contribution of the school. Relative within-school effectiveness is similar to the approaches pursued by other studies that use both principal and school fixed effects. Their third approach, school improvement, attempts to address the possibility that a new principal may not be able to have an immediate and constant effect on student achievement. If, for instance, principals affect student learning through hiring and retention of effective teachers, we might expect that it takes several years for a new principal to affect student outcomes through this channel. The school improvement approach models principal value-added as the coefficient on a principal-specific linear time trend. The model also con- tains a principal-specific intercept, such that a principal’s effectiveness is the average change in student growth relative to this intercept (i.e., average growth prior to the principal enter- ing the school). In addition to examining the distribution of these VA estimates, Grissom et al. (2015a) also show their correlations with external evaluation measures, such as school accountability grades, principals’ evaluation ratings, and school climate measures.
The authors find wide variation in the distribution of principal VA depending on model choice. For math, they find that the standard deviation of principal VA ranges from 0.18 (school effectiveness) to 0.06 (relative within-school effectiveness) or 0.05 (school improve- ment). In general, VA estimates from school FE versus school and principal FE have a fairly large positive correlation, while the principal-by-school time trends (school improve- ment) were uncorrelated with all other VA estimates. School effectiveness estimates were consistently predictive of external evaluation measures, even after controlling for school char- acteristics, and relative within-school effectiveness estimates were also predictive to a lesser
extent. By contrast, school improvement estimates were uncorrelated with all of the exter- nal evaluation measures. The authors conclude that attention to model choice is important, particularly with respect to using principal VA estimates for evaluation purposes. They also note that the most conceptually appealing approaches (i.e., school improvement and relative within-school effectiveness) are the least predictive of external evaluation measures.
Chiang et al. (2016) evaluate the extent to which school VA (i.e., a school FE from a model that does not separate principal effects from school effects) is predictive of principal VA (i.e., principal FE from a model that also includes school FE). To be specific, the authors compare school and principal VA estimates from “close-to-independent” samples of students.
Principal VA estimates are estimated separately by grade and come from a five-year period, while school VA estimates (also estimated separately by grade) come from a year not used to generate principal VA (the year prior to the first year of the principal VA sample). By comparing principal and school VA within each grade and using different years, only a small number of students who repeated grades will contribute to both VA estimates. Chiang et al.
(2016) argue that this approach is advantageous because it prevents finding a mechanical relationship between principal and school VA that reflects common transitory shocks (i.e., components of principal or school effectiveness that are not persistent across years) and com- mon measurement error. The authors find that while principals appear to have substantial effects on student achievement (adjusted standard deviations of 0.14 and 0.11 in math and reading, respectively), school VA is a very poor predictor of principal VA.
Dhuey and Smith (2018) use 12 years of statewide data from North Carolina to estimate principal VA. Specifically, they estimate VA using both school and principal FE, finding that the standard deviation of principal effects is 0.17 for math and 0.12 for reading. They also find that principal education is a weak predictor of value added. The primary contribution of the study is that is utilizes a larger dataset than previous studies, which should improve the quality of the VA estimates.