Chapter II: Was the Chinese Imperial Civil Bureaucracy Meritocratic?
3.3 Empirical Approach and Results
3.3.4 Ratee Qualification and Strategic Rating
As discussed above, our method for detecting strategic reporting behavior involves comparing ratings across employees who differ in their likelihood of
4If there arenpeople and an employee is ranked as thek-th highest, then her percentile rank is n−1k−1. She gets a percentile rank of 1 if she obtains the highest rating, while a percentile rank of 0 corresponds to the poorest rating in the department.
Figure 3.1: Histogram of ∆P Rself
being promoted. To model the likelihood of promotion among employees, we collected information about whether an employee satisfied promotion criteria in the given year from the firm’s personnel archive. As specified in the firm’s guidelines for promotion, for being considered for promotion, each employee needs to pass certain minimum requirements in terms of attendance, academic qualifications, project experience, and tenure. An analysis of the promotion records indicated that employees who failed to pass the promotion criteria had a much lower promotion rate compared with those who passed them. We define the binary variable RateeQual equal to one if the ratee has already passed these promotion criteria, and zero otherwise. We define a second variable, P EER, which equals one if the rater and ratee are of the same rank, and zero otherwise.
Empirical Specification
Our main empirical model is the following:
RAT IN Gijt =β0+β1P EERijt+β2RateeQualjt+β3P EERijt×RateeQualjt +γjtXjt0 +β4DEP Tijt+β5Y EARt+ijt.
In the regression equation,RAT IN Gijt denotes the rating that raterigives to ratee j at year t. The rating scale ranges from zero to ten with 0 as denoting the poorest performance and 10 denoting the highest level of performance.
Xjt0 denotes ratee rank and ratee fixed-effect dummies;DEP Tijt and Y EARt denote dummy variables for department and year fixed-effects.
Some careful analysis is needed for interpreting coefficients in this model. To do so, we introduce two important concepts, “qualification premium”, and “peer difference”, which represent two alternative ways of interpreting and measuring strategic manipulation of ratings using the coefficients in the regression model.
We define the “rating function”
R(P EER, RateeQual) =β1P EER+β2RateeQual+β3P EER×RateeQual.
We introduce the notion of “qualification premium”, which captures the rating premium given to ratees who have already passed promotion requirements.
We define the qualification premium
∆QU AL(P EER) =
R(1,1)−R(1,0) = β2 +β3 if P EER= 1 R(0,1)−R(0,0) = β2 if P EER= 0.
Similarly, we introduce a notion of “peer differnece”:
∆P EER(RateeQual) =
R(1,1)−R(0,1) =β1+β3 if RateeQual= 1 R(1,0)−R(0,0) =β1 if RateeQual= 0.
From the above two definitions, we can see that β1 captures the peer differ- ence when the ratee has failed promotion requirements, and β2 measures the qualification premium when the rater and ratee are not peers.
How do we identify and measure strategic manipulation? We start by con- sidering what coefficients in our model should look like when there is no
strategic manipulation. Without manipulation, we expect peer raters and non-peer raters to behave similarly in recognizing peer premiums. In other words, a rater’s measured qualification premium – that is, how much bet- ter she rates other employees who have passed objective promotion require- ments – should not depend on whether the ratee is a peer or not. That is,
∆QU AL(1) = ∆QU AL(0), which implies that β3 = 0. We can also consider this from the perspective of peer difference. Without manipulation, we expect the peer difference to be independent of whether the peer ratee has passed requirement or not. That is, ∆P EER(1) = ∆P EER(0), which also gives to β3 = 0. To sum up, we expectβ3 = 0, in the absence of strategic manipulation.
Moreover, insofar as employees who have passed these promotion requirements have higher ability or have exhibited better work performance, we expect that ratees with RateeQual = 1 should receive a positive qualification premium from both peer raters and non-peer raters. That is, ∆QU AL(1) > 0 and
∆QU AL(0) > 0, which implies that β2 > 0 and β2 +β3 > 0. If the rater’s evaluation does not depend on whether the rater and ratee are peers, we expect the peer difference to be zero. That is, β1 =β3 = 0.
Whenβ3 <0, we have∆QU AL(1)<∆QU AL(0)and∆P EER(1)<∆P EER(0).
The former condition means that the rater’s attitude towards passing require- ments depends on whether the ratee is a peer. Specifically, the rater gives more generous ratings to peer ratees who have failed requirements, and deni- grates peer ratees who have passed these requirements. These are consistent with strategic manipulation: since peers who have failed requirements are less threatening to the rater compared with peers who have passed, we would ex- pect a self-interested rater to strategically downgrade peers who have passed requirements. We label β3 as “manipulation measure”, since it captures how much a rater strategically downgrades qualified peers who have already passed important promotion requirements, and upgrades less qualified peers who have not yet passed these requirements.
Empirical Results
Table 3.2 presents the estimation results of the OLS regression analysis with robust standard errors clustered by ratee and year. In Figure 3.2 we graph the
implied qualification premium and peer differences, using estimated coefficients from Table 3.2.
Table 3.2: Ratee Qualification and Performance Rating: Difference between Peer and Nonpeer Raters
Ratee’s performance rating (1)
Independent Variables OLS
Peer 0.436***
(0.0610)
RateeQual 0.165***
(0.0418) Peer×RateeQual -0.493***
(0.0792) Ratee rank fixed-effects Yes
Ratee fixed-effects Yes
Year fixed-effects Yes
Department fixed-effects Yes
Observations 7346
AdjustedR2 0.393
Notes: Robust standard errors clustered by ratee and year are reported in parentheses. *, **, *** are significant at 10%, 5%, and 1%, respectively (two-tailed).
Overall, the empirical results suggest that raters do recognize the positive qualification premiums that non-peer ratees deserve (i.e., β2 > 0). However, when the rater is a peer, not only does this qualification premium decrease (i.e., ∆QU AL(0)−∆QU AL(1) = β3 < 0), it indeed becomes negative (i.e.,
∆QU AL(1) < 0). In other words, ceteris paribus, if the ratee passed pro- motion requirements, compared with the case of not passing, he/she would secure higher ratings from non-peer raters, and get lower ratings from peer raters. The regression coefficient of the interaction term between QU AL and RateeQual,β3, captures this difference of qualification premiums between peer raters and non-peer raters.
These results indicate a “discriminatory generosity” on the part of peer raters, leading them to denigrate the relative ranking of peer ratees who have already passed promotion requirements. This is consistent with our notion of strategic reporting.
Figure 3.2: Ratee Qualification Premium and Peer Difference Illustrated