Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=ubes20
Download by: [Universitas Maritim Raja Ali Haji] Date: 12 January 2016, At: 23:47
Journal of Business & Economic Statistics
ISSN: 0735-0015 (Print) 1537-2707 (Online) Journal homepage: http://www.tandfonline.com/loi/ubes20
Comment
Wilbert van der Klaauw
To cite this article: Wilbert van der Klaauw (2005) Comment, Journal of Business & Economic Statistics, 23:2, 154-157, DOI: 10.1198/073500104000000721
To link to this article: http://dx.doi.org/10.1198/073500104000000721
Published online: 01 Jan 2012.
Submit your article to this journal
Article views: 28
some of them could be located by additional matching steps using some of the name and SSN information for blocking. Is it possible that some matches might be located that do not agree on SEIN? Using more compute-intensive matching, it is still possible to match on name? Abowd and Vilhuber ob-serve that some companies may have high turnover that causes many one-quarter job histories. A concern in some of the high-turnover industries (and others) is whether a group of one-quarter job histories may be due to high SSN error rates in certain companies that can greatly increases job loss and cre-ation statistics.
Abowd and Vilhuber have demonstrated that basic coding errors in identifiers such as SSN in seemingly straightforward files can have a profound effect on employment statistics. Their
ideas and methods have substantial implication toward improv-ing the quality of data in files that are used for evaluatimprov-ing em-ployment statistics. It seems that many other projects in which files are linked could suffer from significantly larger errors, be-cause some entities are not linked that should be linked. In situ-ations where records are correctly linked, some of the economic data from one file might be used to locate potential errors in data from the other file.
ADDITIONAL REFERENCE
Winkler, W. E. (1995), “Matching and Record Linkage,” inBusiness Survey Methods, eds. B. G. Cox, D. A. Binder, B. N. Chinnappa, A. Christianson, M. J. Colledge, and P. S. Kott, New York: Wiley, pp. 355–384.
Comment
Wilbert
VAN DERK
LAAUWDepartment of Economics, University of North Carolina, Chapel Hill, NC 27599-3305 (vanderkl@email.unc.edu)
The past two decades have witnessed an increased interest among applied researchers, statisticians, and econometricians in the extent and consequences of measurement error in sur-vey data. The increased availability of administrative data has played an important role in bringing about this change, by pro-viding more precisely defined and arguably more accurate mea-surements of the variable of interest. This has been the case particularly in the study of labor market outcomes, where so-cial security earnings, tax records, and administrative records from employers have been linked to survey data to create vali-dation samples against which the accuracy of survey responses can be assessed. A recent comprehensive survey by Bound, Brown, and Mathiowetz (2001) described how administrative data on employment related outcomes such as earnings, hours of work, unemployment, industry, occupation, and other vari-ables, including benefit receipt, health status, health care use, and education, have been used to assess the magnitude and na-ture of measurement errors in these variables in such surveys as the SIPP, CPS, and PSID. Most of these studies present evi-dence of significant and often nonclassical measurement errors, which can lead to substantial bias in econometric model esti-mates.
The assumed availability of a validation sample also provides the basis of a number of estimation techniques that recently have been proposed to deal with classical and nonclassical measurement errors in linear and nonlinear models (Carroll and Wand 1991; Sepanski and Carroll 1993; Lee and Sepanski 1995; Hu and Ridder 2003; Chen, Hong, and Tamer in press). An important premise of the aforementioned validation stud-ies is that administrative data provide accurate measures of the variables of interest. The article by Abowd and Vilhuber rep-resents an important illustration of how administrative data can be subject to considerable measurement errors themselves, and
demonstrates that these often do not cancel out after aggrega-tion. It also proposes an innovative probabilistic matching pro-cedure that is able to correct a considerable percentage of these errors. Although some of my comments pertain directly to the specifics of their article, I focus mainly on what I consider to be the broader implications of this research.
Because the collection of most administrative data is de-signed primarily to serve the regulatory, enforcement, and ac-counting functions of running a state or federal program, such as unemployment insurance, tax liabilities, and benefit awards, the information can be expected to be more complete, objective, and reliable than that collected through survey interviews. The variables measured are more narrowly defined and less prone to alternative interpretation compared with responses from sur-vey questions. Because administrative databases are generally updated on a more continuous basis, they are also less prone to recall bias, and because of the nature of the data-gathering process, they typically suffer less from attrition and item non-response problems. Another advantage of administrative data is their large sample size and low cost for research, because this information is usually collected as part of the administration of the program.
However, these strengths associated with the gathering of data for the purpose of administering a program are also di-rectly associated with some of its disadvantages. First, the in-formation collected for administrative purposes, although it may be related to the theoretical concept of interest, may not measure what we would like it to measure. For exam-ple, state unemployment insurance (UI) wage records, which
© 2005 American Statistical Association Journal of Business & Economic Statistics April 2005, Vol. 23, No. 2 DOI 10.1198/073500104000000721
employers are required to submit to the state on behalf of employees, may not provide an accurate measure of an in-dividual’s earnings in a given quarter, because it excludes earnings from noncovered employment as well as earnings from jobs in other states. A problem with using tax records is that they do not report the earnings for people with earn-ings low enough to exempt them from payroll taxes (Bound and Krueger 1991). Similarly, UI or social security earnings records may provide a poor proxy of earnings if the theoreti-cal measure of earnings in our analysis is an individual’s per-manent income. UI records may also prove inadequate for the analysis of job flows if our theoretical concept of a job distin-guishes between changes in duties while working for a given employer.
Another important drawback of administrative data is that they typically only contain information relevant to the func-tioning of the program. They often do not include demographic information, such as the individual’s race, ethnicity, and educa-tion level, as well as other pertinent informaeduca-tion that one would typically gather in surveys. For example, UI records lack in-formation about the circumstances leading to job exits (e.g., quits, layoffs, reason for leaving), which play an important role in understanding wage dynamics and job turnover. They also usually lack information on an individual’s activities while not enrolled in the program (e.g., not employed, not on welfare, without Medicaid coverage), and by tracking individuals rather than households, they often provide little context about house-hold decisions.
Administrative data may also contain errors that occur at the data production stage, such as coding and key-in errors. Fur-thermore, they may contain more systematic measurement er-rors. Ardal and Ennis (2001) provided examples of three cases involving the underreporting of an activity among a subset of the population (registration of births by teenage mothers), in-accurate reporting of geographic location, and problems with data entry protocols due to variation in interpretation of vari-able definitions. With regard to social security or UI earnings records, employer tax reports may be inaccurate, individuals may have income from uncovered or under-the-table jobs, or there may be differences between what employees consider to be earnings and what employers report as earnings (Abowd and Stinson 2003). Income tax returns also have been found to con-tain significant reporting errors (Joulfaian and Rider 2003).
Perhaps one of the most attractive features of administra-tive data is that, even though in many cases they were not in-tended for this purpose, they are often longitudinal or can be made longitudinal through matching over time. The longitudi-nal feature of the UI records has been exploited by the Bureau of Labor Statistics (BLS) in generating a national administra-tive database, the ES-202, which combines quarterly payroll and total employment counts for each establishment of cov-ered employment collected in each state. This database was cre-ated to permit an analysis of job creation and destruction across quarters, industries, and counties. However, the longitudinal linking of establishment records based on employer identifiers is hindered by the fact that employer identifiers sometimes change over time. Employer identifiers can change for various reasons that do not involve job destruction or creation, such as company reorganizations that consists of mergers, acquisitions,
or breakouts of company units. Even though such problems af-fect only a relatively small proportion of establishments, the correct or incorrect linking of these records has been shown to have a significant effect on the reported numbers of births and deaths of firms (Spletzer 2000).
In this study, Abowd and Vilhuber show that similar prob-lems arise in linking UI wage records for the state of California in creating job and earnings histories for individual workers. They show that linking based on an individual’s social security number (SSN) leads to a relatively large number of job histo-ries with a single-quarter employment interruption, while also producing a considerable number of records containing only a single job spell lasting exactly one quarter. This appears con-sistent with evidence from a 1997 BLS study, which found both coding errors by employers and data entry errors by state agen-cies to be fairly common, especially because one would expect such errors to take the form of a random digit coding error for a single record in a worker’s job history. Therefore, it seems rea-sonable that many of these single-quarter “holes” and “plugs” actually belong to the same continuous job history.
It is important to note that these problems are not confined to administrative data. For example, Abowd and Stinson (2003) found job identifiers defined in SIPP data to be unreliable for creating individual worker job histories. Similarly, Card (1996) discussed the difficulties in using household panel data from the CPS to link monthly records for individual household mem-bers. Furthermore, errors in employer or employee identifiers not only lead to measurement errors in employee work histo-ries and in establishment birth and death rates derived from these administrative data, but also imply that there will be er-rors when such identifiers are used to match administrative data to other datasets on workers or firms. Because administrative data are not designed for linking, these so-called “broken” ran-dom samples in which the identity of members is observed with error, are likely to be pervasive more generally. As a result, when comparing two measurements from two linked datasets, we often cannot be sure whether any differences that we find are due to measurement errors or due to the incorrect matching of records. For example, Card, Hildreth, and Shore-Sheppard (2001) discussed the implications of coding errors in linking survey data on Medicaid coverage to administrative Medic-aid data, and found that in the administrative data individual SSNs are often misreported. Abowd and Stinson (2003) dis-cussed problems with missing and inaccurate SSNs in the link-ing of Social Security Administration earnlink-ings records to SIPP records.
A useful technique for reducing the probability of incor-rectly matching records due to miscoded case identifiers, is to use probabilistic matching methods (Fellegi and Sunter 1969; Winkler 1995). In the creation of the ES-202 file, for exam-ple, a probability-based statistical matching procedure is used for the longitudinal linking of individual establishment records. After linking records based on available establishment identi-fiers and additional information on organizational changes in firms, apparent births in any given quarter are compared with deaths in the previous quarter to determine statistically whether they in fact may represent a miscoded continuous establishment record. This is done by looking for similarities in the name, ad-dress, and phone number associated with the two establishment
identifiers (Robertson, Huff, Mikkelson, Pivetz, and Winkler 1999; Pivetz, Searson, and Spletzer 2001). Similarly, in lon-gitudinally matching individual records with household panels from the CPS, Card (1996) applied a probability matching pro-cedure that checks for legitimate month-to-month changes and consistency in the reported race, age, sex, and marital status of the individual.
To reduce the likelihood of missed record linkages due to miscoded personal identifiers and the resulting measurement er-rors in derived job histories, Abowd and Vilhuber use a proba-bilistic matching procedure that looks for similarities in SSNs and names between all different employee records provided by a given firm. However, to help identify from among the records containing a single-quarter interruption or a single-quarter job spell (the two most likely to represent a true continuous job spell), their matching procedure also looks for expected earn-ings patterns likely to be associated with such spells. This com-parison of quarterly earnings in the “hole” and “plug” quarters is based on a simple model of earnings dynamics that predicts that whereas in case of a true exit earnings will on average be lower in plug quarters compared to hole quarters, they should be comparable in case of an uninterrupted job spell. More gen-erally, the idea is that accuracy of linkage decisions can be enhanced by finding other patterns in the data that are pre-sumed or known to be consistent with either correct or incorrect matches.
Even though their approach deals only with miscoded single-quarter interruptions and affects only a relatively small propor-tion of job histories, it is found to produce significant reducpropor-tions in worker flow and job turnover rates, as well as in job re-call rates. This result mirrors earlier findings by Poterba and Summers (1986) demonstrating that classification errors in re-ported labor market status can generate transition rates that overstate the degree of dynamics in the labor market. Although this probabilistic matching approach leads to a reduction in measurement errors and more accurate aggregate stock and flow measures, it does not necessarily lead to smaller biases in the estimated relationship between job transition rates on the one hand and individual characteristics and earnings dynamics on the other hand. Even if coding errors are randomly distributed across wage records, by using cross-quarter earnings compar-isons, the probability of correcting a false job interruption will no longer be constant across employees. For example, true con-tinuous spells for individuals with higher wage volatility are less likely to be corrected than those with stable earnings, and the “plugged” quarters in the linked records are more likely to have above-average earnings for these individuals. As a result, relative to a procedure that connects all broken linkages or only those based on SSNs and names, the matching procedure is likely to bias downward the extent of earnings fluctuations in continuous job spells.
Abowd and Vilhuver find that their correction procedure has differential effects on the relative flow rates of various sub-groups of the labor force. However, it is unclear whether the matching procedure will lead to more accurate estimates of differences in flow rates across subgroups of the population, because the corrected flow rate figures will reflect the proba-bilistic matching procedure’s ability to identify false flows in each subgroup. If most of the errors are due to employer coding
errors in reporting employee SSNs, then the relative frequency of these errors may vary across employers (and therefore across subgroups of workers), depending on the resources and data en-try mechanism that they apply to their record keeping, which may vary with the size of the firm. Second, the matching pro-cedure may correct a smaller fraction of false interruptions in one group than in another. For example, the procedure is likely to identify more false flows for those with stable earnings com-pared to those with more variable earnings. As a consequence, observed differences in job turnover rates may reflect differ-ences in the success rate of removing coding errors, instead of true behavioral differences between groups. It is therefore unclear whether the editing procedure will lead to a bias re-duction in parameter estimates in linear and nonlinear models relating job transition rates to worker and firm characteristics. One way to investigate the consequences of matching on vari-ables related to the variable of interest would be to randomly add coding errors of the type suspected here, and then evaluate the matching procedure’s performance in reducing estimation biases.
These quibbles notwithstanding, it is clear that the edit-ing procedure that Abowd and Vilhuber propose leads to a significant bias reduction in aggregate flow statistics. Al-though various public laws prevented the authors from com-bining their data with other data sources, it is important to point out that the idea of full information edits does not have to be limited to the use of within-sample information. For example, information from another longitudinal data source on the relative frequency of true single-quarter interruptions and recalls for different subgroups can potentially be inte-grated into the probabilistic matching model. If such events are much more common in one industry than in others, then a lower-probability cutoff may be assigned for the linking of two employee records in that industry. Similarly, data on true earnings variation accompanying short job interruptions can be used to sharpen the assumed earnings model. Note that this would not necessarily require that the datasets be linked. As long as the auxiliary data are drawn from the same popula-tion as the administrative data, informapopula-tion from either source should prove useful in imputing incomplete data in the other source.
The study leaves one with three important insights. First, measurement errors and the prevalence of coding errors in case identifiers are likely to render most administrative datasets un-suitable for the purpose of validating longitudinal survey data. Instead, such data more appropriately should be treated as pro-viding us an alternative noisy measurement of the true lon-gitudinal data of interest. Much can be learned from linking administrative and survey data. However, like longitudinal sur-vey data, administrative data must be analyzed and interpreted in a way that reflects an understanding of the nature and impli-cations of measurement errors, especially those resulting from incorrect record linkage. Many recently developed methods for estimating linear and nonlinear measurement error models rely on the existence of a validation sample, usually expected to be obtained by linking survey data to administrative data. Others, such as the methods of Hausman, Newey, and Powell (1995), Li and Vuong (1998), Li (2002), Abowd and Stinson (2003), Mahajan (2003), and Schennach (2004) are based on the avail-ability of two independent measurements. I therefore consider
the ongoing development of these latter methods, particularly methods that simultaneously allow for nonclassical measument errors, especially important. Also promising in this re-spect is the development of methods for combining information from different datasets that do not require that records be linked using imprecise identifiers, such as those proposed by Hu and Ridder (2003).
Second, given the large potential gains from longitudinal record linkage within and across datasets and given the inci-dence of coding errors in case identifiers, the study of the conse-quences of matching based on potentially miscoded identifiers, as well as the development of methods that can account for the resulting measurement errors, represents an important area for future research. Relatively little is known about the biases induced by matching errors and the performance of alterna-tive record linkage methods. What variables should we con-dition on, and what are the best weights to assign to an exact match on these variables in calculating overall match probabil-ities? What are the consequences of matching on variables that are directly related to the dependent variable of interest? How should the probability cutoffs be chosen for making linkage decisions? How sensitive are sample statistics and estimated model parameters to variations in the probability matching pro-cedure? A recent study of particular importance that deals with inference in case of record linkage errors is that by Ridder and Moffitt (in press). These authors showed how knowledge about matching errors can be used in a generalized method-of-moments procedure to obtain consistent parameter estimates. Alternatively, their procedure could be used for sensitivity analysis by considering a range of assumptions regarding the misclassification probabilities.
Finally, the ongoing decentralization of and refinements to administrative data gathering and processing procedures have potentially important implications for using these data in policy research. Differences between states in data entry procedures can lead to considerable differences in the rate at which cod-ing errors occur. In terms of the job history patterns analyzed by Abowd and Vilhuber, this can translate into large differ-ences in job turnover rates between states. Similarly, the adop-tion of more reliable key entry procedures could potentially produce large changes in computed job flows. It is not only the likelihood of coding errors that may vary across states and time; even the definition of employer, employee, and employ-ment may differ (Spletzer 2000; Hotchkiss, Pitts, and Robertson 2003). With most evaluation research of public welfare and la-bor market programs and policies being based on cross-state variation in program rules and benefits, and/or changes therein over time, this implies that differences in job mobility due to variations in data processing procedures may be incorrectly at-tributed to differences in program parameters. For example, states using more accurate data recording methods may have more resources, and implementation of new policies may be accompanied by changes in data production procedures (Hotz, Goerge, Balzekas, and Margolin 1998). In absence of a stan-dardized national information collection system, it is therefore important for researchers using administrative data to become better informed of the role of data production procedures affect-ing the quality of data, and to take its effects on data measure-ments into account in their analyses.
ADDITIONAL REFERENCES
Abowd, J. M., and Stinson, M. H. (2003), “Estimating Measurement Error in SIPP Annual Job Earnings: A Comparison of Census Survey and SSA Ad-ministrative Data,” unpublished manuscript, Cornell University.
Ardal, S., and Ennis, S. (2001), “Data Detectives: Uncovering Systematic Er-rors in Administrative Databases,” inProceedings of Statistics Canada Sym-posium on Achieving Data Quality in a Statistical Agency: A Methodological Perspective.
Bound, J., Brown, C., and Mathiowetz, N. (2001), “Measurement Error in Sur-vey Data,” inHandbook of Econometrics, Vol. 5, eds. J. J. Heckman and E. Leamer, Amsterdam: Elsevier, pp. 3705–3843.
Bound, J., and Krueger, A. B. (1991), “The Extent of Measurement Error in Longitudinal Earnings Data: Do Two Wrongs Make a Right?”Journal of Labor Economics, 9, 1–24.
Card, D. (1996), “The Effect of Unions on the Structure of Wages: A Longitu-dinal Analysis,”Econometrica, 64, 957–979.
Card, D., Hildreth, A., and Shore-Sheppard, L. (2001), “The Measurement of Medicaid Coverage in the SIPP: Evidence From California: 1990–1996,” Working Paper 8514, National Bureau of Economic Research.
Carroll, R. J., and Wand, M. P. (1991), “Semiparametric Estimation in Logistic Measurement Error Models,”Journal of the Royal Statistical Society, Ser. B, 53, 573–585.
Chen, X., Hong, H., and Tamer, E. (in press), “Measurement Error Models With Auxiliary Data,”Review of Economic Studies, 27.
Fellegi, I. P., and Sunter, A. B. (1969), “A Theory of Record Linkage,”Journal of the American Statistical Association, 64, 1183–1210.
Hausman, J. A., Newey, W. K., and Powell, J. L. (1995), “Nonlinear Errors in Variables: Estimation of Some Engel Curves,”Journal of Econometrics, 65, 205–233.
Hotchkiss, J. L., Pitts, M. M., and Robertson, J. C. (2003), “The Ups and Downs of Jobs in Georgia: What Can We Learn About Employment Dynamics From State Administrative Data?” Working Paper 2003-38, Federal Reserve Bank of Atlanta.
Hotz, V. J., Goerge, R., Balzekas, J. D., and Margolin, F. (eds.) (1998), “Ad-ministrative Data for Policy-Relevant Research: Assessment of Current Util-ity and Recommendations for Development,” Report of the Advisory Panel on Research Uses of Administrative Data of the Northwestern Univer-sity/University of Chicago Joint Center for Poverty Research.
Hu, Y., and Ridder, G. (2003), “Estimation of Nonlinear Models With Measure-ment Errors Using Marginal Information,” working paper, CLEO, University of Southern California.
Joulfaian, D., and Rider, M. (2003), “Errors in Variables and Estimated Elastic-ities for Charitable Giving,” Working Paper 03-07, Andrew Young School of Public Policy Studies, Georgia State University.
Lee, L. F., and Sepanski, J. H. (1995), “Estimation of Linear and Nonlinear Errors-in-Variables Models Using Validation Data,”Journal of the American Statistical Association, 90, 130–140.
Li, T. (2002), “Robust and Consistent Estimation of Nonlinear Errors-in-Variables Models,”Journal of Econometrics, 110, 1–26.
Li, T., and Vuong, Q. (1998), “Nonparametric Estimation of the Measurement Error Model Using Multiple Indicators,”Journal of Multivariate Analysis, 65, 139–165.
Mahajan, A. (2003), “Misclassified Regressors in Binary Choice Models,” un-published manuscript, Stanford University.
Pivetz, T. R., Searson, M. A., and Spletzer, J. R. (2001), “Measuring Job and Establishment Flows With BLS Longitudinal Microdata,”Monthly Labor Re-view, 124, 13–20.
Poterba, J. M., and Summers, L. H. (1986), “Reporting Errors and Labor Market Dynamics,”Econometrica, 54, 1319–1338.
Ridder, G., and Moffitt, R. (in press), “The Econometrics of Data Combination,” inHandbook of Econometrics, Vol. 6.
Robertson, K., Huff, L., Mikkelson, G., Pivetz, T., and Winkler, A. (1999), “Improvements in Record Linkage Processes for the Bureau of Labor Statis-tics’ Business Establishment List,” inProceedings of an International Work-shop and Exposition, National Academies Press, pp. 212–221; available at
http://www.fcsm.gov.
Schennach, S. M. (2004), “Estimation of Nonlinear Models With Measurement Error,”Econometrica, 72, 33–75.
Sepanski, J. H., and Carroll, R. J. (1993), “Semiparametric Quasi-Likelihood and Variance Function Estimation in Measurement Error Models,”Journal of Econometrics, 58, 223–256.
Spletzer, J. R. (2000), “The Contribution of Establishment Births and Deaths to Employment Growth,” Journal of Business & Economic Statistics, 18, 113–126.
Winkler, W. (1995), “Matching and Record Linkage,” inMethods Business Sur-vey, eds. B. G. Cox et al., New York: Wiley, pp. 355–384.
