REGRESSION MODEL: THE PROBLEM OF ESTIMATION
3.3 PRECISION OR STANDARD ERRORS OF LEAST-SQUARES ESTIMATES
From Eqs. (3.1.6) and (3.1.7), it is evident that least-squares estimates are a function of the sample data. But since the data are likely to change from sample to sample, the estimates will change ipso facto. Therefore, what is needed is some measure of “reliability” or precisionof the estimators βˆ1
andβˆ2.In statistics the precision of an estimate is measured by its standard error (se).17Given the Gaussian assumptions, it is shown in Appendix 3A, Section 3A.3 that the standard errors of the OLS estimates can be obtained
16Mark Blaug, The Methodology of Economics: Or How Economists Explain, 2d ed., Cambridge University Press, New York, 1992, p. 92.
17Thestandard erroris nothing but the standard deviation of the sampling distribution of the estimator, and the sampling distribution of an estimator is simply a probability or fre- quency distribution of the estimator, that is, a distribution of the set of values of the estimator obtained from all possible samples of the same size from a given population. Sampling distri- butions are used to draw inferences about the values of the population parameters on the basis of the values of the estimators calculated from one or more samples. (For details, see App. A.)
as follows:
where var =variance and se =standard error and where σ2is the constant or homoscedastic variance of ui of Assumption 4.
All the quantities entering into the preceding equations except σ2can be estimated from the data. As shown in Appendix 3A, Section 3A.5, σ2itself is estimated by the following formula:
ˆ σ2=
uˆ2i
n−2 (3.3.5)
whereσˆ2 is the OLS estimator of the true but unknown σ2 and where the expressionn−2 is known as the number of degrees of freedom (df),
ˆ u2i being the sum of the residuals squared or the residual sum of squares (RSS).18
Once
ˆ
u2i is known, σˆ2can be easily computed.
ˆ
u2i itself can be com- puted either from (3.1.2) or from the following expression (see Section 3.5 for the proof ):
uˆ2i =
yi2− ˆβ22
xi2 (3.3.6)
Compared with Eq. (3.1.2), Eq. (3.3.6) is easy to use, for it does not require computinguˆifor each observation although such a computation will be use- ful in its own right (as we shall see in Chapters 11 and 12).
Since
βˆ2= xiyi
xi2
(3.3.1) (3.3.2)
(3.3.3)
(3.3.4) var (βˆ2)= σ2
xi2 se (βˆ2)= σ
xi2
var (βˆ1)= X2i n
xi2σ2 se (βˆ1)= Xi2
n xi2σ
18The term number of degrees of freedommeans the total number of observations in the sample (=n) less the number of independent (linear) constraints or restrictions put on them.
In other words, it is the number of independent observations out of a total of nobservations.
For example, before the RSS (3.1.2) can be computed, βˆ1andβˆ2must first be obtained. These two estimates therefore put two restrictions on the RSS. Therefore, there are n−2, notn, in- dependent observations to compute the RSS. Following this logic, in the three-variable regres- sion RSS will have n−3 df, and for the k-variable model it will have n−kdf.The general rule is this:df=(n−number of parameters estimated).
an alternative expression for computing ˆ u2i is
(3.3.7)
In passing, note that the positive square root of σˆ2
(3.3.8)
is known as the standard error of estimate orthe standard error of the regression (se).It is simply the standard deviation of the Yvalues about the estimated regression line and is often used as a summary measure of the “goodness of fit” of the estimated regression line, a topic discussed in Section 3.5.
Earlier we noted that, given Xi,σ2 represents the (conditional) variance of both ui andYi.Therefore, the standard error of the estimate can also be called the (conditional) standard deviation of ui andYi.Of course, as usual, σY2andσY represent, respectively, the unconditional variance and uncondi- tional standard deviation of Y.
Note the following features of the variances (and therefore the standard errors) of βˆ1andβˆ2.
1. The variance of βˆ2is directly proportional to σ2but inversely propor- tional to xi2.That is, given σ2, the larger the variation in the Xvalues, the smaller the variance of βˆ2and hence the greater the precision with which β2 can be estimated. In short, given σ2, if there is substantial variation in the X values (recall Assumption 8), β2 can be measured more accurately than when the Xi do not vary substantially. Also, given xi2, the larger the vari- ance of σ2, the larger the variance of β2. Note that as the sample size n increases, the number of terms in the sum, xi2, will increase. As n in- creases, the precision with which β2can be estimated also increases. (Why?) 2. The variance of βˆ1 is directly proportional to σ2 and X2i but in- versely proportional to xi2and the sample size n.
3. Sinceβˆ1andβˆ2are estimators, they will not only vary from sample to sample but in a given sample they are likely to be dependent on each other, this dependence being measured by the covariance between them. It is shown in Appendix 3A, Section 3A.4 that
(3.3.9) cov (βˆ1,βˆ2)= − ¯Xvar (βˆ2)
= − ¯X σ2
xi2 ˆ
σ = uˆ2i n−2 uˆ2i =
yi2− xiyi
2
xi2
Since var (βˆ2) is always positive, as is the variance of any variable, the nature of the covariance between βˆ1andβˆ2 depends on the sign of X¯.If X¯ is posi- tive, then as the formula shows, the covariance will be negative. Thus, if the slope coefficient β2isoverestimated(i.e., the slope is too steep), the intercept coefficient β1 will be underestimated (i.e., the intercept will be too small).
Later on (especially in the chapter on multicollinearity, Chapter 10), we will see the utility of studying the covariances between the estimated regression coefficients.
How do the variances and standard errors of the estimated regression coefficients enable one to judge the reliability of these estimates? This is a problem in statistical inference, and it will be pursued in Chapters 4 and 5.