ANALYSIS OF VARIANCE AND QUADRATIC FORMS
4.2 Analysis of Variance
108 4. ANALYSIS OF VARIANCEAND QUADRATIC FORMS
BothP and (I−P) are symmetric and idempotent so thatPP =P and (I−P)(I−P) = (I−P). The two middle terms in equation 4.14 are zero since the two quadratic forms are orthogonal to each other:
P(I−P) =P−P =0.
Thus,
YY = YP Y +Y(I−P)Y =YY+ee. (4.15) The total uncorrected sum of squares has been partitioned into two quadratic forms with defining matricesP and (I−P), respectively.YY is that part ofYY that can be attributed to the model being fit and is labeled SS(Model). The second termeeis that part ofYY not explained by the model. It is the residual sum of squares after fitting the model and is labeled SS(Res).
The orthogonality of the quadratic forms ensures that SS(Model) and Degrees of Freedom SS(Res) are additive partitions. The degrees of freedom associated with
each will depend on the rank of the defining matrices. The rank ofP = [X(XX)−1X] is determined by the rank ofX. For full-rank models, the rank ofXis equal to the number of columns inX, which is also the number of parameters inβ. Thus, the degrees of freedom for SS(Model) ispwhen the model is of full rank.
The r(P) is also given by tr(P) since P is idempotent. A result from matrix algebra states that tr(AB) = tr(BA). (See Exercise 2.24.) Note the rotation of the matrices in the product. Using this property, withA=X andB= (XX)−1Xwe have
tr(P) = tr[X(XX)−1X] = tr[(XX)−1XX]
= tr(Ip) =p. (4.16)
The subscript onI indicates the order of the identity matrix. The degrees of freedom of SS(Res), n−p, are obtained by noting the additivity of the two partitions or by observing that the tr(I−P) = tr(In)−tr(P) = (n−p). The order of this identity matrix isn. For each sum of squares, the correspondingmean square is obtained by dividing the sum of squares by its degrees of freedom.
The expressions for the quadratic forms, equation 4.15, are the defini- Computational Forms
tional forms; they show the nature of the sums of squares being computed.
There are, however, more convenient computational forms. The computa- tional form for SS(Model) =YY is
SS(Model) = βXY, (4.17)
and is obtained by substitutingXβfor the the firstY andX(XX)−1XY for the second. Thus, the sum of squares due to the model can be computed
4.2 Analysis of Variance 109
TABLE 4.1.Analysis of variance summary for regression analysis.
Sum of Squares
Source of Degrees of Definitional Computational
Variation Freedom Formula Formula
Total(uncorr) r(I) =n YY
Due to model r(P) =p YY =YP Y βXY Residual r(I−P) = (n−p) ee =Y(I−P)Y YY −βXY
without computing the vector of fitted values or then×nmatrixP. Theβ vector is much smaller thanY, andXY will have already been computed.
Since the two partitions are additive, the simplest computational form for SS(Res)=eeis by subtraction:
SS(Res) = YY −SS(Model). (4.18) The definitional and computational forms for this partitioning of the total sum of squares are summarized in Table 4.1.
(Continuation of Example3.8) The partitioning of the sums of squares is
illustrated using the Heagle ozone example (Table 3.1, page 82). The total Example 4.2 uncorrected sum of squares with four degrees of freedom is
YY = ( 242 237 231 201 )
242237 231201
= 2422+ 2372+ 2312+ 2012= 208,495.
The sum of squares attributable to the model, SS(Model), can be obtained from the definitional formula, usingY from Table 3.1, as
YY = ( 247.563 232.887 221.146 209.404 )
247.563 232.887 221.146 209.404
= 247.5632+ 232.8872+ 221.1462+ 209.4042
= 208,279.39.
The more convenient computational formula gives βXY = ( 253.434 −293.531 )
911 76.99
= 208,279.39.
110 4. ANALYSIS OF VARIANCEAND QUADRATIC FORMS (See the text following equation 3.12 forβandXY.)
The definitional formula for the residual sum of squares (see Table 3.1 fore) gives
ee = (−5.563 4.113 9.854 −8.404 )
−5.563 4.113 9.854
−8.404
= 215.61.
The simpler computational formula gives
SS(Res) = YY −SS(Model) = 208,495−208,279.39
= 215.61.
The total uncorrected sum of squares has been partitioned into that Meaning of SS(Regr) due to the entire model and a residual sum of squares. Usually, however,
one is interested in explaining the variation of Y about its mean, rather than about zero, and in how much the information from the independent variables contributes to this explanation. If no information is available from independent variables, the best predictor ofY is the best available estimate of the population mean. When independent variables are available, the question of interest is how much information the independent variables contribute to the prediction ofY beyond that provided by the overall mean ofY.
The measure of the additional information provided by the indepen- dent variables is the difference between SS(Model) when the independent variables are included and SS(Model) when no independent variables are included. The model with no independent variables contains only one pa- rameter, the overall meanµ. Whenµ is the only parameter in the model, SS(Model) is labeled SS(µ). [SS(µ) is commonly called the correction factor.] Theadditional sum of squares accounted for by the independent variable(s) is called the regression sum of squares and labeled SS(Regr).
Thus,
SS(Regr) = SS(Model)−SS(µ), (4.19) where SS(Model) is understood to be the sum of squares due to the model containing the independent variables.
The sum of squares due toµ alone, SS(µ), is determined using matrix SS(µ) notation in order to show the development of the defining matrices for the
quadratic forms. The model when µis the only parameter is still written in the formY =Xβ+, but nowX is only a column vector of ones and
4.2 Analysis of Variance 111 β=µ, a single element. The column vector of ones is labeled1. Then,
β = (11)−11Y = 1
n
1Y =Y (4.20)
and
SS(µ) = β(1Y) = 1
n
(1Y)(1Y)
= Y(1
n11)Y. (4.21)
Notice that1Y =
Yiso that SS(µ) is (
Yi)2/n, the familiar result for the sum of squares due to correcting for the mean. Multiplication of 11 gives ann×nmatrix of ones. Convention labels this theJ matrix. Thus, the defining matrix for the quadratic form giving the correction factor is
1
n(11) = 1 n
1 1 1 · · · 1 1 1 1 · · · 1 1 1 1 · · · 1 ... ... ... ...
1 1 1 · · · 1
= 1
nJ. (4.22)
The matrix (J/n) is idempotent with rank equal to tr(J/n) = 1 and, hence, the correction factor has 1 degree of freedom.
Theadditionalsum of squares attributable to the independent variable(s) Quadratic form for SS(Regr) in a model is then
SS(Regr) = SS(Model)−SS(µ)
= YP Y −Y(J/n)Y
= Y(P −J/n)Y. (4.23) Thus, the defining matrix for SS(Regr) is (P −J/n). The defining matrix J/nis orthogonal to (P−J/n) and (I−P) (see exercise 4.15) so that the total sum of squares is now partitioned into three orthogonal components:
YY = Y(J/n)Y +Y(P −J/n)Y +Y(I−P)Y
= SS(µ) + SS(Regr) + SS(Res) (4.24)
with 1, (p−1) = p, and (n−p) degrees of freedom, respectively. Usu- ally SS(µ) is subtracted fromYY and only thecorrected sum of squares partitioned into SS(Regr) and SS(Res) reported.
For the Heagle ozone example, Example 4.2, Example 4.3
SS(µ) = (911)2
4 = 207,480.25
112 4. ANALYSIS OF VARIANCEAND QUADRATIC FORMS
TABLE 4.2.Summary analysis of variance for the regression of soybean yield on ozone exposure (Data courtesy A. S. Heagle, N. C. State University).
Source of Mean
Variation d.f. Sum of Squares Squares
Totaluncorr 4 YY = 208,495.00
Mean 1 nY2 = 207,480.25
Totalcorr 3 YY −nY2 = 1,014.75
Regression 1 βXY −nY2 = 799.14 799.14 Residuals 2 YY −βXY = 215.61 107.81
so that
SS(Regr) = 208,279.39−207,480.25 = 799.14.
The analysis of variance for this example is summarized in Table 4.2.
The key points to remember are summarized in the following.
• The rank ofXis equal to the number of linearly independent columns inX.
• The model is a full rank model if the rank ofX equals the number of columns ofX, (n > p).
• Theuniqueordinary least squares solution exists only if the model is of full rank.
• The defining matrices for the quadratic forms in regression are all idempotent. Examples areI,P, (I−P), andJ/n.
• The defining matrices J/n, (P −J/n), and (I −P) are pairwise orthogonal to each other and sum toI. Consequently, they partition the total uncorrected sum of squares into orthogonal sums of squares.
• The degrees of freedom for a quadratic form are determined by the rank of the defining matrix which, when it is idempotent, equals its trace. For a full rank model,
r(I) = n, the only full rank idempotent matrix r(P) = p
r(J/n) = 1 r(P −J/n) = p
r(I−P) = n−p.