Quiz 2- Linear Regression Analysis (Based on Lectures 15-31) Time: 1 Hour

1

Quiz 2- Linear Regression Analysis (Based on Lectures 15-31) Time: 1 Hour

1. The random errors ε in multiple linear regression model y=Xβ ε+ are assumed to be identically and independently distributed following the normal distribution with zero mean and constant variance. Here y is a n×1 vector of observations on response variable, X is a n K× matrix of n observations on each of the K explanatory variables,

β is a K×1 vector of regression coefficients and ε is a n×1 vector of random errors.

The residuals ˆε = −y yˆ based on the ordinary least squares estimator of β have, in general,

(A) zero mean, constant variance and are independent (B) zero mean, constant variance and are not independent (C) zero mean, non constant variance and are not independent (D) non zero mean, non constant variance and are not independent

Answer: (C)

Solution:The residual is

1 1

2

ˆ ˆ

where ( ' ) '

( ) where ( ' ) '

( )

( )ˆ 0

( )ˆ ( )

y y

y Xb b X X X y

I H y H X X X X

I H E

V I H

ε

ε ε

ε σ

−

−

= −

= − =

= − =

= −

=

= −

• Since E( )εˆ =0, so ˆε_i's have zero mean.

• Since I−H is not generally a diagonal matrix, so ˆ 'ε_i sdo not have necessarily the same variances.

• The off-diagonal elements in (I−H) are not zero, in general. So ˆ 'ε_i s are not independent.

2

2. Consider the multiple linear regression model y= Xβ ε+ , ( )E ε =0,

2 2 2

1 2

( ) ( , ,..., _n)

V ε =diag σ σ σ where y is a n×1 vector of observations on response variable, X is a n K× matrix of n observations on each of the K explanatory variables,

β is a K×1 vector of regression coefficients and ε is a n×1 vector of random errors.

Let h_ii be the i^th diagonal element of matrix H =X X X( ' )⁻¹. The variance of the i^th PRESS residual is

(A) σ_i²(1−h_ii) (B) σ_i²(1−h_ii)² (C)

2

1

i

hii

σ

− (D)

2

(1 )2 i

hii

σ

−

Answer: (C)

Solution:

Let e_i = −y_i yˆ_ibe the i^th ordinary residual based on the ordinary least squares estimator of β. The i^th PRESS residual e_(i) is

( ) 1

i i

ii

e e

= h

− .

The variance of e_{( )i} is obtained as follows:

( ) 2

( ) ( )

(1 )

i i

ii

Var e Var e

= h

− and

2

( ) ( ) ( )

( )_{i i} (1 _ii).

V e I H V

Var e h

ε σ

= −

⇒ = −

Thus

2

( ( )) . 1

i i

ii

Var e

h

= σ

−

3

3. Consider the linear model y=β₁X₁+β₂X₂+ε, ( )E ε =0, ( )V ε =Iwhere the study variable y and the explanatory variables X₁ and X₂are scaled to length unity and the correlation coefficient between X₁ and X₂is 0.5. Let b₁ and b₂be the ordinary least squares estimators of β₁ and β₂respectively. The covariance between b₁ and b₂ is

(A) 2/3 (B) -2/3 (C) -0.5 (D) 1/3

Answer: (B)

Solution:

If ris the correlation coefficient betweenX₁ and X₂, then the ordinary least squares estimators b₁ and b₂ of β₁ and β₂ are the solutions of

1 1

2

( ' ) ' where ' 1 . 1

b r

b X X X y X X

b r

  −  

= = = 

 

  Then

1

2

1

2

1 2 2

1 1 ( ' )

1 1

1 1 ( ) ( ' )

1 1

( , ) .

1 X X r

r r V b X X r

r r Cov b b r

r

−

−

 − 

= − − 

 − 

= = − − 

= − −

1 2

When 0.5, cov( , ) 2. r= b b = −3

4

4. Under the multicolinearity problem in the data in the modely=β₁X₁+β₂X₂+ε, where the study variable y and the explanatory variables X₁ and X₂are scaled to length unity and the random error ε is normally distributed with^E

( )

ε =^0,V

( )

ε =σ^2I whereσ²is unknown. What do you conclude about the null hypothesis H₀₁:β₁=0 and H₀₂:β₂ =0 out of the following when sample size is small.

(A) Both H₀₁and H₀₂ are more often accepted.

(B) Both H₀₁and H₀₂ are more often rejected.

(C) H₀₁is accepted more often and H₀₂ is rejected more often.

(D) H₀₁is rejected more often and H₀₂ is accepted more often.

Answer: (A)

Solution:

If ris the correlation coefficient betweenX₁ and X₂, then the ordinary least squares estimators b₁ and b₂are the solutions of

1 1

( ' ) ' where ' . 1

b X X X y X X r

r

−  

= =  

 

Then

2

2 1

2

( )

( ) ( ' ) 1 .

1 1

E b

V b X X r

r r β

σ ⁻ σ

=

 − 

= = − − 

As rbecomes higher, the variance of b₁ and b₂become larger and so the test statistic

 , 1, 2 var( )

i i

i

t b i

b

= =

for testing H_0i:β =_i 0 becomes very small and so H_0iis more often accepted.

5

5. Consider the setup of multiple linear regression model y=Xβ ε+ where y is a n×1 vector of observations on response variable, X is a n K× matrix of n observations on each of the K explanatory variables, β is a K×1 vector of regression coefficients and ε is a n×1 vector of random errors following N(0,σ²I)with all usual assumptions. Let

( ' ) and 1 _ii

H =X X X ⁻ h be the i^th diagonal element of H. A data point is a leverage point if h_iiis

(A) greater than the value of average size of hat diagonal.

(B) less than the value of average size of hat diagonal.

(C) greater or less than the value of average size of hat diagonal.

(D) none of the above.

Answer: (A)

Solution:

The i^thdiagonal element of His h_ii = x (¹_i X X¹ )⁻¹x_i where x_i is the i^th row of X matrix. Theh_iiis a standardized measure of distance of i^th observation from the center (or centroid) of the x-space.

( )

^{1 ( )}

Average size of hat diagonal .

n ii i

h rank H trH K

h n n n n

=

∑

= = = =

Ifh_ii >2hthen the point is remote enough to be considered as a leverage point.