1. The Simple Regression Model Two Variables

(1)

1. The Simple Regression Model Two Variables

Read Wooldridge (2013), Chapter 2

1 The Simple Regression Model . Quantitative Methods of Economic Analysis . 2949605 . Chairat Aemkulwat

Outline

I. Definition

II. Deriving OLS Estimators III. Goodness of Fit

VI. Expected Values and Variances

I. Definition II. Derivation III. R-Squared IV. EV&Var

1 The Simple Regression Model . Quantitative Methods of Economic Analysis . 2949605 . Chairat Aemkulwat

I. Definition

• Regression Analysis

– the study of the dependence of one variable on one or more other variables.

• Simple Regression Analysis – two variables: y and x

• the measure of the marginal effect of x on y

I. Definition II. Derivation III. R-Squared IV. EV&Var 3

1. The Simple Regression Model . Quantitative Methods of Economic Analysis . 2949605 . Chairat Aemkulwat I. Definition

Three issues:

1) What is the functional relationship between x and y?

2) There is never exact relationship between two variables in the real world.

3) How can we find the ceteris paribus relationship between y and x?

(2)

A Simple Regression Model

• A simple linear model can be written as y= ₀+ ₁x+ u

where

y, x : variables

₀, ₁ : parameters u : error term

Example

• Define y be the “wage” rate and x be years of

“education”.

wage = 

₀

+ 

₁

educ + u

• u: unobserved factors

examples: innate ability, experience and other factors.

Assumption: Zero Population Mean versus Zero Conditional Mean

• E(u) = 0

– This implies that the average value of uin the population is zero.

• E(u|x) = 0

– We say the expected value of ugiven xequals zero.

• Example: The average level of innate ability is the same regardless of years of education (x = educ).

eg. educ= 12 , 16

Population Regression Function: PRF

• Understand components:

y= ₀+ ₁x+ u

• Population regression function (PRF) is derived by taking expectation conditional on x.

E(y|x) = ₀+ ₁x

• E(y|x) as a linear function of x, where for any x the distribution of y is centered about E(y|x)

. .

x₁ x₂

E(y|x) = ₀+ ₁x

y

f(y)

(3)

Population Regression Line

• Population regression line, data points and the associated error terms

– n = 38 M – y_i= ₀+ ₁x_i+ u_i

(data points)

– E(y|x) = ₀+ ₁x (PRL) – PRL = population

regression line

• Error terms

 u_i= y_i‐E(y|x)

9

. .

x₁ x₂

E(y|x) = 0+ 1x

y

f(y)

Population regression line, data points and the associated error terms

.

. .

.

y₄

y₁ y₂ y₃

x₁ x₂ x₃ x₄

}

{

u₁ u₂

u₃ u₄

x

y E(y|x) = ₀+ ₁x

– n = 38 M – y_i= ₀+ ₁x_i+ u_i

(data points)

– E(y|x) = ₀+ ₁x (PRL) – PRL = population

regression line

• Error terms

 u_i= y_i‐E(y|x_i)

wage= ₀+ ₁educ+ u

Population vs Sample

– n = 38 M – y_i= ₀+ ₁x_i+ u_i

(data points)

– E(y|x) = ₀+ ₁x (PRL) PRL = population regression line

• Error terms

 u_i= y_i‐E(y|x)

11

• Sample regression line, sample data points and the associated estimated error terms

– n = 526

– y_i= ₀+ ₁x_i+ u_i (sample data points) –

SRL = sample regression line

• Residuals – _i= y_i= _i

x yˆ   ˆ₀ ˆ₁ wage= ₀+ ₁educ+ u

Sample regression line, sample data points and the associated estimated error terms

.

. .

.

y₄

y₁ y₂ y₃

x₁ x₂ x₃ x₄ }

}

{

û₁ û₂

û₃ û₄

x y yˆˆ₀ˆ₁x

• Sample regression line, sample data points and the associated estimated error terms

– n = 526 – y_i= ₀+ ₁x_i+ u_i

(sample data points)

– (SRL)

SRL = sample regression line

• Residuals – _i= y_i= _i

x yˆ   ˆ₀ ˆ₁

wage= ₀+ ₁educ+ u

(4)

Sample regression line, sample data points and the associated estimated error terms

. . .

.

y

₄

y

₁

y

₂

y

₃

x

₁

x

₂

x

₃

x

₄

}

{

û

₁

û

₂

û

₃

û

₄

x

y

_y _x

1

0 ˆ

ˆ 



ˆ 



obs ACT GPA

1 21 2.8

2 24 3.4

3 26 3

4 27 3.5

5 29 3.6

6 25 3

7 25 2.7

8 30 3.7

Some Terminology

• In the simple linear regression model, where

y= ₀+ ₁x+ u,

• We typically refer to yas the

Dependent Variable

Left‐Hand Side Variable

Explained Variable

Regressand

Response Variable

Predicted Value

• We typically refer to xas the

Independent Variable

Right‐Hand Side Variable

Explanatory Variable

Regressor

Control Variables

Predictor Variable

Covariate

II. Deriving the OLS Estimates

• Let {(x_i, y_i): i=1, …, n} be a random sample of size n from the population. For each i

y_i= ₀+ ₁x_i+ u_i

• How to find the estimates of parameters, ₀& ₁?

• Two Methods:

1) Method of Moments 2) Least Squares Method

1. The Simple Regression Model . Quantitative Methods of Economic Analysis . 2949605 . Chairat Aemkulwat II. Deriving the OLS Estimates

Deriving OLS using Method of Moments

• Method of Moments

– Here we replace the population moment with the sample average.

• What does this mean? Recall that for

– E(X), the mean of a population distribution,

– a sample estimator of E(X) is simply the arithmetic mean of the sample, .

(5)

Method of Moments

Four assumptions:

(1) The model is linear.

(2) x_i’s are not the same.

(3) The expected value of uis zero [E(u) = 0].

(4) The covariance between xand uis zero.

[Cov(x,u) = E(xu) = 0] ^{Note that}Cov(X,Y) = E(XY) – E(X)E(Y)

Method of Moments

Given: u= y –₀–₁x From Assumptions,

(1) E(y –₀–₁x) = 0 (2) E[x(y –₀–₁x)] = 0

• These are called moment restrictions

 

1 2

(2) ˆ ⁱ ⁱ

i

x x y x x

 

 

 

0 1 0 1

ˆ ˆ ˆ ˆ

(1) y x, or   y x Derive

 



^ˆ ^ˆ



⁰

ˆ 0 ˆ

1

1 0 1

1 0













 n

i

i i

i n

i

i i

x y

x

x y



More Derivation of OLS

• Given the definition of a sample mean, and properties of summation, we can rewrite the first condition as follows.

x y

1 0

ˆ ˆ

or

ˆ , ˆ











0 1

1

ˆ ˆ

( ) 0

n

i i

i

y

 

x



  



More Derivation of OLS

 

   

  



 



















n

i i i

n

i i

n

i i i n

i

i i

i

x x y

y x x

x x x y

y x

x x y y x

1 2 1

1

1 1 1

1

1 1

ˆ ˆ

ˆ 0 ˆ



• we can rewrite the second condition as follows.

  

   ²

1

1 2 1

1

ˆ provided that 0

n

i i n

i n i

i i

i

x x y y

x x x x

 ^



 

  



 



(6)

Method of Least Squares

• To find sample regression function from the method of least squares

– use graph – fitted values of y_i – residuals of observation i – minimizing the sum of squared

residuals

– find first order equations (# of restrictions)

    



 



 ⁿ

i

i i

n

i

i y x

u

1

2 1 0 1

2 ˆ ˆ

ˆ  

.

. .

.

y₄

y₁ y₂ y₃

x₁ x₂ x₃ x₄ }

}

{

û₁ û₂

û₃ û₄

x y yˆˆ₀ˆ₁x

Alternate approach to derivation

• Given the intuitive idea of fitting a line, we can set up a formal minimization problem

• That is, we want to choose our parameters such that we minimize the following:

 

  



 



 ⁿ

i

i i

n

i

i y x

u

1

2 1 0 1

2 ˆ ˆ

ˆ  

• Intuitively, OLS is fitting a line through the sample points such that the sum of squared residuals is as small as possible, hence the term least squares.

• The residual, û, is an estimate of the error term, u, and is the difference between the fitted line (sample regression function) and the sample point.

Alternate approach, continued

• If one uses calculus to solve the minimization problem for the two parameters you obtain the following first order conditions, which are the same as we obtained before, multiplied by n

 



^ˆ ^ˆ



⁰

ˆ 0 ˆ

1

1 0 1

1 0













 n

i

i i

i n

i

i i

x y

x

x y



Algebraic Properties of OLS Statistics

(1) The sample average of residuals is zero.

(2) The sample covariance between regressors and OLS residuals, , is zero.

(3) The point ( ̅and ) is always on the OLS regression line.

(4) The sample covariancebetween and is zero.



⁰ ¹



1

ˆ ˆ 0

n

i i

i

y  x



  





⁰ ¹



1

ˆ ˆ 0

n

i i i

i

x y  x



  



0 1 0 1

ˆ ˆ , or ˆ ˆ y x   y x

(7)

Summary of OLS slope estimate

• The slope estimate is the sample covariance between xand y divided by the sample variance of x

• If xand yare positively correlated, the slope will be positive

• If xand yare negatively correlated, the slope will be negative

• Only need xto vary in our sample

  

 

1

1 2

1

ˆ

n

i i

i n

i i

x x y y

x x

 ^



 







 

1 2

ˆ ⁱ ⁱ

i

x x y x x

 

 

 

Example ACT scores and the GPA

• y = GPA(grade point average, 0‐4.0) x = ACT(American College Test, 0‐30) n = 8 students

= 0.57 + 0.10ACT

• Interpret =0.57; =0.10.

• What is the predicted value of GPA when ACT=20?

obs GPA ACT

1 2.8 21

2 3.4 24

3 3 26

4 3.5 27

5 3.6 29

6 3 25

7 2.7 25

8 3.7 30

  

 

  

 

1 1

1 2 2

1 1

ˆ

n n

i i i i

i i

n n

i i

x x y y ACT ACT GPA GPA

x x ACT ACT

 ^ ^

 

   

 

 

 

x y

obs ACT GPA

1 21 2.8

2 24 3.4

3 26 3

4 27 3.5

5 29 3.6

6 25 3

7 25 2.7

8 30

3.7

Average 25.875 3.2125

-4.875 23.765625 -0.4125 2.0109375

-1.875 3.515625 0.1875 -0.3515625

0.125 0.015625 -0.2125 -0.0265625

1.125 1.265625 0.2875 0.3234375

3.125 9.765625 0.3875 1.2109375

-0.875 0.765625 -0.2125 0.1859375

-0.875 0.765625 -0.5125 0.4484375

4.125 17.015625 0.4875 2.0109375

56.875 5.8125

0.102197802 (GPA

( ²

( ² )(GPA ²

 )(GPA ²

( ²/ )(GPA ²=

= ̅ 3.21 .102 ∗ 25.875 .57

Graph, Fitted Values and Residuals

Actual Fitted

obs ACT GPA Residual

1 21 2.8 2.7142 0.0857

2 24 3.4 3.0208 0.3791

3 26 3 3.2252 -0.2252

4 27 3.5 3.3274 0.1725

5 29 3.6 3.5318 0.0681

6 25 3 3.1230 -0.1230

7 25 2.7 3.1230 -0.4230

8 30 3.7 3.6340 0.0659

2.6 2.8 3.0 3.2 3.4 3.6 3.8

20 22 24 26 28 30 32

ACT

GPA

GPA vs. ACT

(8)

Eviews: Regress GPA on ACT

Dependent Variable: GPA Method: Least Squares Sample: 1 8

Included observations: 8

Variable Coefficient Std. Error t-Statistic Prob.

C 0.568132 0.928421 0.611933 0.563

ACT 0.102198 0.035692 2.863324 0.0287

R-squared 0.577424 Mean dependent var 3.2125

Adjusted R-squared 0.506994 S.D. dependent var 0.383359 S.E. of regression 0.269173 Akaike info criterion 0.425395

Sum squared resid 0.434725 Schwarz criterion 0.445255

Log likelihood 0.298422 F-statistic 8.198622

Durbin-Watson stat 2.268603 Prob(F-statistic) 0.028677

Example: Effect of wages on education

• n=526 individuals in 1976 wage (in dollars per hour) educ (years of schooling

w = – 0.90 + 0.54educ

• Define

• What is the predicted hourly wage for high school students?

54 ˆ .

; 90 ˆ .

1

0   



Eview: w = ‐0.90 + 0.54educ

Dependent Variable: WAGE Method: Least Squares Sample: 1 526

Included observations: 526

Variable Coefficient Std. Error t-Statistic Prob.

C -0.90485 0.684968 -1.321013 0.1871

EDUC 0.541359 0.053248 10.16675 0

R-squared 0.164758 Mean dependent var 5.896103

Adjusted R-squared 0.163164 S.D. dependent var 3.693086

S.E. of regression 3.37839 Akaike info criterion 5.27647

Sum squared resid 5980.682 Schwarz criterion 5.292688

Log likelihood -1385.71 F-statistic 103.3627

Durbin-Watson stat 1.823686 Prob(F-statistic) 0

III. Goodness of Fit – R 2

• SST = SSE + SSR

SST : total sum of squares

‐the total sample variation in the y_i SSE : explained sum of squares

‐the sample variation in the _i SSR : residual sum of squared

‐the sample variation in the _i. i i

i y u

y  ˆ  ˆ

1. The Simple Regression Model . Quantitative Methods of Economic Analysis . 2949605 . Chairat Aemkulwat III. Goodness of Fit – R2

• Graph and define explained part and unexplained part:

     

 

 

   

 

   









































0 ˆ ˆ that know we and

SSE ˆ

ˆ 2 SSR

ˆ ˆ

ˆ 2 ˆ

ˆ ˆ

2 2

2 2 2

y y u y y u

y y y y u u

y y u

y y y y y

y

i i i i

i i

i i i i

R²= SSE SST

(9)

33

.

y₄

y₁ y₂

x₁ x₂ x₃ x₄

}

{

û₁ û₂

û₄

x y

x yˆˆ₀ˆ₁ y₄

ˆ4

y

explained

Find R 2 and Interpretation

• (Requirement: model with intercept)

• 0 < R²< 1

R²= 1 a perfect fit R²0 a poor fit

• Interpretation: It is the fraction of the sample variation in y explained by x.

R²= SSE SST

We can also think of 2 as being equal to the squared correlation coefficient between the actual and the values ˆ

i i

R

y y

 

2 2

ˆ ˆ ˆ ˆ

i i

y y y y

R

y y y y

   

 

     



 

GPA and wages revisited

• Example ACT scores and GPA

= 0.57 + 0.10ACT n=8; R²=0.577424

• Example: Effect of wages on education w = ‐0.90 + 0.54educ

n=526, R²=0.164758

IV. Expected Values and Variances of the OLS

• Algebraic properties of OLS estimation vs. statistical properties of OLS

• Here, we learn about statistical properties of OLS estimators, and , over different random samples from the

population.

• We want to prove that and are unbiased from 4 assumptions,

– SLR.1‐SLR.4.

1. The Simple Regression Model . Quantitative Methods of Economic Analysis . 2949605 . Chairat Aemkulwat IV. Expected Values and Variances of the OLS

(10)

Unbiasedness – (4 assumptions)

• SLR.1 Linear in parameters y = 

₀

+ 

₁

x + u

• SLR.2 Random sampling

A random sample of size n from the population {(y_i, x_i) i = 1,2,….,n}.

Random sampling

• Random sampling – definition

If Y₁, Y₂, …, Y_nare independent random variables with a common pdf f(y;,²), then {Y₁, Y₂, …, Y_n} is a random sample from the population represented by f(y;,²)

• We also say that

– Y_iare i.i.d. (independent, identically distributed) random variables from a distribution.

X_iare i.i.d. r.v.’s

X₁ X₂ X_n

Explain Y

_i

are i.i.d r.v’s and X

_i

are i.i.d r.v’s intuitively

SLR.2: A random sample of size n from the population {(y_i, x_i) i = 1,2,….,n}.

Replication ( , ( , … ( ,

R=1 (n=526) ( , ( , … ( ,

R=2 (n=526) ( , ( , … ( ,

    

R=m (n=526) ( , ( , … ( ,

39 Y_iare i.i.d. r.v.’s

Y1 Y2 Yn

Note that i = 1, …, n (n=526 observations) R = 1, …, m (m=100 times)

Unbiasedness – SLR.3‐SLR.4

• SLR.3 Sampling variation in x

_i

x

_i

are not all equal.

• SLR.4 Zero conditional Mean E(u

_i

|x

_i

) =0

 

 



  ₂

ˆ1

x x

y x x

i i

 i  

 

 



 

 ₁ ₂

ˆ1

x x

u x x

i i

 i

implies 

Show that …

 

1 2

ˆ ⁱ ⁱ

i

x x y x x

 

 

 

(11)

are unbiased (Theorem 2.1)

• Theorem 2.1 Using assumptions SLR(1) through SLR(4),

for any values of _j

• Proof:

1 1) (ˆ   E

1 1) (ˆ  E

 

1 1 2 ^{ } 1

1 2 1

1 ˆ

then 1 ,

ˆ

that so , let







 









 













i i x

i i x i i

u E s d E

u s d x x d

 





  ₂

ˆ1

x x

y x x

i i

 i  

 

 



 

 ₁ ₂

ˆ1

x x

u x x

i i

 i

implies 

The sampling distribution of our estimate is centered around the true parameter

Summary: Unbiasedness

• The OLS estimators of ₁and ₀are unbiased

• Proof of unbiasedness depends on our 4 assumptions – if any assumption fails, then OLS is not necessarily unbiased.

• Remember unbiasedness is a description of the estimator – in a given sample we may be “near” or “far” from the true parameter.

Assumption SLR.5

43

• Assumption SLR.5: Homoskedasticity – constant error variance VAR(u|x) = ²

The variance of the unobservable u, conditional on x, is constant.

We want to know how far is away from ₁ on average, i.e., study VAR( )

• Thus ²is also the unconditional variance, called the error variance.

•  – The square root of the error variance is called the standard deviation of the error.

• SLR.1-SLR.5 are called Gauss- Markov assumptions.

• Var(u|x) = E(u²|x)‐[E(u|x)]²

= E(u²|x)

= E(u²)

= Var(u) = ²

Mean and Variance of y

• Comparison

SLR(4) Zero conditional mean

E(u|x) = 0 (expected value) SLR(5) homoskedasticity

VAR(u|x) = ² (variance of u)

• Find the conditional mean and conditional variance of y E(y|x) = ₀+ ₁x E(u|x) = 0 SLR(3) VAR(y|x) = ² VAR(u|x) = ² SLR(4)

(12)

Homoskedastic Case

Heteroskedastic Case (violation)

Variances of the OLS Estimators

• Find standard deviation of the estimators.

2

1 2

VAR( )ˆ

(x_i x)

  





2 1 2

0 2

VAR( )ˆ

( )

i

n x

x x

  ^



 

 

   

 

2

1 1

2 2

2

2 2

2 2 2 2

2 2

2

2 2 2

2 2 1

ˆ 1

1 1

ˆ 1

i i x

i i i i

x x

i i

x x

x

x x

Var Var d u

s

Var d u d Var u

s s

d d

s s

s Var

s s

 

 

  

   

     

   

   

   

    

 

    



 

Variance of OLS: Summary

Interpret

• The larger the error variance, ², the larger the variance of the slope estimate

• The larger the variability in the x_i, the smaller the variance of the slope estimate

• As a result, a larger sample size should decrease the variance of the slope estimate

• Problem that the error variance is unknown

2

1 2

VAR( )ˆ

(x_i x)

  





(13)

Efficient Estimates

• OLS estimators have the lowest variances.

• An unbiased estimator with minimum variance is called an efficient estimator.

Estimating the Error Variance

• Can we find the value of var( )?

• Errors (u_i) vs. Residuals ( ) y_i= ₀+ ₁x_i+ u_i y_i= + x_i+

• The unbiased estimator of ²is

• There are n‐2 degrees of freedom. This is because there are two restrictions (two FOCs)

2 n

uˆ

ˆ ⁱ

2 2

 





2

1 2

VAR( )ˆ

(x_i x)

  





Unbiased Estimator

• Proof: E( ) = ²

 

   

0 1

0 1 0 1

0 0 1 1

2

2 2

ˆ ˆ

ˆ

ˆ ˆ

Then, an unbiased estimator of is

ˆ 1 ˆ / 2

2

i i i

i i

i

u y x

x u x

u x

u SSR n n

 

   



  

    

    

  





Standard Error of Regression

• ² error variance

• estimate of error variance

• .

Standard error of the regression Standard error of the estimate Root mean squared error

• Standard deviation of :

• Standard error of :

2 n

u ˆ

ˆ ⁱ

2 2