Predictive Analytics Regression and Classification - Lecture 1

Predictive Analytics Regression and Classification

Lecture 1 : Part 5

Sourish Das

Chennai Mathematical Institute

Aug-Nov, 2019

Linear Regression

mpg=β₀+β₁ wt +

2 3 4 5

1015202530

wt

mpg

Linear Regression

I mpg= β0+β1 wt+

I We write the model in terms of linear models y=Xβ+

wherey= (mpg₁ ,mpg₂ , . . . ,mpg_n)^T;

X=









 1 wt1

1 wt2

... ... 1 wtn











β= (β₀, β₁)^T and= (₁, ₂, . . . , _n)^T

Linear Regression

I Normal Equations:

βˆ = ( ˆβ0 βˆ1)^T = (X^TX)⁻¹X^Ty

=

n Pn i=1wt_i Pn

i=1wti P

i=1wt²_i

−1 Pn i=1mpg_i Pn

i=1wti.mpg_i

Quadratic Regression

mpg=β₀+β₁ hp +β₂ hp² +

50 150 250

1015202530

hp

mpg

Quadratic Regression

I mpg= β₀+β₁ hp+β₂ hp² +

I We write the model in terms of linear models y=Xβ+

wherey= (mpg₁ ,mpg2 , . . . ,mpgn)^T;

X=











1 hp₁ hp²₁ 1 hp₂ hp²₂ ... ... ... 1 hp_n hp²_n









 ,

β= (β0, β1, β2)^T and= (1, 2, . . . , n)^T

I The linear model is linear in parameter.

Linear Regression

I Normal Equations:

βˆ = ( ˆβ0 βˆ1 βˆ2)^T

= (X^TX)⁻¹X^Ty

=





n Pn

i=1hp_i Pn i=1hp²_i P_n

i=1hp_i P

i=1hp²_i P_n

i=1hp³_i Pn

i=1hp²_i Pn

i=1hp³_i Pn i=1hp⁴_i





−1

 Pn

i=1mpg_i P_n

i=1hp_i.mpg_i Pn

i=1hp²_i.mpg_i





Non-linear Regression Basis Functions

I Consideri^th record

y_i =f(x_i) +_i represents f(x) as

f(x) =

K

X

j=1

β_jφ_j(x) =φβ

we sayφis a basis system for f(x).

Representing Functions with Basis Functions

I mpg= β₀+β₁ hp+β₂ hp² +

I Generic terms for curvature in linear regression y =β1+β2x+β3x²+· · ·+i

implies

f(x) =β1+β2x+β3x²+· · ·

I Sometimes in ML φis known as ‘engineered features’ and the process is known as ‘feature engineering’

Fourier Basis

I sine cosine functions of incresing frequencies

y =β1+β2sin(ωx)+β3cos(ωx)+β4sin(2ωx)+β5cos(2ωx)· · ·+_i

I constant ω= 2π/P defines the periodP of oscillation of the first sine/cosine pair. P is known.

I φ={1,sin(ωx),cos(ωx),sin(2ωx),cos(2ωx)...}

I β^T ={β₁, β₂, β₃,· · · }

y =φβ+

I Again in MLφis known as ‘engineered features’

I mpg= β₀+β₁sin(ω hp ) +

Functional Estimation/Learning

I We are writing the function with its basis expansion

y =φβ+

I Lets assume basis (or engineered features) φarefully known.

I Problem is β is unknown - hence we estimateβ.

Regression Line

mpg=β₀+β₁hp+

50 100 150 200 250 300

1015202530

hp

mpg

Feature Engineering

mpg=β₀+β₁hp+β₂ hp²+

50 100 150 200 250 300 350

101520253035

0 20000

40000 60000

80000 100000

120000

hp

hp2mpg

In the next lecture...

I We will discuss sampling distributions and inference of regressions !