Linear Algebra - Lecture 23

Linear Algebra for Computer Science

Lecture 23

Introduction to Machine Learning

and learning from data

Machine Learning

Model

input output

Classification

Classifier ^Apple

Classification

Classifier ^Orange

Object detection

Detector

Speech Recognition

Model .دوﺑﻧ ﯽﮑﯾ دوﺑ ﯽﮑﯾ

Segmentation

Model

Stock Market Prediction

Predictor

Learning from data

https://www.analyticsvidhya.com/blog/2018/03/comprehensive-collection-deep-learning-datasets/

Supervised Learning

http://seansoleyman.com/effect-of-dataset-size-on-image-classification-accuracy/

Supervised Learning

http://seansoleyman.com/effect-of-dataset-size-on-image-classification-accuracy/

Training data:

X₁, y₁ X₂, y₂ X₃, y₃

:X_n, y_n

Supervised Learning

http://seansoleyman.com/effect-of-dataset-size-on-image-classification-accuracy/

Training data:

Apple

Apple

Orange

⋮

Orange

Supervised Learning

http://seansoleyman.com/effect-of-dataset-size-on-image-classification-accuracy/

Training data:

0

0

1

⋮

1

Supervised Learning

Classifier/

Regressor

input output

Classification

Classifier

input features

y ∈ {Class₁, Class₂, …, Class_n}

Classification

Classifier ^Apple

Classification

Classifier ^Orange

Regression

Regressor ^{y ∈ R}

input features

Regression

Regressor ^{y ∈ Rn}

input features

Regression

Learnable Models

Classifier/

Regressor

input output

Learnable Models: Example

Classifier 0

Learnable Models: Example

Classifier 1

Learnable Models: Input-output map

f

x ^{∈ Rm} y ^{∈ Rn}

y = f(x) f: ^{Rm → Rn}

Learnable Models: Example

f

x =

I^.flatten() y ^{= 0}

y = f(x) f: ^{Rm → Rn} I

Learnable Models: Example

f

x =

features(I⁾ y ^{= 0}

y = f(x) f: ^{Rm → Rn} I

Learnable Models: parameters

f θ

x ^{∈ Rm} y ^{∈ Rn}

y = f(^θ, x)

θ: model parameters

Learnable Models: parameters

Learnable Models: parameters

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Parameter Learning:

○ A collection of input-output paris (x₁, y₁), (x₂, y₂), …, (x_N, y_N),

○ choose θ such that y = f(θ, x) is a reasonable output for any input x.

Learning from data

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Parameter Learning:

○ A collection of input-output paris (x₁, y₁), (x₂, y₂), …, (x_N, y_N),

○ choose θ such that y = f(θ, x) is a reasonable output

■ for training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

■ for unseen data (generalization)

Learning from data

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Parameter Learning:

○ A collection of input-output paris (x₁, y₁), (x₂, y₂), …, (x_N, y_N),

○ choose θ such that y = f(θ, x) is a reasonable output

■ for training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

■ for unseen data (generalization)

Learning from data

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

○ choose θ such that f(θ, x_i) is close to y_i

Learning from data: Cost function

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

○ choose θ such that f(θ, x_i) is close to y_i

Cost function

Learning from data: Cost function

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

○ choose θ such that f(θ, x_i) is close to y_i

○ cost function:

C(θ) = 𝚺_i=1..N d( f(θ, x_i), y_i )

Learning from data: Cost function

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

○ choose θ such that f(θ, x_i) is close to y_i

○ cost function:

C(θ) = 𝚺_i=1..n d( f(θ, x_i), y_i )

data output

Learning from data: Cost function

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

○ choose θ such that f(θ, x_i) is close to y_i

○ cost function:

C(θ) = 𝚺_i=1..n d( f(θ, x_i), y_i )

model output given x_i

Learning from data: Cost function

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

○ choose θ such that f(θ, x_i) is close to y_i

○ cost function:

C(θ) = 𝚺_i=1..n d( f(θ, x_i), y_i )

distance

Learning from data: Cost function

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

○ choose θ such that f(θ, x_i) is close to y_i

○ cost function:

C(θ) = 𝚺_i=1..n

ǁ

_{f(θ, xi) - yi}

ǁ

²

distance

Learning from data: Cost function

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

○ choose θ such that f(θ, x_i) is close to y_i

○ cost function:

C(θ) = 𝚺_i=1..n d( f(θ, x_i), y_i )

Learning from data: Cost function

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

○ choose θ such that f(θ, x_i) is close to y_i

○ cost function:

C(θ) = 𝚺_i=1..n d( f(θ, x_i), y_i ) choose θ such that C(θ) is small

Learning from data: Cost function

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

○ choose θ such that f(θ, x_i) is close to y_i

○ cost function:

C(θ) = 𝚺_i=1..n d( f(θ, x_i), y_i ) θ^* = argmin_θ C(θ)

Cost function

Example: Linear Regression

f θ

x ^{∈ Rm} y ⁼ A x + b ^∈ Rⁿ

Example: Linear Regression

f θ

x ^{∈ Rm} y ⁼ A x + b ^∈ Rⁿ

A: ? by ? matrix

b: ?-D vector

Example: Linear Regression

f θ

x ^{∈ Rm} y ⁼ A x + b ^∈ Rⁿ

A: n by m matrix

b: n-D vector

Example: Linear Regression

f θ

x ^{∈ Rm} y ⁼ A x + b ^∈ Rⁿ

y =

f(

^θ,

x)

θ = ?

Example: Linear Regression

f θ

x ^{∈ Rm} y ⁼ A x + b ^∈ Rⁿ

y =

f(

^θ,

x)

θ = (A,b)

Example: Linear Regression

Affine maps

Example: Linear Regression

f θ

x ^{∈ R} y ⁼ a x + b ^∈ R

y =

f(

^θ,

x)

θ = ?

Example: Linear Regression

f θ

x ^{∈ R} y ⁼ a x + b ^∈ R

y =

f(

^θ,

x)

θ = (a,b)

Example: Linear Regression

Example: Linear Regression

Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

Example: Linear Regression

f

_θ

x ^{∈ R} y ⁼ a x + b ^∈ R

Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

Example: Linear Regression

f

_θ

x ^{∈ R} y ⁼ a x + b ^∈ R

Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

cost function:

C(θ) = 𝚺_i=1..N d( f(θ, x_i), y_i )

Example: Linear Regression

f

_θ

x ^{∈ R} y ⁼ a x + b ^∈ R

Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

cost function:

C(a,b) = 𝚺_i=1..n d( f(a,b, x_i), y_i )

Example: Linear Regression

f

_θ

x ^{∈ R} y ⁼ a x + b ^∈ R

Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

cost function:

C(a,b) = 𝚺_i=1..n d( f(a,b, x_i), y_i ) = 𝚺_i=1..n d( a x_i + b, y_i )

Example: Linear Regression

f

_θ

x ^{∈ R} y ⁼ a x + b ^∈ R

Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

cost function (sum of squared errors):

C(a,b) = 𝚺_i=1..n ( a x_i + b - y_i )²

Example: Linear Regression

f

_θ

x ^{∈ R} y ⁼ a x + b ^∈ R

Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

cost function (sum of squared errors):

C(a,b) = 𝚺_i=1..n ( a x_i + b - y_i )²

a^*,b^* = argmin_a,b 𝚺_i=1..n ( a x_i + b - y_i )²

Example: Linear Regression

cost function (sum of squared errors):

C(a,b) = 𝚺_i=1..n ( a x_i + b - y_i )²

a^*,b^* = argmin_a,b 𝚺_i=1..n ( a x_i + b - y_i )²

How to find a*,b*?

Solution 1: Least squares

Solution 1: Least squares

Solution 1: Least squares

Example: Linear Regression

cost function (sum of squared errors):

C(a,b) = 𝚺_i=1..n ( a x_i + b - y_i )²

a^*,b^* = argmin_a,b 𝚺_i=1..n ( a x_i + b - y_i )²

How to find a*,b*?

Solution 2: partial derivatives

cost function (sum of squared errors):

C(a,b) = 𝚺_i=1..n ( a x_i + b - y_i )²

a^*,b^* = argmin_a,b 𝚺_i=1..n ( a x_i + b - y_i )²

∂ C(a,b) / ∂ a = 0

∂ C(a,b) / ∂ b = 0

Solution 2: partial derivatives

cost function (sum of squared errors):

C(a,b) = 𝚺_i=1..n ( a x_i + b - y_i )²

a^*,b^* = argmin_a,b 𝚺_i=1..n ( a x_i + b - y_i )²

∂ C(a,b) / ∂ a = 2 𝚺_i=1..n x_i ( a x_i + b - y_i ) = 0

∂ C(a,b) / ∂ b = 2 𝚺_i=1..n ( a x_i + b - y_i ) = 0

Solution 2: partial derivatives

cost function (sum of squared errors):

C(a,b) = 𝚺_i=1..n ( a x_i + b - y_i )²

a^*,b^* = argmin_a,b 𝚺_i=1..n ( a x_i + b - y_i )²

𝚺_i=1..n x_i ( a x_i + b - y_i ) = 0 𝚺_i=1..n ( a x_i + b - y_i ) = 0

Solution 2: partial derivatives

a^*,b^* = argmin_a,b 𝚺_i=1..n ( a x_i + b - y_i )²

𝚺_i=1..n x_i ( a x_i + b - y_i ) = a 𝚺_i=1..n x_i² + b 𝚺_i=1..n x_i - 𝚺_i=1..n x_i y_i = 0 𝚺_i=1..n ( a x_i + b - y_i ) = a 𝚺_i=1..n x_i + b n - 𝚺_i=1..ny_i = 0

Solution 2: partial derivatives

a^*,b^* = argmin_a,b 𝚺_i=1..n ( a x_i + b - y_i )²

(

^𝚺_i=1..n ^x_i²

)

^{a +}

(

^𝚺_i=1..n ^x_i

)

^{b = 𝚺}_i=1..n ^x_i ^y_i

(

^𝚺_i=1..n ^x_i

)

a + n b = 𝚺_i=1..n y_i

Solution 2: partial derivatives

a^*,b^* = argmin_a,b 𝚺_i=1..n ( a x_i + b - y_i )²

(

^𝚺_i=1..n ^x_i²

)

^{a +}

(

^𝚺_i=1..n ^x_i

)

^{b = 𝚺}_i=1..n ^x_i ^y_i

(

^𝚺_i=1..n ^x_i

)

a + n b = 𝚺_i=1..n y_i

a^*,b^* ⇐ solve system of linear equations

Solution 2: partial derivatives

Example: Linear Regression

a^*,b^* = argmin_a,b 𝚺_i=1..n ( a xⁱ + b - yⁱ )²

a^*,b^* ⇐ solve system of linear equations

y ⁼ a^* x + b^*

f

_a,b

x y ⁼ a x + b

Evaluation

● Find good parameters θ

○ θ^* = argmin_θ 𝚺_i=1..n ( f(θ, x_i) - y_i )²

○ another method

● How good θ^* is?

● How well the regressor works?

y ⁼ a^* x + b^*

f

_θ

x

Evaluation

● Find good parameters θ

○ θ^* = argmin_θ 𝚺_i=1..n ( f(θ, x_i) - y_i )²

○ another method

● How good θ^* is?

● How well the regressor works?

● Given Training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

Error = C(θ^*) = 𝚺_i=1..N ( f(θ^*, x_i) - y_i )²

y ⁼ a^* x + b^*

Learning from data

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Parameter Learning:

○ A collection of input-output paris (x₁, y₁), (x₂, y₂), …, (x_N, y_N),

○ choose θ such that y = f(θ, x) is a reasonable output

■ for training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

■ for unseen data

Learning from data

f θ

x ^{∈ Rm} y ⁼ f(^θ, x)

● Parameter Learning:

○ A collection of input-output paris (x₁, y₁), (x₂, y₂), …, (x_N, y_N),

○ choose θ such that y = f(θ, x) is a reasonable output

■ for training data (x₁, y₁), (x₂, y₂), …, (x_N, y_N)

■ for unseen data

○ Generalization: How well the model works on unseen data