Estimasi Prob. Density Function dengan EM

(1)

Estimasi

Prob. Density Function

dengan

EM

Sumber

:

--

Forsyth & Ponce Chap. 7

--

Standford

Vision & Modeling

Probability Density Estimation

• Parametric Representations

• Non-Parametric Representations

• Mixture Models

(2)

Metode

estimasi

Non

-

parametric

• Tanpa asumsi apapun tentang distribusi

• Estimasi sepenuhnya bergantung ada DATA

• cara mudah menggunakan: Histogram

Histograms

(3)

Histograms

• Butuh komputasi banyak, namun sangat umum

digunakan

• Dapat diterapkan pada sembarang bentuk

densitas (arbitrary density)

Histograms

Permasalahan:

• Higher dimensional Spaces:

- jumlah batang (bins) yg. Exponential

- jumlah training data yg exponential

- Curse of Dimensionality

• size batang ? Terlalu sedikit: >> kasar

Terlalu banyak: >> terlalu halus

(4)

Pendekatan

secara

prinsip

:

• x diambil dari ‘unknown’ p(x)

• probabiliti bahwa x ada dalam region R adalah:

V

x

p

dx

x

p

P

R

)

(

'

)

'

(

≈

=

∫

Pendekatan

secara

prinsip

:

V

x

p

dx

x

p

P

R

)

(

'

)

'

(

≈

=

∫

N

K

P

=

• x diambil dari ‘unknown’ p(x)

(5)

Pendekatan

secara

prinsip

:

V

x

p

dx

x

p

P

R

)

(

'

)

'

(

≈

=

∫

N

K

P

=

NV

K

x

p

≈

⇒

(

)

• x diambil dari ‘unknown’ p(x)

• probabiliti bahwa x ada dalam region R adalah:

Pendekatan

secara

prinsip

:

NV

K

x

p

≈

⇒

(

)

Dengan Fix V Tentukan K Dengan Fix K Tentukan V Metoda Kernel-Based K-nearest

(6)

Metoda

Kernel

-

Based:

NV

K

x

p

≈

⇒

(

)

Parzen Window:

   < = otherwise 0 2 / 1 | u | 1 ) (_u j H

Metoda

Kernel

-

Based:

NV

K

x

p

≈

⇒

(

)

Parzen Window:

   < = otherwise 0 2 / 1 | u | 1 ) (u j H

∑

N

(7)

Metoda

Kernel

-

Based:

NV

K

x

p

≈

⇒

(

)

Parzen Window:

   < = otherwise 0 2 / 1 | u | 1 ) (_u j H

∑

= − = N n n x x H K 1 ) (

∑

= − = N n n d H x x Nh x p 1 ) ( 1 ) (

Metoda

Kernel

-

Based:

NV

K

x

p

≈

⇒

(

)

Gaussian Window:













₋

−

=

∑

= 2 2 1 2 /2

2 ||

||

exp

)

2 (

1

1 )

(

h

x

h

N

x

p

N n n

π

d

(8)

Metoda

Kernel

-

Based:

K

-

nearest

-

neighbor:

NV

K

x

p

≈

⇒

(

)

(9)

K

-

nearest

-

neighbor:

K

-

nearest

-

neighbor:

Klasifikasi secara Bayesian :

V

N

K

C

x

p

k k k

)

=

|

(

NV

K

x

p

(

)

=

N

C

p

₍

₎

₌

k

(10)

K

-

nearest

-

neighbor:

Klasifikasi secara Bayesian :

V

N

K

C

x

p

k k k

)

=

|

(

NV

K

x

p

(

)

=

N

C

p

k k

)

=

(

K

x

C

p

k k

|

)

=

(

“aturan klasifikasi k-nearest-neighbour ”

Probability Density Estimation

• Parametric Representations

• Non-Parametric Representations

• Mixture Models (Model Gabungan)

(11)

Mixture

-Models (Model

-

Models (Model Gabungan

Gabungan):

):

Gaussians:

- Mudah

- Low Memory

- Cepat

- Good Properties

Non-Parametric:

- Umum

- Memory Intensive

- Slow

Mixture Models

Campuran

fungsi

Gaussian (mixture of

Gaussians):

x p(x)

(12)

Campuran

fungsi

Gaussian:

x p(x)

Jumlah dari Gaussians tunggal

Keunggulan: Dapat mendekati bentuk densitas

sembarang (Arbitrary Shape)

Campuran

fungsi

Gaussian:

x p(x)

Generative Model:

z

(13)

Campuran

fungsi

Gaussian:

x p(x)

∑

=

M j

j

P

j

x

p

x

p

1

)

(

)

|

(

)

(

















₋

−

=

₂ _/₂ ₂ 2

2 ||

||

exp

)

2 (

1 )

|

(

j d j

x

j

x

p

σ

µ

πσ

Campuran

fungsi

Gaussian:

Maximum Likelihood:

∑

=

−

=

−

=

N n n

x

p

L

E

1

)

(

ln

(14)

Campuran

fungsi

Gaussian:

Maximum Likelihood:

∑

=

−

=

−

=

N n n

x

p

L

E

1

)

(

ln

0 =

∂

k

E

µ

E k µ

Campuran

fungsi

Gaussian:

Maximum Likelihood:

∑

=

−

=

−

=

N n n

x

p

L

E

1

)

(

ln

0 =

∂

E

_⇒ ₌

∑

= n N n n x x j P 1 ) | ( µ

(15)

Campuran

fungsi

Gaussian:

∑

= = = _N n n n N n n j x j P x x j P 1 1 ) | ( ) | ( µ

Campuran

fungsi

Gaussian:

∑

=

_M k n n n

k

P

k

x

p

j

P

j

x

p

x

j

P

1

)

(

)

|

(

)

(

)

|

(

)

|

(

(16)

Campuran

fungsi

Gaussian:

∑

= = = _N n n n N n n j x j P x x j P 1 1 ) | ( ) | ( µ         − − = ₂ 2 2 / 2 ₂ || || exp ) 2 ( 1 ) | ( j j n d j n x j x p σ µ πσ

∑

=

_M k n n n

k

P

k

x

p

j

P

j

x

p

x

j

P

1

)

(

)

|

(

)

(

)

|

(

)

|

(

Campuran

fungsi

Gaussian:

∑

= = = _N n n n N n n j x j P x x j P 1 1 ) | ( ) | ( µ     − − = 2 || || 1 xn µj

∑

=

_M k n n n

k

P

k

x

p

j

P

j

x

p

x

j

P

1

)

(

)

|

(

)

(

)

|

(

)

|

(

(17)

Campuran

fungsi

Gaussian:

Maximum Likelihood:

∑

=

−

=

−

=

N n n

x

p

L

E

1

)

(

ln

0 =

∂

k

E

µ

E k µ

Tidak ada

solusi pendek !

Campuran

fungsi

Gaussian:

Maximum Likelihood:

∑

=

−

=

−

=

N n n

x

p

L

E

1

)

(

ln

E

Gradient Descent

(18)

Campuran

fungsi

Gaussian:

Maximum Likelihood:

∑

=

−

=

−

=

N n n

x

p

L

E

1

)

(

ln

)

,...,

,

,...,

,

,...,

(

₁ _M ₁ _M ₁ _M k

f

E

_µ

_σ

_α

µ

=

∂

Campuran

fungsi

Gaussian:

Optimasi secara Gradient Descent:

• Complex Gradient Function

(highly nonlinear coupled equations)

• Optimasi sebuah Gaussian tergantung dari seluruh

campuran lainnya.

(19)

Campuran

fungsi

Gaussian:

x p(x)

-> Dengan strategi berbeda:

Observed Data:

Campuran

fungsi

Gaussian:

x p(x)

Observed Data:

(20)

Campuran

fungsi

Gaussian:

x p(x) y Variabel Hidden 1 2 Observed Data:

Campuran

fungsi

Gaussian:

x p(x) y Variabel Hidden 1 2 Observed Data:

(21)

Contoh

populer

ttg. Chicken and Egg

ttg

. Chicken and Egg

Problem:

x p(x) 1 1 1111 1 2 2 2222 2 y

Anggap

kita tahu

Max.Likelihood

Utk. Gaussian #1

Max.Likelihood

Utk. Gaussian #2

Chicken+Egg Problem:

x p(x) Anggap kita tahu

P(y=1|x)

P(y=2|x)

(22)

Chicken+Egg Problem:

x p(x)

1 1 1111 1 2 2 2222 2 y

Tapi yg ini kita tidak tau sama sekali _? ?

Chicken+Egg Problem:

x p(x)

(23)

Clustering:

x

1 1 1111 1 2 2 2222 2 y

Tebakan benar ?

K-mean clustering / Basic Isodata

Pengelompokan

(Clustering):

Procedure: Basic Isodata

1. Choose some initial values for the means

Loop: 2. Classify the n samples by assigning them to the class of the closest mean.

3. Recompute the means as the average of the samples in their class.

4. If any mean changed value, go to Loop;

M

µ µ ,...,1

(24)

Isodata

:

Inisialisasi

1

µ

2

µ

Isodata

:

Menyatu

(Convergence)

1

µ

2

(25)

Isodata

:

Beberapa

permasalahan

Ditebak

Eggs / Terhitung

Eggs /

Terhitung

Chicken

x p(x) Max.Likelihood Utk. Gaussian #1 Max.Likelihood Utk. Gaussian #2

(26)

GaussianAproximasi

yg

.

baik

x p(x)

• Namun tidak optimal!

• Permasalahan: Highly overlapping Gaussians

Expectation Maximization (EM)

• EM adalah formula umum dari problem seperti “Chicken+Egg” (Mix.Gaussians, Mix.Experts, Neural Nets,

HMMs, Bayes-Nets,…)

• Isodata: adalah contoh spesifik dari EM

• General EM for mix.Gaussian: disebut Soft-Clustering • Dapat konvergen menjadi Maximum Likelihood

(27)

Ingat

rumusan

ini

?:

∑

= = = _N n n n N n n j x j P x x j P 1 1 ) | ( ) | ( µ         − − = ₂ 2 2 / 2 ₂ || || exp ) 2 ( 1 ) | ( j j n d j n x j x p σ µ πσ

∑

=

_M k n n n

k

P

k

x

p

j

P

j

x

p

x

j

P

1

)

(

)

|

(

)

(

)

|

(

)

|

(

Soft Chicken and Egg Problem:

x p(x) P(1|x) 0.1 0.3 0.7 0.1 0.01 0.0001 0.99 0.99 0.99 0.5 0.001 0.00001

∑

(28)

Soft Chicken and Egg Problem:

x p(x) P(1|x)

∑

= = = _N n n n N n n j x j P x x j P 1 1 ) | ( ) | ( µ 0.1 0.3 0.7 0.1 0.01 0.0001 0.99 0.99 0.99 0.5 0.001 0.00001 Anggap kita tahu:

Weighted Mean of Data

Soft Chicken and Egg Problem:

x p(x) P(1|x)

∑

= = = _N n n n N n n j x j P x x j P 1 1 ) | ( ) | ( µ 0.1 0.3 0.7 0.1 0.01 0.0001 0.99 0.99 0.99 0.5 0.001 0.00001 Step-2: Hitung ulang posteriors

(29)

Langkah

prosedur

EM:

Procedure: EM

1. Choose some initial values for the means

E-Step: 2. Compute the posteriors for each class and each sample:

M-Step:3. Re-compute the means as the weighted average of their class:

4. If any mean changed value, go to Loop; otherwise, stop. M µ µ ,...,1 ) | (j xn P ∑ ∑ = = = N n n n N n n j x j P x x j P 1 1 ) | ( ) | ( µ

EM

dan

Gaussian mixture

)

,

(

max

arg

( 1) ) (i

₌

_Q

_θ

i−

θ

∑

− = −

=

_N i n N n n i n i j

x

j

p

x

j

p

) 1 ( 1 ) 1 ( ) (

)

,

|

(

)

,

|

(

θ

µ

(30)

EM

dan

Gaussian mixture

)

,

(

max

arg

( 1) ) (i

₌

_Q

_θ

i−

θ

∑

= − = −

₋

=

∑

_N n i n N n T i j n i j n i n i j

x

j

p

x

j

p

1 ) 1 ( 1 ) ( ) ( ) 1 ( ) (

)

,

|

(

)

)(

,

|

(

θ

µ

θ

EM

dan

Gaussian mixture

)

,

(

max

arg

( 1) ) (i

₌

_Q

_θ

i−

θ

∑

= −

=

N n i n i j

p

j

x

N

1 ) 1 ( ) (

1 ₍

_|

_,

_θ

₎

α

(31)

Contoh

-

contoh

EM:

Training Samples

Contoh

(32)

Contoh

-

contoh

EM:

Training Samples End Result of EM

Contoh

(33)

Contoh

-

contoh

EM:

Color Segmentation

Contoh

-

contoh

EM: