Mathematical Methods in Engineering 2 (Machine Learning)

Mathematical Methods in Engineering 2 (Machine Learning)

Dept. of Control & Instrumentation Engineering, Korea Univ. Jooyoung Park

(Textbook: C. Bishop, Pattern Recognition and Machine Learning, Cambridge Univ. Press, 2006)

Lecture #1 2016.3.15

Graphical models (PGM)

Ch8 Graphical models (PGM)

-diagram instead of algebraic manipulation

-useful properties of PGM:

① visualization (e.q. for factorization)

② insights (e.q. for cond. indep.)

③ computation made easy (e.q. sum-product alg.)

-PGM, 𝐺 = 𝑣 , 𝜀

 vertex (node) represents a r.v.

 edge (link) represents probabilistic

Graphical models (PGM)

-Three types of PGM

① directed graphs (Bayesian networks)

② undirected graph (Markov random fields)

③ factor graphs

EX)

a a

b c b c

BN MRF FG

Bayesian networks

8.1 Baysesian networks (DG)

EX) 𝑝 𝑎, 𝑏, 𝑐 = 𝑝 𝑎 𝑝 𝑏 𝑎 𝑝 𝑐 𝑎, 𝑏 = ³_𝑘=1𝑝(𝑥_𝑘|𝑝𝑎 𝑥_𝑘 )

Note: 𝑝 𝑥₁, ⋯ , 𝑥_𝑘 = ^𝐾_𝑘=1𝑝(𝑥_𝑘|𝑝𝑎 𝑥_𝑘 ) joint pdf a factorization

a

b c

𝑥₁

𝑥₂ 𝑥₃

Bayesian networks

EX) Lin. regress prob (Bayesian approach)

(A Bayesian approach for the lin. reg. prob.) trn data D= 𝑥_𝑛, 𝑡_{𝑛 𝑛=1}^𝑁

𝑡_𝑛 = 𝑤^𝑇∅ 𝑥_𝑛 + 𝜖_𝑛, 𝜖_𝑛~𝑁 0, 𝜎² IID In the Bayesian approach,

𝑤~𝑁 0, 𝛼𝐼

more precisely

∴joint pdf 𝑝 𝑡, 𝑤 𝑥, 𝛼, 𝜎² = 𝑝 𝑤 𝛼 𝑝 𝑡 𝑤, 𝑥, 𝜎² = 𝑝 𝑤 𝛼 𝑝 𝑡_𝑛 𝑤, 𝑥_𝑛, 𝜎² Joint dist. : 𝑝(𝑡, 𝑤) = 𝑝(𝑤) 𝑝(𝑡|𝑤) = 𝑝(𝑤) ^𝑁_𝑛=1𝑝(𝑡_𝑛|𝑤)

hyper parameter

b

𝑡₁ … 𝑡_𝑛 𝑡_𝑛 N

w

Conditional indep.

8.2 Conditional indep.

 Def. 𝑎 and 𝑏 are conditionally indep. Given 𝑐,

if 𝑝 𝑎, 𝑏 𝑐 = 𝑝 𝑎 𝑐 𝑝(𝑏|𝑐) (where a, 𝑏, 𝑐 are r.v.)

 Notation : 𝑎 ⊥ 𝑏

 Note : Assume 𝑝(𝑏|𝑐) ≠ 0 𝑎 ⊥ 𝑏|𝑐, we have 𝑝 𝑎 𝑐 = 𝑝(𝑎|𝑏, 𝑐)

∵ 𝑝 𝑎 𝑏, 𝑐 = ^{𝑝 𝑎,𝑏 𝑐}_{𝑝 𝑏 𝑐} = 𝑝(𝑎|𝑐)

Conditional indep.

8.2.2 d-separation property

 Consider a directed graph, where A, B, and C are arbitrary non intersecting sets of nodes.

 Def. Consider all possible (undirected) paths from any node in A to any node in B.

We say that any such path, P, is blocked if at least one of the following hold:

a. P contains either HT or TT node, and the node is in the set C.

(즉, 관찰된 non-consider 노드는 이 노드를 지나는 path 를 block 함) b. P contains a collider(HH node), and neither the node nor any of its

descendants belongs to the set C.

(즉, 관찰되지 않은(latent) HH노드는 이 노드를 지나는 path를 block 가능) 단, 이 collider의 후손이 관찰되면 이 bock 상황이 해제됨.

Means “directed”

Conditional indep.

If all the possible paths are blocked, then A is d-separated from B given C.

(또는 A and B are d-separated C)

 d-separation Thm. :

Let A, B and C can separated sets of node in a directed graph.

If A and B are d-separated by C, then we have 𝑎 ⊥ 𝑏|𝑐