Machine Learning with Graphs

(1)

Revisit: Machine Learning with Graphs

 Node classification (semi-supervised Learning)

 Predict a type of a given node

 Link prediction

 Predict whether two nodes are linked

 Community detection (node clustering, unsupervised learning)

 Identify densely linked clusters of nodes

 Network similarity

 How similar are two (sub)networks

 Ranking

 Page ranking in Web

1 4

2 3

6 5 7

node edge

(2)

Example: Node Classification

? ?

?

Machine Learning

(3)

Example: Node Classification

(4)

Example: Link Prediction

Machine Learning

?

x

(5)

Example: Link Prediction

?

Content recommendation is link prediction!

(6)

Feature Learning in Graphs

Goal:

Efficient feature learning for machine learning in networks!

vector node

𝑓: 𝑢 → ℝ^𝑑

ℝ^𝑑

feature representation, node embedding

u

(7)

Example: Feature Learning in Graphs

representation in feature space (node embedding) Graph input

(8)

Graph Laplacian (Spectral Graph Theory)

𝑳 = 𝑫 − 𝑾

(9)

Graphs and Graph Signals

𝑣₃

𝑣₂

𝑣₄

𝑣₅ 𝑣₆

𝑣₇ 𝑣₈ 𝑣₁

𝒱 = 𝑣_𝑖 𝑖 = 1, … , 𝑁}

ℰ = ℯ_𝑖𝑗 𝑖, 𝑗 = 1, … , 𝑁}

𝒢 𝒱, ℰ = {𝒱, ℰ}

AAAI 2020 Tutorial (Yao Ma, et al.): Spectral graph theory. American Mathematical Soc.; 1997.

(10)

Graphs and Graph Signals (Functions)

𝒱 = 𝑣_𝑖 𝑖 = 1, … , 𝑁}

ℰ = ℯ_𝑖𝑗 𝑖, 𝑗 = 1, … , 𝑁}

𝒢 𝒱, ℰ = {𝒱, ℰ}

Graph Signal:

= 𝑓 =

2 3 … 2

1 . . . 2

5 . . . 6

… 4 . . .

… 3

𝑁 × 𝑑 dim. Matrix 𝐺𝑁𝑁 input Matrix

𝑓₁ 𝑓₂ 𝑓₃ 𝑓₄ 𝑓₅ 𝑓₆ 𝑓

(11)

Graphs and Graph Signals

𝒱 = 𝑣_𝑖 𝑖 = 1, … , 𝑁}

ℰ = ℯ_𝑖𝑗 𝑖, 𝑗 = 1, … , 𝑁}

𝒢 𝒱, ℰ = {𝒱, ℰ}

Graph Signal:

𝑁 × 1 vector for simplicity

= 𝑓 = 2 5 1. ..

6 4

𝑓₁ 𝑓₂ 𝑓₃ 𝑓₄ 𝑓₅ 𝑓₆ 𝑓₇ 𝑓

(12)

Matrix Representations of Graphs

𝑣₃

𝑣₂

𝑣₄

𝑣₅ 𝑣₆

𝑣₇ 𝑣₈ 𝑣₁

𝑨 =

Adjacency Matrix

Adjacency Matrix: 𝐴 𝑖, 𝑗 = 1 if 𝑣_𝑖 is adjacent to 𝑣_𝑗 𝐴 𝑖, 𝑗 = 0, otherwise

or Affinity Matrix

(13)

Matrix Representations of Graphs

𝑣₃

𝑣₂

𝑣₄

𝑣₅ 𝑣₆

𝑣₇ 𝑣₈ 𝑣₁

− =

Adjacency Matrix

Degree Matrix Laplacian Matrix

Degree Matrix:

Adjacency Matrix: 𝐴 𝑖, 𝑗 = 1 if 𝑣_𝑖 is adjacent to 𝑣_𝑗 𝐴 𝑖, 𝑗 = 0, otherwise

(14)

Laplacian Matrix as an Operator

Laplacian matrix is a difference operator:

𝒉 = 𝑳𝒇 = 𝑫 − 𝑨 𝒇 = 𝑫𝒇 − 𝑨𝒇

(15)

Laplacian Matrix as an Operator

𝒉 = 𝑳𝒇 = 𝑫 − 𝑨 𝒇 = 𝑫𝒇 − 𝑨𝒇

ℎ_𝒊 = 𝑳_𝒊 𝒇 = 𝑫_𝒊 − 𝑨_𝒊 𝒇 = 𝑫_𝒊𝒇 − 𝑨_𝒊𝒇

𝑨 𝑫

𝑓 = 𝑓₁ 𝑓₂ ..

𝑓

(16)

Laplacian Matrix as an Operator

𝒉 = 𝑳𝒇 = 𝑫 − 𝑨 𝒇 = 𝑫𝒇 − 𝑨𝒇

𝑨 𝑫

= 𝑫_𝒊,𝒊𝑓_𝒊 − 𝑨_𝒊𝒇

𝑓 = 𝑓₁ 𝑓₂ ..

𝑓

(17)

Laplacian Matrix as an Operator

𝒉 = 𝑳𝒇 = 𝑫 − 𝑨 𝒇 = 𝑫𝒇 − 𝑨𝒇

𝑨 𝑫

ℎ₆= 𝟒𝑓₆ − 𝑓₂ − 𝑓₃ − 𝑓₅ − 𝑓_𝟕

= (𝑓₆−𝑓₂) + (𝑓₆−𝑓₃) + (𝑓₆−𝑓₅) + (𝑓₆−𝑓_𝟕)

𝑓 = 𝑓₁ 𝑓₂ ..

𝑓

(18)

Laplacian Matrix as an Operator

𝒉 = 𝑳𝒇 = 𝑫 − 𝑨 𝒇 = 𝑫𝒇 − 𝑨𝒇

𝑨 𝑫

ℎ_𝒊 = ෍

𝒗_𝒋∈𝓝(𝒗_𝒊)

(𝑓_𝒊 − 𝑓_𝑗) meaning?

𝑓 = 𝑓₁ 𝑓₂ ..

ℎ = 𝟒𝑓 − 𝑓 − 𝑓 − 𝑓 − 𝑓 𝑓

(19)

Laplacian Matrix as an Operator

𝒉 = 𝑳𝒇 = 𝑫 − 𝑨 𝒇 = 𝑫𝒇 − 𝑨𝒇

𝑨 𝑫

meaning?

𝑓 = 𝑓₁ 𝑓₂ ..

𝑓

ℎ_𝒊 = ෍

(𝑓_𝒊 − 𝑓_𝑗)

= (𝑓

(20)

Laplacian Matrix as an Operator

Laplacian quadratic form:

𝑨 𝑫

𝒉 = 𝑳𝒇, ℎ_𝑖 = ෍

(𝑓_𝑖 − 𝑓_𝑗) 𝒇^𝑻𝑳𝒇 = ෍

𝒊

𝑓_𝑖ℎ_𝑖 = ෍

𝒊

෍

𝑓_𝑖(𝑓_𝑖 − 𝑓_𝑗)

𝑓 = 𝑓₁ 𝑓₂ ..

𝑓

(21)

Laplacian Matrix as an Operator

Laplacian quadratic form: ^𝑨

𝒉 = 𝑳𝒇, ℎ_𝑖 = ෍

(𝑓_𝑖 − 𝑓_𝑗) 𝒇^𝑻𝑳𝒇 = ෍

𝒊

෍

= 𝟏

𝟐 ෍

𝒊,𝒋

𝑨(𝒊, 𝒋)𝑓_𝑖(𝑓_𝑖 − 𝑓_𝑗) + 𝟏

𝟐 ෍

𝒊,𝒋

𝑨(𝒋, 𝒊)𝑓_𝑖(𝑓_𝑖 − 𝑓_𝑗)

(22)

Laplacian Matrix as an Operator

Laplacian quadratic form: ^𝑨

𝒉 = 𝑳𝒇, ℎ_𝑖 = ෍

(𝑓_𝑖 − 𝑓_𝑗) 𝒇^𝑻𝑳𝒇 = ෍

𝒊

෍

= 𝟏

𝟐 ෍

𝒊,𝒋

𝑨(𝒊, 𝒋)𝑓_𝑖(𝑓_𝑖 − 𝑓_𝑗) + 𝟏

𝟐 ෍

𝒊,𝒋

= 𝟏

𝟐 ෍

𝒊,𝒋

𝑨 𝒊, 𝒋 𝑓_𝑖 𝑓_𝑖 − 𝑓_𝑗 − 𝟏

𝟐 ෍

𝒊,𝒋

𝑨(𝒊, 𝒋)𝑓_𝑗(𝑓_𝑖 − 𝑓_𝑗)

(23)

Laplacian Matrix as an Operator

Laplacian quadratic form:

𝒉 = 𝑳𝒇, ℎ_𝑖 = ෍ ^𝑨

(𝑓_𝑖 − 𝑓_𝑗) 𝒇^𝑻𝑳𝒇 = ෍

𝒊

෍

= 𝟏

𝟐 ෍

𝒊,𝒋

𝑨(𝒊, 𝒋)𝑓_𝑖(𝑓_𝑖 − 𝑓_𝑗) + 𝟏

𝟐 ෍

𝒊,𝒋

= 𝟏

𝟐 ෍

𝒊,𝒋

𝑨 𝒊, 𝒋 𝑓_𝑖 𝑓_𝑖 − 𝑓_𝑗 − 𝟏

𝟐 ෍

𝒊,𝒋

𝑨(𝒊, 𝒋)𝑓_𝑗(𝑓_𝑖 − 𝑓_𝑗)

𝟏 𝟐

(24)

Summary: Laplacian Matrix as an Operator

𝒉 = 𝑳𝒇 = 𝑫 − 𝑨 𝒇 = 𝑫𝒇 − 𝑨𝒇 𝒉(𝒊) = ෍

(𝑓_𝑖 − 𝑓_𝑗) Laplacian quadratic form:

𝒇^𝑻𝑳𝒇 = 𝟏

𝟐 ෍

𝒊,𝒋

𝑨 𝒊, 𝒋 𝑓_𝑖 − 𝑓_𝑗 ^𝟐 It can represent

“Smoothness” or “Frequency” of the signal 𝑓

Low frequency graph signal