Support Vector Machines

Jonghun Park jonghun@snu.ac.kr Dept. of Industrial Eng.

Seoul National University

Support Vector Machines

464.561A Models and Technologies for Information Services

convex optimization problem

minx φ(x) s.t.

h_i(x) ≥ c_i, i = 1, . . . , l

binary classification problem

3

optimal decision surface

maximum margin

given a linearly separable training set and the optimal decision surface

D = {(x₁, y₁), . . . , (x_l, y_l)} ⊆ Rⁿ × {+1, −1} w^∗ · x = b^∗

the maximum margin is given by m^∗ = max φ(w, b) = φ(w^∗, b^∗)

5

computation of maximum margin

w^∗ · x_p = b^∗ + k

w^∗ · x_q = b^∗ − k (x_q, −1) ∈ D

(x_p, +1) ∈ D

computation of maximum margin

m^∗ = |x_p − x_q| cos γ

= w^∗ · (x_p − x_q)

|w^∗|

= w^∗ · x_p − w^∗x_q

|w^∗|

= (b^∗ + k) − (b^∗ − k)

|w^∗|

= 2k

|w^∗|

7

objective function

m^∗ = max 2k

|w|

= min |w| 2k

⇒ φ(w, b) = 1

2w · w

constraints

w^∗ · x_i ≥ b^∗ + k, ∀(x_i, y_i) ∈ D s.t. y_i = +1 w^∗ · x_i ≤ b^∗ − k, ∀(x_i, y_i) ∈ D s.t. y_i = −1

⇒

w · x_i ≥ 1 + b, ∀(x_i, y_i) ∈ D s.t. y_i = +1

w · (−x_i) ≥ 1 − b, ∀(x_i, y_i) ∈ D s.t. y_i = −1

⇒

w · (y_ix_i) ≥ 1 + y_ib, ∀(x_i, y_i) ∈ D

9

maximum margin classifier

D = {(x₁, y₁), . . . , (x_l, y_l)} ⊆ Rⁿ × {+1, −1} min φ(w, b) = min

w,b

1

2w · w s.t.

w · (y_ix_i) ≥ 1 + y_ib, ∀(x_i, y_i) ∈ D

quadratic programming

(w^∗, b^∗) = arg_w,b min

�1

2w^T Qw − q · w

�

s.t. X^T w ≥ c

where Q = I, q = 0

X =





y₁x¹₁ . . . y_ix¹_i . . . y_lx¹_l

... ... ...

y₁xⁿ₁ . . . y_ixⁿ_i . . . y_lxⁿ_l





c =







1 + y₁b 1 + y₂b

...

1 + y_lb





 x_i = (x¹_i , . . . , xⁿ_i )

11

Lagrangian optimization problem

minx φ(x) s.t.

g_i(x) ≥ 0, i = 1, . . . , l x ∈ Rⁿ

maxα min

x L(α, x) = max

α min

x

�

φ(x) −

�l

i=1

α_ig_i(x)

� s.t.

α ≥ 0, i = 1, . . . , l

nested optimization

13

maxα L(α, x^∗) = max

α

�

φ(x^∗) −

�l

i=1

α_ig_i(x^∗)

�

minx L(α^∗, x) = min

x

�

φ(x) −

�l

i=1

α^∗_i g_i(x)

�

⇒ ∂L

∂x (α, x^∗) = 0

KKT (Karush-Kuhn-Tucker) condition

α^∗ and x^∗ s.t.

maxα min

x L(α, x) = L(α^∗, x^∗) = φ(x^∗) −

�l

i=1

α^∗_i g_i(x^∗)

x^∗is a solution to the primal objective function iff

∂L

∂x (α^∗, x^∗) = 0 α^∗_i g_i(x^∗) = 0

g_i(x^∗) ≥ 0 α^∗ ≥ 0

complementary condition

15

α^∗_i g_i(x^∗) = 0

⇒ L(α^∗, x^∗) = φ(x^∗)

Lagrangian dual

∂L

∂x (α, x^∗) = 0 ⇒ L(α, x^∗) = φ^�(α) maxα φ^�(α)

α_i ≥ 0, i = 1, . . . , l s.t.

⇒

maxα φ^�(α) = φ^�(α^∗) = L(α^∗, x^∗) = φ(x^∗) s.t. x^∗ and α^∗ satisfy KKT conditions.

example

17

min φ(x) = min 1 2x² s.t.

g(x) = x − 2 ≥ 0

example

L(α, x) = 1

2x² − α(x − 2)

example

19

∂L

∂x (α, x^∗) = x^∗ − α = 0

⇒ x^∗ = α

⇒ L(α, x^∗) = 2α − 1 2α²

⇒ max

α φ^�(α) = max

α

�

2α − 1 2α²

�s.t. α ≥ 0

⇒ dφ^�

dα (α^∗) = 2 − α^∗ = 0

⇒ x^∗ = α^∗ = 2, α^∗g(x^∗) = α^∗(x^∗ − 2) = 0

dual maximum margin optimization

minw,b φ(w, b) = min

w,b

1

2w · w

s.t. g_i(w, b) = y_i(w · x_i − b) − 1 ≥ 0, i = 1, . . . , l

maxα min

w,b L(α, w, b) = φ(w, b) −

�l

i=1

α_ig_i(w, b)

= 1

2w · w −

�l

i=1

α_iy_iw · x_i + b

�l

i=1

α_iy_i +

�l

i=1

α_i s.t. 0, i = 1, . . . , l

KKT conditions

21

α^∗, w^∗, and b^∗ s.t. max

α min

w,b L(α, w, b) = L(α^∗, w^∗, b^∗) will satisfy

∂L

∂w(α^∗, w^∗, b^∗) = 0

∂L

∂b (α^∗, w^∗, b^∗) = 0

α^∗_i (y_i(w^∗ · x_i − b^∗) − 1) = 0 y_i(w^∗ · x_i − b^∗) − 1 ≥ 0

α^∗_i ≥ 0

i = 1, . . . , l

derivation of Lagrangian dual

∂L

∂w(α, w^∗, b) = w^∗ −

�l

i=1

α_iy_ix_i = 0

⇒ w^∗ =

�l

i=1

α_iy_ix_i

∂L

∂b (α, w, b^∗) =

�l

i=1

α_iy_i = 0

⇒ becomes a constraint

maximum margin Lagrangian dual

23

maxα φ^�(α) = max

α L(α, w^∗, b^∗)

= max

α





�l i=1

α_i − 1 2

�l i=1

�l j=1

α_iα_jy_iy_jx_i · x_j





s.t.

�l

i=1

α_iy_i = 0 α_i ≥ 0

i = 1, . . . , l

KKT complementary condition

α_j^∗ > 0, for (x_j, y_j) ∈ D

⇒ w^∗ · x_j = b^∗ + 1, if y_j = +1 w^∗ · x_j = b^∗ − 1, if y_j = −1

α_j^∗ = 0, for (x_j, y_j) ∈ D

⇒ w^∗ · x_j > b^∗ + 1, if y_j = +1 w^∗ · x_j < b^∗ − 1, if y_j = −1

implications

  The primal maximum-margin optimization computes the supporting hyperplanes whose margin is limited by the support vectors

  The dual maximum-margin optimization computes the support vectors that limit the size of the margin of the supporting hyperplanes

25

computation of b

^∗

pick a support vector with nonzero Lagrangian multipliers (x_sv⁺, y_sv⁺) s.t. y_sv⁺ = +1

b^∗ = w^∗ · x_sv⁺ − 1 =

�l

i=1

α^∗_i y_ix_i · x_sv⁺–1

∵

w^∗ · x_j = b^∗ + 1, if y_j = +1 w^∗ =

�l

i=1

α_iy_ix_i

dual classification function

27

optimal decision surface: _w^∗ _{· x = b}^∗

⇒ fˆ(x) = sgn(w^∗ · x − b^∗)

⇒ fˆ(x) = sgn

� _l

�

i=1

α^∗_i y_ix_i · x −

�l

i=1

α^∗_i y_ix_i · x_sv⁺ + 1

�

b^∗ =

�l

i=1

α^∗_i y_ix_i · x_sv⁺–1 w^∗ =

�l

i=1

α_iy_ix_i

∵

linear SVM: problem

target function f : Rⁿ → {+1, −1}

D = {(x₁, y₁), . . . , (x_l, y_l)} ⊆ Rⁿ × {+1, −1} labeled, linearly separable training set:

where

x_i ∈ Rⁿ, y_i = f(x_i), i = 1, . . . , l

compute a classifier _f^ˆ _{: R}ⁿ _{→ {+1, −1}} using D s.t.

fˆ(x) � f(x), ∀x ∈ Rⁿ

linear SVM: training

29

α^∗ = arg_α max





�l i=1

α_i − 1 2

�l i=1

�l j=1

α_iα_jy_iy_jx_i · x_j



 s.t.

�l

i=1

α_iy_i = 0

α_i ≥ 0, i = 1, . . . , l

linear SVM: classification

fˆ(x) = sgn

� _l

�

i=1

α_i^∗y_ix_i · x −

�l

i=1

α_i^∗y_ix_i · x_sv⁺ + 1

�

where

(x_sv⁺, +1) ∈ {(x_i, +1)|(x_i, +1) ∈ D and α^∗_i > 0}

SVM for non-linearly separable data

31

nonlinear SVM example

w · x = b

x · x = 1

transformation into feature space

33

Φ(x) = Φ(x₁, x₂) = (x²₁, x²₂, √

2x₁x₂) = (z₁, z₂, z₃) = z where x ∈ R², z ∈ R³

w^∗ · Φ(x) = b^∗

w^∗ � (1, 1, 0), b^∗ � 1

⇒ z₁ + z₂ = 1

classification in feature space

fˆ(x) = sgn(w^∗ · Φ(x) − b^∗)

= sgn(w^∗ · z − b^∗)

= sgn

� _d

�

i=1

w_i^∗z_i − b^∗

�

dual representation

35

w^∗ =

�l

i=1

α_i^∗y_iΦ(x_i)

fˆ(x) = sgn(w^∗ · Φ(x) − b^∗)

= sgn(

�l

i=1

α^∗_i y_iΦ(x_i) · Φ(x) − b^∗)

→ sgn(

�l

i=1

α^∗_i y_i(x_i · x)² − b^∗)

example kernel function

Φ(x) = (x²₁, x²₂, √

2x₁x₂) Φ(x) · Φ(y) = (x²₁, x²₂, √

2x₁x₂) · (y₁², y₂², √

2y₁y₂)

= x²₁y₁² + x²₂y₂² + 2x₁x₂y₁y₂

= (x₁y₁ + x₂y₂)(x₁y₁ + x₂y₂)

= (x · y)(x · y)

= (x · y)²

kernel function

37

k(x, y) = Φ(x) · Φ(y)

where Φ : Rⁿ → R^m s.t. m ≥ n fˆ(x) = sgn

� _l

�

i=1

α_i^∗y_ik(x_i, x) − b^∗

�

standard kernels

kernel name kernel function free

parameters

linear none

homogeneous polynomial

non-

homogeneous polynomial

Gaussian

k(x, y) = x · y

k(x, y) = (x · y)^d d ≥ 2

k(x, y) = (x · y + c)^d d ≥ 2, c > 0

k(x, y) = ^{−(|x−y|2/2σ2)} σ > 0

39

computation of b

^∗

b^∗ = w^∗ · Φ(x_sv⁺) − 1

=

�l

i=1

α^∗_i y_iΦ(x_i) · Φ(x_sv⁺)–1

=

�l

i=1

α_i^∗y_ik(x_i, x_sv⁺)–1

nonlinear SVM: training

s.t.

�l

i=1

α_iy_i = 0

α_i ≥ 0, i = 1, . . . , l α^∗ = arg_α max





�l i=1

α_i − 1 2

�l i=1

�l j=1

α_iα_jy_iy_jk(x_i, x_j)





maximum margin classifier

41

maximum-margin classifier: _f^ˆ_{(x) = sgn(w}^∗ _{· x − b}^∗₎ primal optimization problem: _min φ(w, b) = min 1

2w · w s.t. y_i(w · x_i − b) − 1 ≥ 0, i = 1, . . . , l

(x₁, y₁), . . . , (x_l, y_l) ∈ Rⁿ × {+1, −1}

introducing slack variables

y_i(w · x_i − b) + ξ_i − 1 ≥ 0, i = 1, . . . , l

soft-margin optimization problem

43

w,ξ,bmin φ(w, ξ, b) = min

w,ξ,b

�1

2w · w + C

�l

i=1

ξ_i

�

s.t. y_i(w · x_i − b) + ξ_i − 1 ≥ 0, i = 1, . . . , l ξ_i ≥ 0, i = 1, . . . , l

ξ = (ξ₁, . . . , ξ_l), C > 0

(x₁, y₁), . . . , (x_l, y_l) ∈ Rⁿ × {+1, −1}

note: _f^ˆ_{(x) = sgn(w}^∗ _{· x − b}^∗₎

soft-margin misclassifications

for (x_j, +1), w · x_j = b + (1 − ξ_j)

ξ_j ≤ 1 ξ_j > 1

dual setting for soft-margin classifiers

45

L(α, β, w, ξ, b) = 1

2w · w + C

�l

i=1

ξ_i

−

�l

i=1

α_i(y_i(w · x_i − b) + ξ_i − 1)

−

�l

i=1

β_iξ_i

Lagrangian optimization problem

maxα,β min

w,ξ,b L(α, β, w, ξ, b) s.t. α_i ≥ 0

β_i ≥ 0

i = 1, . . . , l

KKT conditions

47

∂L

∂w(α, β, w^∗, ξ, b) = 0

∂L

∂ξ_i (α, β, w, ξ^∗, b) = 0

∂L

∂b (α, β, w, ξ, b^∗) = 0

α^∗_i (y_i(w^∗ · x_i − b^∗) + ξ_i^∗ − 1) = 0 β_i^∗ξ_i^∗ = 0

y_i(w^∗ · x_i − b^∗) + ξ_i^∗ − 1 ≥ 0 α^∗_i ≥ 0, β_i^∗ ≥ 0, ξ_i^∗ ≥ 0

α^∗, β^∗, w^∗, ξ^∗, and b^∗ will satisfy for i = 1, . . . , l

optimal value

maxα,β min

w,ξ,b L(α, β, w, ξ, b) = L(α^∗, β^∗, w^∗, ξ^∗, b^∗)

= 1

2w^∗ · w^∗ + C

�l

i=1

ξ_i^∗

Langrangian dual

49

∂L

∂w(α, β, w^∗, ξ, b) = w^∗ −

�l

i=1

α_iy_ix_i = 0

⇒ w^∗ =

�l

i=1

α_iy_ix_i

∂L

∂b (α, β, w, ξ, b^∗) =

�l

i=1

α_iy_i = 0 ⇒

�l

i=1

α_iy_i = 0

∂L

∂ξ_i (α, β, w, ξ^∗, b) = C − α_i − β_i = 0

⇒ α_i = C − β_i ⇒ 0 ≤ α_i ≤ C

soft margin Lagrangian dual

i = 1, . . . , l maxα φ^�(α) = max

α





�l i=1

α_i − 1 2

�l i=1

�l j=1

α_iα_jy_iy_jx_i · x_j



 s.t.

�l

i=1

α_iy_i = 0 0 ≤ α_i ≤ C

computation of

51

b

^∗

pick a support vector with a zero-valued slack variable

∵ w^∗ =

�l

i=1

α_iy_ix_i

(x_sv⁺, +1) s.t. ξ_sv^{∗ +} = 0 ⇒ 0 < α^∗_sv⁺ < C

b^∗ = w^∗ · x_sv⁺ − 1 + ξ_sv⁺

=

�l

i=1

α^∗_i y_ix_i · x_sv⁺–1

readings

  Searching Social Media Streams on the Web

  Finding Advertising Keywords on Web Pages

  Optimizing Search Engines using Clickthrough Data

  Hierarchical Document Categorization with Support Vector Machines

  Predicting Structured Objects with Support Vector Machines

  Hidden Markov Support Vector Machines

references

  N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines and Other Kernel-Based

Learning Methods, Cambridge University Press, 2000.

  L. Hamel, Knowledge Discovery with Support Vector Machines, Wiley, 2009.

53