Move-based Algorithms for the Optimization of an Isotropic Gradient MRF Model
Behrooz Nasihatkon Richard Hartley
5 December 2012
NICTA Funding and Supporting Members and Partners
Outline
•
Total Variation•
Current Methods•
The 3-clique Model•
Move-based Algorithms•
Main Theorem•
ConclusionTotal Variation
• goodregularizer.
• For a functionx: Ω⊆Rn →RTotal Variation is defined as TV(x) =
Z
Ω
|∇tx(t)|dt
• discontinuity preserving(edge preserving for images).
x1(t) x2(t) TV(x1) =TV(x2)
Total Variation
• goodregularizer.
• For a functionx: Ω⊆Rn →RTotal Variation is defined as TV(x) =
Z
Ω
|∇tx(t)|dt
• discontinuity preserving(edge preserving for images).
x1(t) x2(t) TV(x1) =TV(x2)
Total Variation
• goodregularizer.
• For a functionx: Ω⊆Rn →RTotal Variation is defined as TV(x) =
Z
Ω
|∇tx(t)|dt
• discontinuity preserving(edge preserving for images).
x1(t) x2(t) TV(x1) =TV(x2)
Modeling TV using MRFs
• Approximate Total Variation using an MRF,
• A set of nodes 1,2, . . . ,n,
• A set of labelsx= [x1,x2, . . . ,xn],xi∈ L.
• Energy function
E(x) =X
i
fi(xi) + ˜TV(x),
Current Models
• Approximate Magnitude of Gradient usingedge-basedpotentials.
TV˜ (x) = X
(i,j)∈C2
wij|xi−xj|
• Magnitude of Gradient (MoG) at each nodei is approximated by
MoG(i) =X
j∈Ni
wij|xi−xj|
b
bbb b
b b bbbb bb
bbb b b bbbb bb
bbb b b
b
bb b b bb
b
b
4Neighbourhood 8Neighbourhood 16Neighbourhood
4-connected model
MoG(i) =|xi−xj|+|xi−xk|
MoG(i) =1
b
b
b
b
b
b
b bbbbbb
b bbbbbb
b bbbbbb
b bbbb bi j
k
Diagonal Edges
MoG(i) =|xi−xj|+|xi−xk|+
√ 2
2 |xi−xl|
MoG(i) =1
b
b
b
b
b
b
b bbbbbb
b bbbbbb
b bbbbbb
b bbbb bi j
k b l
The 3-Clique Model
• The gradient vector≈
xj−xi xk−xi
,
• For ordered labelsxi ∈ {1,2, . . . ,M} MoG(i) =
q
(xi−xj)2+ (xi−xk)2
• For general labelsxi ∈ Lwith a semi-metricd
MoG(i) = q
d(xi,xj)2+d(xi,xk)2
b
b
b
b
b
b
b bbbbbb
b bbbbbb
b bbbbbb
b bbbb b
i j
k
The 3-Clique Model
• The gradient vector≈
xj−xi xk−xi
,
• For ordered labelsxi ∈ {1,2, . . . ,M} MoG(i) =
q
(xi−xj)2+ (xi−xk)2
• For general labelsxi ∈ Lwith a semi-metricd
MoG(i) = q
d(xi,xj)2+d(xi,xk)2
b
b
b
b
b
b
b bbbbbb
b bbbbbb
b bbbbbb
b bbbb b
i j
k
The 3-Clique Model
• The gradient vector≈
xj−xi xk−xi
,
• For ordered labelsxi ∈ {1,2, . . . ,M} MoG(i) =
q
(xi−xj)2+ (xi−xk)2
• For general labelsxi ∈ Lwith a semi-metricd
MoG(i) = q
d(xi,xj)2+d(xi,xk)2
b
b
b
b
b
b
b bbbbbb
b bbbbbb
b bbbbbb
b bbbb b
i j
k
The 3-clique Model
4 neighbours 8 neighbours 3-cliques
Move-Based Algorithms
min
x∈L
X
i
fi(xi) +γ X
(i,j,k)∈C3
q
d(xi,xj)2+d(xi,xk)2
• Move-based approachis a popular way of optimizing Multi-label MRFs.
• Optimizing the multi-label MRF iteratively by solvinga series of binary MRF optimizations.
b
b
b
b
b
b
b bbbbbb
b bbbbbb
b
x1x2x3x4· · ·
b
b
b
b
b
b
b bbbbbb
b bbbbbb
b
u1u2u3u4· · ·
The Alpha-Expansion Algorithm
• Nodes have a choice to switch toαor stay unchanged:
lα0(xi) =xi lα1(xi) =α
• luα(x) = [lαu1(x1),lαu2(x2), . . . ,lαun(xn)]
procedureALPHA-EXPANSION(x,L) repeat
for eachα∈ Ldo u∗ ←argminuE(luα(x)) x ←luα∗(x)
end for untilconvergence return x
end procedure
The Alpha-Beta Swap Algorithm
• Nodes with labelsαorβ have a chance to swap.
lα,β0 (xi) =
α ifxi ∈ {α, β}
xi otherwise lα,β1 (xi) =
β ifxi∈ {α, β}
xi otherwise
procedureALPHA-BETA-SWAP(x,L) repeat
for eachα, β∈ L×Ldo u∗ ←argminuE(luα,β(x)) x ←luα,β∗ (x)
end for untilconvergence
General Move Algorithm
• Take arbitraryl0andl1
l0(xi) =arbitrary l1(xi) =arbitrary
• The pair of functions(l0,l1)is called theupdate policy.
• State Preservation Property
∀x∈ L l0(x) =x or l1(x) =x
General Move Algorithm
• Take arbitraryl0andl1
l0(xi) =arbitrary l1(xi) =arbitrary
• The pair of functions(l0,l1)is called theupdate policy.
• State Preservation Property
∀x∈ L l0(x) =x or l1(x) =x
Solving the Binary Problem
• How to solve
u∗ ←argminuE(lu(x))
• Energy functions consisting ofquadraticandcubicterms are solvable by graph-cuts if and only if they aresubmodular1.
• The functionf:{0,1}×{0,1} →Rissubmodularif f(0,1) +f(1,0)≥f(0,0) +f(1,1).
• A pseudo-Boolean function ofnvariables is submodular ifany restrictionto any pair of variables is submodular.
1Kolmogorov and Zabih 2004.
Central Question
• Main Question: Given
E(x) = X
(i,j,k)∈C3
q
d(xi,xj)2+d(xi,xk)2
what choice of policy(l0,l1)results in a submodularE(lu(x))as a function ofu, so we can solve
u∗ ←argminuE(lu(x))
Main Theorem (General Case)
E(x) = X
(i,j,k)∈C3
q
d(xi,xj)2+d(xi,xk)2
Theorem
Assume d:L × L →Ris asemi-metricand the update policy has the state preservation property, the energy function Ex0(u) =E(lu(x))is submodular for allx,if and only iffor any three labels x,y,z∈ L
d(x,y1)−d(x,y0)
d(x,z1)−d(x,z0)
≥0, where xu is a compact form for lu(x).
Main Theorem (Ordered Labels)
E(x) =P
(i,j,k)∈C3
p(xi−xj)2+ (xi−xk)2
(Middlebury Dataset)
Proposition
WithL={0,1, . . . ,M−1}and d(x,y) =|x−y|(ordered labels), and havingstate preservation propertyfor the update policy, the energy function Ex0(u) =E(lu(x))is submodular for allxif and only if(l0,l1)is a mirrored update policy.
Mirrored Policy
Definition
An update policy(l0,l1)is calledmirroredif
(i) ∀x ∈ A l0(x)<l1(x)or∀x∈ A l0(x)>l1(x), (ii) ∃µ∈ Lsuch that∀x ∈ A
l0(x)+l1(x)
2 ∈ {µ, µ+1
2, µ+1}.
µ
y1 z0 z1 y0
v0 v1
w1 w0
µ+1
Main Theorem (Unordered Labels)
E(x) =P
(i,j,k)∈C3
p1(xi 6=xj) +1(xi6=xk)
(Buffalo-Xiph.org)
Proposition
With d(x,y) =1(x6=y)(unordered labels), and assuming thestate preservation propertyfor the update policy, the energy function Ex0(u) =E(lu(x))is submodular for allxif and only if for any pair of active labels y,z ∈ A, we have l0(y)6=l1(z).
Mirrored Swap
ls0(x) =
min(x,s−x) 0≤s−x<M,
x otherwise, l
1 s(x) =
max(x,s−x) 0≤s−x<M,
x otherwise.
procedureMIRRORED-SWAP(x,M) repeat
for eachs∈ {1,2, . . . ,2n−3}do u∗ ←argminuE(lus(x))
x ←lus(x) end for untilconvergence return x
end procedure
s/2
0 1 2 3 4 5 6 7 8
Conclusion
• Suits MRFs with thecontinuousandorderedlabels,
• Alpha-expansion cannot be applied toordered labels,
• Mirrored Swap algorithm for theordered labels,
• Forunordered labels, the submodularity holds for vaster types of binary moves, including alpha-expansion and alpha-beta swap.
Thanks
Questions?
??
