Characterizations of the Local Solutions

Chapter 3. Qualitative Properties of the Minimum Sum-of-Squares

3.4 Characterizations of the Local Solutions

to the value given by the centroid system in (d)).

Proposition 3.4 One has

J_i(x) = j ∈ J | aⁱ ∈ A[x^j] . (3.17) Proof. From the formula of h_i,j(x) it follows that

h_i,j(x) = ^X

q∈J

kaⁱ−x^qk²− kaⁱ−x^jk². Therefore, by (3.15) we have

ϕ_i(x) = max

j∈J

h X

q∈J

kaⁱ−x^qk²− kaⁱ−x^jk²ⁱ

= ^X

q∈J

kaⁱ−x^qk²+ max

j∈J

− kaⁱ −x^jk²

= ^X

q∈J

kaⁱ−x^qk²−min

j∈J kaⁱ−x^jk².

Thus, the maximum in (3.15) is attained when the minimum min

j∈J kaⁱ−x^jk² is achieved. So, by (3.16),

J_i(x) =ⁿj ∈ J | kaⁱ−x^jk= min

q∈J kaⁱ−x^qk^o.

This implies (3.17). 2

Invoking the subdifferential formula for the maximum function (see [20, Proposition 2.3.12] and note that the Clarke generalized gradient coincides with the subdifferential of convex analysis if the functions in question are convex), we have

∂ϕ_i(x) = co{∇h_i,j(x)|j ∈ J_i(x)} = co2 x_e^j −_ea^i,j | j ∈ J_i(x) , (3.18) where

xe^j = x¹, . . . , x^j−1,0_Rⁿ, x^j+1, . . . , x^k (3.19) and

ea^i,j = aⁱ, . . . , aⁱ, 0_Rⁿ

j−th position|{z}

, aⁱ, . . . , aⁱ. (3.20) By the Moreau-Rockafellar theorem [84, Theorem 23.8], one has

∂f²(x) = 1 m

X

i∈I

∂ϕ_i(x) (3.21)

with ∂ϕ_i(x) being computed by (4.50).

Now, suppose x = (x¹, ..., x^k) ∈ _R^nk is a local solution of (3.2). By the necessary optimality condition in DC programming (see, e.g., [31] and [77]), which can be considered as a consequence of the optimality condition obtained by Dem’yanov et al. in quasidifferential calculus (see, e.g., [25, Theorem 3.1]

and [26, Theorem 5.1]), we have

∂f²(x) ⊂ ∂f¹(x). (3.22)

Since ∂f¹(x) is a singleton, ∂f²(x) must be a singleton too. This happens if and only if ∂ϕi(x) is a singleton for every i ∈ I. By (4.50), if |J_i(x)| = 1, then

|∂ϕ_i(x)| = 1. In the case where |J_i(x)| > 1, we can select two elements j1

and j₂ from J_i(x), j₁ < j₂. As ∂ϕ_i(x) is a singleton, by (4.50) one must have xe^j1−_ea^i,j1 = x_e^j2−_ea^i,j2. Using (3.19) and (3.20), one sees that the latter occurs if and only if x^j1 = x^j2 = aⁱ. To proceed furthermore, we need to introduce the following condition on the local solution x.

(C1) The components of x are pairwise distinct, i.e., x^j1 6= x^j2 whenever j₂ 6= j₁.

Definition 3.2 A local solution x = (x¹, ..., x^k) of (3.2) that satisfies (C1) is called a nontrivial local solution.

Remark 3.5 Proposition 3.3 shows that every global solution of (3.2) is a nontrivial local solution.

The following fundamental facts have the origin in [71, pp. 346]. Here, a more precise and complete formulation is presented. In accordance with (3.17), the first assertion of the next theorem means that if x is a nontrivial local solution, then for each data point aⁱ ∈ A there is a unique component x^j of x such that aⁱ ∈ A[x^j].

Theorem 3.3 (Necessary conditions for nontrivial local optimality) Suppose that x = (x¹, ..., x^k) is a nontrivial local solution of (3.2). Then, for any i ∈ I, |J_i(x)| = 1. Moreover, for every j ∈ J such that the attraction set A[x^j] of x^j is nonempty, one has

x^j = 1

|I(j)|

X

i∈I(j)

aⁱ, (3.23)

where I(j) ={i ∈ I | aⁱ ∈ A[x^j]}. For any j ∈ J with A[x^j] = ∅, one has

x^j ∈ A[x],/ (3.24)

where A[x] is the union of the balls B(a¯ ^p,ka^p − x^qk) with p ∈ I, q ∈ J satisfying p∈ I(q).

Proof. Suppose x = (x¹, ..., x^k) is a nontrivial local solution of (3.2). Given any i ∈ I, we must have |J_i(x)| = 1. Indeed, if |J_i(x)| > 1 then, by the analysis given before the formulation of the theorem, there exist indexes j₁ and j₂ from J_i(x) such that x^j1 = x^j2 = aⁱ. This contradicts the nontriviality of the local solution x. Let J_i(x) = {j(i)} for i ∈ I, i.e., j(i) ∈ J is the unique element of J_i(x).

For each i ∈ I, observe by (3.15) that

hi,j(x) < hi,j(i)(x) = ϕi(x) ∀j ∈ J \ {j(i)}.

Hence, by the continuity of the functions h_i,j(x), there exists an open neigh- borhood U_i of x such that

h_i,j(y) < h_i,j(i)(y) ∀j ∈ J \ {j(i)}, ∀y ∈ U_i. It follows that

ϕ_i(y) =h_i,j(i)(y) ∀y ∈ U_i. (3.25) So, ϕ_i(.) is continuously differentiable on U_i. Put U = ^\

i∈I

U_i. From (3.14) and (3.25) one can deduce that

f²(y) = 1 m

X

i∈I

ϕ_i(y) = 1 m

X

i∈I

h_i,j(i)(y) ∀y ∈ U.

Therefore, f²(y) is continuously differentiable function on U. Moreover, the formulas (4.50)–(3.20) yield

∇f²(y) = 2 m

X

i∈I

(y_e^j(i) −_ea^i,j(i)) ∀y ∈ U, (3.26) where

ye^j(i) = y¹, ..., y^j(i)−1,0_Rⁿ, y^j(i)+1, ..., y^k and

ea^i,j(i) = aⁱ, . . . , aⁱ, 0_Rⁿ

|{z}

j(i)−th position

, aⁱ, . . . , aⁱ.

Substitutingy = xinto (3.26) and combining the result with (3.22), we obtain X

i∈I

(x_e^j(i) −_ea^i,j(i)) =m(x¹−a⁰, ..., x^k −a⁰). (3.27)

Now, fix an index j ∈ J with A[x^j] 6= ∅ and transform the left-hand side of (3.27) as follows:

X

i∈I

(x_e^j(i) −_ea^i,j(i)) = ^X

i∈I, j(i)=j

(x_e^j(i) −_ea^i,j(i)) + ^X

i∈I, j(i)6=j

(x_e^j(i) −_ea^i,j(i))

= ^X

i∈I, j(i)=j

(x_e^j(i) −_ea^i,j(i)) + ^X

i /∈I(j)

(x_e^j(i) −_ea^i,j(i)).

Clearly, if j(i) = j, then the j-th component of the vector x_e^j(i) −_ea^i,j(i), that belongs to _R^nk, is 0_Rⁿ. If j(i) 6= j, then the j-th component of the vector xe^j(i) −_ea^i,j(i) is x^j −aⁱ. Consequently, (3.27) gives us

X

i /∈I(j)

(x^j −aⁱ) =m(x^j −a⁰).

Since ma⁰ = a¹+· · ·+a^m, this yields ^X

i∈I(j)

aⁱ = |I(j)|x^j. Thus, formula (3.23) is valid for any j ∈ J satisfying A[x^j] 6= ∅.

For any j ∈ J with A[x^j] = ∅, one has (3.24). Indeed, suppose to the contrary that there exits j₀ ∈ J with A[x^j0] = ∅ such that for some p ∈ I, q ∈ J, one has p ∈ I(q) and x^j0 ∈ B¯(a^p,ka^p−x^qk). If ka^p−x^j0k = ka^p−x^qk, then Jp(x) ⊃ {q, j₀}. This is impossible due to the first claim of the theorem.

Now, if ka^p − x^j0k < ka^p − x^qk, then p /∈ I(q). We have thus arrived at a contradiction.

The proof is complete. 2

Roughly speaking, the necessary optimality condition given in the above theorem is a sufficient one. Therefore, in combination with Theorem 3.3, the next statement gives a complete description of the nontrivial local solutions of (3.2).

Theorem 3.4 (Sufficient conditions for nontrivial local optimality) Suppose that a vector x = (x¹, ..., x^k) ∈ _R^nk satisfies condition (C1) and |J_i(x)| = 1 for every i ∈ I. If (3.23) is valid for any j ∈ J with A[x^j] 6= ∅ and (3.24) is fulfilled for any j ∈ J with A[x^j] = ∅, then x is a nontrivial local solution of (3.2).

Proof. Let x = (x¹, ..., x^k) ∈ _R^nk be such that (C1) holds, J_i(x) = {j(i)}

for every i ∈ I, (3.23) is valid for any j ∈ J with A[x^j] 6= ∅, and (3.24) is satisfied for any j ∈ J with A[x^j] = ∅. Then, for all i ∈ I and j⁰ ∈ J \ {j(i)},

one has

kaⁱ−x^j(i)k < kaⁱ−x^j0k.

So, there exist ε > 0, q ∈ J, such that

kaⁱ −x_e^j(i)k < kaⁱ−_ex^j0k ∀i ∈ I, ∀j⁰ ∈ J \ {j(i)}, (3.28) whenever vector x_e = (x_e¹, ...,x_e^k) ∈ _R^nk satisfies the conditionkx_e^q−x^qk < ε for all q ∈ J. By (3.24) and by the compactness of A[x], reducing the positive number ε (if necessary) we have

xe^j ∈ A[/ x]_e (3.29)

whenever vector x_e = (x_e¹, ...,x_e^k) ∈ _R^nk satisfies the condition kx_e^q − x^qk < ε for all q ∈ J, where A[x] is the union of the balls ¯_e B(a^p,ka^p−x_e^qk) with p∈ I, q ∈ J satisfying p ∈ I(q) = {i ∈ I | aⁱ ∈ A[x^q]}.

Fix any vector x_e = (x_e¹, ...,x_e^k) ∈ _R^nk with the property that kx_e^q −x^qk < ε for all q ∈ J. Then, by (3.28) and (3.29), J_i(x) =_e {j(i)}. So,

minj∈J kaⁱ −x_e^jk² = kaⁱ−x_e^j(i)k². Therefore, one has

f(x) =_e 1 m

X

i∈I

minj∈J kaⁱ−x_e^jk²

= 1 m

X

i∈I

kaⁱ −x_e^j(i)k²

= 1 m

X

j∈J

X

i∈I(j)

kaⁱ−_ex^j(i)k²

= 1 m

X

j∈J

X

i∈I(j)

kaⁱ−_ex^jk²

≥ 1 m

X

j∈J

X

i∈I(j)

kaⁱ−x^jk² = f(x), where the inequality is valid because (3.23) obviously yields

X

i∈I(j)

kaⁱ−x^jk² ≤ ^X

i∈I(j)

kaⁱ−x_e^jk²

for every j ∈ J such that the attraction set A[x^j] of x^j is nonempty. (Note that x^j is the barycenter of A[x^j].)

The local optimality of x = (x¹, ..., x^k) has been proved. Hence, x is a

nontrivial local solution of (3.2). 2

Example 3.2 (A local solution need not be a global solution) Consider the clustering problem described in Example 3.1. Here, we have I = {1,2,3}and J = {1,2}. By Theorem 3.1, problem (3.2) has a global solution. Moreover, if x = (x¹, x²) ∈ _R^2×2 is a global solution then, for everyj ∈ J, the attraction set A[x^j] is nonempty. Thanks to Remark 3.5, we know that x is a nontrivial local solution. So, by Theorem 3.3, the attraction sets A[x¹] and A[x²] are disjoint. Moreover, the barycenter of each one of these sets can be computed by formula (3.23). Clearly, A = A[x¹]∪A[x²]. Since A[x^j] ⊂ A= {a¹, a², a³}, allowing permutations of the components of each vector x = (x¹, x²) ∈ _R^2×2 (see Remark 3.4), we can assert that the global solution set of our problem is contained in the set

nx¯:= (1 2,1

2), (0,0), xˆ:= (0,1

2), (1,0), x_e := (1

2,0),(0,1)^o. (3.30) Since f(¯x) = ¹₃ and f(ˆx) = f(x) =_e ¹₆, we infer that ˆx and x_e are global solutions of our problem. Using Theorem 3.4, we can assert that ¯x is a local solution. Thus, ¯x is a local solution which does not belong to the global solution set, i.e., ¯x is a local-nonglobal solution of our problem.

Example 3.3 (Complete description of the set of nontrivial local solutions) Again, consider the MSSC problem given in Example 3.1. Allowing permu- tations of the components of each vector in _R^2×2, by Theorems 3.3 and 3.4 we find that the set of nontrivial local solutions consists of the three vectors described in (3.30) and all the vectors of the form x = (x¹, x²) ∈ _R^2×2, where x¹ = (¹₃,¹₃) and

x² ∈/ B(a¯ ¹,ka¹−x¹k)∪B(a¯ ²,ka²−x¹k)∪B(a¯ ³,ka³ −x¹k).

This set of nontrivial local solutions is unbounded and non-closed.

Example 3.4 (Convergence analysis of the k-means algorithm) Consider once again the problem described in Example 3.1. By the results given in Ex- ample 3.3, the centroid systems in items (a), (b), (c) and (d) of Example 3.1 are local solutions. In addition, by Example 3.2, the centroid systems in the just mentioned items (a) and (c) are global solutions. Concerning the centroid system in item (e) of Example 3.1, remark that x := (¹₃,¹₃),(1 +

√5

3 ,0) is not a local solution by Theorem 3.3, because a² ∈ A[x¹]∩A[x²], i.e.,J₂(x) ={1,2}

(see Figure 3.1). In general, with x¹ = (¹₃,¹₃) and x² ∈ _R^2×2 belonging to the boundary of the set

B(a¯ ¹,ka¹−x¹k)∪B¯(a²,ka²−x¹k)∪B(a¯ ³,ka³ −x¹k),

x := (x¹, x²) is not a local solution of the MSSC problem under consideration.

The above analysis shows that the k-means algorithm is very sensitive to the choice of starting centroids. The algorithm may give a global solution, a local-nonglobal solution, as well as a centroid system which is not a local solution. In other words, the quality of the obtained result greatly depends on the initial centroid system.

Figure 3.1: The centroids in item (e) of Example 3.1