We start by defining several relevant concepts.
Definition 6.4.AhyperplaneH inRnis an(n−1)-dimensional affine subset ofRn, that is, H ={x∈Rn :`(x) =α} is the level set of a nontrivial linear function`:Rn→R. If`is given by `(x) =ha, xifor somea6= 0inRn, then
H =H(a,α):={x∈Rn:ha, xi=α}. A hyperplaneH partitionsRn into two half-spaces.
Definition 6.5.LetH =H(a,α)be a hyperplane inRn. The closed half-spaces associated withH are the two closed sets
H¯(a,α)+ ={x∈E:ha, xi ≥α}, H¯(a,α)− ={x∈E:ha, xi ≤α}.
Similarly, the open half-spaces associated withH are the two open sets H(a,α)+ ={x∈E:ha, xi> α},
H(a,α)− ={x∈E:ha, xi< α}.
Definition 6.6.Let C andD be two nonempty sets and H := H(a,α) a hy- perplane inRn.
H is called a separating hyperplanefor the setsC andDifCis contained in one of the closed half-spaces determined by H and D in the other, say C⊆H¯(a,α)+ andD⊆H¯(a,α)− .
H is called a strictly separating hyperplanefor the sets C and D if C is contained in one of the open half-spaces determined byH andD in the other, sayC⊆H(a,α)+ andD⊆H(a,α)− .
H is called a strongly separating hyperplanefor the setsC andD if there exist β andγ satisfying β > α > γ,C⊆H¯(a,β)+ , andD⊆H¯(a,γ)− .
H is called a properly separating hyperplane for the sets C andD if H separatesCandD andCandDare not bothcontained in the hyperplane H.
H is called a support hyperplane of C at a point x ∈ C if x ∈ H and C⊆H¯+.
If there exists a hyperplane H separating the sets C and D in one of the senses above, we say thatC andD can be separated,strictly separated, strongly separated,properly separated, respectively.
We are ready to study the separation properties of convex sets inRn. We begin by considering the separation of a single point from a convex set.
6.2 Separation of Convex Sets in Finite-Dimensional Vector Spaces 145 Theorem 6.7.IfC⊂Rn is a nonempty convex set andx /∈ri(C), then there exists a hyperplaneH(a,α) such that x∈H(a,α) andC⊆H¯(a,α)+ , that is,
ha, xi ≥ ha, xi for all x∈C.
Proof. We first assume thatx /∈C. The variational inequality (6.1) gives hΠCx−x, x−ΠCxi ≥0 for all x∈C;
defininga:=ΠC(x)−x6= 0, and writing
x−ΠCx=x−x+ (x−ΠCx) =x−x−a, we getha, x−x−ai ≥0, orha, x−xi ≥ kak2>0. Therefore,
ha, xi ≥ ha, xi for all x∈C, and the theorem is proved in this case.
Ifx∈C\ri(C), then there exists a sequence{xk}of points not inCsuch thatxk→x. It follows from (6.1) that
hΠC(xk)−xk, x−ΠC(xk)i ≥0 for all x∈C.
Sincexk ∈/C andΠC(xk)∈C, we haveΠC(xk)6=xk; defining ak:= ΠC(xk)−xk)
kΠC(xk)−xkk, we have
hak, x−ΠC(xk)i ≥0 for all x∈C. (6.2) Since the sequence{ak}is bounded, it has a convergent subsequence; to avoid cumbersome notation, we assume that the sequence{ak}itself converges, say ak→a, where kak= 1. Sincexk →xandΠC is continuous,
ΠC(xk)→ΠC(x) =x;
thus lettingk→ ∞in (6.2) gives
ha, xi ≥ ha, xi for all x∈C,
and the theorem is proved. ut
Theorem 6.7 immediately implies the following.
Theorem 6.8. (Support hyperplane theorem)If C ⊆Rn is a nonempty convex set andx∈C\ri(C), then there exists a support hyperplane toC atx.
The following theorem provides the weakest separation result for two con- vex sets. It follows easily from Theorem 6.7, because a useful trick reduces the problem of separating two convex sets to the separation of a point from a suitably defined convex set.
Theorem 6.9.Let C and D be two nonempty convex sets in Rn. If C and D are disjoint, then there exists a hyperplaneH(a,α) that separatesC andD, that is,
ha, xi ≤α≤ ha, yi for all x∈C, y∈D.
Proof. The trick is to define the set
A:=C−D={x−y:x∈C, y∈D},
and note that 0∈/ A, due to C∩D =∅. The setA is convex, because A= C+ (−D) is the Minkowski sum of the convex setsCand−D; it follows from Theorem 6.7 that there exists a hyperplaneH(a,α) with α=ha,0i= 0 such thatha, ui ≤0 for allu∈A. SinceA=C−D, this means thatha, x−yi ≤0 for allx∈C and ally∈D, or
ha, xi ≤ ha, yi for all x∈C, y∈D.
Then, any hyperplaneH(a,α)withαsatisfying sup
x∈Cha, xi ≤α≤ inf
y∈Dha, yi
separates the setsCandD. ut
00000000000000 00000000000000 00000000000000 00000000000000 00000000000000 00000000000000 00000000000000
11111111111111 11111111111111 11111111111111 11111111111111 11111111111111 11111111111111 11111111111111
a
C
H
D
H
+H
−Fig. 6.2.Separation of two convex sets by a hyperplane.
Theorem 6.10. (Strong separation theorem)Let C, Dbe two nonempty, disjoint, closed, convex sets inRn. If one of them is compact, thenC andD can be strongly separated.
6.2 Separation of Convex Sets in Finite-Dimensional Vector Spaces 147 Proof. Let us assume for definiteness that D is compact. We note that the theorem is equivalent to the existence of a hyperplane H(a,α) satisfying the condition
x∈Cinfha, xi> α >max
y∈Dha, yi, or equivalently the condition
ha, xi> α >ha, yi for all x∈C, y∈D.
Define A :=C−D. The set A is convex, and we claim that it is closed.
Letuk →ube a convergent sequence withuk∈A; we will show thatu∈A.
Writeuk =xk−yk, wherexk ∈C andyk ∈D. SinceD is compact, we can extract a convergent subsequence from{yk}. To avoid cumbersome notation, assume that yk → y ∈ D. Since xk −yk → u and yk → y, we see that xk →x:=u+y∈C. Thus,u=x−y∈A, and the claim is proved.
Since C∩D =∅, we have 0 ∈/ A =A, and it follows from the first part of the proof of Theorem 6.7 that there exists a hyperplane H(a,0) such that ha, ui ≥ kak2>0 for all u∈A, orha, x−yi ≥ kak2 for allx∈Cand y∈D.
This implies
ha, xi ≥ ha, xi −kak2
2 ≥ ha, yi+kak2
2 >ha, yi for all x∈C, y∈D;
it is easy to see that the theorem holds withα=kak2/2 + maxy∈Dha, yi. ut Remark 6.11.The strong separation theorem breaks down if neitherCnorD is compact. For example, consider the closed convex setsC={(0, y)} (they- axis) andD={(x, y) :y≤lnx}. Neither set is compact, and it is easy to see that the only hyperplane separatingCandDis they-axis. SinceC coincides with the separating hyperplane, there exists no hyperplane separatingC and D strictly, let alone strongly.
Theorem 6.12.If C ⊆ Rn is a nonempty closed convex set, then C is the intersection of all the closed half-spaces containing it, that is,
C= \
(a,α)
H¯(a,α)+ :C⊆H¯(a,α)+ .
Proof. Denote byD the intersection set above. It is clear thatD is a closed, convex set containingC, so it remains to show thatD⊆C.
If this is not true, then there exists a point x0 ∈D that does not lie in C. Applying Theorem 6.10 to the convex sets{x0}andC, we see that there exists a hyperplaneH :=H(a,α) such thatx0 ∈H− andC ⊆H+; but then H¯+ is one of the closed half-spaces intersected to obtainD, and soD⊆H¯+. Sincex0 ∈D, we obtainx0 ∈H¯+, which contradicts x0∈H−. The theorem
is proved. ut
Remark 6.13.The result above provides an “external” or “from outside” char- acterization of closed convex sets as intersections of closed half-spaces. That is, all closed convex sets are obtained from half-spaces using theintersection operation. In contrast, the convex hull operation generates a convex set by enlargement, “from inside.” This is an instance of a “duality,” which is a common phenomenon in convexity.
The most useful separation result for two convex sets in a vector space is perhaps theproper separation theorem (Theorem 6.15 below and its general version Theorem 6.33 on page 161). It is here that the properties of the relative interior developed in Chapter 5 prove most useful.
We need the following result in its proof.
Lemma 6.14.Two nonempty convex sets C and D in Rn can be properly separated if and only if the origin and the convex set K := C−D can be properly separated.
Proof. LetH := H(a,α) be a hyperplane properly separating C and D such that C ⊆H¯+, D ⊆H¯−, and assume without loss of generality that C does not lie onH. Then
ha, xi ≥α≥ ha, yi for all x∈C, y∈D,
andha, x0i> α for somex0∈C; it follows thatha, zi ≥0 for allz∈K, with strict inequality holding for some z0 ∈ K. This proves that the hyperplane H(a,0)properly separates the sets{0}andK.
Conversely, suppose that the sets {0} and K are properly separated by a hyperplane H(a,α) such that K ⊆H¯(a,α)+ . Then ha, x−yi ≥ α ≥0 for all x∈Candy∈D, or
ha, xi ≥α+ha, yi for all x∈C, y∈D,
and either the first inequality is strict for some x0 ∈ C and y0 ∈ D, or else α >0. In the first case, any hyperplaneH(a,γ)with γ∈Rsatisfying
x∈Cinfha, xi ≥γ≥α+ sup
y∈Dha, yi
properly separatesCandD; in the second case we haveα >0 andha, xi=α+
ha, yifor anyx∈Candy∈D, so the hyperplaneH(a,γ)withγ=α/2 +ha, yi
properly separatesCandD. ut
Theorem 6.15. (Proper separation theorem) Two nonempty convex sets C and D in Rn can be properly separated if and only if ri(C) and ri(D) are disjoint.
Proof. Define the convex setK := C−D. It follows from Lemma 5.11 and Corollary 5.21 that ri(K) = ri(C−D) = ri(C)−ri(D); thus, ri(C)∩ri(D) =∅
6.2 Separation of Convex Sets in Finite-Dimensional Vector Spaces 149 and 0∈/ ri(K) are equivalent statements. Consequently, Lemma 6.14 reduces the proof of the theorem to proving that the sets {0} and K are properly separable if and only if 0∈/ri(K).
Suppose that the origin andKare properly separated by a hyperplaneH, such that 0∈ H¯− and K ⊆H¯+. We claim that 0∈/ ri(K). If 0 ∈/ H, then ri(K) ⊆H¯+, so that 0 ∈/ ri(K). Otherwise, 0 ∈ H and there exists a point x∈ K\H. If we had 0∈ri(K), there would exist a pointy ∈K such that 0∈(x, y), giving the contradictiony∈C∩H−=∅. This proves the claim.
Conversely, suppose that 0∈/ri(K). Write
L:= aff(K) =u0+ span{u1, . . . , uk},
where{ui}k1 is linearly independent. If 0∈/ L, then{ui}k0 is linearly indepen- dent, and we can extend it to a basis{ui}n−10 of Rn. Then the hyperplane H := u0 + span{u1, . . . , un−1} does not contain the origin, so it properly separates{0}andK.
If 0 ∈ L, we apply Theorem 6.9 within the vector space L to the sets {0}and ri(K), and obtain a hyperplaneP inLseparating 0 andKsuch that ri(K)⊆P+; seeFigure 6.3.We may assume that 0∈P; otherwise the transla- tion ofPso that it passes through the origin also satisfies the same separation properties. Extending P to the hyperplane H = span{P, uk+1, . . . , un−1}, it is evident thatH properly separates{0}andK. ut
P L
0 K
H
Fig. 6.3.Proper separation of convex sets.
We use Theorem 6.15 to obtain an improved version of Theorem 6.8.
Theorem 6.16.LetC be a nonempty convex set inRn. If D is a nonempty convex subset of the relative boundary ofC (D⊆C¯\ri(C)), then there exists a support hyperplane toC containingD but not all points of C.
Proof. Since ri(C)∩ri(D)⊆ri(C)∩D=∅, Theorem 6.15 implies that there exists a hyperplaneH :=H(a,α)properly separating CandD, say
ha, xi ≤α≤ ha, yi for all x∈C, y∈D.
Ify∈D, thenha, yi ≥α, but sincey∈C, we also have¯ ha, yi ≤α. This means thatha, yi=αfor ally ∈D, that is, D ⊆H. SinceH properly separatesC
andD, we must haveC6⊆H. ut
Finally, we present a proper separation theorem involving two convex sets one of which is an affine set.
Theorem 6.17.LetC⊂Rn be a nonempty convex set. IfM is an affine set such that ri(C) andM are disjoint, thenM can be extended to a hyperplane H such thatri(C)andH are disjoint.
Proof. Since M is affine, ri(M) = M, and Theorem 6.15 implies that there exists a hyperplaneH properly separatingC andM. IfM ⊆H, we are done;
if not, there exists a pointx∈M \H 6=∅. The affine setsH andM must be disjoint; otherwise, there would exist a pointu∈M∩H, and the line passing throughxanduwould intersect both half-spaces ¯H+and ¯H−, a contradiction because the line stays inM, which is included in one of these half-spaces.
If the hyperplaneH is shifted parallel to itself so that the new hyperplane
H˜ includes M, then ˜H∩ri(C) =∅. ut
y
x
z
Fig. 6.4.Separation of an affine set and a convex set.
It is perhaps surprising that even when the affine set M and the convex set C in Theorem 6.17 are disjoint, the hyperplane H ⊃ M may contain points of C. The example in Figure 6.4 is such a case in which M ⊂R3 is