Revisiting Rockafellar’s Theorem on Relative Interiors of Convex Graphs with Applications to Convex Generalized Differentiation

Portland State University Portland State University

PDXScholar PDXScholar

Mathematics and Statistics Faculty

Publications and Presentations Fariborz Maseeh Department of Mathematics and Statistics

1-2024

Revisiting Rockafellar’s Theorem on Relative Revisiting Rockafellar’s Theorem on Relative Interiors of Convex Graphs with Applications to Interiors of Convex Graphs with Applications to Convex Generalized Differentiation Convex Generalized Differentiation

Boris S. Mordukhovich Wayne State University Nguyen Mau Nam

Portland State University, mau.nam.nguyen@pdx.edu Dang Van Cuong

DuyTan University, DaNang, Vietnam G. Sandine

Portland State University

Follow this and additional works at: https://pdxscholar.library.pdx.edu/mth_fac Part of the Physical Sciences and Mathematics Commons

Let us know how access to this document benefits you.

Citation Details Citation Details

Published as: Van Cuong, D., Mordukhovich, B., Nam, N. M., & Sandine, G. (2022). Revisiting Rockafellar's Theorem on Relative Interiors of Convex Graphs with Applications to Convex Generalized Differentiation.

arXiv preprint arXiv:2201.10689.

This Pre-Print is brought to you for free and open access. It has been accepted for inclusion in Mathematics and Statistics Faculty Publications and Presentations by an authorized administrator of PDXScholar. Please contact us if we can make this document more accessible: pdxscholar@pdx.edu.

arXiv:2201.10689v2 [math.OC] 4 Apr 2023

Revisiting Rockafellar’s Theorem on Relative Interiors of Convex Graphs with Applications to Convex Generalized Differentiation

Dang Van Cuong ¹,B. S. Mordukhovich²,Nguyen Mau Nam³,G. Sandine⁴ Dedicated to Roger Wets with the highest respect

Abstract. In this paper we revisit a theorem by Rockafellar on representing the relative interior of the graph of a convex set-valued mapping in terms of the relative interior of its domain and function values. Then we apply this theorem to provide a simple way to prove many calculus rules of generalized differentiation for set-valued mappings and nonsmooth functions in finite dimensions.

Using this important theorem by Rockafellar allows us to improve some results on generalized dif- ferentiation of set-valued mappings in [13] by replacing the relative interior qualifications on graphs with qualifications on domains and/or ranges.

Key words. convex analysis, generalized differentiation, geometric approach, relative interior, nor- mal cone, subdifferential, coderivative, calculus rules

AMS subject classifications.49J52, 49J53, 90C31

1 Introduction

The notion of relative interior for convex sets in finite-dimensional spaces goes back to Steinitz [18] and then has been systematically studied and applied in finite-dimensional convex analysis and related areas; see, e.g., the seminal monograph by Rockafellar [16] and the subsequent publications including the books by Borwein and Lewis [4] and by Hiriart- Urruty and Lemar´echal [8] with further references and commentaries. In contrast to the interior, the relative interior isnonemptyfor any nonempty convex set inRⁿ, while the latter notion shares many important properties of the interior being very useful in applications.

Relative interiors play a crucial role in many aspects of convex analysis and optimization in finite dimensions such as convex separation, generalized differential calculus, Fenchel conjugate, and Fenchel and Lagrange duality; see, e.g., [1,2,4,5,7,8,10,12,14,15,17] and the references therein.

Among many important results involving relative interiors is the theorem by Rockafellar [16, Theorem 6.8] allowing us to represent the relative interior of a convex set G in Rⁿ×R^m in terms of the the relative interiors of the image D of G under the projection mapping

1Department of Mathematics, Faculty of Natural Sciences, Duy Tan University, Da Nang, Vietnam (dvcuong@duytan.edu.vn). This research is funded by Vietnam National Foundation for Science and Tech- nology Development (NAFOSTED) under grant number 101.02-2020.20.

2Department of Mathematics, Wayne State University, Detroit, Michigan 48202, USA (boris@math.wayne.edu). Research of this author was partly supported by the USA National Science Foundation under grant DMS-1808978, by the Australian Research Council under grant DP-190100555, and by Project 111 of China under grant D21024.

3Fariborz Maseeh Department of Mathematics and Statistics, Portland State University, Portland, OR 97207, USA (mnn3@pdx.edu). Research of this author was partly supported by the USA National Science Foundation under grant DMS-2136228.

4Fariborz Maseeh Department of Mathematics and Statistics, Portland State University, Portland, OR 97207, USA (gsandine@pdx.edu).

(x, u) → x and the set S(x) := {u ∈ R^m | (x, u) ∈ G} for x ∈ D. In the language of set-valued analysis, this theorem gives a representation of the relative interior of the graph of a convex set-valued mapping in terms of the relative interiors of its domain and the function’s values. The first proof of this result was provided in the aforementioned book by Rockafellar as a consequence of several other results involving relative interiors of convex sets. A more self-contained proof was given by Rockafellar and Wets [17, Proposition 2.43].

In a recent paper [6], we provided the third proof of this important theorem and explored its generalization to locally convex topological vector spaces using a generalized relative interior concept called thequasi-relative interior introduced in [3].

The first goal of the present paper is to revisit Rockafellar’s theorem and derive a new result on relative interiors of graphs of generalized epigraphical mappings. Then we employ these results and the geometric approach to convex analysis developed in [13,14] to provide a simple way to access many calculus rules of generalized differentiation for set-valued map- pings under new qualification conditions. The usage of Rockafellar’s theorem and related developments allows us to improve, in particular, a number of calculus rules obtained in [13]

for coderivatives of convex set-valued mappings. Our developments have a great potential for further implementations in the field of set-valued optimization; see, e.g., the books [9,11]

and the references therein for this and related areas of optimization theory and applications.

This paper is organized as follows. Section 2 contains basic concepts of convex analysis in finite dimensions used throughout the paper. In Section 3, we revisit Rockafellar’s theorem on relative interiors of convex graphs and derive a number of new results with detailed proofs. Section 4 is devoted to employing this theorem and the geometric approach to con- vex analysis in the study of generalized differentiation for convex set-valued mappings and nonsmooth functions in finite dimensions. Some applications to convex generalized equa- tions, convex constraint systems, and optimal value functions are presented in Section 5.

In Section 6, we develop the convex coderivative calculus for set-valued mappings obtained under relative interior qualification conditions imposed on domains or ranges. These de- velopments significantly improve calculus rules in [13, Section 11] under relative interior qualification conditions imposed on graphs. Throughout the paper, we use standard nota- tion of convex analysis in finite dimensions; see [4,8,14,16]. In particular,hx, yidenotes the inner product ofx, y∈Rⁿ;B(x;γ) signifies the closed ball centered atxwith radiusγ ≥0;

the closure of a set Ω ⊂Rⁿ is denoted by Ω; the convex hull of a set Ω is co(Ω); the cone generated by a set Ω is cone(Ω) :={tw|t≥0, w∈Ω}.

2 Preliminaries

In this section, we recall a number of concepts and results of convex analysis in finite dimensions used throughout the paper; see, e.g., [4,8,14,16] and the references therein.

A subset Ω of Rⁿ is calledconvex if

λx+ (1−λ)y∈Ω for allx, y∈Ω and λ∈(0,1).

It follows directly from the definition that Ω is convex if and only if for any two points

x, y∈Ω, the line segment (x, y) :={λx+ (1−λ)y |λ∈(0,1)} is a subset of Ω. A subset Ω of Rⁿ is calledaffine if for anyx, y∈Ω and for anyλ∈Rwe have

λx+ (1−λ)y∈Ω,

which means that Ω is affine if and only if the line through any two points x, y ∈ Ω is a subset of Ω. It follows directly from the definition that any affine set is a convex set. In addition, the intersection of any collection of affine sets is an affine set and thus allows us to define theaffine hull of a setS by

aff(S) :=\

{Ω|Ω is affine and S ⊂Ω}.

Therelative interior ri(Ω) of a set Ω inRⁿ is defined as its interior within the affine hull of Ω, i.e., by

ri(Ω) :={x∈Ω| ∃γ >0 satisfying B(x;γ)∩aff(Ω)⊂Ω}.

Let Ω be a nonempty convex subset ofRⁿ with ¯x∈Ω. Thenormal coneto Ω at ¯x is N(¯x; Ω) :=

v∈Rⁿhv, x−xi ≤¯ 0 for all x∈Ω withN(¯x; Ω) :=∅if ¯x /∈Ω.

The following theorem provides several characterizations for the relative interior of a convex set; see, e.g., [6, Theorem 2.2].

Theorem 2.1 (characterizations of relative interior for convex sets in Rⁿ). Let Ω be a nonempty convex set in Rⁿ and let x¯∈Rⁿ. The following properties are equivalent:

(a) ¯x∈ri(Ω).

(b) ¯x∈Ω and for every x∈Ω withx6= ¯x there existsu∈Ωsuch that x¯∈(x, u).

(c) ¯x∈Ω andcone(Ω−x)¯ is a linear subspace of Rⁿ. (d) ¯x∈Ω andcone(Ω−x)¯ is a linear subspace of Rⁿ. (e) ¯x∈Ω and the normal coneN(¯x; Ω) is a subspace of Rⁿ.

The relative interior possesses several nice algebraic and topological properties, some of which are presented in the theorem below; see, e.g., [16].

Theorem 2.2 (properties of relative interiors). Let Ω and Ωi for i = 1, . . . , m be nonempty convex subsets ofRⁿ. Then

(a) ri(Ω) is nonempty and convex.

(b) [a, b) ⊂ri(Ω) for any a∈ri(Ω) and b∈Ω, where [a, b) :={ta+ (1−t)b |0 < t≤ 1}

defines the half-open interval connectinga, b∈Rⁿ. (c) Ω = ri(Ω) andri(Ω) = ri(Ω).

(d) ri(ri(Ω)) = ri(Ω).

(e) ri(Pm

i=1Ωi) =Pm

i=1ri(Ωi).

(f) ri(A(Ω)) =A(ri(Ω)), where A:Rⁿ→R^m is a linear mapping.

(g) ri(∩^m_i=1Ωi) =∩^m_i=1ri(Ωi) provided that ∩^m_i=1ri(Ωi)6=∅.

It is worth noting that the relative interior may not inherit all properties of the interior.

For example, for two nonempty convex sets Ω₁ and Ω₂ inRⁿ with Ω₁ ⊂Ω₂, it is not true in general that ri(Ω₁)⊂ri(Ω₂).

Another important role of the relative interior is in the study of convex proper separation inRⁿ. Recall that two nonempty convex sets Ω₁,Ω₂ ⊂Rⁿcan beproperly separatedif there existsv∈Rⁿfor which the following two inequalities hold:

sup

hv, w1iw₁ ∈Ω₁ ≤inf

hv, w2iw₂ ∈Ω₂ , (2.1) inf

hv, w1iw1 ∈Ω1 <sup

hv, w2iw2 ∈Ω2 . (2.2) Observe that condition (2.1) can be equivalently rewritten as

hv, w1i ≤ hv, w2iwhenever w₁ ∈Ω₁, w₂∈Ω₂, while (2.2) means that there exist wb₁∈Ω₁ and wb₂∈Ω₂ such that

hv,wb₁i<hv,wb₂i.

As a central theorem of convex analysis in finite dimensions, the following theorem uses the relative interior to provide necessary and sufficient conditions for properly separating two nonempty convex sets; see, e.g., [16, Theorem 11.3].

Theorem 2.3 (relative interior and proper separation in finite dimensions). Let Ω1 andΩ2 be two nonempty convex subsets ofRⁿ.ThenΩ1 andΩ2can be properly separated if and only if ri(Ω₁)∩ri(Ω₂) =∅.

The relative interior plays a crucial role in many other issues of convex analysis. For instance, a direct application of Theorem2.3for Ω and the single-point set{x}¯ shows that if ¯x ∈ Ω\ri(Ω), then N(¯x; Ω) 6= {0}. The relative interior and the proper separation theorem can also be used in the statement and proof of the normal cone intersection rule in the theorem below; see, e.g., [16, Corollary 23.8.1] for more details.

Theorem 2.4 (normal cone intersection rule in finite dimensions). LetΩ1, . . . ,Ωm ⊂ Rⁿ be convex sets satisfying the relative interior condition

\m i=1

ri(Ωi)6=∅,

where m≥2. Then we have the normal cone intersection rule N

¯ x;

\m i=1

Ω_i

= Xm

i=1

N(¯x; Ω_i) for all ¯x∈

\m i=1

Ω_i.

Given a set-valued mappingF:Rⁿ→→R^m, thegraph of F is the set gph(F) :=

(x, y)∈Rⁿ×R^my ∈F(x) ,

and it is called convexif gph(F) is a convex subset of the product space Rⁿ×R^m. We also consider the domain and range of F defined by

dom(F) :=

x∈RⁿF(x)6=∅ and rge(F) := [

x∈Rⁿ

F(x),

respectively. It is easy to see that if F is a convex set-valued mapping, then dom(F) and rge(F) are convex sets as well.

The coderivative of a convex set-valued mappingF at (¯x,y)¯ ∈gph(F) is defined by D^∗F(¯x,y)(v) :=¯

u∈Rⁿ(u,−v)∈N((¯x,y); gph(F))¯ , v∈R^m. (2.3) The coderivative can be used to define thesubdifferential of an extended-real-valued convex functionf:Rⁿ→(−∞,∞]. Given ¯x∈dom(f) :={x∈Rⁿ|f(x)<∞}, define

∂f(¯x) :=D^∗E_f(¯x, f(¯x))(1) =

v∈Rⁿhv, x−xi ≤¯ f(x)−f(¯x) for all x∈Rⁿ , (2.4) where Ef(x) := [f(x),∞) forx∈Rⁿ is the epigraphical mapping/multifunction associated withf with gph(Ef) = epi(f) :={(x, λ)∈Rⁿ×R|f(x)≤λ}.

3 Rockafellar’s Theorem on Relative Interiors of Convex Graphs

In this section, we revisit the aforementioned theorem by Rockafellar on representing relative interiors of graphs of convex set-valued mappings.

Theorem 3.1 (Rockafellar’s theorem on relative interiors of convex graphs). Let F:Rⁿ→→R^m be a convex set-valued mapping. Then we have the representation

ri gph(F)

=

(x, y)∈Rⁿ×R^mx∈ri dom(F)

, y∈ri F(x) . (3.1) Proof. We first prove the inclusion “⊂” in (3.1). Consider the projection mappingP:Rⁿ× R^m→Rⁿ given by

P(x, y) :=xfor (x, y)∈Rⁿ×R^m. It follows from Theorem2.2(f) that

P(ri(gph(F)) = ri(P(gph(F))) = ri(dom(F)). (3.2) Now, take any (¯x,y)¯ ∈ ri(gph(F)) and get from (3.2) that ¯x ∈ri(dom(F)). Since (¯x,y)¯ ∈ ri(gph(F)) ⊂ gph(F), we have ¯y ∈ F(¯x). Fix any y ∈ F(¯x) with y 6= ¯y. Then (¯x, y) ∈ gph(F) with (¯x, y) 6= (¯x,y). By the equivalence of (a) and (b) from Theorem¯ 2.1, there exists (u, z)∈gph(F) and t∈(0,1) such that

(¯x,y) =¯ t(¯x, y) + (1−t)(u, z).

Then ¯x = t¯x+ (1−t)u, which implies (1−t)¯x = (1 −t)u and so ¯x = u. In addition,

¯

y = ty+ (1−t)z ∈ (y, z), where z ∈ F(¯x). Using the equivalence of (a) and (b) from Theorem2.1 again yields ¯y∈ri(F(¯x)).

To verify next the reverse inclusion in (3.1), fix ¯x∈ri(dom(F)) and ¯y∈ri(F(¯x)). Arguing by contradiction, suppose that (¯x,y)¯ ∈/ ri(gph(F)) and then find by Theorem 2.3 a pair (u, v)∈Rⁿ×R^m such that

hu, xi+hv, yi ≤ hu,xi¯ +hv,yi¯ whenever (x, y)∈gph(F). (3.3) In addition, it follows from the proper separation of {(¯x,y)}¯ and gph(F) that there exists a pair (x₀, y₀)∈gph(F) satisfying

hu, x0i+hv, y0i<hu,xi¯ +hv,yi.¯ (3.4) Letting x := ¯x in (3.3) yields hv, yi ≤ hv,yi¯ for all y ∈ F(¯x). Since ¯x ∈ ri(dom(F)) and x0 ∈ dom(F), we deduce from Theorem 2.1 that there exists ˜x ∈ dom(F) such that

¯

x=tx₀+ (1−t)˜x for somet∈(0,1), which is is true even ifx₀ = ¯x. Choose ˜y∈F(˜x) and consider the convex combination

y^′:=ty₀+ (1−t)˜y,

wherey^′ ∈F(¯x) since gph(F) is convex. Since (˜x,y)˜ ∈gph(F) we use (3.3) and (3.4) to get hu,xi˜ +hv,yi ≤ hu,˜ xi¯ +hv,yi,¯

hu, x₀i+hv, y₀i<hu,xi¯ +hv,yi.¯

Multiplying the first inequality above by 1−t and the second one by t, and then adding them together give us the condition

hu,xi¯ +hv, y^′i<hu,xi¯ +hv,yi,¯

which yields hv, y^′i < hv,yi. From this along with (3.3) when¯ x = ¯x, we conclude that the sets {¯y} and F(¯x) can be properly separated. Applying Theorem 2.3 tells us that

¯

y /∈ri(F(¯x)), a contradiction that verifies (¯x,y)¯ ∈ri(gph(F)).

Given a set-valued mapping F:Rⁿ →→ R^m, recall that the inverse of F is the set-valued mappingF⁻¹:R^m→→Rⁿ defined by

F⁻¹(y) :=

x∈Rⁿy∈F(x) , y∈R^m. The next corollary is a direct consequence of Theorem 3.1.

Corollary 3.2 (relative interiors of convex graphs and ranges). Let F:Rⁿ→→R^m be a convex set-valued mapping. If (¯x,y)¯ ∈ri(gph(F)), theny¯∈ri(rge(F)).

Proof. It is not hard to show that dom(F⁻¹) = rge(F) and that (¯x,y)¯ ∈ri(gph(F)) if and only if (¯y,x)¯ ∈ri(gph(F⁻¹)). Thus the conclusion follows directly from Theorem3.1.

4 Relative Interiors and Coderivatives of Generalized Epi- graphical Mappings

Let us now apply Theorem3.1to derive a representation for relative interiors ofgeneralized epigraphical mappings defined by

F(x) :=Ef1(x)×Ef2(x)× · · · ×Efm(x), x∈Rⁿ, (4.1) where Efi(x) := [fi(x),∞) for x ∈ Rⁿ, and fi: Rⁿ → (−∞,∞] for i = 1, . . . , m are extended-real-valued convex functions.

Theorem 4.1 (relative interiors of generalized convex epigraphical graphs). Let f_i:Rⁿ→(−∞,∞]for i= 1, . . . , mbe extended-real-valued convex functions satisfying

\m i=1

ri(dom(f_i))6=∅.

Then we have the following representation for the generalized epigraphical mapping (4.1):

ri gph(F)

=n

(x, λ₁, . . . , λm)∈Rⁿ×R^m x∈Tm

i=1ri(dom(fi)),

f_i(x)< λ_ifor all i= 1, . . . , mo .

Proof. It follows from the definition of the generalized epigraphical mapping (4.1) that dom(F) =Tm

i=1dom(fi) and gph(F) =

(x, λ) ∈Rⁿ×R^m x ∈dom(F), λ∈F(x)

=

(x, λ₁, . . . , λ_m)∈Rⁿ×R^m x ∈

\m i=1

dom(f_i), f_i(x)≤λ_i for alli= 1, . . . , m . For anyx∈dom(F), it readily follows that

ri(F(x)) = ri [f₁(x),∞)× · · · ×[fm(x),∞)

= (f₁(x),∞)× · · · ×(fm(x),∞).

Under the assumption that Tm

i=1ri(dom(fi))6=∅, we employ Theorem2.2 and get ri\^m

i=1

dom(fi)

=

\m i=1

ri(dom(fi)).

Applying finally Theorem 3.1gives us ri(gph(F)) =

(x, λ1, . . . , λm)∈Rⁿ×R^m x∈ri

\m i=1

(dom(fi)

, (λ1, . . . , λm)∈ri(F(x))

=

(x, λ₁, . . . , λm)∈Rⁿ×R^m x∈

\m i=1

ri(dom(fi)), fi(x)< λifor all i= 1, . . . , m ,

which thus completes the proof of this theorem.

As a direct consequence of Theorem 3.1, we obtain in the corollary below a representation for the relative interior of the epigraph of an extended-real-valued convex function; see, e.g., [8, Proposition 1.1.9].

Corollary 4.2 (relative interiors of convex epigraphs). Let f:Rⁿ→(−∞,∞]be an extended-real-valued convex function. Then

ri(epi(f)) =

(x, λ)∈Rⁿ×Rx∈ri(dom(f)), f(x)< λ .

Given an extended-real-valued convex functionf:Rⁿ→(−∞,∞], for each ¯x∈dom(f) we define the operation

α⊙∂f(¯x) :=

(α∂f(¯x) ifα >0,

∂^∞f(¯x) ifα= 0,

where∂f(¯x) is the subdifferential off at ¯xdefined in (2.4) and where∂^∞f(¯x) is thesingular subdifferentialof f at ¯x defined by

∂^∞f(¯x) :=

v∈Rⁿ(v,0)∈N((¯x, f(¯x)); epi(f)) .

In the proof of the next result, we employ a well-known representation of the subdifferential of a convex function via the normal cone to its epigraph saying that

∂f(¯x) =

v∈Rⁿ(v,−1)∈N((¯x, f(¯x)); epi(f)) , x¯∈dom(f).

Proposition 4.3 (coderivative of epigraphs for extended-real-valued functions).

Let f:Rⁿ → (−∞,∞] be an extended-real-valued convex function, and let F:Rⁿ →→ R be the set-valued mapping defined by F(x) := [f(x),∞) for x ∈ Rⁿ. Then for any x¯ ∈ dom(F) = dom(f) we have the representation

D^∗F(¯x, f(¯x))(α) =

(α⊙∂f(¯x) if α≥0,

∅ if α <0.

Proof. It is easy to see that gph(F) = epi(f).The coderivative definition yields D^∗F(¯x, f(¯x))(α) =

v∈Rⁿ(v,−α)∈N((¯x, f(¯x)); epi(f)) , α∈R. (4.2) We consider the following three possible choices for α:

• Ifα >0,then (v,−α)∈N((¯x, f(¯x)); epi(f)) if and only if v

α,−1

∈N (¯x, f(¯x)); epi(f) .

This means that v/α∈∂f(¯x),and hencev∈α∂f(¯x).

•Ifα= 0,then the definition of the singular subdifferential yields (v,−α)∈N((¯x, f(¯x)); epi(f)) if and only if v∈∂^∞f(¯x).

• If α <0, then we can show that D^∗F(¯x, f(¯x))(α) = ∅. Indeed, suppose that this is not the case and find (v,−α)∈N((¯x, f(¯x)); epi(f)). Then

hv, x−xi −¯ α λ−f(¯x)

≤0 whenever (x, λ)∈epi(f).

Choosing (¯x, f(¯x) + 1) ∈epi(f), we can see that −α ≤0. This contradiction verifies that D^∗F(¯x, f(¯x))(α) =∅.

Therefore, the representation ofD^∗F(¯x, f(¯x))(α) follows from (4.2) and the above definition

of the operation α⊙∂f(¯x).

Now we are ready to obtain a useful coderivative representation for generalized epigraphical set-valued mappings under the relative interior condition.

Theorem 4.4 (coderivatives of generalized epigraphical mappings). Let fi:Rⁿ→ (−∞,∞]fori= 1, . . . , mbe extended-real-valued convex functions. Consider the generalized epigraphical set-valued mapping F defined in (4.1) and suppose that

\m i=1

ri(dom(fi))6=∅.

Then for any x¯∈dom(F) we have the representation

D^∗F(¯x,y)(α) =¯







 Xm

i=1

αi⊙∂fi(¯x) if αi≥0 for all i= 1, . . . m,

∅ if αi<0 for some i= 1, . . . , m, where y¯:= (f₁(¯x), . . . , fm(¯x))and α:= (α₁, . . . , αm)∈R^m.

Proof. Define the sets Ωi :=

(x, λ₁, . . . , λm)∈Rⁿ×R^mx∈Rⁿ, λi ≥fi(x) , i= 1, . . . , m, and observe that gph(F) =Tm

i=1Ωi. It follows from Corollary4.2that ri(Ωi) =

(x, λ1, . . . , λm)∈Rⁿ×R^m x∈ri(dom(fi)), λi > fi(x) , i= 1, . . . , m.

Choose x₀ ∈ Tm

i=1ri(dom(f_i)) and let ¯λ_i := f_i(x₀) + 1 > f_i(x₀) for i = 1, . . . , m. Then (x₀,λ¯₁, . . . ,λ¯m)∈Tm

i=1ri(Ωi), and henceTm

i=1ri(Ωi)6=∅.

Applying now the normal cone intersection rule from Theorem2.4 gives us N (¯x,y); gph(F¯ ))

=N (¯x,y);¯

\m i=1

Ωi)

= Xm i=1

N (¯x,y); Ω¯ i .

It is easy to see that Ω1= epi(f1)×R^m−1, which gives us by [12, Proposition 2.11] that N (¯x,y); Ω¯ ₁

=N (¯x, f₁(¯x)); epi(f₁)

× {0}.

The latter means that (v,−α)∈N (¯x,y); Ω¯ ₁

if and only if (v,−α1)∈N (¯x, f₁(¯x)); epi(f₁) andα_j = 0 forj = 2, . . . , m. In general, we observe that (v,−α)∈N (¯x,y); Ω¯ _i

if and only if (v,−αi)∈N (¯x, fi(¯x)); epi(fi)

and αj = 0 for j∈ {1, . . . , m} \ {i}.

Finally, it follows from the coderivative construction and the proof of Proposition4.3that D^∗F(¯x,y)(α) =¯ n

v∈Rⁿ

(v,−α)∈N((¯x,y); gph(F¯ ))o

=n

v∈Rⁿ |(v,−α)∈ Xm i=1

N (¯x,y); Ω¯ i)o

=







 Xm i=1

αi⊙∂fi(¯x) if αi ≥0 for all i= 1, . . . , m,

∅ if αi <0 for some i= 1, . . . , m,

which therefore completes the proof of the theorem.

5 Optimal Value Functions and Generalized Chain Rules

Given a set-valued mappingF:Rⁿ→→ R^m and an extended-real-valued function ϕ:R^m → (−∞,∞], define the associated optimal value function by

µ(x) := inf

ϕ(y)y∈F(x) , x∈Rⁿ. (5.1)

Throughout this section, we assume thatµ(x)>−∞for all x∈Rⁿ. For any ¯x∈dom(µ), consider the argminimum set

S(¯x) :=

y ∈F(¯x) µ(¯x) =ϕ(y) .

We have the following exact formula for subdifferentiation of the optimal value function under the relative interior qualification condition.

Proposition 5.1 (subdifferentials of optimal value functions). Let µbe the optimal value function defined in (5.1), where F is a convex set-valued mapping, and where ϕis an extended-real-valued convex function. Then the function µ is convex. In addition, for any

¯

x∈dom(µ) and any y¯∈S(¯x) we have

∂µ(¯x) = [

v∈∂ϕ(¯y)

D^∗F(¯x,y)(v)¯

provided that there exists x₀ ∈ri(dom(F)) such that ri(F(x₀))∩ri(dom(ϕ))6=∅.

Proof. Define the functionψ:Rⁿ×R^m→(−∞,∞] byψ(x, y) :=ϕ(y) for (x, y)∈Rⁿ×R^m. Thenψis clearly convex with dom(ψ) =Rⁿ×dom(ϕ), and hence we get ri(dom(ψ)) =Rⁿ× ri(dom(ϕ)). Choosey₀∈ri(F(x₀))∩ri(dom(ϕ)). Then (x₀, y₀)∈ri(dom(ψ)), and it follows from Theorem3.1that (x0, y0)∈ri(gph(F)). Therefore, ri(gph(F))∩ri(dom(ψ))6=∅. Now

we deduce the claimed result from [13, Theorem 9.1].

Recall that ϕ:R^m →(−∞,∞] isnondecreasing componentwise if we have x_i≤u_i for alli= 1, . . . , m] =⇒[ϕ(x₁, . . . , x_m)≤ϕ(u₁, . . . , u_m)

.

The next theorem gives us a generalization of [8, Theorem 4.3.1] for a broad class of compos- ite extended-real-valued functions. We also provide a simpler proof of this theorem applying the coderivative of generalized epigraphical mappings.

Theorem 5.2 (subdifferentials of a composition with increasing extended-real- valued functions of several variables). Let f_i:Rⁿ→R for i= 1, . . . , m be real-valued convex functions, and let ϕ:R^m →(−∞,∞] be nondecreasing componentwise and convex.

Consider the composite function

g(x) :=ϕ(f₁(x), . . . , fm(x)), x∈Rⁿ.

Then the function g: Rⁿ → (−∞,∞] is convex. Suppose in addition that there exists (x₀, λ₁, . . . , λm) ∈ Rⁿ×R^m such that λi > fi(x₀) for all i = 1, . . . , m and (λ₁, . . . , λm) ∈ ri(dom(ϕ)). Then for any x¯∈dom(g) we have the subdifferential formula

∂g(¯x) = X^m

i=1

γi∂fi(¯x)

(γ₁, . . . , γm)∈∂ϕ(¯y)

,

where y¯:= (f₁(¯x), . . . , fm(¯x)).

Proof. Define the set-valued mapping F(x) := [f1(x),∞)× · · · ×[fm(x),∞) for x ∈ Rⁿ and then deduce from the nondecreasing componentwise property of ϕthat

µ(x) =g(x) for all x∈Rⁿ,

where µ is the optimal value function (5.1) generated by F and ϕ. Observe that in this case we have that each function fi is continuous, and that γ⊙∂fi(¯x) = γ∂f(¯x) whenever γ ≥0 and ¯x∈Rⁿ. Using the representation

ri(F(x)) = (f₁(x),∞)× · · · ×(fm(x),∞) for any x∈Rⁿ,

it follows from the imposed assumptions that there exists x₀ ∈ri(dom(F)) =Rⁿ such that ri(F(x₀))∩ri(dom(ϕ))6=∅. Furthermore, Proposition5.1 tells us that

∂g(¯x) =∂µ(¯x) = [

γ∈∂ϕ(¯y)

D^∗F(¯x,y)(γ).¯

The rest of the proof follows from the coderivative formula for F in Theorem4.4.

6 Coderivative Calculus in Finite-Dimensional Spaces

In this section, under the relative interior conditions imposed on domains and ranges of map- pings, we establish major formulas of coderivative calculus including sum rule, chain rule, and intersection rule for set-valued mappings in finite-dimensional spaces. The obtained results improve those in [13] under more restrictive qualification conditions.

Given two set-valued mappingsF₁, F₂:Rⁿ→→R^m, their sum is defined by (F1+F2)(x) =F1(x) +F2(x) :=

y1+y2y1 ∈F1(x), y2 ∈F2(x) .

It is easy to see that dom(F1 +F2) = dom(F1)∩dom(F2), and that F1 +F2 is convex provided that bothF₁ andF₂ have this property.

Our first calculus result concerns representing coderivatives of sumsF₁+F₂at a given point (¯x,y)¯ ∈gph(F₁+F₂). To formulate this result, consider the nonempty set

S(¯x,y) :=¯

(¯y₁,y¯₂)∈R^m×R^my¯= ¯y₁+ ¯y₂, y¯_i∈F_i(¯x) for i= 1,2 .

The following theorem gives us the coderivative sum rule for set-valued mappings on finite- dimensional spaces. In this version, we use the relative interior qualification condition on domains replacing the condition on graphs known from [13, Theorem 11.1].

Theorem 6.1 (coderivative sum rule via qualification condition on domains). Let F₁, F₂:Rⁿ→→R^m be convex set-valued mappings. Imposing the relative interior condition

ri(dom(F1))∩ri(dom(F2))6=∅, (6.1) we have the coderivative sum rule

D^∗(F₁+F₂)(¯x,y)(v) =¯ \

(¯y1,¯y2)∈S(¯x,¯y)

D^∗F₁(¯x,y¯₁)(v) +D^∗F₂(¯x,y¯₂)(v)

for all (¯x,y)¯ ∈gph(F₁+F₂) andv ∈R^m.

Proof. Fix any u ∈ D^∗(F₁ +F₂)(¯x,y)(v) and (¯¯ y₁,y¯₂) ∈ S(¯x,y) for which we have the¯ inclusion (u,−v)∈N((¯x,y); gph(F¯ 1+F2)). Consider the convex sets

Ω₁ :=

(x, y₁, y₂)∈Rⁿ×R^m×R^m y₁∈F₁(x) , Ω₂ :=

(x, y₁, y₂)∈Rⁿ×R^m×R^m y₂∈F₂(x) and deduce from the normal cone definition that

(u,−v,−v)∈N((¯x,y¯₁,y¯₂); Ω₁∩Ω₂).

Now we intend to verify the inclusion

(u,−v,−v)∈N((¯x,y¯₁,y¯₂); Ω₁) +N((¯x,y¯₁,y¯₂); Ω₂). (6.2) Indeed, it follows from (6.1) that there exists x ∈ ri(dom(F₁))∩ri(dom(F₂)), and hence Theorem2.2 implies that ri(F1(x))6=∅and ri(F2(x))6=∅. Theorem 3.1ensures that

ri(Ω₁) =

(x, y₁, y₂)∈Rⁿ×R^m×R^mx∈ri(dom(F₁)), y₁ ∈ri(F₁(x)) , ri(Ω₂) =

(x, y₁, y₂)∈Rⁿ×R^m×R^mx∈ri(dom(F₂)), y₂ ∈ri(F₂(x)) ,

which shows in turn that condition (6.1) yields ri(Ω₁)∩ ri(Ω₂) 6= ∅. This tells us by Theorem2.4 that (6.2) is satisfied, and therefore we get the relationships

(u,−v,−v) = (u1,−v,0) + (u₂,0,−v) with (ui,−v)∈N((¯x,y¯_i); gph(F_i)) fori= 1,2.

This implies by the coderivative definition that

u=u₁+u₂∈D^∗F₁(¯x,y¯₁)(v) +D^∗F₂(¯x,y¯₂)(v)

as desired. The reverse inclusion is obvious, and thus we verify the claimed sum rule.

Next we present the well-known subdifferential sum rule (see, e.g., [16, Theorem 23.8]), which can also be derived from Theorem6.1.

Corollary 6.2 (subdifferential sum rule). Let f_i:Rⁿ →R, i= 1,2, be extended-real- valued convex functions. Suppose that the relative interior qualification condition

ri(dom(f₁))∩ri(dom(f₂))6=∅ (6.3) is satisfied. Then for all x¯∈dom(f₁)∩dom(f₂) we have the subdifferential sum rule

∂(f₁+f₂)(¯x) =∂f₁(¯x) +∂f₂(¯x). (6.4) Proof. Define the convex set-valued mappingsF₁, F₂:X→→R by

Fi(x) :=

fi(x),∞

for i= 1,2.

It is easy to see that gph(Fi) = epi(fi) and dom(Fi) = dom(fi) fori= 1,2. Furthermore, the qualification condition (6.3) clearly implies the fulfillment of (6.1).

To proceed further, fix any ¯x ∈dom(f₁)∩dom(f₂), and let ¯y := f₁(¯x) +f₂(¯x). For every x^∗ ∈∂(f₁+f₂)(¯x) we have the coderivative inclusion

x^∗ ∈D^∗(F₁+F₂)(¯x,y)(1).¯

Applying to the latter Theorem 6.1with ¯yi=fi(¯x) for i= 1,2 gives us x^∗ ∈D^∗F₁(¯x,y¯₁)(1) +D^∗F₂(¯x,y¯₂)(1) =∂f₁(¯x) +∂f₂(¯x),

which verifies the inclusion “⊂” in (6.4). The reverse inclusion is obvious.

Now we define thecompositionof two set-valued mappingsF:Rⁿ→→R^m andG:R^m→→R^q by

(G◦F)(x) = [

y∈F(x)

G(y) :=

z∈G(y)y ∈F(x) , x∈Rⁿ,

and observe that G◦F is convex provided that both F and G have this property. Given

¯

z∈(G◦F)(¯x), we consider the set

M(¯x,z) :=¯ F(¯x)∩G⁻¹(¯z).

The following theorem establishes the coderivative chain rule for set-valued mappings in finite-dimensional spaces. In this version, we use the relative interior qualification condition on domains and ranges replacing the one on graphs known from [13, Theorem 11.2].

Theorem 6.3 (coderivative chain rule via qualification condition on domains).

Let F:Rⁿ →→ R^m and G:R^m →→ R^q be convex set-valued mappings satisfying the relative interior qualification condition

ri rge(F)

∩ri dom(G)

6=∅. (6.5)

Then for any (¯x,z)¯ ∈gph(G◦F) and w∈R^q we have the coderivative chain rule D^∗(G◦F)(¯x,z)(w) =¯ \

¯ y∈M(¯x,¯z)

D^∗F(¯x,y)¯ ◦D^∗G(¯y,z)(w).¯ (6.6)

Proof. Picking u∈ D^∗(G◦F)(¯x,z)(w) and ¯¯ y ∈ M(¯x,z) gives us the inclusion (u,¯ −w) ∈ N((¯x,z); gph(G¯ ◦F)), which means that

hu, x−xi − hw, z¯ −zi ≤¯ 0 for all (x, z)∈gph(G◦F).

Define two convex subsets ofRⁿ×R^m×R^q by

Ω₁:= gph(F)×R^q and Ω₂ :=Rⁿ×gph(G).

It is easy to see that

Ω₁−Ω₂=Rⁿ× rge(F)−dom(G)

×R^q. (6.7)

Using (6.7), we have the representation

ri(Ω₁−Ω₂) =Rⁿ×ri rge(F)−dom(G)

×R^q.

It follows from (6.5) due to the definitions of the sets Ω₁ and Ω₂ that

0∈ri(Ω₁−Ω₂), and so ri(Ω₁)∩ri(Ω₂)6=∅. (6.8) We can directly deduce from the definitions that

(u,0,−w)∈N((¯x,y,¯ z); Ω¯ ₁∩Ω₂).

Applying Theorem2.4with qualification (6.8) tells us that

(u,0,−w)∈N((¯x,y,¯ z); Ω¯ ₁∩Ω₂) =N((¯x,y,¯ z); Ω¯ ₁) +N((¯x,y,¯ z); Ω¯ ₂).

Thus using further the definitions of the sets Ω₁ and Ω₂based on gph(F) and gph(G), there exists a vector v∈R^m such that we have the representation

(u,0,−w) = (u,−v,0) + (0, v,−w),

where (u,−v)∈N((¯x,y); gph(F¯ )), (v,−w)∈N((¯y,z); gph(G)). This shows by the coderiva-¯ tive definition (2.3) that

u∈D^∗F(¯x,y)(v) and¯ v∈D^∗G(¯y,z)(w),¯

and so we verify the inclusion “⊂” in (6.6). The reverse inclusion is trivial.

Let F:Rⁿ →→R^m be a set-valued mapping, and let Θ⊂R^m be a given set. The preimage or inverse imageof Θ under the mapping F is given by

F⁻¹(Θ) =

x∈Rⁿ F(x)∩Θ6=∅ .

The next result gives us a representation of the normal cone toF⁻¹(Θ) via the normal cone of Θ and the coderivative of F. We use here the relative interior qualification condition on ranges replacing the condition on graphs known from [13, Proposition 10.1].

Proposition 6.4 (representation of the normal cone to preimages). LetF:Rⁿ→→R^m be a convex set-valued mapping, and let Θ⊂R^m be a convex set. Suppose that

ri rge(F)

∩ri(Θ)6=∅. (6.9)

Then for any x¯∈F⁻¹(Θ) andy¯∈F(¯x)∩Θ we have the representation N(¯x;F⁻¹(Θ)) =D^∗F(¯x,y)¯ N(¯y; Θ)

.

Proof. In the setting of Theorem6.3, consider the set-valued mapping G:R^m →→R^q given by

G(x) := ∆_Θ(x) =

(0 ifx∈Θ,

∅ ifx /∈Θ.

It is clear that dom(∆_Θ) = Θ, gph(∆_Θ) = Θ× {0} and that for any ¯x ∈ Θ we get N (¯x,0); gph(∆_Θ)

=N(¯x; Θ)×R^q. Therefore,

D^∗∆_Θ(¯x,0)(v) =N(¯x; Θ) for allv∈R^q. It is easy to check the composite representation

∆_F−1

(Θ)(x) = ∆_Θ◦F

(x) for allx∈Rⁿ,

where ∆_F−1(Θ):Rⁿ→→R^p is given by ∆_F−1(Θ)(x) := 0 ifx∈F⁻¹(Θ), and ∆_F−1(Θ)(x) :=∅ otherwise. Observe that the imposed relative interior qualification condition (6.9) guaran- tees that ri(rge(F))∩ri(dom(G))6=∅. Then the claimed formula for N(¯x;F⁻¹(Θ)) follows from the coderivative chain rule of Theorem6.3 with the outer mappingG:= ∆Θ. The last result of this section provides a precise representation formula for the normal cone to sublevel sets of extended-real-valued convex functions.

Corollary 6.5 (representation of the normal cone to sublevel sets). Letf:Rⁿ→R be a convex function. For λ∈R, consider the sublevel set

Lλ :=

x∈Rⁿf(x)≤λ

Assume that f(¯x) =λ, and that there exists xˆ ∈ri(dom(f)) such that f(ˆx) < λ. Then we have the representation

N(¯x;Lλ) = [

α≥0

α⊙∂f(¯x).

Proof. Let F(x) := Ef(x) for x ∈Rⁿ, and let Θ := (−∞, λ]. Then Lλ =F⁻¹(Θ). Since ˆ

x∈ri(dom(f)) andf(ˆx)< f(¯x) =λ, we can show that ri rge(F)

∩ri(Θ)6=∅.

Indeed, choose γ ∈ R such that f(¯x) < γ < λ. By Corollary 4.2, we see that (¯x, γ) ∈ ri(epi(f)) = ri(gph(F)), so γ ∈ ri(rge(F)) by Corollary 3.2. Thus γ ∈ri rge(F)

∩ri(Θ).

Since N(f(¯x); Θ) =N(λ; Θ) = [0,∞), by Proposition 4.3and Proposition 6.4we have N(¯x;Lλ) =N(¯x;F⁻¹(Θ)) =D^∗F(¯x,y)¯ N(λ; Θ)

=D^∗F(¯x, f(¯x)) [0,∞)

= [

α≥0

D^∗F(¯x, f(¯x))(α) = [

α≥0

α⊙∂f(¯x),

which completes the proof of the corollary.

Acknowledgements. The authors are very grateful to the anonymous referees for their valuable remarks and suggestions that allowed us to improve the original presentation.

References

[1] H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, Springer, New York, 2011.

[2] D. P. Bertsekas, A. Nedi´c and A. E. Ozdaglar, Convex Analysis and Optimization, Athena Scientific, Belmont, MA, 2003.

[3] J. M. Borwein, and A. S. Lewis, Partially finite convex programming, Part I: quasi-relative interiors and duality theory, Math. Program.57(1992), 15–48.

[4] J. M. Borwein and A. S. Lewis, Convex Analysis and Nonlinear Optimization, Springer, New York, 2000.

[5] S. Boyd and L. Vandenberghe,Convex Optimization, Cambridge University Press, New York, 2004.

[6] D. V. Cuong, B. S. Mordukhovich and N. M. Nam, Quasi-relative interiors for graphs of convex set-valued mappings,Optim. Lett.15(2021), 933–952.

[7] F. Giannessi,Constrained Optimization and Image Space Analysis, I: Separation of Sets and Optimality Conditions, Springer, Berlin, 2005.

[8] J.-B. Hiriart-Urruty and C. Lemar´echal,Fundamentals of Convex Analysis, Springer, Berlin, 2001.

[9] A. A. Khan, C. Tammer and C. Z˘alinescu,Set-valued Optimization, Springer, New York, 2015.

[10] A. G. Kusraev and S. S. Kutateladze,Subdifferentials: Theory and Applications, Kluwer, Dor- drecht, The Netherlands, 1995.

[11] B. S. Mordukhovich,Variational Analysis and Applications, Springer, Cham, Switzerland, 2018.

[12] B. S. Mordukhovich and N. M. Nam, An Easy Path to Convex Analysis and Applications, Morgan & Claypool Publishers, San Rafael, CA, 2014; reprint in Springer, Cham, Switzerland, 2022.

[13] B. S. Mordukhovich and N. M. Nam, Geometric approach to convex subdifferential calculus, Optimization.66(2017), 839–873.

[14] B. S. Mordukhovich and N. M. Nam,Convex Analysis and Beyond, I: Basic Theory, Springer, Cham, Switzerland, 2022.

[15] D. Pallaschke and S. Rolewicz, Foundation of Mathematical Optimization: Convex Analysis without Linearity, Kluwer, Dordrecht, The Netherlands, 1998.

[16] R. T. Rockafellar,Convex Analysis, Princeton University Press, Princeton, NJ, 1970.

[17] R. T. Rockafellar and R. J-B. Wets.Variational Analysis, Springer, Berlin, 1998.

[18] E. Steinitz, Bedingt konvergente reihen und konvexe systeme, I,J. Math.143(1913), 128–175.