Notes on Optimization
A.2 An Attempt to classify Optimization Problems
In the previous section, we have described uses of the objective function, gradient and Hessian.
The Taylor series expansion also gives us also a way to roughly classify optimization problems by difficulty. All things being equal, we can sort objective functions by increased unpredictability of the error term. For now we assume an unconstrained objective function, where allxican be chosen arbitrarily fromΩ =Rn(examples are given in Figure A.1). Ideally the objective function is very smooth (at least two continuous derivatives).
1. Without constraints, affine objective functions (defined by zero second and higher order derivatives) have no interesting minimizers. Solving constrained affine problems on the other hand has many important applications for instance in Operations Research and is called linear programming [Van01].
2. Quadratic functions are the simplest unconstrained objectives. These functions are character- ized by constant, positive definite Hessian matrices. (Otherwise the function shape would be a saddle and unbounded from below.) Quadratic objectives are usually minimized with linear equation systems. This means these problems are simple, well understood and very popular in comparison to problems with variable or no Hessian.
3. Objective functions with varying, but positive definite Hessian (everywhere). Such functions
are convex (or “bowl”-shaped). (But convex functions in general are only C0.) Convex functions have a unique minimum. Convexity is not easy to identify, but once known a broad theory with efficient solution methods exist [BV04]. Nice convex functions can be solved at the equivalent cost of a few linear solves.
4. Non-convex functions with unique minima. The Hessian is indefinite in some regions away from the minimum. As a consequence minimization approaches will have to deal with zero or negative curvature directions (multiple valleys in Figure A.1, upper right) Because of these negative curvature directions, methods specialized on convex functions will often fail, be- cause for efficiency reasons they often rely on methods like the Cholesky decomposition of the Hessian. In regions away from the minimizer multiple narrow valleys might exist, with the potential of increasing the ill-conditioning of the problem. While there might be a single minimizer, computationally this problem has to be solved with the same methods as the next.
5. The last fairly nice function class is that of objectives with arbitrary (but hopefully smooth and bounded) Hessians. These objective functions are assumed to have multiple minima. Given a reasonable starting guess many algorithms using descent directions will produce a solution that is in some sense near the starting point. But in general there is no control over which of the local minimizers is picked. For an efficiently solved problem in this class see [FMS03].
6. Objectives with piecewise discontinuous or unbounded Hessians lead in the experience of the author to less well conditioned problems (as similar objective functions with nicer Hessians).
In practice using this Hessian information to precondition the gradient is often still desirable.
7. Objective functions that are only C1 continuous with no access to Hessian information for preconditioning. Simple problems of this class can still be solved with high accuracy using first order (gradient descent) methods.
8. A very hard case are C0 objectives with piecewise continuous or no gradient at all. Here access to the main stopping criterion (a vanishing gradient) is lacking. With no additional information (like subgradients [Pol87]) one can only expect to obtain a relaxation of the objective value.
9. It is even harder to deal with stochastic objective functions. These are assumed to be com- posed of a deterministic function that is perturbed by an independently, identically distributed zero mean noise term [Spa03].
Figure A.2: Examples of simple constraints (from left to right): identity constraints, linear equal- ity constraints, nonlinear equality constraints, ball (or distance) constraints and linear inequality constraints.
Functions that resist above classification scheme are for instanceC0convex or discontinuous quasi- convex. The global knowledge implied by the convexity arguments leads to more efficient modeling than using the Taylor series expansion alone (see Chapter 5 in [Pol87] and Chapter 11 in [CGT00]).
Above classification is mostly done by structure into. Additional difficulty is arises if a particular problem is ill-conditioned. With this we mean the appearance of long and thin (and even worse:
curved) valleys in the objective function landscape. These valleys are tracked by many methods with high precision along the bottom, which makes the search for the minimum quite expensive.
Optimization problems can be further classified by their type of constraints. Constraints are used to describe the domain Ωwhich is usually called the feasible set. (Examples are given in Figure A.2.) Constraints significantly complicate finding the solution of optimization problems. For this reason we expect that a feasible setΩis described in a simple way. In particular one wants to have efficient tests for feasibilityx∈?Ωand often it helps to know how to navigate on the boundary∂Ω.
1. Identity constraints: some variables are assumed to be constantxj =cj. This is the simplest form of constraints and often implemented using Lagrange multipliers by modifying gradient and Hessian of the objective. Such modifications can be numerically rigid and visibly disturb the gradient and error residual in a neighborhood of the affected variables.
2. Linear equality constraints: A linear combination of variable is assumed to be constant Ax =c. This constraint is often numerically nicer than the identity constraint, because the disturbance is distributed over multiple variables. Eliminating equality constraints can make a sparse Hessian denser, hence some care needs to be taken in their implementation [BV04]
(Chapter 10).
3. Nonlinear equality constraints: The solution vector xhas to satisfy a set of nonlinear equa- tionsc(x) = 0. Solving nonlinear equations is general at least as difficult as minimizing an
objective, so this is a hard problem. This is complicated by the fact that there are now two residuals (one for the optimization and one for the nonlinear equations) leading in general to (slightly) suboptimal and (slightly) infeasible solutions. Sometimes it is possible to combine both nonlinear problems into either a single objective function or nonlinear system [CGT00].
4. Inequality constraints: Variables are assumed to be bounded:0≤Ax+b. Multiple constraints of this type will form the faces of a simplex [BV04] (Chapter 11).
5. Ball constraints: Given a normk.k, centercand radiusra ball is defined byB ={kx−ck ≤ r}. Solutions of this problem for simple (often quadratic) objective functions form the basis of trust-region methods [CGT00].
Finding an initial pointx0∈Ωcan be a difficult task. Combinations of many constraints can shrink the domainΩ⊆Rninto the empty set and make the optimization problem infeasible.