Choosingβ =0.01 andσ =0.4, the upper limit dictated by (3.49) is obtained from
4.14α2−25.54α+40≤52+0.01α(−50.12)
which yields α ≤ 6.45. Now, the lower limit dictated by (3.51) is obtained from
|8.28α−25.54| ≤ −0.4(−50.12)
which yieldsα≤5.506 andα≥0.663. These limits and the limitα≤6.45 yield the interval
0.663≤α≤5.506
Note that an exact line search based on equating f(α)=0 yieldsα=3.08.
A final note on line search strategies. The focus here has been onmaintain- ing descent. That is, on maintaining thatf(xk+1)<f(xk). However, referring to Fig. 3.5, it can be seen that a large step that violates this monotonic decrease followed by a steepest descent step can yield a significant decrease inf. Barzilai has shown such a two-point technique, although theoretically proven only on quadratic functions [Barzilai and Borwein 1988].
3.10 Using MATLAB
TheN-D unconstrained gradient-based minimization program in Matlab isfminunc.
It is recommended to use Matlab optimization routines using a “user subroutine” to evaluateffor a givenx. That is, a user subroutine “GETFUN.” For example, we first create a file, “testfun.m” consisting of the code
function [f] = getfun(X)
f = 100 * (X(1) * X(1) - X(2)) ˆ 2 + (1 - X(1)) ˆ 2;
Then, Matlab is executed with the command:
[Xopt,fopt,iflag,output] = fminunc(’testfun’, X);
which produces the desired solution. To see the default parameters type optimset(’fminunc’)
For instance, we can supply the gradient in the user subroutine and avoid the possibly expensive automatic divided difference scheme (the default) by switching on the corresponding feature as
options=optimset(’GradObj’, ‘on’)
fminuncis then executed using the command
[Xopt,fopt,iflag,output] = fminunc(’testfun’, X, options)
with a subroutine getfun that provides the analytical gradient in the vector DF as function [f, Df] = getfun(X)
f = ...
Df(1) = ...; Df(2) = ...; Df(N) = ...;
COM PUTER PROGRAM S STEEPEST, FLREEV, DFP
PROBLEM S
P3.1. Plot contours of the function f =x12x2−x1x2+8, in the range 0<x1<3,0<
x2 <10. You may use Matlab or equivalent program.
P3.2. For the functions given in the following, determine (a) all stationary points and (b) check whether the stationary points that you have obtained are strict local minima, using the sufficiency conditions:
(i)f=3x1+ 100 x1x2 +5x2
(ii)f=(x1−1)2+x1x2+(x2−1)2 (iii)f= x1+x2
3+x12+x22+x1x2
P3.3. (a) What is meant by a “descent direction”? (Answer this using an inequality.)
(b) Ifdis a solution ofW d= −∇f, then state a sufficient condition onWthat guarantees thatdis a descent direction. Justify/prove your statement.
Problems 3.4–3.9 125 P3.4. Given f =4x21+3x22−4x1x2+x1, the point x0=(−1/8,0)T and the direc- tion vectord0= −(1/5, 2/5)T,
(i) Isd0a descent direction?
(ii) Denoting f(α)= f(x0+αd0), find df(1)/dα, or equivalently, f(1).
P3.5. State True/False OR circle the correct answer OR fill in the blanks for the following questions:
(a) (T/F?) Conditions in Wierstraas theorem are satisfied for the following problem:
minimize f =x1+x2
subject to x1x2 ≥1,x1≥0,x2≥0 (b) (T/F?) The problem in (a) has a solution.
(c) A C2 continuous function f is convex if and only if its Hessian is .
(d) Iffis convex, then a local minimum is also . P3.6. (Fill in the blanks)
With respect to the problem: minimize f(x), wherefisC2continuous:
(i) A point ˆxis a “candidate point” if
(ii) To determine whether a candidate point corresponds to a local minimum, we need to check the condition
P3.7. Two plots of f(α)= f(x0+αd0) are shown here. Identify which plot corre- sponds to steepest descent and which plot corresponds to Newton’s method.
f
α Figure P3.7
P3.8. (Fill in the blanks)
(a) The problem f =x12+10000x22 can be more effectively solved using the
steepest descent method by .
(b) Preconditioning the Hessian in conjugate gradient methods means
to .
(c) Preconditioning the Hessian in conjugate gradient methods helps
to .
P3.9. Given f =x12+5x22,xk=[7, 2]T,k=iteration index:
(i) Evaluate current value,f(xk). Then, evaluatef(xk+1), wherexk+1has been obtained withoneiteration of steepest descent method.
(ii) Show the current point, direction vector, and new point on a sketch of the contour map of the function (draw, say, three contours off).
(iii) Repeat above for one iteration of Newton’s method.
P3.10. Given f =x12+25x22, a pointx0=(5, 1)T,f(x0)=50:
(i) Construct f(α) along the steepest descent direction.
(ii) Perform a line search (i.e., minimize f(α) with respect toα), and obtain a new pointx1. Give value off(x1).
(iii) Provide a plot showing contour(s) of the function, steepest descent direc- tion,x0andx1.
P3.11. A cardboard box (see Fig. P3.11) is to be designed to have a volume of 1.67 ft3. Determine the optimal values ofl, w,andhso as to minimize the amount of cardboard material. (Hint: use the volume constraint equation to eliminate one of the variables.)
h
w 2
2
w Figure P3.11
P3.12. Prove that ifAis a positive definite matrix, then conjugate vectors – that is, vectors satisfying (3.25) – are linearly independent.
P3.13. A sequence {f(xk)}is generated using an optimization program as: fk = [10.01, 4.01, 1.60, 0.639, 0.252 . . . ] converging to f∗ =0. Determine whether the con- vergence rate is linear or nonlinear.
Problems 3.14–3.18 127 P3.14. [Courtesy: Dr. H.J. Sommer] A cylindrical coordinate robot is to be used for palletizing a rectangular area. Find the maximum rectangular area available within the annular footprint of the robot workspace (Fig. P3.14). Taker1=12and r2=24.
r1
r2
w
Figure P3.14
P3.15. (i) Construct linear and quadratic approximations to the function f =x1x22 at the pointx0=(1, 2)T. Plot the original function and its approximations (as in Example 1.21, Chapter 1).
(ii) For the functionf=x1x2, determine expressions forf(α) along the line x1=x2and also along the line joining (0, 1) to (1, 0).
P3.16. Consider Example 3.1 in the text withk1=k2=1 lb/in andF1=0. Determine the vertical displacementQ2forF2=2.5 lb, 3 lb, 6 lb, 10 lb. PlotQ2(vertical axis) versusF2.
P3.17. Perform two iterations, by hand calculations, using the methods of steepest descent, Fletcher–Reeves, DFP, and Newton’s for the quadratic function
f =(x1+2x2−6)2+(2x1+x2−5)2, x0=(0,0)T.
P3.18. It is desired to fit an equation of the formf=a x1c1x2c2, through some data obtained by a computer simulation, given below. Specifically, our task is to deter- mine the coefficientsa, c1, andc2(which are our “variables”) in the correlation given in the preceding text by minimizing the sum of square of the deviations between simulated and predicted values off.
fsimulated x1 x2
2.122 5.0 10.0
9.429 3.0 1.0
23.57 0.6 0.6
74.25 0.1 2.0
6.286 3.0 1.8
P3.19. Introduce scaling,x=T y, into Program Steepest. Then, choose aTmatrix to reduce the number of iterations needed to solve Rosenbrock’s problem, starting from (−1.2, 1)T. (Hint: Use a diagonalTmatrix. Include a listing of your modified code highlighting the changes to the code.)
P3.20. Implement the Polak–Ribiere algorithm (see (3.32)) and compare its perfor- mance with the Fletcher–Reeves algorithm for the problems in Example 3.12.
P3.21. Determine the function y(x) that leads to a stationary value of the functional
f = ' 2
0
dy dx
2
+2ydy dx +4x2
dx y(0)=1,y(2) is free
(Hint: discretize the functiony(x) as in Example 3.13).
P3.22. Implement the BFGS formula and compare with the DFP code. You may consider the test functions given in Example 3.12 and also Brook’s function given by
f =x12exp[1−x21−20.25(x1−x2)2] with starting pointsx0=(1.9, 0.1)Tandx0=(0.1, 0.1)T.
P3.23. Compare the results in Table 3.1 with Matlabfminuncby providing analytical gradients as discussed in Section 3.11.
P3.24. Figure P3.24 shows arigidplate supported by five springs and subjected to a load P as shown. Obtain the spring forces by minimizing the potential energy in the system which is given by=5
i=11
2kiδi2−Pδ, whereδi=vertical deflection of the ith spring andδ=the vertical deflection under the load. Assume only axial spring deflections and ignore any side loading on the springs caused by plate tipping. (Hint:
theδiand theδare related by the fact that the rigid plate remains flat).
P3.25. Implement Newton’s with Hessian computed only once at the starting point x0 and compare with the regular Newtons’s algorithm on two problems of your choice. (Choose problems from the literature with known solutions).
P3.26. Implement Levenberg–Marquardt algorithm discussed in the text on two least-squares problems of your choice and compare with FLREEV code. (Choose problems from the literature with known solutions).
P3.27. Repeat Example 3.13 and solve the Brachistochrone problem with friction – what happens to shape of path asμincreases in value? Give physical interpretation.
P3.28. Implement approximate line search based on sufficient decrease in Eq.(3.49) into STEEPEST, FLREEV, or DFP codes and resolve problems in Example 3.12.
Study the effect of parameterβ.
References 129
P = 1500 1b (−6, 2)
(−6, −2)
(−3, −1)
(0, 0) 2
6 1b
50in 3
(6, −2) (6, 2)
5
4
1
Thin Rigid Plate z
y
1b x 50in
1b 50in 1b
50in
1b 75in
Figure P3.24
P3.29. Implement approximate line search based on both Eqs. (3.49) and (3.51) into STEEPEST, FLREEV, or DFP codes and resolve problems in Example 3.12.
REFERENCES
Armijo, L. Minimization of functions having Lipschitz-continuous first partial derivatives, Pacific Journal of Mathematics,16, 1–3, 1966.
Barzilai, J. and Borwein, M., Two-point step size gradient methods,IMA Journal of Numeri- cal Analysis, 19888(1):141–148.
Cauchy, A., Methode generale pour la resolution des systemes d’equations simultanes,C. R.
Acad. Sci. Par., 25, 536–538, 1847.
Chandrupatla, T.R. and Belegundu, A.D.Introduction to Finite Elements in engineering.
Prentice-Hall, Englewood Cliffs, NJ, 1991.
Curry, H., The method of steepest descent for nonlinear minimization problems,Quarterly of Applied Mathematics,2, 258–261, 1944.
Davidon, W.C., Variable metric method for minimization, Research and Development Report ANL-5990 (Revised), Argonne National Laboratory, U.S. Atomic Energy Com- mision, 1959.
Dennis, J.E. and Schnabel, R.B.,Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice-Hall, Englewood-Cliffs, NJ, 1983.
Fletcher, R.,Practical Methods of Optimization, 2nd Edition, Wiley, New York, 1987.
Fletcher, R. and Powell, M.J.D. A rapidly convergent descent method for minimization.The Computer Journal,6, 163–168, 1963.
Fletcher, R. and Reeves, C.M., Function minimization by conjugate gradients.The Computer Journal,7, 149–154, 1964.
Gill, P.E. and Murray, W., Quasi-Newton methods for unconstrained optimization.Journal of the Institute of Mathematics and Its Applications,9, 91–108, 1972.
Goldstein, A.A., On steepest descent,SIAM Journal on Control,3, 147–151, 1965.
Kardestuncer, H. (Ed.),Finite Element Handbook, McGraw-Hill, New York, 1987.
Luenberger, D.G., Introduction to Linear and Nonlinear Programming, Addison-Wesley, Reading, MA, 1965.
Polak, E.,Computational Methods in Optimization: A Unified Approach, Academic Press, New York, 1971.
Pshenichny, B.N. and Danilin, Y.M.,Numerical Problems in Extremal Problems, MIR Pub- lishers, Moscow, 1978, 1982.
Wolfe, P., Convergence conditions for ascent methods,SIAM Review,11, 226–235, 1968.