1
Chapte
r
5
Chapter 5
2
3
Chapte
r
5
Unidimensional Search
(1) If have a search direction, want to minimize in that direction by numerical methods
(2) Search Methods in General
2.1. Non Sequential – Simultaneous evaluation of f at n points – no good (unless on parallel computer).
4
Chapte
r
5
(3) Types of search that are better or best is often problem dependent. Some of the types are:
a. Newton, Quasi-Newton, and Secant methods. b. Region Elimination Methods (Fibonacci, Golden Section, etc.).
c. Polynomial Approximation (Quadratic Interpolation, etc.).
d. Random Search
(4) Most methods assume
(a) a unimodal function, (b) that the min is
5
Chapte
r
5
To Bracket the Minimum
)
(
)
(
until
x
doubling
Continue
)
(
)
(
Compute
and
)
(
Compute
NEW
OLD NEW
OLD NEW
6
Chapte
r
5
points)
closest
the
(using
f(x)
minimum
the
giving
point
on the
bracket
a
keep
you to
enables
point that
the
Discard
.
,
,
,
points
spaced
equally
4
have
now
You
)
(
Compute
3.
) 1 ( )
2 ( )
2 1 2 ( )
3 ( )
2 ( )
1 (
x
x
x
x
x
x
7
8
Chapte
r
5
1. Newton’s Method
Newton’s method for an equation is
)
Application to Minimization
9
Chapte
r
5
Examples
Minimize
2
Minimize
Continue
x
10
Chapte
r
5
Advantages of Newton’s Method
(1) Locally quadratically convergent (as long as f (x) is positive – for a minimum).
(2) For a quadratic function, get min in one step.
Disadvantages
(1) Need to calculate both f (x) and f (x) (2) If f (x)→0, method converges slowly
11
Chapte
r
5
2. Finite-Difference Newton Method
Replace derivatives with finite differences
2 )
( )
1 (
)
(
)
(
2
)
(
)
(
)
(
h
h
x
f
x
f
h
x
f
h
x
f
h
x
f
x
x
k k
Disadvantage
12
Chapte
r
5
3. Secant(Quasi-Newton) Method
Analogous equation to (A) is
)
The secant approximates f (x) as a straight line
13
Chapte
r
5
Start the Secant method by using 2 points spanning x at which first derivatives are of opposite sign.
14
Chapte
r
5
Order of Convergence
Can be expressed in various ways. Want to consider how
1
0
* )
(
* )
1 (
* )
(
c
c
x
x
x
x
Linea r
k
a s
x
x
k k k
15
Chapte
r
5
1
0
* )
(
* )
1 (
p
c
c
x
x
x
x
P
Order
p k
k
Fastest in practice
If p = 2, quadratic convergence p = 1.32 ?
)
0
(
0
lim
* )
(
* )
1 (
x x r c a nd c a s k
x x
r Superlinea
k k
k k
k
Usually fast in practice
16
17
18
19
20
21
Chapte
r
5
Quadratic Interpolation
Approximate f(x) by a quadratic function. Use 3 points Minimize
cx for equations us
simultaneo 3
Solve
points 3
the at
22
Chapte
r
5
2
1 1 1 1
2
2 2 2 2
2
3 3 3 3
2 2
1 1 1 1
2 2
2 2 2 2
2 2
3 3 3 3
1
( )
1
( )
1
( )
1
( )
1
( )
1
( )
1
1
1
1
1
1
f x
x
x
f x
f x
x
x
f x
f x
x
x
f x
b
c
x
x
x
x
x
x
x
x
x
x
x
x
23
Chapte
r
5
2 2 2 2
2 3 3 2 1 3 3 1
2 2
1 2 2 1
2 2 2 2 2 2
1 3 2 2 3 1 3 2 1
2 2 2 2 2
1 2 3 2 3 1 3 2
1
(
)
( )
1
( )
( )
1
( )
(
)
:
Numerator
( )
(
)
( ) (
)
( )(
)
(
)(
)
( )(
) :
Denominator
b
f x x
f x x
f x x
f x x
f x x
f x x
c
f x
x
x
f x
x
x
f x
x
x
f x
x
x
f x
x
x
f x
x
x
c b x
24
25
26
27
28
29
30
