Roundoff-Error Analysis for Gaussian Elimination

Systems of Linear Equations

4.5 Roundoff-Error Analysis for Gaussian Elimination

as well as the following for i = 1, 2, ... , n:

or

that is,

Pi = Pi - _j=l

f ^(Xij~j

⁼

^(I1

^Pi

⁺

_j=l

f

^l1(Xijl

^{~jl) f!.i}

_(Ji

n

= -bPi

+ L

b(Xij~j, j=l

L

n ^((Xij

+

b(XiJ~j = Pi

+

^bPi,

j=l

Ai = b,

which was to be shown. D

The criterion expressed in Theorem (4.4.19) permits us to draw conclu- sions about the fitness of a solution from the smallness of its residual. For example, if all components of Ao and bo have the same relative accuracy s,

then (4.4.19) is satisfied if

IAox-bol :(s(lbol + IAoll·xl)·

From this inequality, the smallest s can be computed for which a given

.x

^can

still be accepted as a useable solution.

4.5 Roundoff-Error Analysis for Gaussian

using 2-place floating-point arithmetic. If the element a21 = 1 is taken as the pivot, then 2-place floating-point arithmetic yields

y = 0.50, x = 0.50.

In the second case, the accuracy of the result is considerably higher. This could lead to the jmpression that the largest element in magnitude should be chosen as a pivot from among the candidates in a column to get the best numerical results. However, a moment's thought shows that this cannot be unconditionally true. If the first row of the equation system (4.5.1) is mul- tiplied by

200,

for example, the result is the system

jl

(4.5.2)

which has the same solution as (4.5.l). The element

all

= 1 is now just as large as the element

a21

= 1. However, the choice of

all

as pivot element leads to the same inexact result as before. We have replaced the matrix A of (4.5.1) by

A

= DA, where D is the diagonal matrix

=

f200 0]

D

l °

^1·

Obviously, we can also adjust the column norms of A~i.e., replace A by

A

^{= AD} (where D is a diagonal matrix)~without changing the solution x to Ax = b in any essential way. If x is the solution to Ax = b, then y = D-1x is the solution of

Ay

= (AD)(D-IX) = b. In general, we refer to a scaling of a matrix A if A is replaced by Dl ADz, where D1, Dz are diagonal matrices.

The example shows t~at it is not reasonable to propose a particular choice of pivots unless assumptions about the scaling of a matrix are made. Unfor- tunately, no one has yet determined satisfactorily how to carry out scaling so that partial pivot selection is numerically stable for any matrix A. Practical experience, however, suggests the following scaling for partial pivoting:

Choose Dl and D2 so that

n n

L !aik!

~

L !ajl!

k=l j=l

holds approximately for all i, 1= 1,2, ... , n in the matrix

A

= Dl ADz. The sum of the absolute values of the elements in the rows (and the columns) of

A

should all have about the same magnitude. Such matrices are said to be equilibrated. In general, it is quite difficult to determine Dl and Dz ^{so that} Dl ADz is equilibrated. Usually we must get by with the following: Let Dz = I, Dl = diag(sb ... , sn}, where

1

then for

it

= Dl AD2 , it is true at least that

L /

n ^{aik /} = 1 for i = 1, 2, ... , n.

k~1

Now, instead of replacing A by

it,

i.e., instead of actually carrying out the transformation, we replace the rule for pivot selection instead in order to avoid the explicit scaling of A. The pivot selection for thejth elimination step A(j-1) -> A(j) is given by the following:

(4.5.3). Determine r ~ j so that

/a(j.-l)/s _{r) r} = max/aV-I)/s. _{lJ l} =1=

a

_,

i?-j

and take a~)-l) as the pivot.

The example above shows that it is not sufficient, in general, to scale A prior to carrying out partial pivot selection by making the largest element in absolute value within each row and each column have roughly the same magnitude:

(4.5.4 ) max

/a

ik / ~ max

/ajl/

for all i, 1= 1,2, ... ,

n.

k

For, if the scaling matrices

DI =

[~OO ~ j,

are chosen, then the matrix A of our example (4.5.1) becomes

- [1 1 ]

A = DIADz = 1 0.005'

The condition (4.5.4) is satisfied, but inexact answers will be produced, as before, if

all

⁼ 1 is used as a pivot.

We would like to make a detailed study of the effect of the rounding errors which occur in Gaussian elimination or direct triangular decomposi- tion (see Section 4.1). We assume that the rows of the n x n matrix A are already so arranged that A has a triangular decomposition of the form A = LR. Hence, Land R can be determined using the formulas of (4.1.12). In fact, we only have to evaluate expressions of the form

bn

:=n(C -

^{alb1 -}

.~~

^- an-Ibn-I),

which were analyzed in (1.4.4)-( 1.4.11). Instead of the exact triangular de- composition LR = A, (4.1.12) shows that the use of floating-point arithmetic will result in matrices

L =

^(lik),

R =

^(fik) for which the residual F

:=

(/;d =

A - LR is not zero in general. Since, according to (4.1.13), the jth partial sums

are exactly the elements of the matrix A(j), which is produced instead of A(j)

of (4.1.6) from the jth step of Gaussian elimination in floating-point arith- metic, the estimates (1.4.7) applied to (4.1.12) yield

(4.5.5)

I

^{hkl = I aik -}

jtll;/jk

^I

~

¹

~P;ps ~t: ^{(Iali) I +} Il;jll rjkl}

^{for k}

~

^i,

Ihil

⁼

la

ki -

_j=l ±~/jil~

^{1 eps}

_-

_eps

[1al:i-1)1 + _j=l ii1(laM)I + IT,.jllfj ;!}]

for k > i.

Further, (4.5.6)

since the first j

+

1 rows of ^A(j) [or ^A(j) of (4.1.6)] do not change in the subsequent elimination steps, and so they already agree with the correspond- ing rows of R. We assume, in addition, that Il;k

I

~ 1 for all i, k (which is satisfied, for example, when partial or complete pivoting is used). Setting

a·'=max J

I

^ali) lk ,

I

i, k

O<i<n-l

it follows immediately from (4.5.5) and (4.5.6) that

Ihkl

^{~ 1} _{- eps} eps ^(ao

+

2al

+

2a2

+ ... +

2ai-2

+

ai-d

~ 2(i - 1}a eps for k ~ i, 1 - eps

(4.5.7)

I

_{hk/ ~ 1} ^eps _(ao

₊

_2al

_{+ ... +}

_2ak-2

₊

_2ak-d

- eps

~ 2ka eps for k < i.

1 - eps

For the matrix F, then, the inequality

(4.5.8)

IF /

^{~ 2a -1 ---=ep_s_}

- eps

000 1 1 1 1 2 2 1 2 3 1 2 3

o

1 2 3

o

1 2 3 n-1 n-1

holds, where 1 F 1 := ( 1 fjk ^{1 ).} If a has the same order of magnitude as ao , that is, if the matrices AU) do not grow too much, then the computed matrices Li{

form the exact triangular decomposition of the matrix A - F, which differs little from A. Gaussian elimination is stable in this case.

The value a can be estimated with the help of ao = max" s

1

a,s

I.

^For

partial pivot selection it can easily be shown that ak-l ~2kao,

and hence that a ~ 2n-1ao. This bound is much too pessimistic in most cases; however, it can be attained, for example, in forming the triangular decomposition of the matrix

1 0 0 1

-1 1 0 1

-1 -1 0 1

A=

-1 -1 0 1

-1 -1 1 1

-1 -1 -1 1

Better estimates hold for special types of matrices. For example in the case of upper Hessenberg matrices, that is, matrices of the form

A~ r:

^x

n

the bound a ~ (n - l)ao can be shown. (Hessenberg matrices arise in eigenvalue problems.)

For tridiagonal matrices

1X1 P2 0

A=

Y2

0 _Yn Pn IXn

it can even be shown that

a = max 1 ak ¹ ~ 2ao

k

holds for partial pivot selection. Hence, Gaussian elimination is quite numerically stable in this case.

For complete pivot selection, Wilkinson (1965) has shown that ak-l ~ f(k)ao

with the function

This function grows relatively slowly with k:

k 10 20 50 100

f(k)

19 67 530 3300

Even this estimate is too pessimistic in practice. Up until now, no matrix has been found which fails to satisfy

k = 1, 2, ... , n - 1,

when complete pivot selection is used. This indicates that Gaussian elimin- ation with complete pivot selection is usually a stable process. Despite this, partial pivot selection is preferred in practice, for the most part, because:

(1) Complete pivot selection is more costly than partial pivot selection. (To compute ^A(i), the maximum from among (n - i

+

1)2 elements must be determined instead of n - i

+

1 elements as in partial pivot selection.) (2) Special structures in a matrix, i.e. the band structure of a tridiagonal

matrix, are destroyed in complete pivot selection.

If the weaker estimates (1.4.11) are used instead of (1.4.7), then the follow- ing bounds replace those of (4.5.5) for the

hk:

Ihkl ~·l_e:~

^eps

fjt/I~jllfjkl - Ifikl],

^{k ~ i,}

eps

[i. - - J

l1kd ~

1-

n'

^eps

j~/Ilkjllrjil '

^{k ~ i}

⁺

^1,

or

(4.5.9)

I

^F

I ^~.

^{1 eps} _-n'eps ^[

I LID I ^R I - I ^R I ],

where