Interval Arithmetic; Statistical Roundoff Estimation

Error Analysis

1.5 Interval Arithmetic; Statistical Roundoff Estimation

In view of (1.4.9) this ensures the existence of quantities ej with (1.4.1O) C = ^.-1 I ajbAl + j . ej} + a.b.(l + (n - 1 + <5')e.},

j=l

I

^e·

I .::;

^_-,ep,-s_

} 1 - n . eps' <5' ^:=

1

0 if a. = 1,

\ 1 otherwise.

For r = c - a1 b1 - a2 b2 - ... - a. b. we have consequently (1.4.11) ^Irl'::;l _{- n} eps ^. _eps

r·-

^{Ijla1 jbj}l+(n-1+<5'}la.b.l·

¹

j= 1

In particular, (1.4.8) reveals the numerical stability of our algorithm for comput- ing P •. The roundoff error rxm contributes the amount

c - a1b1 - a2b2 - ... - ambm rxm •

to the absolute error in

P •.

This, however, is at most equal to

I

^{c· ee} - a1b1e.'a~ ... - ambme· .. ¹

& (

iel

+

;t

1 la;b;j )eps

'" I

^a.

I '

which represents no more than the influence of the input errors ee and e., of c and a;, i = 1, ... , m, respectively, provided I ee I, I e., I .::; eps. The remaining roundoff errors J1k and <5 are similarly shown to be harmless.

The numerical stability of the above algorithm is often shown by interpreting (1.4.1 O) in the sense of backward analysis: The computed approximate solution b. is the exact solution of the equation

whose coefficients

a.

^{:= a.(l} + (n - 1 + <5')e.}

have been changed only slightly from their original values a j. This kind of analysis, however, involves the difficulty of having to define how large n can be so that errors of the form ne,

I

^e

I .::;

eps can still be considered as being of the same order of magnitude as the machine precision eps.

1.5 Interval Arithmetic; Statistical Roundoff

sequently the number of individual roundoff errors, is very large, and the corresponding algorithm is too complicated to permit the estimation of the total effect of all roundoff errors in this fashion.

A technique known as interval arithmetic offers an approach to determin- ing exact upper bounds for the absolute error of an algorithm, taking into account all roundoff and data errors. Interval arithmetic is based on the realization that the exact values for all real numbers a E R which either enter an algorithm or are computed as intermediate or final results are usually not known. At best one knows small intervals wich contain a. For this reason, the interval-arithmetic approach is to calculate systematically in terms of such intervals

a

= [a', a"],

bounded by machine numbers a', a" E A, rather than in terms of single real numbers a. Each unknown number a is represented by an interval

a

⁼ [a', a"] with a E

a.

The arithmetic operations (I) E {E9,

e,

®, CD} be- tween intervals are defined so as to be compatible with the above interpreta- tion. That is,

c = a

^{® jj} is defined as an interval (as small as possible) satisfying

c::>

{a r b

I

^{a E}

a

^{and bE}

^b}

and having machine number endpoints.

In the case of addition, for instance, this holds if E9 is defined as follows:

[c', c"]

=

[a', a"] E9 [b', b"], where

C' := max{y' ^E A

I

^{y' ~ a'}

+

^b'},

C" := min{y" ^E A

I

y" ?= a"

+

^b"},

with A denoting again the set of machine numbers. In the case of multiplica- tion ®, assuming, say, a' > 0, b' > 0,

[c', c"] = [a', a"] ® [b', b"]

can be defined by letting

c' ^:= max{y' ^E A

I

^{y' ~ a'} x b'}, e" ^:= min{y" ^E A

I

^{y" ?= a" x b"}.}

Replacing, in these and similar fashions, every quantity by an interval and every arithmetic operation by its corresponding interval operation-this is readily implemented on computers-we obtain interval algorithms which produce intervals guaranteed to contain the desired exact solutions. The data for these interval algorithms will be again intervals, chosen to allow for data errors.

It has been found, however, that an uncritical utilization of interval arith- metic techniques leads to error bounds which, while certainly reliable, are in

most cases much too pessimistic. It is not enough to simply substitute inter- val operations for arithmetic operations without taking into account how the particular roundoff or data errors enter into the respective results. For example, it happens quite frequently that a certain roundoff error c impairs some intermediate results U 1, •.. , Un of an algorithm considerably,

I ~:i ^I ~

^{1 for i =} 1, ... , n,

while the final result y = f(u1, .•• , un) is not strongly affected,

I ~~ ^I ~

^1,

even though it is calculated from the highly inaccurate intermediate values

U b ... , Un: the algorithm shows error damping.

EXAMPLE 1. Evaluate y = cp(x) ⁼ x3 - 3x² + 3x = ((x - 3) x x + 3) x x using Horner's scheme:

W :=v + 3, y:=w X x.

The value x is known to lie in the interval

X E X = [0.9, 1.1].

Starting with this interval and using straight interval arithmetic, we find

u

⁼

x e

^[3,3] = [-2.1, -1.9],

v

=

u

^®

x

= [-2.31, -1.71], W =

v e

^{[3, 3] =} [0.69, 1.29],

Y

= W ®

x

= [0.621, 1.419].

The interval

y

is much too large compared to the interval {cp(x)/x E x} = [0.999,1.001], which describes the actual effect of an error in x on cp(x).

EXAMPLE 2. Using just ordinary 2-digit arithmetic gives considerably more accurate results than the interval arithmetic suggests:

x = 0.9 x = 1.1

u -2.1 -1.9

v -1.9 -2.1

w 1.1 0.9

y 0.99 0.99

For the successful application of interval arithmetic, therefore, it is not sufficient merely to replace the arithmetic operations of commonly used algorithms by interval operations: It is necessary to develop new algorithms producing the same final results but having an improved error-dependence pattern for the intermediate results.

EXAMPLE 3. In Example 1 a simple transformation of cp(x) suffices:

y=cp(x)=1+(x-1t

When applied to the corresponding evaluation algorithm and the same starting interval

x

= [0.9, 1.1], interval arithmetic now produces the optimal result:

UI

'=x e

[1,1] = [-0.1,0.1], Uz :=UI@UI = [-0.01,0.01], U3 ,= Uz @ UI = [ -0.001, 0.001],

Y

^{,= U3} EB [1, 1] = [0.999, 1.001]'

As far as ordinary arithmetic is concerned, there is not much difference between the two evaluation algorithms of Example 1 and Example 3. Using two digits again, the results are practically identical to those in Example 2:

x = 0.9 x = 1.1

UI -0.1 0.1

Uz ^{0.Q1 0.Q1}

U3 -0.001 0.001

Y 1.0 1.0

For an in-depth treatment of interval arithmetic the reader should consult, for instance, Moore (1966) or Kulisch (1969).

In order to obtain statistical roundoff estimates [Rademacher (1948)], we assume that the relative roundoff error [see (1.2.6)] which is caused by an elementary operation is a random variable with values in the interval [ - eps, eps]. Furthermore we assume that the roundoff errors 8 attributable to different operations are independent random variables. By Jie we denote the expected value and by

a;

the variance of the above round-off distribution.

They satisfy the general relationship

a; ⁼

^{E(8 - E(8W}

⁼

^{E(82 ) - (E(8W}

⁼

^{Jie2 - Ji;.}

Assuming a uniform distribution in the interval [- eps, eps], we get 1 eps

a; =

E(8^{2 )}

= - - r

t² dt

= t

eps2

=:8

2.

2 eps . -eps

(1.5.1) Jie:= E(8)

=

0,

Closer examinations show the roundoff distribution to be not quite uniform [see Sterbenz (1974), Exercise 22, p. 122]. It should also be kept in mind that

the ideal roundoff pattern is only an approximation to the roundoff patterns observed in actual computing machinery, so that the quantities Ile and a;

may have to be determined empirically.

The results x of algorithms subjected to random roundoff errors become random variables themselves with expected values Ilx and variances a~,

connected again by the basic relation

a; = E(x - E(xW = E(x2) - (E(xW = Ilx2 - Il~·

The propagation of previous roundoff effects through elementary operations is described by the following formulas for arbitrary independent random variables x, y and constants 11., f3 E IR:

ll,x±Py = E(rxx

±

^f3y) = rxE(x)

±

^f3E(y) = rxllx

±

f3lly, (1.5.2) a;X±Py = E((rxx

±

^{f3y)2) - (E(rxx}

±

^f3yW

= rx2E(x - E(xW

+

^f32E(y - E(yW = rx2a~

+

f32 a;.

The first of the above formulas follows by the linearity of the expected-value operator. It holds for arbitrary random variables x, y. The second formula is based on the relation E(xy) = E(x)E(y), which holds whenever x and yare independent. Similarly, we obtain for independent x and y

Ilxxy = E(x x y) = E(x)E(y) = Ilxlly,

(1.5.3) _a~ x y = E(x x y - E(x )E(y

nz

^{= Ilx21ly2 - 1l~1l;}

22 22 22

= axay

+

Ilxay

+

^Ilyax'

EXAMPLE. For calculating y = a2 - b² (see Example 2 in Section 1.3) we find, under the assumptions (l.5.1), E(a) = a, O"~ = 0, E(b) = b, O"t = 0, and using (1.5.2) and (1.5.3), that

'11 = a 2(1 +

I:d,

E('1d = a2,

0";,

= a4i;2, '12 = b2(l + 1:2)' E('12) = b2,

0";2

= b4i;2,

Y = ('11 - '12)(1 + 1:3)' E(y) = E('11 - '12)E(1 + 1:3) = a2 - b2 ('11> '12, <'3 are assumed to be independent),

= (a4 + ^b4)i;4 + [(a2 _ b2)2 + ^a4 + ^b4]i;2.

Neglecting i;4 compared to i;2 yields

0";

== ((a2 - b2)2 + ^a4 + ^b4)i;2.

F or a := 0.3237, b ^:= 0.3134, eps := 5 x 10 - 4 (see Example 5 in Section 1.3), we find

O"y == 0.144i; = 0.000 0415

which is close in magnitude to the true error ~y = 0.000 01787 for 4-digit arithmetic.

Compare this with the error bound 0.000 10478 furnished by (1.3.17).

We denote by M(x) the set of all quantities which, directly or indirectly, have entered the calculation of the quantity x. If M(x) (1 M(y) =1=

0

for the algorithm in question, then the random variables x and yare in general dependent.

The statistical roundoff error analysis of an algorithm becomes extremely complicated if dependent random variables are present. It becomes quite easy, however, under the following simplifying assumptions:

(1.5.4)

(a) The operands of each arithmetic operation are independent random variables.

(b) In calculating variances all terms of an order higher than the smallest one are neglected.

(c) All variances are so small that for elementary operations ^'l: in first-order approximation, E(x 'l: y) == E(x) 'l: E(y) = J.1x 'l: J.1y •

If in addition the expected values J.1x are replaced by the estimated values x, and relative variances &; ^:= a;/J.1; ~ a;/x² are introduced, then from (1.5.2) and (1.5.3) [compare (1.2.6), (1.3.5)],

z = fl(x

±

y):

z = fl(x x y):

z

=

fl(x / y):

&; ==

(~r

^&;

⁺ (~r

^&;

⁺

^£2,

&2 ..:.. &2 _{z - x}

+

⁸² _y

+

/;2 _,

8;

⁼⁼

8; ⁺ 8; ⁺ /;2.

It should be kept in mind, however, that these results are valid only if the hypotheses (1.5.4), in particular (1.5.4a), are met.

It is possible to evaluate above formulas in the course of a numerical computation and thereby to obtain an estimate of the error of the final results. As in the case of interval arithmetic, this leads to an arithmetic of paired quantities (x, 8;) for which elementary operations are defined with the help of the above or similar formulas. Error bounds for the final results r are then obtained from the relative variance &; , assuming that the final error distribution is normal. This assumption is justified inasmuch as the distribu- tions of propagated errors alone tend to become normal if subjected to many elementary operations. At each such operation the nonnormal roundoff error distribution is superimposed on the distribution of previous errors.

However, after many operations, the propagated errors are large compared to the newly created roundoff errors, so that the latter do not appreciably affect the normality of the total error distribution. Assuming the final error distribution to be normal, the actual relative error of the final result r is bounded with probability 0.9 by 2&,.

EXERCISES FOR CHAPTER 1

1. Show that with floating-point arithmetic of t decimal places rd(a)=-I-+e a with lel'::;5xlO-¹

holds in analogy to (1.2.2). [In parallel with (1.2.6), as a consequence, fl(a

*

b) = (a

*

b)/(1 + e) with lei'::; 5 x 10-¹ for all arithmetic operations

* = +, -, x, l]

2. Let a, b, c be fixed-point numbers with N decimal places after the decimal point, and suppose 0< a, b, c < 1. A substitute product a * b is defined as follows: Add 1O-N/2 to the exact product a . b, and then delete the (N + 1)-st and subsequent digits.

(a) Give a bound for I(a * b) * c - abel.

(b) By how many units of the Nth place can (a

*

b)

*

c and a

*

(b

*

c) differ?

3. Evaluating

L

^aj in floating-point arithmetic may lead to an arbitrarily large

i= 1

relative error. If, however, all summands aj are of the same sign, then this relative error is bounded. Derive a crude bound for this error, disregarding terms of higher order.

4. Show how to evaluate the following expressions in a numerically stable fashion:

1 1 - x

1 + 2x - 1 + x for

I

x

I

~ 1,

J

^{x +}

^~

^-

J

^{x -}

^~

^forx

^~

^1,

1 - cos x

for x =1= 0,

I

x

I

^{~ l.}

x

5. Suppose a computer program is available which yields values for arcsin y in floating-point representation with t decimal mantissa places and for

I

y

I .::;

1 subject to a relative error e with Ie

I .::;

5 ^X 10-1. In view of the relation

. x

arctan x = arCSIn

J

_{1 + x} ^{2 '}

this program could also be used to evaluate arctan x. Determine for which values x this procedure is numerically stable by estimating the relative error.

6. For given z, the function tan(z/2) can be computed according to the formula

Z

(1 -

^{cos Z)1/2}

tan-=+ .

2 - l+cosz

Is this method of evaluation numerically stable for z ~ 0, z ~ n/2? If necessary, give numerically stable alternatives.

7. The function

is to be evaluated for 0 ::;; cp ::;; n/2, 0 < kc ::;; 1.

The method

avoids the calculation of cos cp and is therefore faster. Compare this with the direct evaluation of the original expression for f(cp, kc ) with respect to numerical stability.

8. For the linear function f (x) := a + bx, where a =F 0, b =F 0, compute the first derivative Dh f(O)

=

1'(0)

=

b by the formula

Dh f(O) = f(h) ~[( - h)

in binary floating-point arithmetic. Suppose that a and b are binary machine numbers, and h a power of 2. Multiplication by h and division by 2h can be therefore carried out exactly. Give a bound for the relative error of Dh f(O). What is the behavior of this bound as h --+ O?

9. The square root ± (u + iv) of a complex number x + iy with y =F 0 may be calculated from the formulas

u= +Jx+F+7

_{- 2 '}

v = 2u. y

Compare the cases x ~ 0 and x < 0 with respect to their numerical stability.

Modify the formulas if necessary to ensure numerical stability.

10. The variance 8² of a set of observations x ^{b .•• , Xn} is to be determined. Which of the formulas

8² =

~1

_{n -} ⁽ _i=l

± ^{xf - nx}

^{2 ),}

is numerically more trustworthy?

11. The coefficients an br(r = 0, ... , n) are, for fixed x, connected recursively:

bn :=an ;

for r = n - 1, n - 2, ... , 0: br^:= xbr^{+ 1} + ar.

(a) Show that the polynomials

• •

A(z):=

I

^a,z', B(z):= Ib,z'-1

r=O r=1

satisfy

A(z) = (z - x)B(z) + bo.

(b) Suppose A(x) = bo is to be calculated by the recursion (*) for fixed x in floating-point arithmetic, the result being b'o. Show, using the formulas (compare Exercise 1)

the inequality

fl(u + v) = - -u+v I + (J' fl(u· v) = - -u·v

I +

1t'

I(JI

^~ eps,

11t I

^~ eps,

IA(x)-b'ol ~-1 eps (2eo-lb'oI), - eps

where eo is defined by the following recursion:

e.:= lanl/2;

for r = n - 1, n - 2, ... , 0; e,:=

I

x

I

^{er+ 1} +

I

b;

I·

Hint: From

fl( b' ) xb;+ 1

p,:= ^{X ,+1} = 1

+ ^{7tr+ 1 r} = n - 1, ... ,0,

derive

b; := fl(p, + a,) = PI' + + (J, a, = xb;+ 1 + a, + Or

• b' 1t,+ 1

u, = -x ^,+1 1 + 1t,+ 1 - (J,b; (r = n - 1, ... , 0);

then show b'o =

D=o

^(ak + Ok)Xk, O. := 0, and estimate

D

^10k

II

^x

I

^k.

12. Assuming Earth to be spherical, two points on its surface can be expressed in Cartesian coordinates

Pi = [Xi, Yi, Zi] = [r cos rJ.i cos Pi, r sin rJ.i cos Pi, r sin Pi]' i = 1,2, where r is the earth radius and rJ.i, Pi are the longitudes and latitudes of the two points Pi, respectively. If

T

cos b = Pl;2 = COS(rJ.l - rJ.2) cos PI cos P2 + sin PI sin P2,

r

then rb is the great-circle distance between the two points.

(a) Show that using the arccos function to determine (j from the above expres- sion is not numerically stable.

(b) Derive a numerically stable expression for (j.

References for Chapter 1

Ashenhurst, R. L., Metropolis, N.: Dnnormalized floating-point arithmetic. J. Assoc.

Comput. Mach. 6, 415-428, 1959.

Bauer, F. L.: Computational graphs and rounding error. SIAM J. Numer. Anal. 11, 87-96, 1974.

---, Heinhold, J., Samelson, K., Sauer, R.: Moderne Rechenanlagen. Stuttgart:

Teubner 1965.

Henrici, P.: Error Propagation for Difference Methods. New York: Wiley 1963.

Knuth, D. E.: The Art of Computer Programming. Vol. 2. Seminumerical Algorithms.

Reading, Mass.: Addison-Wesley 1969.

Kulisch, D.: Grundziige der Intervallrechnung. In: D. Laugwitz: Uberblicke Math- ematik 2, 51-98. Mannheim: Bibliographisches Institut 1969.

Moore, R. E.: Interval Analysis. Englewood Cliffs, N.J.: Prentice-Hall 1966.

Neumann, J. von, Goldstein, H. H.: Numerical inverting of matrices. Bull. Amer.

Math. Soc. 53,1021-1099,1947.

Rademacher, H. A.: On the accumulation of errors in processes of integration on high-speed calculating machines. Proceedings of a symposium on large-scale digital calculating machinery. Annals C omput. Labor. Harvard U niv. 16, 176-185, 1948.

Scarborough, J. B.: Numerical Mathematical Analysis. Baltimore: Johns Hopkins Press 1930, 2nd edition 1950.

Sterbenz, P. H.: Floating Point Computation. Englewood Cliffs, N.J.: Prentice-Hall 1974.

Wilkinson, J. H.: Error analysis of floating-point computation. Numer. Math. 2, 219-340, 1960.

- - - : Rounding Errors in Algebraic Processes. New York: Wiley 1963.

- - - : The Algebraic Eigenvalue Problem. Oxford: Clarendon Press 1965.