Block SOR methods for rank-decient least-squares problems
C.H. Santos∗, B.P.B. Silva, J.Y. Yuan1
Departamento de Matematica - UFPR, Centro Politecnico, CP: 19.081, CEP: 81531-990, Curitiba, Parana, Brazil
Received 31 August 1997; received in revised form 2 July 1998
Abstract
Many papers have discussed preconditioned block iterative methods for solving full rank least-squares problems. How-ever very few papers studied iterative methods for solving rank-decient least-squares problems. Miller and Neumann (1987) proposed the 4-block SOR method for solving the rank-decient problem. Here a 2-block SOR method and a 3-block SOR method are proposed to solve such problem. The convergence of the block SOR methods is studied. The optimal parameters are determined. Comparison between the 2-block SOR method and the 3-block SOR method is given also. c 1998 Elsevier Science B.V. All rights reserved.
AMS classication:65F10
Keywords:Preconditioned iterative method; Block SOR method; Convergence; Least-squares problem; Rank-decient least-squares problem; Optimal parameter
1. Introduction
We frequently meet rank-decient least-squares problems dened by
min
x∈rnkAx−bk2; (1.1)
whereAism×n matrix withm¿nand rank (A)¡ n, when we solve the actual problems in statistics, economics, genetics, dierential equations, and image and signal processing. Recently, least-squares methods have got more attention in application areas, and also in applied mathematics society.
Many papers have studied the solvers of the rank-decient least-squares problems. The general and ecient ways are singular-value decomposition and QR decomposition. It is well-known that the iterative methods are preferable for large sparse problems. For rank-decient least-squares problems,
∗Corresponding author.
1Partially supported by grant 301035/93-8 of CNPq, Brazil.
there are very few papers to study the iterative methods for solving rank-decient problems. Miller and Neumann discussed the SOR method to solve the problem (1987) [6]. They partitioned the matrix A into four parts and then applied the SOR method to solve the new system. In fact, the big problem for iterative methods to solve the rank-decient problems is the determination of the rank of the matrix A. The problem is simple in theory, but not in applications.
Recently, Bjorck and Yuan [2] proposed three algorithms to nd linearly independent rows of the matrix A by LU factorization, Luo, et al. [4] used the basic solution method (Benzi and Meyer called direct-projection method [1]) to nd rank of A, and Silva and Yuan [10] applied the QR decomposition in column-wise to nd the set of linearly independent rows of A. Those methods motivate us to consider iterative methods for solving the rank-decient problems. We can nd the desired preconditioner for rank-decient least-squares problems by their algorithms.
Since there are many solutions for the rank-decient problems and we are generally interested in just the minimum 2-norm solution, we shall derive a new system for the minimum 2-norm solution. We shall study block SOR methods to solve the new equation by preconditioning technique. Our block SOR methods are dierent from Miller and Neumann’s SOR method. We shall study the convergence and optimal relaxation parameter of the block SOR methods, and give some comparison result between our block SOR methods. Like full rank case, we show the 2-block SOR method always converges for certain value of the relaxation parameter.
The outline of this paper is as follows. The new system of normal equation for the rank-decient least-squares problems is derived in Section 2. The 3-block SOR method is proposed in Section 3. Its convergence and optimal parameter are also discussed in the section. The convergence theory shows the 3-block SOR method just converges for some certain case with certain conditions. In Section 4, the 2-block SOR method is studied. The convergence of the 2-block SOR method tells us the 2-block SOR method is always convergent for rank-decient problems with certain relaxation parameter like the block SOR method for full rank problems. The optimal parameter for the 2-block SOR method is given as well in Section 4. The comparison between the 2-2-block SOR method and the 3-block SOR method is investigated. The result shows the 2-block SOR method is better than the 3-block SOR method for rank-decient problems as full rank problems. Throughout the paper, we always assume that the matrix A with rank (A) =k ¡ n has the partition
A=
A1 A2
; (1.2)
where A1∈Rk×n is full row rank, and A2 ∈R(m−k)×n.
2. New system of normal equation
For the treatment of problem (1.1), we need the following lemma.
Lemma 2.1. Assume that the matrix A has the structure of (1.2). Then
N(A) =N(A1) and R(AT) =R(AT
1); (2.1)
Proof. ∀x∈N(A), it follows from Ax= 0 that A1x = 0, that is x∈N(A1). Hence
N(A)⊂N(A1): (2.2)
Since A has the structure of (1.2) and rank (A) =k, there exists one nonsingular matrix P such that
PA=
Since the minimum 2-norm solution x of the rank-decient least-squares problem of (1.1) is in
R(AT), that is in R(AT
1) by Lemma 2.1, We can consider the transformation x=AT
1y; (2.5)
where y ∈Rk, to obtain the minimum 2-norm solution of the problem of (1.1). Substituting (2.5)
into (1.1), we obtain the new system of the normal equation of the problem (1.1) for rank-decient case as follows:
A1ATAAT1y=A1ATb: (2.6)
By the structure of A in (1.2), we can rewrite the system (2.6) as an augmented form
have the same partition as A in (1.2). Therefore we have shown the following theorem.
Theorem 2.2. Suppose that the matrix A in the problem of (1.1) has the partition of (1.2) with
rank (A) = k = rank (A1) ¡ n. Then the minimum 2-norm solution x of the rank-decient
least-squares problems of (1.1) is given by
x=AT1y;
Moreover, the general solution x of the rank-decient problem of (1.1) is given by
x=AT
1y+z; ∀z ∈N(A1):
The remaining of the paper will discuss block SOR methods to solve the system (2.8). For the SOR method, there is a well-known eigenvalue functional between the Jacobi iteration matrix and the SOR iteration matrix.
Theorem 2.3. (Young [11]). Assume that the Jacobi iteration matrix J is weakly p-cyclic and
consistently ordered. Suppose is eigenvalue of J. If satises the functional
(+!−1)p=!ppp−1; (2.9)
then is eigenvalue of the SOR iteration matrix L!. On the contrary, if is eigenvalue of L!,
and satises (2.9), then is eigenvalue of J.
3. The 3-block SOR method
3.1. The 3-block SOR algorithm
Consider the 3-block diagonal matrix D = diag (A1AT1; I; A1AT1) for the system of (2.8). We can obtain the 3-block SOR method as follows:
Hence, we propose the following 3-Block SOR algorithm for solving problem (1.1) as follows.
Algorithm 3.1.
1. Set y(0);
2. Calculate r(0)1 and r(0)2 ;
3. Calculate the iterative parameter !;
3.2. Convergence and optimal factor
Now we shall discuss the convergence of the 3-block SOR method for rank-decient least-squares problems. In this case, the Jacobi matrix J3 is
J3 =
0 0 −(A1AT 1)
−1 −A2AT1 0 0
0 −(A1AT 1)
−1A1AT
2 0
: (3.3)
Lemma 3.1. The eigenvalues of J3 in (3.3) lie in the real interval
I3 := [−2=3;0]; (3.4)
where =kA2AT1(A1AT1) −1k
2.
Proof. Suppose that is eigenvalue of J and the corresponding eigenvector is (xT; yT; zT)T. Then by the denition of eigenvalue, there is the following relation:
J3
x y z
=
x y z
;
which is
−(A1AT1)
−1z=x; −A2AT1x=y; −(A1AT1)
−1A
1AT2y=z:
(3.5)
It follows from (3.4) that there is
−PTPz =3z; (3.6)
where
P=A2AT1(A1AT1)
−1: (3.7)
Since PTP is symmetric and semi-positive denite, and 3 is eigenvalue of −PTP, −26360.
Similar to the proof given by Niethammer, et al. [7], we can show the following convergence result for the 3-block SOR Algorithm 3.1.
Theorem 3.2. The 3-block SOR method of (3.1) for rank-decient least-squares problem of (1.1)
converges for ! in some interval if and only if
¡33=2
≈5:196152; (3.8)
where =kA2AT1(A1AT1) −1k
In particular; if ¡23=2≈2:828427; then the method of (3.1) converges for all ! such that
and diverges for all other values of !. Furthermore, if ¡33=2, then optimal 3-block SOR
relax-ation parameter !(3)b is given by
!b(3)= 3 [(+
√
1 +2)1=3+ (
−√1 +2)1=3]; (3.11)
and the spectral radius of the optimum iterative matrix L(3)
!(3)b is given by
4. The 2-block SOR method
4.1. The 2-block SOR algorithm
Consider the 2-block diagonal matrix D2 as
D2=
for the system of (2.8). Then the corresponding Jacobi matrix J2 is
J2 =
−1 and the 2-block SOR method dened as follows:
Algorithm 4.1.
1. Set y(0);
2. Calculate r(0)1 and r(0)2 ;
3. Calculate the iterative parameter !;
4. Iterate for l= 1;2; : : : ; until “Convergence”
y(l+1)= (1
−!)y(l)+!(A 1AT1)
−1(b
1−r(1l));
r2(l+1)= (1−!)r2(l)+!(b2−A2A1Ty(l)) +A2AT1(y(l+1)−y(l));
r1(l+1)= (1−!)r1(l)−!(A1AT1) −1A
1AT2r (l+1) 2 :
4.2. Convergence and optimal factor
Lemma 4.1. Let be eigenvalue of the Jacobi matrix J2 in (4.1) for the 2-block SOR method.
Then the spectrum of J2 is pure imaginary; that is;
260:
Proof. Since
J2 2 =
0 (A1AT1)
−1P 0 0 −PPT 0 0 0 −PTP
;
the results is true.
Lemma 4.2. (Young [11]). Let x be any root of the real quadratic equation x2−bx+c= 0. Then
|x|¡1 if and only if
|c|¡1; |b|¡1 +c;
where b and c are real.
The convergence result of the 2-block SOR method for the rank decient problem follows from these two lemmas and Theorem 2.3.
Theorem 4.3. The 2-block SOR method of (4.2) for the rank-decient least-squares problem of
(1.1) converges for all ! in the interval
0¡ ! ¡ 2
1 +; (4.4)
where =kPk2 =kA2AT1(A1AT1)
−1k2 =(A
1)(A1; A2), (A1) = (1)
A1=
(k)
A1 and (A1; A2) =
(1)
A2=
(k)
A1;
and A2, (i)
A is the ith singular value of A. Furthermore; the optimum SOR relaxation parameter
!(2)b is given by
!b(2)= 2
1 +√1 +2; (4.5)
and the spectral radius of the optimum iteration matrix L(2)
!(2)b is given by
L(2)
!(2)b
=
1 +√1 +2
2
= ! (2)
b
2
!2
: (4.6)
Proof. The proof of the convergence region is the same as the proof of Markham et al. in [5]. We will omit the proof. Similar to the proof of Markham et al. [5] or of Young [11], we can show the optimal factor results. Here we want to give the dierent proof on this issue. We dene the functions s; t : [0; ]−→R as
s() = [2(1−!)−! 2]
2 (4.7)
and
t() = [!
2−4(1−!)]
4 : (4.8)
It follows from the eigenvalue functional of (2.9) in Theorem 2.3 that
=s(2)+ −
q
t(2)!||: (4.9)
It is evident that s() and t() are an ane decreasing function and an increasing function of , respectively, and !2||t() is quadratic function of . If t ¡0, then ||6|s|+!||p
|t|; if t ¿0, then ||6|s|+!||√t. It is well-known that (L(2)
! )¿|1−!|. Then if there exists ! such that
||6|1−!| for all and all 2 ∈[0; 2], then we obtain the optimal factor !(2)
b and spectral radius
(L
!(2)b ). Since = 0 as = 0, we can get ||6|1−!| if we restrict s(2) = −(1−!) and
t(2) = 0. In view of the above consideration, the following conditions
!22
−4(1−!) = 0 (4.10)
and
(1−!)−!22=2 =−(1−!) (4.11)
imply ||6|1−!|. If the condition of (4.10) holds, then the condition of (4.11) must hold. Then the optimal factor !b is the unique positive root of Eq. of (4.10). Solving (4.10), we obtain (4.5)
and (4.6).
5. Comparison and conclusion
Theorem 5.1. The spectral radii (L(2)
!(2)b )and (L
(3)
!(3)b ) for the 2-block optimum SOR method and
the 3-block optimum SOR method dened in (4.6) and (3.12), respectively, for the rank-decient
least-squares problem of (1.1) satisfy the following inequality for all ¿0:
(L(2)
!(2)b )¡ (L
(3)
!(3)b ): (5.1)
Analyzing two algorithms, we know that Algorithms 3.1 and 4.1 have the same requirement on storage and multiplications. In terms of the convergence theory, we know that the 2-block SOR method always converges for certain interval of !, while the 3-block SOR method does not. It follows from Theorem 5.1 that the 2-block optimum SOR method converges faster than the 3-block optimum SOR method. Therefore, the 2-block SOR method is better than the 3-block SOR method for the rank-decient problem of (1.1) like the full rank case.
References
[1] M. Benzi, C.D. Meyer, A direct projection method for sparse linear systems, SIAM J. Sci. Comput. 16(1995) 1159–1176.
[2] A. Bjorck, J.Y. Yuan, Preconditioner for Least Squares Problems by LU Decomposition, Lecture Notes on Scientic Computing, Springer, Berlin, 1998, accepted.
[3] R. Freund, A note on two block SOR methods for sparse least-squares problems, LAA, 88/89 (1987) 211–221. [4] Z. Luo, B.P.B. Silva, J.Y. Yuan, Notes on direct projection method, Technical Report, Department of Mathematics,
UFPR, 1997.
[5] T.L. Markham, M. Neumann, R.J. Plemmons, Convergence of a direct-iterative method for large-scale least-squares problems, LAA 69(1985) 155–167.
[6] V.A. Miller, M. Neumann: Successive overrelaxation methods for solving the rank decient least squares problem, LAA 88/89(1987) 533–557.
[7] W. Niethammer, J. de Pillis, R.S. Varga, Convergence of block iterative methods applied to sparse least squares problems, LAA 58(1985) 327–341.
[8] C.H. Santos, Iterative methods for rank decient least squares problems, Master Dissertation, UFPR, Curitiba, Brazil, 1997.
[9] B.P.B. Silva, Topics on numerical linear algebra, Master Dissertation, UFPR, Curitiba, Brazil, 1997.
[10] B.P.B. Silva, J.Y. Yuan, Preconditioner for least squares problems by QR decomposition, Technical Report, Department of Mathematics, UFPR, Brazil, 1997.
[11] D.M. Young, Iterative Solution of large linear systems, Academic Press New York, 1971.
