Some Results on the Range-Restricted GMRES Method

In this paper we reconsider the range-restricted GMRES (RRGMRES) method for solving nonsymmetric linear systems. We first review an important result for the usual GMRES method. Then we give an example to show that the range-restricted GMRES method does not admit such a result. Finally, we give a modified result for the range-restricted GMRES method. We point out that the modified version can be used to show that the range-restricted GMRES method is also a regularization method for solving linear ill-posed problems.

• Nonsymmetric Linear System
• Krylov Subspace Method
• Arnoldi Process
• GMRES
• RRGMRES

1. Introduction

We consider the problem of finding a solution x ∈ X to the nonsymmetric linear systems [1] [2]

A x = b , (1)

where X is a real separable Hilbert space with inner product ( ⋅ , ⋅ ) and norm ‖   ⋅   ‖ = ( ⋅ , ⋅ ) 1 / 2 and A : X → X is a bounded linear operator. We further assume that A is invertible on its range R ( A ) , that is, for any b ∈ R ( A ) , the Equation (1) has a unique solution x ∈ X .

The generalized minimal residual (GMRES) method, proposed by Saad and Schultz [3] , is one of the most popular iterative methods for solving large linear systems of equations with a square nonsymmetric matrix. It is an extension of the minimal residual method (MINRES) for symmetric systems. In the past four decades, numerous variants of GMRES appeared. In 1988, Walker [4] proposed the Householder GMRES, which uses an algorithm that uses the House-holder reflections to orthogonalize the basis vectors and thus has better numerical stability. Saad [5] in 1993 proposed to accelerate the GMRES by using the variable preconditioner at each iteration step. Morgan [6] established the GMRES with deflated restarting by deflating the eigenvalues of small magnitude, which maybe hampers the convergence of GMRES.

For a nonzero vector v ∈ X , the Krylov subspace K m ( A , v ) is defined by

K m ( A , v ) = span { v , A v , A 2 v , ⋯ , A m − 1 v } ,   m = 1,2, ⋯ .

Let the initial guess x 0 = 0 . At the m-th step, the GMRES method obtains an approximation x m , where x m solves the linear least-squares problem

min x ∈ K m ( A , b ) ‖ b − A x ‖ .

In the implementations of GMRES, the Arnoldi process [7] is used to establish an orthonormal basis of the Krylov subspace K m ( A , v ) . The Arnoldi process based on the modified Gram-Schmidt procedure is described as follows.

Algorithm 1 Arnoldi process

1) Let v 1 = v / ‖ v ‖ .

2) For j = 1,2, ⋯ , m

3) w j = A v j .

4) For i = 1,2, ⋯ , j

5) h i j = ( v i , w j ) .

6) w j = w j − v i h i j .

7) End For

8) h j + 1, j = ‖ w j ‖ .

9) v j + 1 = w j / h j + 1, j .

10) End For

Obviously, if

dim K m ( A , v ) = dim K m + 1 ( A , v ) = m ,

then w m = 0 and the Arnoldi process breaks down after the basis { v j } j = 1 m of K m ( A , v ) has been determined.

Calvetti, Lewis, and Reichel have showed that the GMRES method has the following important property ( [8] , Lemma 2.3).

Theorem 1. Let the linear operator A : X → X be invertible on R ( A ) . Assume that dim K m ( A , b ) = dim K m + 1 ( A , b ) = m . Then the iterate x m generated by the GMRES method applied to the Equation (1) with the initial approximate solution x 0 = 0 satisfies

A x m = b .

Conversely, assume that A x m = b with x m ∈ K m ( A , b ) . Then the Arnoldi process breaks down after the orthonormal basis { v j } j = 1 m of K m ( A , b ) has been determined.

The range-restricted GMRES (RRGMRES) method [9] [10] [11] [12] is also an important iterative method for solving general nonsymmetric linear systems. This method uses the Krylov subspace K m ( A , A b ) and has several advantages over the GMRES method especially for linear ill-posed problems. Since K m ( A , A b ) ⊂ R ( A ) , the RRGMRES method restricts the computational solution to R ( A ) .

However, the following example shows that the second half of Theorem 1 does not hold for the RRGMRES method.

Example. Let

A = [ 0 0 0 1 1 0 0 0 0 1 0 0 0 0 1 0 ] ,   b = [ 0 0 0 1 ] .

The matrix A is invertible, and thus the equation A x = b has a unique solution. The solution is

x = [ 0 0 1 0 ] .

We have

A b = [ 1 0 0 0 ] ,   A 2 b = [ 0 1 0 0 ] ,   A 3 b = [ 0 0 1 0 ] ,   A 4 b = [ 0 0 0 1 ] .

Clearly, x ∈ K 3 ( A , A b ) . However, since dim K 4 ( A , A b ) = 4 , the Arnoldi process does not break down after the orthonormal basis { v 1 , v 2 , v 3 } of K 3 ( A , A b ) has been generated.

2. Main Results

In this section we shall show that a result slightly different from Theorem 1 holds for the RRGMRES method. For deducing the result, we require the following lemma. Although its proof is similar to that of ( [8] , Lemma 2.2), we include the proof for completeness.

Lemma 2. Assume that the linear operator A : X → X is invertible on its range R ( A ) . Then

dim A K m ( A , A b ) = dim K m ( A , A b ) ,   m = 1 , 2 , 3 , ⋯ .

Proof. It is obvious that dim A K m ( A , A b ) ≤ dim K m ( A , A b ) . Now we assume that dim A K m ( A , A b ) < dim K m ( A , A b ) . Then, there is a w ∈ K m ( A , A b ) , w ≠ 0 such that A w = 0 . Since A is invertible on its range R ( A ) , it follows that A w = 0 if and only if w = 0 . This contradiction shows that dim A K m ( A , A b ) = dim K m ( A , A b ) .

We are in a position to present the main result of this note.

Theorem 3. Let the linear operator A : X → X be invertible on R ( A ) . Assume that dim K m ( A , A b ) = dim K m + 1 ( A , A b ) = m . Then the iteration x m generated by the RRGMRES method applied to the Equation (1) with the initial approximate solution x 0 = 0 satisfies

A x m = b .

Conversely, we assume that A x m = b with x m ∈ K m ( A , A b ) and dim K m ( A , A b ) = m . Then the Arnoldi process breaks down after the orthonormal basis { v j } j = 1 m of K m ( A , A b ) or the orthonormal basis { v j } j = 1 m + 1 of K m + 1 ( A , A b ) has been generated.

Proof. It is clear that K m ( A , A b ) ⊂ K m + 1 ( A , A b ) . Under the assumption that dim K m ( A , A b ) = dim K m + 1 ( A , A b ) = m , we have

K m ( A , A b ) = K m + 1 ( A , A b ) .

It follows from Lemma 2 that dim A K m ( A , A b ) = m = dim K m + 1 ( A , A b ) , which together with A K m ( A , A b ) ⊂ K m + 1 ( A , A b ) shows that A K m ( A , A b ) = K m + 1 ( A , A b ) . Thus, we have K m ( A , A b ) = A K m ( A , A b ) and A b ∈ K m ( A , A b ) = A K m ( A , A b ) . It shows that there is a w m ∈ K m ( A , A b ) = A K m ( A , A b ) such that A b = A w m , i.e., A ( b − w m ) = 0 . Since A is invertible on R ( A ) , it follows that b − w m = 0 . Note that w m ∈ A K m ( A , A b ) . Thus, there exists an x m ∈ K m ( A , A b ) such that A x m = w m = b .

Conversely, we assume that there exists an x m ∈ K m ( A , A b ) such that A x m = b . If dim K m ( A , A b ) = dim K m + 1 ( A , A b ) = m , the result holds naturally, i.e., the Arnoldi process breaks down after the orthonormal basis { v j } j = 1 m of K m ( A , A b ) has been generated. Thus, we only need to consider the case dim K m + 1 ( A , A b ) = m + 1 . Since x m ∈ K m ( A , A b ) , it follows that b ∈ A K m ( A , A b ) ⊂ K m + 1 ( A , A b ) . Then, A b ∈ A K m + 1 ( A , A b ) , which shows that dim A K m + 1 ( A , A b ) = dim K m + 2 ( A , A b ) and K m + 2 ( A , A b ) = A K m + 1 ( A , A b ) . Moreover, by Lemma 2, we obtain

dim A K m + 1 ( A , A b ) = dim K m + 1 ( A , A b ) .

Therefore,

dim K m + 2 ( A , A b ) = dim K m + 1 ( A , A b ) = m + 1

and K m + 1 ( A , A b ) = K m + 2 ( A , A b ) , which proves that the Arnoldi process breaks down after the orthonormal basis { v j } j = 1 m + 1 of K m + 1 ( A , A b ) has been generated.

We note that the first half of Theorem 3 has been given out in ( [13] , Theorem 2.3) as A is a nonsingular matrix. However, the second half of Theorem 3 is a new result, which shows a main difference between GMRES and RRGMRES.

The example from the previous section can verify the second part of Theorem 3. In this example, x ∈ K 3 ( A , A b ) and dim K 4 ( A , A b ) = 4 . Thus, the Arnoldi process in RRGMRES don’t break down until the orthonormal basis of K 4 ( A , A b ) has been generated.

We can validate the other case of the second part of Theorem 3 by setting the coefficient matrix A as the identity matrix. In this case, x = b , x ∈ K 1 ( A , A b ) = K 2 ( A , A b ) . Thus, the Arnoldi process in RRGMRES breaks down after the orthonormal basis of K 1 ( A , A b ) has been generated.

In linear discrete ill-posed problems, the right-hand side vector of the nonsymmetric linear systems (1) is usually contaminated by an error. We denote the perturbed linear system by

A x = b δ , (2)

where e = b − b δ is an error vector, and ‖ e ‖ ≤ δ with δ > 0 . If ‖ e ‖ or its fairly accurate estimate is known, the discrepancy principle is used to estimate a regularization parameter. When the GMRES method is applied to solve the perturbed linear system (2), the iterations will be terminated as soon as

‖ b δ − A x m δ ‖ ≤ α δ , (3)

where x m δ is the m δ -th iterate, and α is an appropriate positive number.

The following theorem [8] shows that the usual GMRES method is a regularization method for solving linear ill-posed problems.

Theorem 4. Let δ satisfy 0 < δ ≤ ε with ε being an appropriate positive number, and let ‖ e ‖ ≤ δ . Choose the initial solution to be x 0 = 0 . Let x m δ be determined by the usual GMRES method with the discrepancy principle (3). Then

l i m δ → 0 sup ‖ b − b δ ‖ ≤ δ ‖ x − x m δ ‖ = 0,

where x is the solution of (1).

We point out that Theorem 1 is an essential result for proving Theorem 4, see [8] .

Extensive numerical experiments have shown that the RRGMRES method may yield better approximate solutions than the usual GMRES method, see, for example, [9] [11] [14] [15] [16] . However, as far as we know, analysis of the regularization property of the RRGMRES method has not been done theoretically. We find out that by making use of Theorem 3 and following almost the same arguments in [8] , it can be shown that when the associated error-free right-hand side lies in a finite-dimensional Krylov subspace, the RRGMRES method is also a regularization method for solving linear ill-posed problems. So, we present the result in the following theorem and omit its proof.

Theorem 5 Let δ satisfy 0 < δ ≤ ε with ε being an appropriate positive number, and let ‖ e ‖ ≤ δ . Choose the initial solution to be x 0 = 0 . Let x m δ be determined by the RRGMRES method with the discrepancy principle (3). Then

l i m δ → 0 sup ‖ b − b δ ‖ ≤ δ ‖ x − x m δ ‖ = 0,

where x is the solution of (1).

3. Conclusion

The RRGMRES method uses the range-restricted Krylov subspace, and has some advantages over the usual GMRES method for linear ill-posed problems. In this paper, we have shown that the result about the break-down of the Arnoldi process in the RRGMRES may be different from the one in the usual GMRES. The result can be used to show that the RRGMRES is a regularization iterative method.

Acknowledgements

This research was funded by the Natural Science Foundation of Hunan Province under grant 2017JJ2102.

Conflicts of Interest

The author declares no conflicts of interest regarding the publication of this paper.

References

1. Colton, D. and Kress, R. (2013) Inverse Acoustic and Electromagnetic Scattering Theory. Springer-Verlag, Berlin. https://doi.org/10.1007/978-1-4614-4942-3

2. Diao, H., Li, H., Liu, H. and Tang, J. (2023) Spectral Properties of an Acoustic-Elastic Transmission Eigenvalue Problem with Applications. Journal of Differential Equations, 371, 629-659. https://doi.org/10.1016/j.jde.2023.07.002

3. Saad, Y. and Schultz, M. (1986) GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems. SIAM Journal on Scientific Computing, 7, 856-869. https://doi.org/10.1137/0907058

4. Walker, H. (1988) Implementation of the GMRES Method Using Householder Transformations. SIAM Journal on Scientific Computing, 9, 152-163.https://doi.org/10.1137/0909010

5. Saad, Y. (1993) A Flexible Inner-Outer Preconditioned GMRES Algorithm. SIAM Journal on Scientific Computing, 14, 461-469. https://doi.org/10.1137/0914028

6. Morgan, R. (2002) GMRES with Deflated Restarting. SIAM Journal on Scientific Computing, 24, 20-37. https://doi.org/10.1137/S1064827599364659

7. Saad, Y. (2003) Iterative Methods for Sparse Linear Systems. SIAM, Philadelphia.https://doi.org/10.1137/1.9780898718003

8. Calvetti, D., Lewis, B. and Reichel, L. (2002) On the Regularizing Properties of the GMRES Method. Numerische Mathematik, 91, 605-625. https://doi.org/10.1007/s002110100339

9. Calvetti, D., Lewis, B. and Reichel, L. (2000) GMRES-Type Methods for Inconsistent Systems. Linear Algebra and Its Applications, 316, 157-169.https://doi.org/10.1016/S0024-3795(00)00064-1

10. Cao, Z. and Wang, M. (2002) A Note on Krylov Subspace Methods for Singular Systems. Linear Algebra and Its Applications, 350, 285-288.https://doi.org/10.1016/S0024-3795(02)00310-5

11. Bellalij, M., Reichel, L. and Sadok, H. (2015) Some Properties of Range Restricted GMRES Methods. Journal of Computational and Applied Mathematics, 290, 310-318.https://doi.org/10.1016/j.cam.2015.05.008

12. Lin, Y., Bao, L. and Cao, Y. (2013) Augmented Arnoldi-Tikhonov Regularization Methods for Solving Large-Scale Linear Ill-Posed Systems. Mathematical Problems in Engineering, 2013, 1-11. https://doi.org/10.1155/2013/548487

13. Baglama, J. and Reichel, L. (2007) Augmented GMRES-Type Methods. Numerical Linear Algebra with Applications, 14, 337-350. https://doi.org/10.1002/nla.518

14. Buccini, A., Onisk, L. and Reichel, L. (2023) Range Restricted Iterative Methods for Linear Discrete Ill-Posed Problems. Electronic Transactions on Numerical Analysis, 58, 348-377. https://doi.org/10.1553/etna_vol58s348

15. Calvetti, D., Lewis, B. and Reichel, L. (2001) On the Choice of Subspace for Iterative Methods for Linear Discrete Ill-Posed Problems. International Journal of Applied Mathematics and Computer Science, 11, 1069-1092.

16. Neuman, A., Reichel, L. and Sadok, H. (2012) Implementations of Range Restricted Iterative Methods for Linear Discrete Ill-Posed Problems. Linear Algebra and Its Applications, 436, 3974-3990. https://doi.org/10.1016/j.laa.2010.08.033