Linear Algebra

Universit degli Studi di Trento Dipartimento dIngegneria Civile ed Ambientale Prof. Dr.-Ing.
Michael Dumbser
Numerical Methods for Free Surface Hydrodynamics Solution of linear algebraic equation systems
Winter School on Advanced Numerical Methods Part I - Free Surface Hydrodynamics

Solution of Linear Algebraic Equation Systems
Prof. Dr.-Ing. Michael Dumbser
M. Dumbser 1 / 18
30 /01/2012
Universit degli Studi di Trento Dipartimento dIngegneria Civile ed Ambientale Prof. Dr.-Ing. Michael Dumbser
Solution of Systems of Linear Algebraic Equations

A standard task in linear algera is finding the solution of a system of linear equations of the general form
A X ! B
with an N x N matrix Example:
A and an N x M matrix B
x1 x2 4 x1 4 x2 x1 2 x2
x3 4 x3 x3
! 1 ! 2 ! 1 1 B ! 2 1
1 1 1 A ! 4 4 4 1 2 1
M. Dumbser 2 / 18
N ! 3, M ! 1
Solution of Systems of Linear Algebraic Equations

Recall: The system of linear equations
T T A x ! b
may have One unique solution if the determinant of A is not zero:
A {0
An infinite number of solutions if D, the determinant of A is zero and the determinants Dk of all matrices formed by the matrix A in which one column k has been substituted by b are also zero:
D ! A ! 0 Dk ! Aij bi ! 0 with
j { k k _,2..., N a 1
No solution if at least one of the determinants Dk is non zero.
A !0
M. Dumbser 3 / 18
A b { 0
ij i
with
j { k for some k _,2,..., N a 1
Main Classes of Solution Algorithms for Systems of Linear Algebraic Equations

Task: solution of the linear algebraic equation system Direct Methods Solve the problem directly using A and b and produce eventually some intermediate matrices Gau elimination / LU-decomposition general, stable when pivoting is used QR-decomposition general, more expensive than Gau, problem with roundoff errors Cholesky-decomposition only for symmetric positive definite matrices, but cheaper than Gau Thomas algorithm only for tri-diagonal matrices, very cheap: computational effort proportional to N
M. Dumbser 4 / 18
A X ! B
Iterative Methods (not exhaustive) Solve the problem starting with an initial guess x0 and generating a sequence of iterative solutions xi+1=L(xi) Jacobi, Gau-Seidel and SOR method simple and general, do not converge for all matrices A. Conjugate Gradient (CG) method Works only for symmetric positive definite matrices. Optimal method (convergence to exact solution after N iterations). GMRES method [Saad & Schulz, 1986] Works for general matrices. Optimal method (convergence to exact solution after N iterations). Much more complex and expensive than CG.
Linear Algebra
The systematic solution of this problem is given by the so-called Gau-algorithm, named after the German mathematician Carl-Friedrich Gau (1777-1855) Define an auxiliary matrix C as
C ! A, B
Now the matrix C is modified by a sequence of operations on its rows to transform its left part into the unit matrix. The admissible row-operations are: (1) addition / subtraction of multiples of rows (2) multiplication / division of a row by a constant factor (3) exchange of rows When the left part of the matrix C has been transformed into the unit matrix, the right part contains the solution X of the equation system.
M. Dumbser 5 / 18
The Gau-Algorithm Without Pivoting I

In detail, the simple Gau-algorithm without pivoting for an N x N matrix A and an N x M matrix B proceeds in two parts as follows: Part 1 is a forward loop, which transforms the matrix A into a normalized upper triangular form. DO i = 1, N (1) IF C(i,i) = 0 THEN Find row j > i with nonzero element in column i, i.e. where C(j,i) is not 0 IF there is no such row, THEN Matrix is singular. EXIT. ELSE Exchange rows i and j in matrix C. END IF END IF (2) Divide the entire row i by the diagonal element C(i,i). (3) DO j = i+1, N Subtract the multiple C(j,i) times row i from row j to eliminate all entries in column i of row j. END DO END DO
M. Dumbser 6 / 18
The Gau-Algorithm Without Pivoting II

In detail, the simple Gau-algorithm without pivoting for an N x N matrix A and an N x M matrix B proceeds in two parts as follows: Part 2 is a very simple backward loop, which transforms the matrix A into the unit matrix. DO i = N, 1, -1 DO j = i-1, 1, -1 Subtract the multiple C(j,i) times row i from rows j in order to eliminate all entries in column i of those rows j. END DO END DO All operations have been performed on the augmented matrix C. The type of row-operations is completely determined by the left part if matrix C, i.e. by the original matrix A. The right part, i.e. matrix B, is transformed with the same operations in a passive way. At the end of the Gau-algorithm, the right part of the matrix C, i.e. where B was located initially, we find the solution X of the equation system. To compute the inverse of a matrix using the Gau-algorithm, the right hand side B is simply set to the the N x N unit matrix, since we have:
M. Dumbser 7 / 18
A X ! I
X !A
1
Memory and Computational Effort of the Gau Algorithm

The memory storage requirements for the matrix C in double precision are
s ! 8N ( N M )
The computational effort (i.e. the necessary number of floating point operations) for the execution of the Gau algorithm is proportional to N to the cube. This makes the algorithm very expensive and not feasible for very large matrices (apart from the above-mentioned memory storage problem).
# flop ~ N 3
In practice, very often iterative solution methods are used, in particular when the matrix A is sparse, i.e. it contains a very large number of zero elements.
M. Dumbser 8 / 18
Problems of the Gau-Algorithm Without Pivoting

In theory, the Gau-algorithm presented before is able to solve any linear system with non-singular matrix A. In practice, however, when using finite precision computer hardware, we encounter problems with roundoff errors. Recall: On finite precision computer hardware real numbers x are represented digitally as floating point numbers, for example according to the IEEE 754 double precision standard
f 1
s
as
r
x ! s f 2e 1s V ! 1
p
with
e ! E B, B ! 2 r 1 1
Hence, there exist very small but finite numbers V < I for which
M. Dumbser 9 / 18
with I =2-(p+1)=2-(52+1) 1e-16 for double precision arithmetics.
Problems of the Gau-Algorithm Without Pivoting

Example: Consider the following equation system
Vx1 2 x2 ! 1 x1 x2 ! 1 2 1 V 1 1 1 V 1
1 V 1 1 1 } x1 ! } , x2 ! 2 V 2 2 V 2
1 0 1 V 1 V 2V
Applying the Gau-algorithm without pivoting on finite precision hardware yields
V C ! 1
2 1 1 1
2 1 1 V V 2 1 0 1 1 V V
2 V 1
1 0 0 1
M. Dumbser 10 / 18
1 2 1 V V 2 1 2
1 0 0 1
1 1 V V 1 2
1 0 0 1 0 1 2
Remedy: The Gau-Algorithm With Pivoting

Example: Consider the following equation system
Vx1 2 x2 ! 1 x1 x2 ! 1
1 V 1 1 1 } x1 ! } , x2 ! 2 V 2 2 V 2
Applying the Gau-algorithm with pivoting on finite precision hardware yields
V C ! 1
2 1 1 1
1 V
1 1 2 1 1 1 0 2 0 1 1 2
1 1 1 0 2 V 1 V
1 1 1 0 2 1
1 1 1 1 0 1 2
M. Dumbser 11 / 18
The Gau-Algorithm With Column Pivoting I

The modified first part of the Gau-algorithm with pivoting reads as follows. The second part remains unchanged! DO i = 1, N (1) Find row j > i with p=|C(j,i)| = max(|C(i+1:N,i)|) IF p=0 THEN Matrix is singular. EXIT. ELSE Exchange rows i and j in matrix C. END IF (2) Divide the entire row i by the diagonal element C(i,i). (3) FOR j from i+1 to N Subtract the multiple C(j,i) times row i from row j to eliminate all entries in column i of row j. END DO END DO
M. Dumbser 12 / 18
A Special Case of Gau-Eliminiation: The Thomas-Algorithm

In the case where the matrix A is tridiagonal, i.e. with the special structure
b1 a2 0 A! 0 0 0
c1 0 b2 c2 a3 b3 0 ai 0 0 0 0
0 0 c3 bi a N 1 0
0 0 0 0 0 0 0 ci bN 1 cN 1 aN bN
d1 d2 B ! di d N 1 d N
there is a very fast and efficient special case of the Gau-algorithm, called the Thomas-algorithm. Since it is a variation of the Gau-algorithm, the Thomas algorithm is a direct method to solve general linear tridiagonal systems. As the original Gau-algorithm, it proceeds in two stages, one forward elimination and one back-substitution.
M. Dumbser 13 / 18
The Thomas-Algorithm
Part I: Forward elimination.
c1 :! c1 / b1 ; d1 :! d1 / b1
DO i = 2, N
K :! 1 / bi ci 1ai ci :! ci K d i :! d i ai d i 1 K
END DO
Part II: Back substitution
xN ! d N
DO i = N-1, 1, -1
xi ! d i ci xi 1
END DO
M. Dumbser 14 / 18
The Conjugate Gradient Method

For large matrices A, the number of operations of the Gau algorithm grows with N3. For large but sparse matrices (many entries are zero), it may be more efficient to use an iterative scheme for the solution of the system. For symmetric, positive definite matrices A, a very efficient algorithm can be constructed, which is the so-called conjugate gradient method, which we have already seen for nonlinear, multi-variate optimization. We start with some definitions: (1) A matrix is symmetric positive definite, if
A !A
T
and
T T T x T A x " 0 x { 0 T T A x ! b as
(2) Two vectors u and v are called conjugate with respect to matrix A if
T T uT A v ! 0
(3) If we knew N conjugate directions pj (j = 1 .. N), we could write the solution x of
T T T T x ! E1 p1 E 2 p2 ... E N pN
The coefficients E are given by:
TT pT b Ej ! Tj T pT A p j j
M. Dumbser 15 / 18
(4) The term
T T T r ! b A x is called the residual

The idea of the CG method is to minimize the functional
T 1 T T TT g ( x ) ! x T Ax b T x 2 T T T x ! xk P p
Its gradient is:
T T T xg T! Ax b ! r xx
Using the steepest descent method in direction p, we obtain the minimum of g starting from xk as follows:
TT xg xx xg T T T ! ! T pT Ax b ! 0 xP xP xx T T T T pT AxK Pp b ! 0
TT T TT pT b AxK pT rk P! T T ! T T p T Ap pT Ap
M. Dumbser 16 / 18

A preliminary version of the conjugate gradient method now reads as
T x0 ! 0
T T T r0 ! b Ax0
T T p0 ! r0
DO k = 0... N - 1
T T vk ! Apk TT T pk rk Pk ! T T 1D minimization along direction pk T pk vk T T T Move point x into the minimum along direction pk xk 1 ! xk Pk pk T T T T T T rk 1 ! b Axk 1 ! b Axk Pk vk Compute new residual ( = negative gradient T T X = steepest descent direction) rk 1 ! rk Pk vk T T k p j Ark 1 T T T Do not move in the steepest descent direction, pk 1 ! rk 1 T T p j but compute a new conjugate direction pk+1 that pj Apj j !0 takes into account all the previous search
From definition (3) we know that the algorithm will terminate with the exact solution after at most N iterations
ENDDO
M. Dumbser 17 / 18
directions. As in nonlinear minimization, this guarantees faster convergence.

After some algebra, the final conjugate gradient method is given as follows
T X x0 ! b
T T T r0 ! b Ax0
T T p0 ! r0
TT E 0 ! r0T r0
DO k = 0... N - 1
T T vk ! Apk Ek Pk ! T T T pk vk T T T xk 1 ! xk Pk pk T T X rk 1 ! rk Pk vk TT E k 1 ! rkT 1rk 1 T T E T pk 1 ! rk 1 k 1 pk Ek

ENDDO
M. Dumbser 18 / 18

Linear Algebra

Transféré par

Informations du document

Description originale:

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Linear Algebra

Transféré par

Droits d'auteur :

Formats disponibles

Universit degli Studi di Trento Dipartimento dIngegneria Civile ed Ambientale Prof. Dr.-Ing.

Winter School on Advanced Numerical Methods Part I - Free Surface Hydrodynamics

Solution of Systems of Linear Algebraic Equations

Solution of Systems of Linear Algebraic Equations

may have One unique solution if the determinant of A is not zero:

No solution if at least one of the determinants Dk is non zero.

j { k for some k _,2,..., N a 1

Main Classes of Solution Algorithms for Systems of Linear Algebraic Equations

The Gau-Algorithm Without Pivoting I

The Gau-Algorithm Without Pivoting II

Memory and Computational Effort of the Gau Algorithm

Problems of the Gau-Algorithm Without Pivoting

with I =2-(p+1)=2-(52+1) 1e-16 for double precision arithmetics.

Problems of the Gau-Algorithm Without Pivoting

Applying the Gau-algorithm without pivoting on finite precision hardware yields

Remedy: The Gau-Algorithm With Pivoting

Applying the Gau-algorithm with pivoting on finite precision hardware yields

The Gau-Algorithm With Column Pivoting I

A Special Case of Gau-Eliminiation: The Thomas-Algorithm

Part II: Back substitution

The Conjugate Gradient Method

(3) If we knew N conjugate directions pj (j = 1 .. N), we could write the solution x of

(4) The term

T T T r ! b  A x is called the residual

The Conjugate Gradient Method

Its gradient is:

The Conjugate Gradient Method

directions. As in nonlinear minimization, this guarantees faster convergence.

The Conjugate Gradient Method

T T vk ! Apk Ek Pk ! T T T pk vk T T T xk 1 ! xk  Pk pk T T X rk 1 ! rk  Pk vk TT E k 1 ! rkT 1rk 1  T T E T pk 1 ! rk 1  k 1 pk Ek

Vous aimerez peut-être aussi

T T T r ! b A x is called the residual

T T vk ! Apk Ek Pk ! T T T pk vk T T T xk 1 ! xk Pk pk T T X rk 1 ! rk Pk vk TT E k 1 ! rkT 1rk 1 T T E T pk 1 ! rk 1 k 1 pk Ek