Introduction to Simulation - Lecture 7

Krylov-Subspace Matrix Solution Methods

Part II
Jacob White

Thanks to Deepak Ramaswamy, Michal Rewienski,

and Karen Veroy

Outline
Reminder about GCR
Residual minimizing solution
Krylov Subspace
Polynomial Connection

Review Eigenvalues and Norms

Induced Norms
Spectral mapping theorem

Estimating Convergence Rate

Chebychev Polynomials

Preconditioners
Diagonal Preconditioners
Approximate LU preconditioners

With Normalization

Generalized
Conjugate Residual
Algorithm

r 0 = b Ax 0

For j = 0 to k-1
pj = r j
Residual is next search direction
For i = 0 to j-1
Orthogonalize
T
p j p j ( Mp j ) ( Mpi ) pi Search Direction
pj

pj

( Mp ) ( Mp )
T

x j +1 = x j + ( r
r j +1 = r j

Normalize

) ( Mp ) p
( r ) ( Mp ) M p
j T

Update Solution

j T

Update Residual

SMA-HPC 2003 MIT

Generalized
Conjugate Residual
Algorithm
1) orthogonalize the Mr i ' s

rp00

rp11

With Normalization
Algorithm Steps by Picture

pr 22

rp33

2) compute the r minimizing solution x k

r k+1

SMA-HPC 2003 MIT

rk
M pk

First Few Steps

Generalized
Conjugate Residual
Algorithm

r0
First search direction r = b Mx = b, p0 =
Mr 0
0

Residual minimizing x1 =
solution
Second Search
Direction

( r 0 ) Mp0 p0
T

r1 = b Mx1 = r 0 1Mr 0
p1 =

r1 1,0 p0

M r1 1,0 p0

SMA-HPC 2003 MIT

Generalized
Conjugate Residual
Algorithm
Residual minimizing
solution

First few steps

Continued

x 2 = x1 + ( r1 ) Mp1 p1
T

Third Search Direction

r 2 = b Mx 2 = r 0 2,1Mr 0 2,0 M 2 r 0

p2 =

r1 2,0 p0 2,1 p1

M r1 2,0 p0 2,1 p1

SMA-HPC 2003 MIT

The kth step of GCR

Generalized
Conjugate Residual
Algorithm
k 1

pk = r ( Mr k )
k

j =0

pk =

pk
Mpk

k = ( r k ) ( Mpk )
T

x k +1 = x k + k pk

r k +1 = r k k Mpk

( Mp ) p
j

Orthogonalize and
normalize search
direction

Determine optimal stepsize in

kth search direction
Update the solution
and the residual

SMA-HPC 2003 MIT

Polynomial view

Generalized
Conjugate Residual
Algorithm

If j 0 for all j k in GCR, then

1) span { p0 , p1 ,..., pk } = span r 0 , Mr 0 ,..., Mr k

2) x k +1 = k ( M ) r 0 , k is the k th order poly

2
minimizing r k +1
2
k +1
k +1
0
3) r = b Mx = r M k ( M ) r 0
= ( I M k ( M ) ) r 0 k +1 ( M ) r 0
th
where k +1 ( M ) r 0 is the ( k + 1) order poly
2
minimizing r k +1 subject to k +1 ( 0 ) =1
SMA-HPC 2003 MIT

Residual Minimization

Krylov Methods

Polynomial View

If x k +1 span r 0 , Mr 0 ,..., Mr k

minimizes r k +1

2
2

= k ( M )r , k is the k order poly

2
minimizing r k +1
2
k +1
k +1
2) r = b Mx = ( I M k ( M ) ) r 0 =k +1 ( M ) r 0
th
where k +1 ( M ) r 0 is the ( k + 1) order poly
k +1 2
minimizing r
subject to k +1 ( 0 ) =1

1) x

k +1

th

Polynomial Property only a function of

solution space and residual minimization
SMA-HPC 2003 MIT

Krylov Methods
Nodal Formulation

No-leak Example
Insulated bar and Matrix

Incoming Heat

T (1)

T (0)
Near End
Temperature

Discretization

m
SMA-HPC 2003 MIT

Far End
Temperature

2 1

1 2
Nodal

1 Equation
Form

1
2

10

Krylov Methods
Nodal Formulation

leaky Example
Conducting bar and Matrix

T (1)

T (0)
Near End
Temperature

Discretization

Far End
Temperature

2.01 1

1 2.01
Nodal

Equation

1 Form

1 2.01

m
SMA-HPC 2003 MIT

11

GCR Performance(Random Rhs)

10

R
E
S
I
D
U
A
L

10

10

10

10

Insulating
-1

Leaky
-2

-3

-4

10

20

Iteration

30

40

50

60

Plot of log(residual) versus Iteration

SMA-HPC 2003 MIT

12

GCR Performance(Rhs = -1,+1,-1,+1.)

0

10

R
E
S
I
D
U
A
L

-1

10

-2

10

-3

10

Insulating

-4

10

Leaky
-5

10

10

15

20

25

Iteration

30

35

40

45

50

Plot of log(residual) versus Iteration

SMA-HPC 2003 MIT

13

Krylov Methods

Residual Minimization
Optimality of poly

Residual Minimizing Optimality Property

r k +1 k+1 ( M )r 0 k+1 ( M ) r 0
k+1 is any k th order poly such that k+1 ( 0 ) =1

Therefore
Any polynomial which satisfies
the constraints can be used to
get an upper bound on

r k +1
r0

SMA-HPC 2003 MIT

14

Induced Norms

Matrix Magnification
Question

Suppose y = Mx
How much larger is y than x?
OR

How much does M magnify x?

SMA-HPC 2003 MIT

15

Vector Norm
Review

Induced Norms
L2 (Euclidean) norm :
x

i=1

L1 norm :
x

xi

xi

i=1

< 1

< 1

L norm :
x

= max
i

xi

< 1

SMA-HPC 2003 MIT

16

Standard Induced
l-norms

Induced Matrix
Norms

Definition:
M l max

Mx
x

Examples

M
SMA-HPC 2003 MIT

max
1

= max

i
j =1

max

M ij

M
j
i =1

ij

x l =1

Mx

Max Column
Sum
Max Row
Sum

17

Standard Induced
l-norms continued

Induced Matrix
Norms

= m ax
j

i =1

Why? Let x =

= m ax
i

Why? Let

[1

j =1

x =

= max abs column sum

ij

0
ij

[ 1

= max abs column sum

1]

Not So easy to compute

SMA-HPC 2003 MIT

As the algebra on the slide shows the relative changes in the solution x is bounded
by an A-dependent factor times the relative changes in A. The factor

|| A1 || || A ||
was historically referred to as the condition number of A, but that definition has
been abandoned as then the condition number is norm-dependent. Instead the
condition number of A is the ratio of singular values of A.
cond ( A) =

max( A)
min( A)

Singular values are outside the scope of this course, consider consulting Trefethen
& Bau.

18

Useful
Eigenproperties

Spectral Mapping
Theorem

Given a polynomial

f ( x ) = a0 + a1 x + + a p x p

Apply the polynomial to a matrix

f ( M ) = a0 + a1M + + a p M p
Then

spectrum ( f ( M ) ) = f ( spectrum ( M ) )

SMA-HPC 2003 MIT

19

Krylov Methods

u N
=
u1

eigenvectors of M

k (M )

u
1

u N

u
1

Convergence Analysis
Norm of matrix polynomials

k ( 1 )

u N

k ( N )
1

Cond(U)

u
1

k ( 1 )

u N

k ( N )

condition number of
M's eigenspace
SMA-HPC 2003 MIT

20

Krylov Methods
k ( 1 )

Convergence Analysis
Norm of matrix polynomials

= max x =1
k ( N )
2

( ) x
k

= max i k ( i )
k ( M )

cond (V ) max i k ( i )

SMA-HPC 2003 MIT

21

Krylov Methods

Convergence Analysis
Important Observations

1) A residual minimizing Krylov subspace algorithm

converges to the exact solution in at most n steps

Proof: Let n ( x ) = ( x 1 )( x 2 ) ... ( x n )

where i ( M ) . Then, max i n ( i ) = 0,

n ( M ) = 0 and therefore r n = 0

2) If M has only q distinct e-values, the residual

minimizing Krylov subspace algorithm converges
in at most q steps

Proof: Let q ( x ) = ( x 1 )( x 2 ) ... ( x q )

SMA-HPC 2003 MIT

22

Convergence for M =MT

Krylov Methods

Residual Polynomial

If M = MT then
1) M has orthonormal eigenvectors

cond (V ) =

u
1

u N

u
1

u N

=1

k ( M ) = max i k ( i )

2) M has real eigenvalues

If M is postive definite, then ( M ) > 0

SMA-HPC 2003 MIT

23

Residual Poly Picture for Heat Conducting Bar Matrix

No loss to air (n=10)

* = evals(M)
- = 5th order poly
- = 8th order poly

SMA-HPC 2003 MIT

24

Residual Poly Picture for Heat Conducting Bar Matrix

No loss to air (n=10)

Keep k ( i ) as small as possible:

Strategically place zeros of the poly
SMA-HPC 2003 MIT

25

Convergence for M =MT

Krylov Methods

Polynomial Min-Max Problem

Consider ( M ) [ min , max ] , min > 0

Then a good polynomial ( pk ( M ) is small)

can be found by solving the min-max problem

min kth order max x [min ,max ] pk ( x )

polys s . t .
pk ( 0 ) =1

The min-max problem is exactly

solved by Chebyshev Polynomials
SMA-HPC 2003 MIT

26

Convergence for M =MT

Krylov Methods

Chebyshev Solves Min-Max

The Chebyshev Polynomial

Ck ( x ) cos ( k cos 1 ( x ) ) x [ 1,1]

min kth order max x [min ,max ] k ( x )

polys s .t .
k ( 0 ) =1

max x [min ,max ]

x
Ck 1 + 2 min
max min

min
Ck 1 + 2
max min

SMA-HPC 2003 MIT

27

Chebyshev Polynomials minimizing over [1,10]

SMA-HPC 2003 MIT

28

Convergence for M =MT

Krylov Methods

Chebychev Bounds

min kth order max x [min ,max ] k ( x )

polys s .t .
k ( 0 ) =1

max
Ck 1 2
max min

max

min

2
max

+ 1

min

SMA-HPC 2003 MIT

29

Convergence for M =MT

Krylov Methods

Chebychev Result

If ( M ) [ min , max ] , min > 0

k

rk

max

min

r0
2
max

+ 1

min

SMA-HPC 2003 MIT

30

Preconditioning

Krylov Methods

1
0

0
1
0

Diagonal Example

1
0

0
2
0

For which problem will GCR Converge Faster?

SMA-HPC 2003 MIT

31

Preconditioning

Krylov Methods

Diagonal Preconditioners

Let M = D + M nd
(

Apply GCR to D 1M x = I + D 1M nd x = D 1b
The Inverse of a diagonal is cheap to compute
Usually improves convergence
SMA-HPC 2003 MIT

32

Heat Conducting
Bar example
x
x1

x2

x
100

xi

xi +1

Discretized system

one small x
xn

2 + 1
u1 f (x1)

1 2 +

1 1+ +100
100

100 1+ +100 1

1
1

u f (x )
1
2

n n

max
> 100
min

SMA-HPC 2003 MIT

33

Which Convergence Curve is GCR?

rk
r0

Iteration
SMA-HPC 2003 MIT

34

Heat Conducting
Bar example

Preconditioned Matrix
Eigenvalues

Residual Minimizing
Krylov-subspace
Algorithm can
eliminate outlying
eigenvalues by
placing polynomial
zeros directly on
them.

SMA-HPC 2003 MIT

35

The World According

to Krylov

Heat Flow Comparison Example

Dimension Dense GE Sparse GE

GCR

O ( m3 )

O (m)

O ( m2 )

O ( m6 )

O ( m3 )

O ( m3 )

O ( m9 )

O ( m6 )

O ( m4 )

GCR faster than banded GE in 2 and 3 dimensions

Could be faster, 3-D matrix only m3 nonzeros.
GCR converges too slowly!
SMA-HPC 2003 MIT

36

Preconditioning

Krylov Methods

Approximate LU
Preconditioners

Let M L U
Applying GCR to

((

LU

( )

M x = LU

Use an Implicit matrix representation!

Forming y =

(( LU )

M x is equivalent to

solving LUy = Mx
SMA-HPC 2003 MIT

37

Preconditioning

Krylov Methods

Approximate LU
Preconditioners Continued

Nonzeros in an exact LU Factorization

Filled-in LU factorization
Too expensive.

Ignore the fillin!

SMA-HPC 2003 MIT

38

Factoring 2-D Grid Matrices

Generated Fill-in Makes Factorization Expensive

SMA-HPC 2003 MIT

39

Preconditioning

Krylov Methods

Approximate LU
Preconditioners Continued

THROW AWAY FILL-INS!

Throw away all fill-ins
Throw away only fill-ins with small values
Throw away fill-ins produced by other fill-ins
Throw away fill-ins produced by fill-ins of
other fill-ins, etc.

40

Summary
Reminder about GCR
Residual minimizing solution
Krylov Subspace
Polynomial Connection

Review Norms and Eigenvalues

Induced Norms
Spectral mapping theorem

Estimating Convergence Rate

Chebychev Polynomials

Preconditioners
Diagonal Preconditioners
Approximate LU preconditioners
SMA-HPC 2003 MIT

41