ROOT FINDING TECHNIQUES

FOR SOLVING
NON-LINEAR EQUATIONS

ROOT FINDING TECHNIQUES FOR SOLVING

NON-LINEAR EQUATIONS
1. Introduction
Life would be extremely boring if all physical systems could be described by linear systems of equations. Most
engineering problems are non-linear in nature although, in some instances, reasonable approximations may be
obtained by linearisation. However, in a lot of cases linear approximations are useless with little or no possibility of
obtaining a reasonable solution with them. A particular good example is turbulent flowing liquid which is chaotic
in nature and cannot be described with linear equations. Non-linear equations are required to adequately describe
plastic and viscoplastic deformation of solids. Even a simple pendulum requires non-linear equations to describe its
behaviour for relatively large oscillations.
In this course of lectures we will focus on algebraic equations and in particular arranging the equations so that
their solution can be simply viewed as a root finding exercise. Attention is focused on the so called, fixed-point
methods as these are easy to implement and, if constructed properly, can be highly convergent. A fixed-point

( )

= x ( k ) , to solve f (x ) = 0 , i.e. to
determine the values of x that are the roots of f (x ) . If properly constructed a root of f (x ) should be a fixed point
of the iterative scheme, i.e. if f ( ) = 0 , then = ( ) .
method involves the construction of an iterative procedure of the form, x

( k +1)

It is expected that the student should familiarise him/herself with the direct search techniques: Bisection Method,
False-Position Method, Secant(or Chord) Method, as these are based on relatively simple ideas so will not be
covered in this lecture course.

2. Root Multiplicity
An equation of the form ,

f (x ) = 0 , is said to have a root at x = if f ( ) = 0 .

If the function, f , can be written as f (x ) = (x ) g (x ) , with g( ) 0 , then f (x ) is said to have a root of

m

multiplicity(or order) m at

x = .

Consider a Taylor series expansion of f (x ) at

x = , i.e.

(x ) d 2 f ( ) + (x ) d 3 f ( ) + (x ) d 4 f ( ) + ...
df
f (x ) = f ( ) + (x ) ( ) +
dx
2! dx 2
3! dx 3
4! dx 4
2

or more succinctly,
2
3
4
(
(
(
x )
x )
x ) 4
f (x ) = f ( ) + (x )f ( ) +
f ( ) +
f ( ) +
f ( ) + ...

2!

where f

( ) = d fs ( ) , i.e. d fs
dx

f (x ) = (x ) f ( ) +

3!

dx

evaluated at

4!

x = . If is a root of f (x ) then

(x ) f ( ) + (x )2 f ( ) + (x )3 f 4 ( ) + ...
2!

(1)

3!

4!

= (x )g 1 (x )

and note that, g 1 ( ) = f ( ) , so g 1 ( ) 0 if and only if f ( ) 0 . If, however, f ( ) = 0 , then

NUMERICAL METHODS FOR ENGINEERS

f (x ) = (x )
and

(x ) f 4 ( ) + ... = (x )2 g (x )
f ( ) (x )
+
f ( ) +
2
2!
3!
4!
2

g 2 ( ) = f ( ) / 2! , so g 2 ( ) 0 if and only if f ( ) 0 .

In general, f (x ) has a root of multiplicity m at

f ( ) = f ( ) = f ( ) = f ( ).......... .. = f

x = if and only if
( ) = 0 and f m ( ) 0 .

m1

(2)

If m = 2 then the root is called a double root and this occurs when f ( ) = f ( ) = 0 and f ( ) 0 .

Example 2.1
Determine the roots of p(x ) = x + 2 x + 1 as well as the multiplicity of each root.
2

This polynomial can be factorised to give p(x ) = (x + 1) and so p (x ) has a double root at
2

p (x ) = 2(x + 1) and p (x ) = 2 and thus p( 1) = p ( 1) = 0 with p (x ) 0 .

x = 1. Note that,

3. Single Variable Fixed-Point Iteration

One possible approach to obtaining a root of f (x ) is via a graphical construction.
If, for example, we define f1 (x ) and f 2 (x ) so that f (x ) = f 1 (x ) f 2 (x ) , then graphs for f1 (x ) and f 2 (x ) can
be constructed on a single plot.
Clearly, f (x ) = 0 at the points where f1 (x ) = f 2 (x ) .
Thus, the intersection points of the two graphs for f1 (x )
and f 2 (x ) are roots of f (x ) .

f 1(x)=x
2

f2 (x)= x

Example 3.1
Use a graphical construction to find the roots of

f (x ) = x x 2 . Let f1 (x ) = x and f 2 (x ) = x 2 and the

graph follows. Clearly, the roots are at x = 0 and x = 1 .
Constructions of this type should only be used as a rough
guide.

NUMERICAL METHODS FOR ENGINEERS

The Newton-Raphson Method

Before going on to examine fixed-point methods in general it is worth, at this stage, introducing the NewtonRaphson method which is an extremely popular fixed-point method. To do this consider the geometrical
construction shown in the figure.

( )

(k )
=
Note that, f x

f(x)
(k)

f'(x )

give

the

( )

f x (k )
which can be rearranged to
x (k ) x ( k +1)

classical

x (k +1) = x ( k )

f (x )
.
f (x ( k ) )
(k )

Newton-Raphson

method,

i.e.

This is clearly, a fixed point method, since it is of the form,

x ( k +1) = (x ( k ) ) , where (x (k ) ) = x ( k )

(k)

f(x )

(k+1)

(k)

f (x ( k ) )
.
f (x ( k ) )

Moreover, if f (x ) has a root of multiplicity 1 at

( ) =

f ( )
0
=
=
f ( )
f ( )

x = , then

Thus, any root of f (x ) is a fixed point of the Newton-Raphson method.

Example 3.2
Use the Newton-Raphson method to determine the root of f (x ) = e 2 and take your initial guess to be
x

x (0 ) = 0 .

In this case, f (x ) = e and so x

x

Thus, setting

x (2 )

k = 0 gives, x

(k +1)

(1)

=x

=x

(0 )

(k )

( )
( )

(k )

x
f x (k )
2
(k ) e
.

=
x

(k )
(k )
x
f x
e

ex

(0 )

x (0 )

= 0

e0 2
1 2
=
= 1 , and setting k = 1 gives,
0
1
e

e1 2
= 1 1 = 0.75576 .
e

Continuing

in

similar

fashion

x (4 ) = 0.69507

e 0.69507 2
= 0.69315 .
e 0.69507

The exact answer is

ln 2 0. 69315.

provides:

x (3 ) = 0.75576

e 0.75576 2
= 0.69507
e 0.75576

and

This example demonstrates just how rapidly the Newton-Raphson method can converge to an answer.

NUMERICAL METHODS FOR ENGINEERS

3.1 General Concepts

Fixed-point iteration can be visualised via a graphical construction called a cobweb. This construction involves
forming graphs for x and (x ) on the same plot.
The points where, x = (x ) , are the fixed points of the iterative scheme. This plot can then be used to follow the
iterative process.
y

y= (x)
y=x

(x (o))

(1)

(x )

x(5) x

(4)

x (3)

(2)

x (1)

x (o)

xx

This figure is typical of a problem that has two fixed points where one is a fixed point of attraction (i.e.
the other is a fixed point of repulsion (i.e. ).
Although a fixed point of repulsion, , satisfies = () the iterative procedure, x
converge to .

( k +1)

) and

( )

= x ( k ) , does not

One of the distinct differences between the two fixed points, identified in the figure, is that, () > 1 whilst

( ) < 1 .

It turns out that a necessary and sufficient condition for a fixed point,

( ) < 1 .

, to be a fixed point of attraction is that

To prove this we need to make use of the Mean Value Theorem.

In addition, if
scheme,

is a fixed point and ( ) < 1 , then provided x (0 ) is sufficiently close to , the iterative

( )

x ( k +1) = x ( k ) , will converge.

READ SUPPLEMENTARY MATERIAL: PROOFS ARE NON-EXAMINABLE

(PAGES 10 TO 11)

NUMERICAL METHODS FOR ENGINEERS

Example 3.1.1
Investigate the behaviour of the following fixed-point iterative schemes,
(i) x

(k +1)

= x (k ) + 2 / 3 ,

(k +1)

= 3x ( k ) 2 ,

(ii) x

at the roots of the polynomial, p(x ) = x 3x + 2 , i.e. at

2

x = 1 and x = 2 .

Note that, these schemes were obtained by simply manipulating the equation x 3x + 2 = 0 to give: (i)
2

x = (x 2 + 2 )/ 3 , (ii) x = 3x 2 .

= x (k ) + 2 / 3 = (x (k ) ) and note that, by construction, the fixed points of this scheme, satisfies
= ( ) = ( 2 + 2 )/ 3 2 3 + 2 = 0 , and so are roots of p(x ) .

(i) Let x

(k +1)

Moreover, (x ) = 2 x / 3 and (1) = 2 / 3 < 1 , (2 ) = 4 / 3 > 1 , so 1 is a fixed point of attraction whilst 2 is

a fixed point of repulsion.
In addition, (x ) < 1
if a 1 = 1 / 2 < x
(ii) Let

(0 )

1 < 2 x / 3 < 1

3 / 2 < x < 3 / 2 , and so convergence to this point is guaranteed

< 3 / 2 = b1 .

( )

x (k +1) = 3x ( k ) 2 = x (k ) and note that, by construction, the fixed points of this scheme, satisfies

= ( ) = 3 2

3 + 2 = 0 , and so are roots of p(x ) .

2

Moreover, (x ) = 3 / 2 3x 2 and (1) = 3 / 2 > 1 , (2 ) = 3 / 4 < 1 , so 2 is a fixed point of attraction

whilst 1 is a fixed point of repulsion in this case.

NUMERICAL METHODS FOR ENGINEERS

4. Rates of Convergence
It is important to appreciate that some fixed-point iterative schemes will converge to a solution at a faster rate (i.e.
in less iterations) than others. To understand what influences convergence consider the following:

( )

( k +1) = x ( k +1) = x (k ) = + (k ) =
= ( ) + (k ) ( ) +

2!

( ) +

(k )
( ) + ...........
3!
3

(k )
= ( ) +
( ) +
( ) + ...........
2!
3!
(k )

where +
If x

(k )

(k )

(k )2

(k )2

) has been replaced by its Taylor series.

is close to the fixed point then higher powers of

(k )

will be small and so

( k +1)

( k )( ) .

( )

It is therefore clearly advantageous to have ( ) as small as possible and ideally = 0 is best.

A fixed point iterative method is said to be of 2nd order if ( ) = 0 and ( ) 0 where in this case

( k +1)

(k )

( ) .
2!
2

Similarly, if ( ) = ( ) = 0 and ( ) 0 then the method is said to be 3rd order, etc.

Example 4.1
Show that, if is a root of f (x ) of multiplicity 1, then the Newton-Raphson method, x

(k +1)

= x (k )

( )
( )

f x (k )
, is
f x (k )

at least a 2nd order method.

It is clear that each root of f (x ) is a fixed point of x

( k +1)

f ( ) / f ( ) = .

( ) ( ) ( )

= x (k ) f x ( k ) / f x (k ) = x (k )

since

Note that,

(x ) =

d
f (x )
f (x ) f (x )f (x )
f (x )f (x )
x
=1

=
2
dx
f (x )
f (x ) (f (x ))
(f (x ))2

and ( ) = 0 because f ( ) = 0 .

NUMERICAL METHODS FOR ENGINEERS

5. Fixed Points for Functions of Several Variables

A system of non-linear equations can generally be written as,

f1 (x 1 , x 2 ,..., x n ) = 0

f 2 (x 1 , x 2 ,..., x n ) = 0
f 3 (x 1 , x 2 ,..., x n ) = 0

f n (x 1 , x 2 ,..., x n ) = 0 ,
where

the

.
.
.

(5)

objective

is

the

determination

of

the

x = (x 1 , x 2 ,..., x n ) = ( 1 , 2 ,..., n ) = , which satisfy this system.

T

vector

roots

of

the

form,

It is convenient to use a vectorial representation for this system, i.e.

F(x ) = F(x 1 , x 2 ,..., x n ) = (f1 (x ), f 2 (x ),..., f n (x )) =

T

= (f1 (x 1 , x 2 ,..., x n ), f 2 (x 1 , x 2 ,..., x n ),..., f n (x 1 , x 2 ,..., x n )) = (0,0...,0 ) = 0

T

where the functions, f1 , f2 ,........, f n , are called the co-ordinate functions of F.

A fixed point iterative scheme for this system takes the vectorial form:

x (k +1)

x1
x2
= .
.
xn

(k +1)

1 (x (k ) )
1
(k )
2 (x )
2
=
= .
.
.
.
(k )
n (x )
n

(k )

= (x 1(k ) , x (2k ) ,..., x (nk ) ) = (k )

and a fixed point satisfies

1
1 ( )
2
2 ( )
= . =
.
= 1 , 2 ,..., n = ( )
.
.
n
n ( )

and, of course, we required the iterative scheme to be constructed so that its fixed points are the roots of the above
non-linear system.

NUMERICAL METHODS FOR ENGINEERS

Example 5.1
Confirm that = (0,1) is a root of the 2-variable system
T

f1 (x 1 , x 2 ) = x 12 x 2 x 12 4 x 2 x 1 + 4 x 2 + 4 x 1 4 = 0
f 2 (x 1 , x 2 ) = x 1 x 2 2 x 1 = 0

This is true because

f1 (1 , 2 ) = f 1 (0,1) = 0 2 1 0 2 4 0 1 + 4 1 + 4 0 4 = 0
f 2 (1 , 2 ) = f 2 (0,1) = 0 1 2 0 = 0
5.1 Newton's Method
Newton's method is just an extension of the single variable Newton-Raphson method. Its derivation can be
obtained by considering a Taylor series expansion of the co-ordinate functions. In the single variable case we have

))

( ) (

((

)( )

f1 x ( k +1) = f1 x ( k ) + x (k +1) x (k ) = f1 x (k ) + x (k +1) x ( k ) f 1 x (k ) + O x (k +1) x ( k )

and if

))
2

x (k +1) is close to a root then f1 (x (k +1) ) 0 and (x (k +1) x (k ) ) 0 .

2

Setting these to zero in the above equation gives

( ) (

)( )

( )

0 = f 1 x (k ) + x ( k +1) x (k ) f 1 x (k )

( )

( )

f1 x (k ) x ( k +1) = f1 x (k ) x (k ) f1 x (k )

which is, of course, the Newton-Raphson method.

Consider then, a two variable system, where the corresponding Taylor expansion is

(
(

) (
) (

) (
) (

)
)

) (
) (

f1 x 1(k +1) , x (2k +1) f 1 x 1( k ) , x 2(k ) + x 1(k +1) x 1( k ) f 1 / x 1 x 1(k ) , x (2k ) + x 2(k +1) x (2k ) f 1 / x 2 x 1(k ) , x (2k )
f 2 x 1( k +1) , x (2k +1) f 2 x 1( k ) , x 2(k ) + x 1(k +1) x 1( k ) f 2 / x 1 x 1( k ) , x (2k ) + x (2k +1) x (2k ) f 2 / x 2 x 1(k ) , x (2k )

(k +1)

Setting f1 x 1

f 1 / x 1
f 2 / x 1
or J

(k ) (k +1)

, x (2k +1) = f 2 x 1( k +1) , x (2k +1) = 0 provides, in matrix form,

f1 / x 2
f 2 / x 2

(k )

x1
x2

( k +1)

f 1 / x 1
=
f 2 / x 1

f 1 / x 2
f 2 / x 2

(k )

x1
x2

(k )

f
1
f2

(k )

= J (k ) x ( k ) F ( k ) where J is the partial derivative matrix and is known as the Jacobian matrix, and

F (k ) = F(x (k ) ) = (f1 (x (k ) ), f 2 (x (k ) )) .
T

NUMERICAL METHODS FOR ENGINEERS

These ideas can be generalised to n variables where Newton's method takes the form J
and J is an n n matrix with coefficients (J )ij = f i / x j .

(k ) (k +1)

x ( k +1) = x ( k ) J (k ) F (k ) = ( k ) and
F( ) = 0 , i.e. the fixed points of the method are roots of F.

It is clear that Newton's method is a fixed point scheme since

J = J F( )

= J (k ) x ( k ) F ( k )

Note that, it is essential that the Jacobian is at least non-singular in the vicinity of the root.

Example 5.1.1
Apply Newton's Method to the system in example 5.1 starting with x

(0 )

= (1,1) .
T

Taking partial derivatives of the co-ordinate functions provides

f 1 / x 1 (x 1 , x 2 ) = 2x 1 x 2 2x 1 4x 2 + 4
f 1 / x 2 (x 1 , x 2 ) = x 12 4 x 1 + 4
f 2 / x 1 (x 1 , x 2 ) = x 2 2 = 0
f 2 / x 2 (x 1 , x 2 ) = x 1
and so J

(0 )

0 1
0 1 x1
, giving
=
1 1
1 1 x 2

x1
and this implies that
x2

(1)

0 1
=
1 1

Proceeding in a similar fashion gives,

0 4 x1
1 0 x 2

(2)

0 4 x1
=
1 0 x 2

0 1 x1
=
1 1 x2

(0 )

f
1
f2

(0 )

0 1 1
0
1

=
1 1 1
1
1

1
1 1 1
0
=
.
=
1
1 0 1
1

J (1) =
(1)

(1)

f
1
f2

0 4
and
1 0
(1)

0 4 0
x1
. Thus,
=
1 0 1
x2

(2 )

x
= 1
x2

(1)

0
and the
1

method has in fact converged.

In general convergence will take more that two iterations. However, Newton's Method can be show to have 2nd
order convergence and this is the prime reason for its use in modern numerical analysis.

NUMERICAL METHODS FOR ENGINEERS

SUPPLEMENTARY MATERIAL: PROOFS NON-EXAMINABLE

3.1 Mean Value Theorem

h (x ) is a continuous function over the interval, a x b , then there exists an in this interval such that,

If
b

h (x )dx = h ( )(b a ) .

h(x)

Although it is relatively easy to formally prove this, the

figure illustrates the concept equally well. The basic idea
is that can be found so that A 2 + A 3 = A 2 + A 1
which is clearly achievable for continuous functions.

h()
2

3.2 Uniqueness and convergence proofs

This theorem can be used with

h (x ) = (x ) provided, of course, (x ) is a continuous function.

In this case, the Mean Value Theorem gives,

b

(x )dx = (b ) (a ) = ( )(b a )

(3)

where a b .
If we assume that, (x ) is a continuous function then:

(i) There is no more than one unique fixed point for the iterative scheme, x

(x ) < 1 .

Proof: Assume that both

with .

( k +1)

( )

= x ( k ) , in an interval where

and are fixed points of the scheme, x ( k +1) = (x ( k ) ) , i.e. = ( ) and = ()

The mean value theorem can be applied between points and to give

= () ( ) = ( )( ) = ( ) <
where the inequality was generated by making use of the fact that and so ( ) < 1 .
Note that, it is not possible for the inequality,
only one fixed point in the interval.

< , to be true, and so we must have = , i.e. there is

NUMERICAL METHODS FOR ENGINEERS

10

(ii) If

is a fixed point and ( ) < 1 , then provided x (0 ) is sufficiently close to , the iterative scheme,

( )

x ( k +1) = x ( k ) , will converge.

Proof: Although a proof for (ii) is not conceptually difficult it is somewhat involved. The main problem is the need
for a rigorous definition of the statement; ' x
Continuity of (x ) at

(0 )

is sufficiently close to

'.

x = ensures the existence of an interval, a x b , in which belongs and

(x ) < 1 . Let us assume that, for the time-being, each x (k +1) belongs to this interval so that x ( k +1) < 1 .

It is useful to introduce the positive number, u, with the property that, x

Moreover, let us define the kth error to be
Value Theorem, we have,

(k )

(k +1)

) u < 1.

= x ( k ) = x ( k 1) and thus, with application of the Mean

( )(
)
u (x ( ) ) = u (

)(

(k ) = x ( k ) = x ( k 1) = ( k 1) x (k 1) = (k 1) x (k 1)
k 1)

k 1

from

(k )

which

we

u 0 ,
k

deduce
and

that,
y

hence

lim ( k ) (0 ) lim u k = 0 .
k

y=x

It clear, therefore, that convergence is

guaranteed with the assumption

( k +1)

) < 1 for all k.

y=(x)
(1)
(x )

To complete the proof we need to define

an interval that ensures x

( k +1)

) <1

(x(0) )

for all k.
One rather conservative possibility is
where
simply,
a 1 x b1
and
a1 = d ,
b1 = + d

d = min{b , a } .

d
a

NUMERICAL METHODS FOR ENGINEERS

a1 x(1)

d
(2)

x(0) b=b1

xx

11