MNchap1

7/29/2019 MNchap1

1/46

Solving Nonlinear Equation

0x1x2x

)( 1xf

)( 0xf

Root r

7/29/2019 MNchap1

2/46

Numerical Methods Wen-Chieh Lin 2

Nonlinear Equations

Given function f, we find value x for which

Solution x is root of equation, or zero of

function f So problem is known as root finding or zero

finding

0)( xf

7/29/2019 MNchap1

3/46


Nonlinear Equations: two cases

Single nonlinear equation in one unknown, where

Solution is scalar x for which f(x) = 0

System ofn coupled nonlinear equations in nunknowns, where

Solution is vector x for which all components off

are zero simultaneously, f(x) = 0

7/29/2019 MNchap1

4/46


Example of 1-D nonlinear equation

for which x = 0.3604 is one approximate solution

Example of system of nonlinear equations in twodimensions

for which x = [-1.8, 0.8] is one approximatesolution vector

1

4

2

2

2

2

1

1

xe

xx

x

Examples: Nonlinear Equations

0)sin(3 xexx

7/29/2019 MNchap1

5/46


Multiplicity

Iff(R) = f(R) = f(R) = = f(m-1)(R) = 0 butf(m)(R) 0, then root R has multiplicity m

Ifm = 1 (f(R) = 0, f(R) 0 ), then R is simpleroot

7/29/2019 MNchap1

6/46


Bisection method begins with initial bracket andrepeatedly halves its length until solution hasbeen isolated as accurately as desired

http://www.cse.uiuc.edu/iem/nonlinear_eqns/Bisection/

Interval Halving (Bisection)

while |b-a| > tol,

m = a + (b-1)/2;

If f(a)*f(m) < 0,

b = m;

else

a = m;

end;end;

a b

m

7/29/2019 MNchap1

7/46


Bisection (cont.)

Simple and guaranteed to work if fis continuous in [a, b]

[a, b] brackets a root

Needed iterations to achieve a specified accuracy is

known in advance

Error after n iterations < |b - a| /2n

Slow to converge

Good for initial guess for other root finding

algorithms Finding the initial bracket may be a problem iffis not

given explicitly

7/29/2019 MNchap1

8/46


Use graphing to assist root finding

Set the initial bracket

Detect multiple roots

7/29/2019 MNchap1

9/46


Can we find a root in a better way?

Bisection only utilizes function values f(x)

We can find a root with fewer iterations if

other information is used

linear approximation off(x) Secant line secant method

Tangent line Newtons method

Polynomial approximation off(x)

Mullers method

7/29/2019 MNchap1

10/46


Secant Method

Approximate a function by a straight line

Compute the intersection of the line and x-axis

)()(

)(

)(

)(

10

10

1

21

xfxf

xx

xf

xx

)()(

)()(

10

10112

xfxf

xxxfxx

0x1x2x

)( 1xf

)( 0xf

Root r

7/29/2019 MNchap1

11/46


Secant Method (cont.)

Update endpoints

Repeat

)()()()(

1

11

nn

nnnnn

xfxfxxxfxx

0

x

1x

)( 1xf

)( 0xfRoot r

withswap,)()(if 1010 xxxfxf

7/29/2019 MNchap1

12/46


Example

f(x) = 3x + sin(x)

exp(x) Find the root in [0, 1]

http://www.cse.uiuc.edu/iem/nonlinear_eqns/Secant/

0 0.5 1 1.5 2-1

-0.5

0

0.5

1

1.5

7/29/2019 MNchap1

13/46


Method of False Position

Problem of secant method

Remedy

Always bracket a root in the interval [x0, x1]

How to do this?

7/29/2019 MNchap1

14/46


Newtons method

Better approximation using the first derivative

)( 0xf

10 xx 0x1x

10

00

)()(')tan(

xx

xfxf

)('

)(

0

001

xf

xfxx

)('

)(1

n

nnn

xf

xfxx

7/29/2019 MNchap1

15/46


Interpretation of Newtons method

Truncated Taylor series

is a linear function ofh approximating fnear x

Replace nonlinear function f by this function,whose zero is h = - f(x)/f(x)

Zeros of original function and linear

approximation are not identical, so repeat

process

)(')()( xhfxfhxf

)('

)(1

n

nnn

xf

xfxx

7/29/2019 MNchap1

16/46


Example: Newtons method

http://www.cse.uiuc.edu/iem/nonlinear_eqns/Newton/

7/29/2019 MNchap1

17/46


Comparison of Secant and

Newtons methods

)()(

)()(

1

11

nn

nnnnn

xfxf

xxxfxx

)('

)(1

n

nnn

xf

xfxx

Secant method Newtons method

7/29/2019 MNchap1

18/46


Pros and Cons of Newtons method

Pros

efficient

Cons

Need to know the derivative function

7/29/2019 MNchap1

19/46


)('

)(

1n

n

nn xf

xf

xx

When will Newtons method not converge?

x1=x6, loop

passing

maximum/minimum

7/29/2019 MNchap1

20/46


Mullers method Instead of linear approximation, Mullers

method uses quadratic approximate

Evaluation of derivatives are not required

See the textbook for details

7/29/2019 MNchap1

21/46


Fixed-point Iteration Method

Rearrange f(x)=0 into an equivalent form x=g(x)f(x) = x g(x) = 0

Ifris a root off, then f(r) = r-g(r) = 0r=g(r)

In iterative form,

Also called function iteration

For given equation f(x)=0, there may be manyequivalent fixed-point problems x=g(x) with different

choice ofg Will the method always converge?

)(1 nn xgx

7/29/2019 MNchap1

22/46


Example: Fixed-point Iteration

032)( 2 xxxf

32)(1 xxg

2

3)(2

xxg

7/29/2019 MNchap1

23/46


Example: Fixed-point Iteration (cont.)

032)( 2 xxxf

2

3)(

2

3

xxg

Diver

ge!

7/29/2019 MNchap1

24/46


Convergence Rate

For general iterative methods, define error atiteration n by

en = xn R

where xn is approximate solution and R is true

solution

For methods that maintain interval known to

contain solution, rather than specific

approximate value for solution, take error to be

length of interval containing solution

7/29/2019 MNchap1

25/46


Convergence Rate (cont.)

Sequence converges with rate rif

for some finite nonzero constant C Some cases of interest

r=1: linear (C1: superlinear

r=2: quadratic

Ce

er

n

n

n

1lim

7/29/2019 MNchap1

26/46


Convergence Rate of Bisection

Length of interval containing solution reduced

by half at each iteration

Linearly convergent

r = 1 C = 0.5

7/29/2019 MNchap1

27/46


Convergence of Fixed-point Iteration

IfR = g(R) and |g(R)| < 1, then there is aninterval containing R such that iteration

xn+1 = g(xn)

converges to R if started within that interval

If |g(R)| > 1, then iterative scheme diverges

7/29/2019 MNchap1

28/46


Proof of Convergent Condition

nnn ege )('1

)()()(1 nnn xgRgxgRxR

n

n

nn xR

xR

xgRgxR

)()(1

))(('1 nnn xRgxR

Mean Value Theorem

),( nn xR

Ce

er

n

n

n

1lim

Fixed-point iteration is linearly convergent

)(' RgC

7/29/2019 MNchap1

29/46


Convergence of Newtons Method

Represent Newtons method in fixed-pointiteration form

Condition for convergence

)()('

)(1 n

n

nnn xg

xf

xfxx

1)]('[

)(")()('

2

xf

xfxfxg

7/29/2019 MNchap1

30/46


Convergence rate of Newtons method

)()(1 nn xgRgxR 2)()(")()(')()( 2nnn xRgxRRgRgxg

0)('

)]('[

)(")()('

0)(

2 Rg

xf

xfxfxg

Rf

2)()(")()( 2nn xRgRgxg

2)(")()(2

11 nnnn egxgRgxRe

Ce

er

n

n

n

1lim

Newtons method isquadratically convergent!

Recall that

7/29/2019 MNchap1

31/46


Question from the last class

Is the initial solution important for theconvergence of fixed-point iteration?

7/29/2019 MNchap1

32/46


Answer to the convergence problem of

fixed-point Iteration

|g(R)|1, the fixed-point iteration will diverge

even if the initial condition is very close to a rootsince the iteration will eventually reach the region

causing divergence

Initial solution is important but less critical

The algorithm may not converge if the initial

solution is far from the true solution Recall the conditions that Newtons method does

not converge?

7/29/2019 MNchap1

33/46


Example: Newtons Method for Finding

Complex Roots

f(x) = x3 + 2x2 x + 5

7/29/2019 MNchap1

34/46


Start Newtons method with a complex value

ix 10 ixf 52)( 0 ixf 103)(' 0

i

i

ii

xf

xfxx 04587.1486238.0

103

521

)('

)(

0

001

52)( 23 xxxxf 143)(' 2 xxxf

ixf

xfxx 23665.1448139.0

)('

)(

1

112

7/29/2019 MNchap1

35/46


Newtons Methods for Multiple Roots

Quadratically convergent for simple root,

Linearly convergent for multiple roots as

0)(')]('[

)(")()(' 0)(

2 Rg

xf

xfxfxg

Rf

0

0

)]('[

)(")()('

2

Rf

RfRfRg

7/29/2019 MNchap1

36/46


Remedies for Multiple Roots with

Newtons method

Iff(x) has a root of multiplicity kat x=R, wecan factor out (x-R)k from f(x) to get

With a slightly modified Newtons method

It can be proved that and

Newtons method still converges quadratically

)()()( xQRxxfk

0)(' Rgk

)()('

)(1 nk

n

nnn xg

xf

xfkxx

7/29/2019 MNchap1

37/46


Remedies for Multiple Roots with

Newtons method (cont.)

In practice, we dont know kin advance

Remedies

Try and error

Deflate f(x)f(x)/(x-s) where s is an approximate R

Be warned that an indeterminate form at x=R is

created

7/29/2019 MNchap1

38/46


Systems of Nonlinear Equations

Solving systems of nonlinear equations ismuch more difficult than scalar case because

Wide variety of behavior is possible, so

determining existence and number of solutions or

good starting guess is much more complex In general, there is no simple way to guarantee

convergence to desired solution or to bracket

solution to produce absolutely safe method

Computational overhead increases rapidly withdimension of problem

7/29/2019 MNchap1

39/46


Example: Systems in Two Dimensions

From Michael T. Heath

7/29/2019 MNchap1

40/46


Newtons Method

In n dimensions, Newtons method has form

where J(x) is Jacobian matrix off

In practice, we do not explicitly invert J(xn),but instead solve linear system

for Newton step sn, then take as next iterate

)()( 1 nnn1n xfxJxx

)()( nnn xfsxJ

nn1n sxx

7/29/2019 MNchap1

41/46


Example: Newtons Method

01

4

),(

),(

)(2

2

2

2

1

212

211

1

xe

xx

xxf

xxfxxf

1

22)(

1

21

2

2

1

2

2

1

1

1

xe

xx

xf

xf

x

f

x

f

xJ

T7.110 x

0183.0

11.0)( 0xf

17183.2

4.32

)( 0xJ

)()( 000 xfsxJ

0183.0

11.0

17183.2

4.320s

T]7298.10043.1[00 sxx1

7/29/2019 MNchap1

42/46


Example: Newtons Method

01

4

),(

),(

)(2

2

2

2

1

212

211

1

xe

xx

xxf

xxfxxf

1

22)(

1

21

2

2

1

2

2

1

1

1

xe

xx

xf

xf

x

f

x

f

xJ

T7298.10043.11 x

49651.1

42653.8)( 1

e

exf

17300.2

4596.30086.2

)( 1xJ

)()( 111 xfsxJ

49651.1

42653.8

17300.2

4596.30086.21

e

es

T]729637.1004169.1[112 sxx

7/29/2019 MNchap1

43/46


Fixed-Point Iteration

Fixed-point problem for is to find

vector x such that

Corresponding fixed-point iteration is

Converges if

starts close enough to solution

)(xgx

)(1 nn xgx

nnRR :g

1))(( RJ

A

A

R

J

matrixaofseigenvaluethe

ofnormcomplexmaximum:)(

solutiontrue:

matrixJacobian:

7/29/2019 MNchap1

44/46


Fixed-Point Iteration (cont.)

Convergence rate is normally linear, with

constant C given by

If then convergence rate is at least

quadratic, e.g., Newtons method

nnRR :g

0)( RJ

))(( RJ

7/29/2019 MNchap1

45/46


Example: Fixed-point Iteration

014

),(

),()(

2

2

2

2

1

212

2111

xe

xxxxf

xxfxxf

2

1

2

2

1

4

)1ln(

x

x

x

x T

7.110 x

1.00000000000000 -1.73205080756888

1 .00505253874238 -1.72912388057290

1 .00398063482304 -1.72974647995027

1 .00420874039761 -1.72961406264780

1 .00416023020985 -1.72964222660783

7/29/2019 MNchap1

46/46


Next Monday

Introduction to Matlab/Octave

Matlab offers a student version with discounted

price

Octave is a shareware that has similar environmentas Matlab!

You are encouraged to bring your laptop with

Octave or Matlab installed next Monday

Documents

MNchap1