View
218
Download
0
Category
Preview:
Citation preview
September 12, 2005 Copyright©2001-5 Erik D. Demaine and Charles E. Leiserson L2.1
Introduction to Algorithms
LECTURE2
Asymptotic Notation
•O-, Ω-, and Θ-notation
Recurrences
•Substitution method
•Iterating the recurrence
•Recursion tree
•Master method
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.2
Asymptotic notation
O-notation (upper bounds):
We write f(n) = O(g(n)) if there Exist constants c>0, n0>0 suchThat 0 ≤f(n) ≤cg(n) for all n≥n0.
We write f(n) = O(g(n)) if there Exist constants c>0, n0>0 suchThat 0 ≤f(n) ≤cg(n) for all n≥n0.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.3
Asymptotic notation
O-notation (upper bounds):
We write f(n) = O(g(n)) if there Exist constants c>0, n0>0 suchThat 0 ≤f(n) ≤cg(n) for all n≥n0.
We write f(n) = O(g(n)) if there Exist constants c>0, n0>0 suchThat 0 ≤f(n) ≤cg(n) for all n≥n0.
EXAMPLE: 2n2= O(n3) (c= 1, n0= 2)
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.4
Asymptotic notation
O-notation (upper bounds):
We write f(n) = O(g(n)) if there Exist constants c>0, n0>0 suchThat 0 ≤f(n) ≤cg(n) for all n≥n0.
We write f(n) = O(g(n)) if there Exist constants c>0, n0>0 suchThat 0 ≤f(n) ≤cg(n) for all n≥n0.
EXAMPLE: 2n2= O(n3) (c= 1, n0= 2)
functions,Not values
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.5
Asymptotic notation
O-notation (upper bounds):
We write f(n) = O(g(n)) if there Exist constants c>0, n0>0 suchThat 0 ≤f(n) ≤cg(n) for all n≥n0.
We write f(n) = O(g(n)) if there Exist constants c>0, n0>0 suchThat 0 ≤f(n) ≤cg(n) for all n≥n0.
EXAMPLE: 2n2= O(n3) (c= 1, n0= 2)
functions,Not values
Funny, “one-way”equality
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.6
asymptotic upper bound
} )()(0s.t.,|)({))(( 00 nnncgnfncnfngO
n 0
n
cg(n)
f(n)f n O g n( ) ( ( ))
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.7
Ω-notation (lower bounds)
O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n2).
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.8
Ω-notation (lower bounds)
O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n2).
Ω(g(n))= { f(n) :there exist constants c> 0, n0> 0such
that 0 ≤ cg(n) ≤ f(n) for all n≥n0}
Ω(g(n))= { f(n) :there exist constants c> 0, n0> 0such
that 0 ≤ cg(n) ≤ f(n) for all n≥n0}
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.9
Ω-notation (lower bounds)
O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n2).
Ω(g(n))= { f(n) :there exist constants c> 0, n0> 0such
that 0 ≤ cg(n) ≤ f(n) for all n≥n0}
Ω(g(n))= { f(n) :there exist constants c> 0, n0> 0such
that 0 ≤ cg(n) ≤ f(n) for all n≥n0}
EXAMPLE:
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.10
asymptotic lower bound
} )()(0 s.t. ,|)({))(( 00 nnnfncgncnfng Ω
n 0
n
cg(n)
f(n)f n g n( ) ( ( ))Ω
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.11
-notation (tight bounds)
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.12
-notation (tight bounds)
EXAMPLE:
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.13
• The definition of required every member of be asymptotically nonnegative.
n 0
n
c 2g(n)
c 1g(n)
f(n)f n g n( ) ( ( ))
Graphic Example of Θ
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.14
Theorem 3.1.
• For any two functions f(n) and g(n), if and only if and .
f n g n( ) ( ( ))f n O g n( ) ( ( ))f n g n( ) ( ( ))Ω
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.15
ο-notation and ω-notation
O-notation and Ω-notation are like ≤ and ≥. o-notation and ω-notation are like < and >.
ο(g(n))= { f(n) : for any constant c> 0 there is a constant n0> 0 such that 0 ≤ f(n) < cg(n)
for all n≥ n0}
ο(g(n))= { f(n) : for any constant c> 0 there is a constant n0> 0 such that 0 ≤ f(n) < cg(n)
for all n≥ n0}
2n2= o(n3) (n0= 2/c)EXAMPLE:
But 2n2 ≠ o(n2)
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.16
ο-notation and ω-notation
O-notation and Ω-notation are like ≤ and ≥. o-notation and ω-notation are like < and >.
(g(n))= { f(n) : for any constant c> 0 there is a constant n0> 0 such that 0 ≤ cg(n) < f(n)
for all n≥ n0}
(g(n))= { f(n) : for any constant c> 0 there is a constant n0> 0 such that 0 ≤ cg(n) < f(n)
for all n≥ n0}
EXAMPLE:
n2/2 =w (n), but n2 /2 ≠ w(n2)
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.17
Solving recurrences
• The analysis of merge sort from Lecture 1 required us to solve a recurrence.
• Recurrences are like solving integrals, differential equations, etc.
。 Learn a few tricks.
• Lecture 3: Applications of recurrences to divide-and-conquer algorithms.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.18
Substitution method
The most general method:
1.Guess the form of the solution.
2.Verify by induction.
3.Solve for constants.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.19
Substitution method
The most general method:
1.Guess the form of the solution.
2.Verify by induction.
3.Solve for constants.
EXAMPLE:
• [Assume that T(1) = Θ(1).]• Guess O(n3). (Prove O and Ωseparately.)• Assume that T(k) ≤ ck3 for k< n .• Prove T(n) ≤cn3 by induction.
T(n) = 4T(n/2) + n
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.20
Example of substitution
T(n) = 4T(n/2) + n ≤ 4c(n/2)3 + n = (c/2)n3 + n = cn3 –((c/2)n3 – n)← desired – residual ≤ cn3 ← desired
Whenever (c/2)n3 – n ≥ 0, for example,if c ≥ 2 and n ≥ 1.
residual
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.21
Example (continued)
•We must also handle the initial conditions, that is, ground the induction with base cases.
•Base: T(n) = Θ(1) for all n< n0, where n0is a suitable constant.
•For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if we pick c big enough.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.22
Example (continued)
•We must also handle the initial conditions, that is, ground the induction with base cases.
•Base: T(n) = Θ(1) for all n< n0, where n0is a suitable constant.
•For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if we pick c big enough.
This bound is not tight!
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.23
A tighter upper bound?
We shall prove that T(n) = O(n2).
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.24
A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
T(n)= 4T(n/2) + n ≤ 4c(n/2)2 + n = cn2 + n = O(n2)
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.25
A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
T(n)= 4T(n/2) + n ≤ 4c(n/2)2 + n = cn2 + n = O (n2) Wrong! We must prove the I.H.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.26
A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
T(n)= 4T(n/2) + n ≤ 4c(n/2)2 + n = cn2 + n = O (n2) = cn2 – (- n) [ desired - residual] ≤ cn2 for no choice of c > 0. Lose!
Wrong! We must prove the I.H.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.27
A tighter upper bound!
IDEA: Strengthen the inductive hypothesis.
• Subtract a low-order term.
Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.28
A tighter upper bound!
IDEA: Strengthen the inductive hypothesis.
• Subtract a low-order term.
Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.
T(n)= 4T(n/2) + n = 4(c1(n/2)2–c2(n/2))+ n = c1n2–2c2n+ n = c1n2–c2n–(c2n–n) ≤ c1n2–c2n if c2≥1.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.29
A tighter upper bound!
IDEA: Strengthen the inductive hypothesis.
• Subtract a low-order term.
Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.
T(n)= 4T(n/2) + n
= 4(c1(n/2)2–c2(n/2))+ n
= c1n2–2c2n+ n
= c1n2–c2n–(c2n–n)
≤ c1n2–c2n if c2≥1.Pick c1 big enough to handle the initial conditions.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.30
Recursion-tree method
• A recursion tree models the costs (time) of a recursive execution of an algorithm.
• The recursion-tree method can be unreliable, just like any method that uses ellipses (…).
• The recursion-tree method promotes intuition, however.
• The recursion tree method is good for generating guesses for the substitution method.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.31
Example of recursion tree
Solve T(n) = T(n/4) + T(n/2)+ n2:
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.32
Example of recursion tree
Solve T(n) = T(n/4) + T(n/2)+ n2:
T(n)
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.33
Example of recursion tree
Solve T(n) = T(n/4) + T(n/2)+ n2:
n2
T(n/4) T(n/2)
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.34
Example of recursion tree
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.35
Example of recursion tree
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.36
Example of recursion tree
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.37
Example of recursion tree
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.38
Example of recursion tree
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.39
Example of recursion tree
geometric seriesgeometric series
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.40
The master method
The master method applies to recurrences of the form
T(n) = aT(n/b) + f(n) ,
where a≥1, b> 1, and f is asymptotically positive.
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.41
Three common cases
Compare f (n)with n logba:
1. f (n) = O (n logba–ε)for some constant ε> 0.
• f (n) grows polynomially slower than n logba
(by an nεfactor).
Solution: T(n) = (n logba)
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.42
Three common cases
Compare f (n)with n logba:
1. f (n) = O (n logba–ε)for some constant ε> 0.
• f (n) grows polynomially slower than n logba
(by an nεfactor).
Solution: T(n) = (n logba)
2. f (n) = (n logbalgkn)for some constant k≥0.
• f (n) and nl ogba grow at similar rates.
Solution: T(n) = (n logbalgk+1n).
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.43
Three common cases (cont.)
Compare f (n)with n logba:
3. f (n) = Ω(nlogba+ ε)for some constant ε> 0.
• f (n) grows polynomially faster than n logba (by an nεfactor),
and f (n) satisfies the regularity condition that
af(n/b) ≤ cf(n) for some constant c< 1.
Solution: T(n) = (f(n)).
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.44
Examples
EX. T(n) = 4T(n/2) + n
a =4, b= 2 ⇒n logba =n2; f(n) = n.
CASE1: f(n) = O(n2–ε)for ε= 1.
∴T(n) = (n2).
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.45
Examples
EX. T(n) = 4T(n/2) + n
a =4, b= 2 ⇒n logba =n2; f(n) = n.
CASE1: f(n) = O(n2–ε)for ε= 1.
∴T(n) = (n2).
EX.T(n) = 4T(n/2) + n2
a =4, b= 2 ⇒ n logba =n2; f(n) = n2.
CASE2: f(n) = (n2lg0n), that is, k = 0.
∴T(n) = (n2lgn).
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.46
Examples
EX.T(n) = 4T(n/2) + n3
a =4, b= 2 ⇒n logba=n2; f(n) = n3.
CASE3: f(n) = Ω(n2+ ε)for ε= 1
and 4(n/2)3≤cn3 (reg. cond.) for c= 1/2.
∴T(n) = (n3).
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.47
Examples
EX.T(n) = 4T(n/2) + n3
a =4, b= 2 ⇒n logba=n2; f(n) = n3. CASE3: f(n) = Ω(n2+ ε)for ε= 1 and 4(n/2)3≤cn3 (reg. cond.) for c= 1/2. ∴T(n) = (n3).
EX.T(n) = 4T(n/2) + n2/lgn a =4, b= 2 ⇒ n logba =n2; f(n) = n2/lgn. Master method does not apply. In particular, for every constant ε> 0, we have nε= ω(lgn).
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.48
Idea of master theorem
Recursion tree:
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.49
Idea of master theorem
Recursion tree:
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.50
Idea of master theorem
Recursion tree:
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.51
Idea of master theorem
Recursion tree:
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.52
CASE1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight.
Idea of master theorem
Recursion tree:
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.53
CASE2: (k= 0) The weight is approximately the same on each of the logbn levels.
Idea of master theorem
Recursion tree:
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.54
Idea of master theorem
CASE3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight.
Recursion tree:
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.55
Polynomials v.s. Exponentials
• Polynomials: – A function is polynomial bounded if
• Exponentials: – Any positive exponential function grows faster
than any polynomial.
P n a ni i
i
d( )
0)()( )1(OnOnf
n o a ab n ( ) ( )1
)1(0lim aa
nn
b
n
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.56
Logarithms
• A polylogarithmic function in n is a polynomial in the logarithm of n,
• A function f(n) is polylogarithmically bounded if • for any constant a > 0.• Any positive polynomial function grows faster t
han any polylogarithmic function.
)(log)( )1( nOnf O)(log ab non
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.57
Function iteration
.0))((
,0)( )1(
)(
ifinff
ifinnf i
i
For example, if , thennnf 2)( nnf ii 2)()(
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.58
• Since the number of atoms in the observable universe is estimated to be about , which is much less than , we rarely encounter a value of n such that .
8010 655362
5lg* n
lg* 2 1lg* 4 2lg*16 3lg* 65536 4lg* 2 565536
Slowly Growing function
September 12, 2005
Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.59
Appendix: geometric series
Return to last slide viewed.
Recommended