1

Asymptotes and Algorithms

By Gary ShortGibraltar Software

2

Agenda

• Introduction• Performance, does it matter?• How do we measure performance?• Analysis of Insertion Sort• Simplifying things with asymptotic notation• Designing algorithms• Solving recurrences• Questions.

3

Introduction

• Gary Short• Head of Gibraltar Labs• C# MVP• gary.short@gibraltarsoftware.com• @garyshort• http://www.facebook.com/theothergaryshort

4

Performance – Does it Matter?

Performance is the most important thing in software engineering today...

5

... Apart from everything else!

6

So Why Bother About Performance?

7

How do we Measure Performance?

• What do we care about?– Memory?– Bandwidth?– Computational time?

8

We Need a Model to Work With

• RAM Model– Arithmetic – add, subtract, etc– Data movement – load, copy, store– Control – branching, subroutine call, return– Data – Integers, floats

• Instruction are run in series– And take constant time• Not really, but shhh! –Ed.

9

Analysis of Insertion Sort

InsertionSort(A)for j = 2 to A.length

key=[Aj]i = j - 1while i > 0 and A[i] > key

A[i+1] = A[i]i = i - 1

A[i+1] = key

10

That Makes no Sense, Show me!

11

So What’s The Running Time?

12

Sum Running Time for Each Statement...

T(n) = c1n+c2(n-1)+c3(n-1)+c4 sum(tj) j=2..n+c5 sum(tj-1) j=2..n+c6sum(tj-1) j=2..n+c7(n-1)

13

Best Case Running Time

If the input (A) is already sorted then...A[i] <= key when has initial value of j-1 thus tj=1.And so...T(n) = c1n+c2(n-1)+c3(n-1)+c4(n-1)+c7(n-1)= (c1+c2+c3+c4+c7)n-(c2+c3+c4+c7)Which can be expressed as an+b for constants a

and b that depend on ciSo T(n) is a linear function of n

14

15

Side Note: No One Cares About Best Case

16

Worst Case Scenario

If the input (n) is in reverse sort order then...We have to compare each A[j] with each

element in the sub array A[1..j-1].And so...T(n) = (c4/2+c5/2+c6/2)n^2 +(c1 +c2+c3+c4/2-

c5/2-c6/2+c7)n-(c2+c3+c4+c7)Which can be expressed as an^2 + bn + cSo T(n) is a quadratic function of n

17

18

In Short...

In worst case insertion sort sucks!

19

Man That Was a Lot of Maths!

20

Simplifying Things With Asymptotic Notation

• Asymptotic notation characterises functions by their growth rates

• Functions with the same growth rates have the same Asymptotic notation

21

How Does That Help Us?

Let’s say we have a function with running timeT(n) = 4n^2 - 2n + 2If n = 500 then 4n^2 is 1000 times bigger than 2nSo...We can ignore smaller order terms and

coefficientsT(n) = 4n^2 -2n +2 can be written O(n) = n^2

22

A Short Note on The Abuse of “=“

If T(n) = 4n^2 -2n +2Then saying T(n) = O(n^2) is not strictly correctRather T(n) is in the set O(n^2) and the above

should be read as T(n) is O(n^2) and not T(n) equals O(n^2)

But really on Maths geeks care – Ed.

23

So Back to Insertion Sort

So now we can say of Insertion Sort that...Best case it’s O(n)And worst case it’s O(n^2)And since we only care about worst case...We say that Insertion Sort has O(n^2)Which sucks! – Ed.

24

Designing Algorithms

So can we do better?

25

Optimizing Algorithms is Child’s Play

• Sit at table• Foreach item in itemsOnPlate– Eat item

• Wait(MealComplete)• Foreach dish in dishesUsed– WashDish– DryDish

• Resume Play

26

Child Will Optimize To…

• Pause Game• Set Speed = MaxInt• Run to table• Take sliceBread(1)• Foreach item on Plate– Place item on bread

• Take sliceBread(2)• Run Outside• Resume Game

27

Divide And Conquer

• Divide– Divide the problem into sub problems

• Conquer– Solve the sub problems recursively

• Combine– Add the solutions to the sub problems into the

solution for the original problem.

28

Merge Sort

• Divide– Divide the n elements into two n/2 element arrays

• Conquer– Sort the two arrays recursively

• Combine– Merge the two sorted arrays to produce the

answer.

29

Analysis of Merge Sort

MergeSort(A,p,r)if(p<r)

q = [(p+r)/2]MergeSort(A,p,q)MergeSort(A,q+1,r)Merge(A,p,q,r)

Initial call MergeSort(A,1,A.length)

30

Dancers, or it Never Happened!!

31

So What’s The Running Time?

In the general case...If the divide step yields ‘a’ sub problemsEach 1/b the size of the originalIt takes T(n/b) time to solve one problem of n/b sizeSo it takes aT(n/b) to solve ‘a’ of themThen, if it takes D(n) time to divide the problemAnd C(n) time to combine the resultsThen we get the recurrence...T(n) = aT(n/b) + D(n) + C(n).

32

Apply That to Merge Sort...

• Divide– Computes the middle of the subarray, taking

constant time so, D(n) = O(1)• Conquer– Recursively solve two sub problems each of size

n/2 contributing 2T(n/2) to the running time• Combine– Merge procedure O(n)

• Giving us a recurrence of 2T(n/2)+O(n)

33

Solve The Recurrence Using The Master Method

For a Recurrence in the form T(n) = aT(n/b) + f(n)ThenIf f(n) = O(nlogba-k) then T(n) = O(nlogba)If f(n) = O(nlogba) then T(n) = O(nlogba log n)if f(n) = Omega(n log b a+k) and if af(n/b) <=

cf(n) then T(n) = O(f(n))

34

What?!

• More simply we are comparing f(n) with the function n log ba and intuitively understanding that the bigger of the two determines the solution to the recurrence.

35

And So...

• With Merge Sort we are in the third case of the Master Method thus...

• T(n) = O(n log n)• Which is much better than the O(n^2) of

Insertion Sort

36

37

What We Learned

• Performance is important• Therefore algorithmic optimization is too• We have a model to benchmark• And a syntax• Divide and conquer• Master Method• Other resources.

38

39

Questions?