“Sorting networks and their applications”, AFIPS Proc. of 1968 Spring Joint Computer Conference, Vol. 32, pp 307-314.

http://www.cs.kent.edu/~batcher

Kenneth E. Batcher

Professor, Kent State University

SJTU

2013

陈贵

海

BackgroundSorting is fundamentalLow bound of any sequential sorting algorithms is O(nlogn)Can we improve the time complexity further?– Parallel algorithms– Circuit/Network Design– Parallel Computing Models

SJTU

2013

陈贵

海

①Bitonic Sequence 双调序列sequence of elements {a0, a1, …, an-1} where either– (1) there exists an index, i, 0 i n-1, such that {a0,

…, ai} is monotonically increasing, and {ai+1, …, an-1} is monotonically decreasing,

– e.g. {1, 2, 4, 7, 6, 0}

Or– (2) there exists a cyclic shift of indices so that (1) is

satisfied– e.g. {8, 9, 2, 1, 0, 4} {0, 4, 8, 9, 2, 1}

SJTU

2013

陈贵

海

Value of element

aia0 a1 a2 a3 a4 a5 a6 a7

{ 3, 5, 7, 9, 8, 6, 4, 2 }

Value of element

aia0 a1 a2 a3 a4 a5 a6 a7

{ 8, 6, 4, 2, 3, 5, 7, 9}

①Bitonic Sequence : Examples

SJTU

2013

陈贵

海

Value of element

aia0 a1 a2 a3 a4 a5 a6 a7

{ 3, 5, 7, 9, 11, 13, 15, 17 }

①Bitonic Sequence : Examples

Value of element

aia0 a1 a2 a3 a4 a5 a6 a7

{ 5, 3, 1, 2, 4, 6, 8, 7 }

SJTU

2013

陈贵

海

Bitonic Sort: basic idea

Consider a bitonic sequence S of size n where– the first half ( {a0, a1, …, an/2-1} ) is increasing, and

the second half ( {an/2, an/2+1, …, an-1} ) is decreasing

Value of element

aia0 a1... an/2-1 an/2 an/2 +1 … an-1

SJTU

2013

陈贵

海

②“Bitonic Split” 双调分裂Pair-wise min-max comparison

– s1 = {min(a0, an/2), min(a1, an/2+1), … , min(an/2-1, an-1)}

– s2 = {max(a0, an/2), max(a1, an/2+1), … , max(an/2-1, an-1)}

an/2

a0

an/2-1

an-1

value

aia0 a1... an/2-1 an/2 an/2 +1 … an-1

Compare and exchange

S2

S1

value

SJTU

2013

陈贵

海

There exists– an element b in S1 such that all elements before b is

increasing and all elements after b is decreasing– an element c in S2 such that all elements before c is

decreasing and all elements after c is increasing

S1 and S2

– Both S1 and S2 are bitonic sequences– Any elements in S1 < any elements in S2 (because b <

c and b is the maximum value in S1 and c is the minimum value in S2)

S2

S1

value cb

SJTU

2013

陈贵

海

pair-wise min-max comparison

e.g. { 2, 4, 6, 8, 7, 5, 3, 1}

{ 2, 4, 6, 8

7, 5, 3, 1 }

=> S1={2, 4, 3, 1}

S2={7, 5, 6, 8}

bitonic sequence of size 8

=> 2 bitonic sequence of size 4

Compare and exchange

SJTU

2013

陈贵

海

②Bitonic Split

The split is applicable to any bitonic sequence.

Need not to have the 1st half to be increasing/decreasing and the 2nd half to be decreasing/increasing:

Bitonic(n) 2 Bitonic(n/2)Bitonic Split

SJTU

2013

陈贵

海

Sorting a bitonic sequence

By using bitonic split recursively,INPUT: a bitonic sequence of size n

Phase 1: 2 bitonic sequence of size n/2 Phase 2: 4 bitonic sequence of size n/4 … … Phase (log n): n bitonic sequence of size 1 a sorted sequence can be generated by

concatenating the n bitonic sequence of size 1

SJTU

2013

陈贵

海

③Bitonic Merge 双调合并

sort a bitonic sequence using bitonic splits

1 2 3 4 5 6 7 8 9 10111213141516length

8

16

4

2

SJTU

2013

陈贵

海

Bitonic Merge Circuit : BM[16]

SJTU

2013

陈贵

海

Questions ?

How can we convert an unsorted sequence to a bitonic sequence ? (then, by using bitonic split recursively, a sorted sequence can be formed).

SJTU

2013

陈贵

海

1 2 3 4 5 6 7 8 9 10111213141516length

8

4

16

Turn an unsorted sequence into a bitonic sequence: Bitonic Merge (BM) Operation③

At every phase, sort a bitonic sequence of size 2, 4, 8, 16

into a monotonically increasing or decreased sequence

SJTU

2013

陈贵

海Turn an unsorted sequence into a bitonic sequence

SJTU

2013

陈贵

海

1 2 3 4 5 6 7 8 9 10111213141516length

8

4

16

④Bitonic Sort

SJTU

2013

陈贵

海

Sort (any ordered of) sequence

Using bitonic merge repeatedly

Definition: BM[n]: increasing bitonic merge of size n

• bitonic merge : sort a bitonic sequence of size n into a monotonically increasing sequence

BM[n]: decreasing bitonic merge of size n• bitonic merge that sort a bitonic sequence of size n into a

monotonically decreasing sequence

SJTU

2013

陈贵

海

Steps:

Divide the sequence into a group of 2– any sequence of size 2 is a bitonic sequence: either

the increasing part is of size 2 and the decreasing part is of size 0, or vice versa

Using BM[2] on a group to form an increasing sequence, and BM[2] on the adjacent group to form an decreasing sequence

Concatenate the two group to form a bitonic sequence of size 4

SJTU

2013

陈贵

海

Repeat the above steps on other groups

Repeat the above steps recursively, until a bitonic sequence of size n is formed

Using bitonic merge again to turn the bitonic sequence into a sorted sequence

Steps:

SJTU

2013

陈贵

海 BM[n]: increasing bitonic merge of size n BM[n]: decreasing bitonic merge of size n

Bitonic Sorting Circuit: BS(18)

SJTU

2013

陈贵

海

Sort (any ordered of) sequence

Hence,

n unsorted numbers

n/2 group of 2-number bitonic sequence

n/4 group of 4-number bitonic sequence

…

1 group of n-number bitonic sequence

a sorted sequence

SJTU

2013

陈贵

海

⑤Complexity of Bitonic Sort

Parallel bitonic sort with n processor– The last stage of an n-element bitonic sorting need

to merge n-element, and has a depth of log(n)

– Other stages perform a complete sort of n/2 elements

– Depth, d(n) = d(n/2) + log(n)

– d(n) = 1 + 2 + 4 + … + log(n) = (log2n)

– Complexity: T(n) = (log2n)

SJTU

2013

陈贵

海

⑤Complexity of Bitonic Sort

Parallel sorting with a block of elements per processor– sort the local block of elements first (using any

sorting algorithm such as quicksort, bitonic sort)– sort the elements among processors using parallel

bitonic sort– T(n) = T(local_sort) + T(comparisons)

+T(communication)

Only computation time is considered here (you need to consider all communication time also)

SJTU

2013

陈贵

海

⑥Concluding Remarks

Bitonic Sorting: Common Sense

Regression to Computer Science

One of 10 Most Important Papers

Parallel Algorithm: Ascend/Descend– Another example: Prefix sum

Network Model:

SJTU

2013

陈贵

海27

Bitonic Sorting Network

Hypercube connections!Try to Write Bitonic Sorting algorithm on hypercube.

SJTU

2013

陈贵

海28

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海29

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海30

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海31

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海32

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海33

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海34

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海35

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海36

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海37

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海38

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海39

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海40

Bitonic Sort on Butterfly

SJTU

2013

陈贵

海

PRAM Model

P1 P2 P3 Pn…

Memory

Access time from any processor to any memory unit is equal

It is impossible in practice

So it is an ideal model for parallel computing

SJTU

2013

陈贵

海

PRAM Model

Program for Sum= a(1)+a(2)+…+a(N)

for i = 1 to log N

for j= 1 to n/ 2i

parallel do a(j) = a(j) + a(N/ 2i + j)

endpar

endfor

endfor

Finally a(1) is the sum

SJTU

2013

陈贵

海

Hypercube Model

Suppose node N(i) holds element a(i), where i is the value of node index x1x2…xn

for i = 1 to n

for j= i to n

parallel do

N(00…0 (xj=0) xj+1…xn) N(00…0 (xj=1) xj+1…xn);

a(00…0 (xj=0) xj+1…xn) =

a(00…0 (xj=0) xj+1…xn) + a(00…0 (xj=1) xj+1…xn)

endpar

endfor

endfor

Finally node 00…0 holds the sum

SJTU

2013

陈贵

海

Hypercube Model

000

001

010

011

100

101

110

111

Suppose node 000 holds element a(0) and 111holds element a(7)

a(4)

a(1)

a(3)

a(7)

a(2)

a(0)

a(5)

a(6)

000

001

010

011

100

101

110

111

a(1)+a(5)

a(3)+a(7)a(2)+a(6)

a(0)+a(4)

000

001

010

011

100

101

110

111

a(1)+a(5)+a(3)+a(7)

+a(2)+a(6)a(0)+a(4)

000

001

010

011

100

101

110

111

+a(1)+a(5)+a(3)+a(7)+a(2)+a(6)a(0)+a(4)