Randomized Algorithmstcs.nju.edu.cn/slides/random2013/random1.pdfTextbooks Rajeev Motwani and...

Randomized Algorithms

南京大学

尹一通

Course Info

• 尹一通

yitong.yin@gmail.com yinyt@nju.edu.cn

• office hour:

804, Wednesday 2-4pm

• course homepage: http://tcs.nju.edu.cn/wiki

Textbooks

Rajeev Motwani and Prabhakar Raghavan.Randomized Algorithms.

Cambridge University Press, 1995.

Michael Mitzenmacher and Eli Upfal.Probability and Computing:

Randomized Algorithms and Probabilistic Analysis.

Cambridge University Press, 2005.

ReferencesCLRS

Feller An Introduction to Probability Theory and Its Applications

Häggström Finite Markov Chains and Algorithmic Applications

Alon and SpencerThe Probabilistic Method

“algorithms which use randomness in computation”

Randomized Algorithms

Why?

• Simpler.

• Faster.

• Can do impossibles.

• Leads to clever deterministic algorithms.

• Random input.

• Deterministic problem with random nature.

• ... ...

“algorithms which use randomness in computation”

Randomized Algorithms

How?

• To hit a witness.

• To fool an adversary.

• To simulate random samples.

• To construct a solution.

• To break symmetry.

• ... ...

Probability Principles• Basics: probability space, events, independence,

conditional probability;

• Moments and deviation: random variables, expectation, Markov’s inequality, variance, second moment method;

• The probabilistic method: threshold phenomena, Lovász local lemma;

• Concentration: Chernoff bound, martingales, Azuma’s inequality, bounded difference method;

• Random walks: Markov chain, hitting/cover time, mixing time, coupling, conductance.

Probability Space

�Sample space: Ω

Probability measure:

Pr : � � [0, 1]�

e��

Pr(e) = 1s.t.

event A � �

Pr(A) =�

e�A

Pr(e)probability

Probability SpaceKolmogorov (1933)

Sample space Ω: set of all elementary events (samples)

Set of events Σ: each event is a subset of Ω

� is closed under �, �, \.

�, � � �.

Probability measure

Pr(�) = 1

A � B = � � Pr(A � B) = Pr(A) + Pr(B)

limn��

Pr(An) = 0

for A1 � · · · � An � · · · with�

n An = �

impossible event, certain eventσ-algebra

(K1)(K2)

(K3)(K4)

(K5*)

Pr : � � [0, 1]

Probability Space

�

E : the outcome is even.Pr[E ]

= Pr[ ⇥ ⇥ ]= Pr[ ] + Pr[ ] + Pr[ ]

Notation:

predicate E

Pr[E ] = Pr(A)

event:

A = {x � � | E(x)}set

probability:

�, �¬ �\

��

�, �

� is closed under �, �, \.

�, � � �.

Pr(�) = 1

A � B = � � Pr(A � B) = Pr(A) + Pr(B)

(K1)(K2)(K3)(K4)

Pr[¬E ] = 1 � Pr[E ]

Pr[E1 � E2] = Pr[E1] + Pr[E2] � Pr[E1 � E2]

If E1 � E2, then Pr[E1] � Pr[E2].

� is closed under �, �, \.

�, � � �.

Pr(�) = 1

A � B = � � Pr(A � B) = Pr(A) + Pr(B)

(K1)(K2)(K3)(K4)

Pr(� \ A) = 1 � Pr(A)

If A � B, then Pr(A) � Pr(B).

Pr(A � B) = Pr(A) + Pr(B) � Pr(A � B)

Independence

Definition:Events E1, E2, . . . , En are mutually independentif and only if, for any subset I � {1, 2, . . . , n},

Pr

��

i�I

Ei

�=

�

i�I

Pr[Ei].

Definition:Two events E1 and E2 are independent if and only if

Pr [E1 � E2] = Pr[E1] · Pr[E2].

Model of Computation

Turing machine,random-access machine (RAM)

with access to a perfect coin

(a sequence of uniform and independent random bits)

standard model:

Monte Carlo vs Las Vegas

Monte Carlo Las Vegas

running time is fixedcorrectness is random

always correctrunning time is random

Two types of randomized algorithms:

Monte Carlo algorithmsrunning time is boundedcorrectness is random

• MC algorithm with one-sided error:

if answer=yes: always returns “yes”;if answer= no: Pr[returns “no”]≥ε;

for decision problems (with “yes-or-no” answer):(false positive)

repeat independently for t times“yes” if all t trials return “yes”;“no” if otherwise;return {

if answer=yes: always returns “yes”;if answer= no: Pr[returns “yes”] � (1 � �)t = �

t = O( 1� log 1

� )

Monte Carlo algorithmsrunning time is boundedcorrectness is random

• MC algorithm with two-sided error:

if answer=yes: Pr[returns “yes”]≥1/2+ε;if answer= no: Pr[returns “no”]≥1/2+ε;

for decision problems (with “yes-or-no” answer):

repeat independently for t times

return majority answer;

Monte Carlo algorithms

• MC algorithm with two-sided error:

Pr[correct]≥1/2+ε;

for decision problems (with “yes-or-no” answer):

repeat independently for t times

Pr[exact i trials are correct]

Pr[≤t/2 trials are correct]

=

�t/2��

i=0

�t

i

� �1

2+ �

�i �1

2� �

�t�i

��t/2��

i=0

�t

i

��1

2+ �

�t/2 �1

2� �

�t/2

=

�1

4� �2

�t/2 �t/2��

i=0

�t

i

��

�1

4� �2

�t/2

2t = (1 � 4�2)t/2 = �

t = O( 1�2 log 1

� )

=�t

i

� �12 + �

�i �12 � �

�t�i

Las Vegas algorithmsalways correct

running time is random

LV algorithm MC algorithm

truncation

?

Checking Matrix Multiplication

A B C× =?

three n×n matrices A, B, C:

best matrix multiplication algorithm: O(n2.376)

O(n2.3727)

Checking Matrix Multiplication

A B C× =?

three n×n matrices A, B, C:

best matrix multiplication algorithm: O(n2.376)

O(n2.3727)

O(n�)

best matrix multiplication algorithm:

Checking Matrix Multiplication

A B C× =?

three n×n matrices A, B, C:

Freivald’s Algorithmpick a uniform random r ∈{0,1}n;check whether A(Br) = Cr ;

time: O(n2) if AB = C, always correct

O(n2.3727)

Freivald’s Algorithmpick a uniform random r ∈{0,1}n;check whether A(Br) = Cr ;

if AB = C, always correct

if AB ≠ C,

Pr[ABr = Cr] = Pr[Dr = 0]

D = AB � C �= 0n�nlet

D ri

(Dr)i =n�

k=1

Dikrk =0

rj = � 1

Dij

n�

k �=j

Dikrk

2n

2n�1

=1

2

Dij �= 0say

�

Freivald’s Algorithmpick a uniform random r ∈{0,1}n;check whether A(Br) = Cr ;

if AB = C, always correct

If AB ⇥= C, for a uniformly random r � {0, 1}n,

Pr[ABr = Cr] � 12.

Theorem (Freivald, 1979)

repeat independently for 100 times

time: O(n2) if AB ≠ C, error probability ≤ 2�100

Polynomial Identity Testing (PIT)

Input: two polynomials f, g � F[x] of degree dOutput: f � g?

Input: a polynomial of degree dOutput:

f � F[x]

f � 0?

f is given in product form or as black-box

simple deterministic algorithm:

check whether f(x)=0 for all x � {1, 2, . . . , d + 1}

A degree d polynomial has at most d roots. Fundamental Theorem of Algebra:

Randomized Algorithmpick a uniform random r ;check whether f(r) = 0 ;

∈S S � F

Input: a polynomial of degree dOutput:

f � F[x]

f � 0?

A degree d polynomial has at most d roots. Fundamental Theorem of Algebra:

Randomized Algorithmpick a uniform random r ;check whether f(r) = 0 ;

∈S S � F

f �� 0if

Pr[f(r) = 0] � |S|d

|S| = 2d

=1

2

Checking Identity

database 1

database 2

Are they identical?

北京

南京

Communication Complexity(Yao 1979)

Li LeiHan Meimei

EQ : {0, 1}n ! {0, 1}n " {0, 1}

# of bitscommunicated

a b

f(a, b)

EQ(a, b) =

!1 a = b

0 a != bThere is no deterministic communication protocol solving EQ with less than n bits in the worst-case.

Theorem (Yao, 1979)

Communication Complexity

Li LeiHan Meimei

a b!{0, 1}n !{0, 1}n

f =n�1�

i=0

aixi g =

n�1�

i=0

aixi

pick uniform random r ∈[2n]

r, g(r)

f(r)=g(r) ?

one-sided error � 1

2

by PIT:

# of bit communicated: too large!

Communication Complexity

Li LeiHan Meimei

a b!{0, 1}n !{0, 1}n

f =n�1�

i=0

aixi g =

n�1�

i=0

aixi

pick uniform random r ∈[p]

r, g(r)

f(r)=g(r) ?

k = �log2(2n)�

p � [2k, 2k+1]choose a prime f, g � Zp[x]let