1

主讲教师：张德富 Tel: 18959217108 Email: dfzhang@xmu.edu.cn

课件下载地址： http://algorithm.xmu.edu.cn:10000/Download.aspx#p3

2

教材及参考书 教材 张德富 . 算法设计与分析（高级教程） , 国防工业出版社 ,

2007 参考书

T. H. Cormen, C. E. Leiserson, R. L. Rivest and C. Stein, Introduction to Algorithms (the second edition) ， The MIT Press ， 2001 ，中文名《算法导论（第二版）》（影印版），高等教育出版社， 2003

Brassard, G., Bratley, P., Fundamentals of algorithmics, Prentice hall, 2003

D. S. Hochbaum, Approximation Algorithms for NP-hard Problems, PWS Pulishing Company, 1995

Rajeev Motwani and Prabhakar Raghavan, Randomized Algorithms, Cambridge University Press, 1995

3

要求 :

考试 70% 15%(8 个大实验任选 4 个做，并报告） 提交一篇小论文（ 15% ） 额外加分 5%: 对书中问题或者课堂讨论 , 回答问题 ,

有新的想法或者建议的

Introduction

Why do you study advanced algorithm?

The world is of random and approximation

5

Summary

Introduction Mathematics Fundaments Probability Fundaments Optimization problem Algorithm Linear Programming vs Integer programming Computational complexity Dealing with NP problems Performance Measurements

6

1.1 Mathematics Fundaments

Floors and ceiling

Modular arithmetic

Exponential and Logarithm functions

Harmonic number

.11 xxxxx

./ mod nnaana

.1||for 11 2 xxxex x

. allfor )1(lim xen

x xn

n

.1for )1ln(1

xxxx

x

.1ln1

)()1ln(1

nk

nHnn

k

7

Probability fundaments

Discrete random variable

A (discrete) random variable X is a function from a finite or countably infinite sample space S to the real numbers.

For a random variable X and a real number x, we define the event X = x to be {s∈S : X(s) =x}; thus,

Especially, suppose we are given a sample space S and an event s. Then the random variable I{s} associated with event s is defined as

})(:{

}Pr{}Pr{xsXSs

sxX

otherwise0

occurs if1}{

ssI

8

Conditional probability formalizes the notion of having prior partial knowledge of the outcome of an experiment. The conditional probability of an event s1 given that another event s2 occurs is defined to be

Pr(s1|s2)=Pr(s1∩s2)/Pr(s2) Expected value of a random variable

The simplest and most useful summary of the distribution of a random variable is the "average" of the values it takes on. The expected value of a discrete random variable X is

x

xXxXE }Pr{][

9

Variance and standard deviation The notion of variance mathematically expresses how

far from the mean a random variable‘s values are likely to be. The variance of a random variable X with mean E[X] is

Var[X]=E[(X-E[X])2] The standard deviation δ of a random variable X is

the positive square root of the variance of X. Namely,δ2=Var[X].

10

Normal distribution Binomial distribution Central Limit Theorem

It turns out that for large samples the sampling distribution of the mean is approximately Normal whatever the distribution of the observations. This result is called the Central Limit Theorem. It allows us to use the Normal distribution as a good approximation for large sample size.

Confidence Intervals

11

For example

Flipping a coin.

. if0

, if1{}{

TY

HYHYIX H

The expected number of heads obtained in one flip of the coin is simply the expected value of our random variable XH:

E[XH] = E[I{Y = H}]

= 1 · Pr{Y = H} + 0 · Pr{Y = T}

= 1 · (1/2) + 0 · (1/2)= 1/2.

12

Lemma

Given a sample space S and an event A in the sample space S, let XA = I{A}. Then E[XA] =Pr{A}.

Proof :By the definition of a random variable and the definition of expected value, we have

E[XA] = E[I{A}]

= 1 · Pr{A} + 0 · Pr{Ā}

= Pr{A},

13

Let Xi be the indicator random variable associated with the event in which the ith flip comes up heads. Letting Yi be the random variable denoting the outcome of the ith flip,we have that Xi = I{Yi = H}. Let X be the random variable denoting the total number of heads in the n coin flips, so that

n

iiXX

1

14

We wish to compute the expected number of heads, so we take the expectation of both sides of the above equation to obtain

n

i

n

ii

n

ii

n

XE

XEXE

1

1

1

.2/2/1

][

][][

15

1.2 Computational complexity

Optimization problem

Definition 1 An optimization problem Q is characterized by three components

Instances D: a set of input instances

Solutions S(I): the set of all feasible solutions for an instance I∈D

Value f: a function which assigns a value to each solution ie f: S(I) →R

16

For example

The maximization problem Q:

namely, any given I∈D, find a solution s0∈S(I) such that

Similarly, one can define the minimization problem. For convenience, we will also refer to the value f(s0)

(maximization or minimum value) of the optimal solution as OPT(I), namely OPT(I)= f(s0).

)()( ),( 0 sfsfISs

)(s )(max ISsf

17

Algorithm

An algorithm is a sequence of computational steps that transform the input into the output to solve a given problem.

Algorithm—problem (optimization problem).

A mapping/relation between a set of input instances (domain) and an output set (range)

18

Uncomputable problems vs Intractable problems Genuinely uncomputable problems that just can’t be

solved Intractable or impractical problems that we don’t

have enough computational power to solve in general.

Intractable problems: Easy and hard problems Reductions Decision problem vs. optimization problem

19

P,NP NP-hard, NP-complete Some NP-complete problems

SAT, TSP, 0/1 knapsack problem, n queens problem, packing and covering problem, scheduling problem, timetabling problem, network routing problem et al.

20

Dealing with NP problems

In practice, we quite often encounter instances of intractable problems that we actually need solutions for: The problem might be generally intractable, but we

have a specific instance. For example, scheduling and networking problems

often fit into this class. Even when finding optimal solutions is intractable,

finding near-optimal solutions might be possible.

21

Dealing with NP problems

Your boss asks you to implement an algorithm for an NP-hard optimization problem. What do you do?

I couldn’t find a polynomial-time algorithm; I guess I’m too dumb.

22

Dealing with NP problems

I couldn’t find a polynomial-time algorithm, but neither could all these other smart people.

23

Dealing with NP problems

I couldn’t find a polynomial-time algorithm, because no such algorithm exists!

24

Coping with NP problems

Your boss asks you to implement an algorithm for an NP-hard optimization problem. What do you do?

-Exact algorithms / Global optimization Dynamic Programming, Backtracking, Branch-and-

bound and variants (either hope that the input is very small or that worst

case manifests itself very rarely) (Special cases maybe solvable in polynomial time) -Linear programming algorithms

25

-Heuristic algorithms / Local optimization

Greedy, Local Search, Meta-heuristics (Tabu search, GA, SA, Ant Algorithms, NN) etc

Approximation algorithms

Polynomial-time algorithms with provable performance bounds

Randomized algorithms

DNA and quantum computation

26

Other problems except NP problems

The world behaves randomly and fuzzily, so randomization and approximation are themes that cut across many areas of science and engineering.

Other problems except NP problems can be solved by making use of randomized and approximation algorithms, for example

Deadlock and symmetry problems Computational number theory( cryptography) Random sampling pattern matching ……

27

Performance Measurements

Heuristic algorithm

What do we expect from a “good” heuristic:

1. It should do so quickly

2. It should find a solution of good quality Tools to measure heuristic performances:

1. Run time analysis

2. Analysis on solution quality (experiment)

28

Performance Measurements

Approximation algorithms

What do we expect from a “good” approximation:

1. It runs quickly

2. It should find a solution of good quality Tools to measure approximation performances:

1. Run time analysis

2. Solution quality with performance guarantee( approximation ratio)

29

Performance Measurements

Randomized algorithms

What do we expect from a “good” randomized algorithm:

1. It runs quickly

2. It should make a mistake as small as possible Tools to measure randomized performances:

1. Run time analysis

2. Analysis on solution quality ( The error probability)

30

1.3 Linear programming problem(LP)

Specify the objective of the optimization problem as a linear function of

certain variables, and specify the constraints on resources as equalities or inequalities on those variables.

31

General linear programs

Given a set of real numbers a1, a2, ..., an and a set of variables x1, x2,..., xn, a linear function f on those variables is defined by

linear equality: linear inequalities: and

bxxf n ),,( 1

bxxf n ),,( 1 bxxf n ),,( 1

n

jjjnnn xaxaxaxxf

1111 ),,(

32

Linear constraints equalities or linear equalities. Formally a linear-programming problem is the

problem of either minimizing or maximizing a linear function subject to a finite set of linear constraints.

Minimization linear program Maximization linear program

33

Example

0

42

7

s.t.

32min

1

21

21

21

x

xx

xx

xx

34

Standard forms

Given n real numbers c1, c2, ..., cn; m real numbers b1, b2, ..., bm; and mn real numbers aij for i = 1, 2, ..., m and j = 1, 2, ..., n. We wish to find n real numbers x1, x2,..., xn that

Subject to

j

n

jj xc

1

max

.,,1for 0

,,1for1

njx

mibxa

j

jj

n

jij

35

Feasible solution Infeasible solution Objective value Optimal solution Optimal objective value Unbounded

36

Example

0,,

422

7

7

s.t.

332max

321

321

321

321

321

xxx

xxx

xxx

xxx

xxx

37

Slack forms

.0

,

i

jNj

ijii

Njjj

x

xabx

xcvz

38

Example

0,,,,,

224

7

7

332

654321

3216

3215

3214

321

xxxxxx

xxxx

xxxx

xxxx

xxxz

0,,

422

7

7

s.t.

332max

321

321

321

321

321

xxx

xxx

xxx

xxx

xxx

Standard forms Slack forms

39

The simplex algorithm

The simplex algorithm takes as input a linear program and returns an optimal solution. It starts at some vertex of the simplex and performs a sequence of iterations. In each iteration, it moves along an edge of the simplex from a current vertex to a neighboring vertex whose objective value is no smaller than that of the current vertex (and usually is larger.)

40

The simplex algorithm terminates when it reaches a local maximum, which is a vertex from which all neighboring vertices have a smaller objective value. Because the feasible region is convex and the objective function is linear, this local optimum is actually a global optimum.

41

Integer Programming (IP)

Integer Programming is simply Linear Programming with an added condition:

All variables must be integers Many problems can be stated as Integer Programs. For example, the Vertex Cover problem can be stated

as an integer programming problem.

Given an undirected graph G = (V, E), find a minimum subset V' of V such that if (u, v)∈E,

then u∈V' or v ∈V' (or both).

42

Suppose that we associate a variable x(v) with each vertex v∈V , and let us require that x(v) {0, 1} for ∈each v∈V. 0-1 integer program for finding a minimum vertex cover(VC):

s.t.

for any edge (u, v) E∈ for each v V∈

1)()( vxux

'

)( minVv

vx

}1,0{)( vx

43

0-1 linear programming relaxation :

s.t.

for any edge (u, v) E∈ for each v V∈ for each v V∈

1)()( vxux1)( vx0)( vx

'

)( minVv

vx

44

Homework

Exercise 1.3

2